* [PATCH net v4 10/12] net: sit: Fix one possbile memleak when fail to register_netdevice
From: gfree.wind @ 2017-05-02 13:39 UTC (permalink / raw)
To: davem, jiri, mareklindner, sw, a, kuznet, jmorris, yoshfuji,
kaber, steffen.klassert, herbert, netdev
Cc: Gao Feng
In-Reply-To: <cover.1493699451.git.gfree.wind@foxmail.com>
From: Gao Feng <gfree.wind@foxmail.com>
The ipip6 allocates some resources in its ndo_init func, and
free some of them in its destructor func. Then there is one memleak
that some errors happen after register_netdevice invokes the ndo_init
callback. Because only the ndo_uninit callback is invoked in the error
handler of register_netdevice, but destructor not.
Now create one new func ipip6_destructor_free to free the mem in
the destructor, and ndo_uninit func also invokes it when fail to
register the ipip6 device.
It's not only free all resources, but also follow the original desgin
that the resources are freed in the destructor normally after
register the device successfully.
Signed-off-by: Gao Feng <gfree.wind@foxmail.com>
---
net/ipv6/sit.c | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index 99853c6..28c1649 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -464,6 +464,14 @@ static void prl_list_destroy_rcu(struct rcu_head *head)
return ok;
}
+static void ipip6_destructor_free(struct net_device *dev)
+{
+ struct ip_tunnel *tunnel = netdev_priv(dev);
+
+ dst_cache_destroy(&tunnel->dst_cache);
+ free_percpu(dev->tstats);
+}
+
static void ipip6_tunnel_uninit(struct net_device *dev)
{
struct ip_tunnel *tunnel = netdev_priv(dev);
@@ -477,6 +485,10 @@ static void ipip6_tunnel_uninit(struct net_device *dev)
}
dst_cache_reset(&tunnel->dst_cache);
dev_put(dev);
+
+ /* dev is not registered, perform the free instead of destructor */
+ if (dev->reg_state == NETREG_UNINITIALIZED)
+ ipip6_destructor_free(dev);
}
static int ipip6_err(struct sk_buff *skb, u32 info)
@@ -1329,10 +1341,7 @@ static bool ipip6_valid_ip_proto(u8 ipproto)
static void ipip6_dev_free(struct net_device *dev)
{
- struct ip_tunnel *tunnel = netdev_priv(dev);
-
- dst_cache_destroy(&tunnel->dst_cache);
- free_percpu(dev->tstats);
+ ipip6_destructor_free(dev);
free_netdev(dev);
}
--
1.9.1
^ permalink raw reply related
* [PATCH net v4 09/12] net: ip_tunnel: Fix one possbile memleak when fail to register_netdevice
From: gfree.wind @ 2017-05-02 13:37 UTC (permalink / raw)
To: davem, jiri, mareklindner, sw, a, kuznet, jmorris, yoshfuji,
kaber, steffen.klassert, herbert, netdev
Cc: Gao Feng
In-Reply-To: <cover.1493699451.git.gfree.wind@foxmail.com>
From: Gao Feng <gfree.wind@foxmail.com>
The ip_tunnel allocates some resources in its ndo_init func, and
free some of them in its destructor func. Then there is one memleak
that some errors happen after register_netdevice invokes the ndo_init
callback. Because only the ndo_uninit callback is invoked in the error
handler of register_netdevice, but destructor not.
Now create one new func ip_tunnel_destructor_free to free the mem in
the destructor, and ndo_uninit func also invokes it when fail to
register the ip_tunnel device.
It's not only free all resources, but also follow the original desgin
that the resources are freed in the destructor normally after
register the device successfully.
Signed-off-by: Gao Feng <gfree.wind@foxmail.com>
---
net/ipv4/ip_tunnel.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/net/ipv4/ip_tunnel.c b/net/ipv4/ip_tunnel.c
index 823abae..96a3005 100644
--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -954,13 +954,18 @@ int ip_tunnel_change_mtu(struct net_device *dev, int new_mtu)
}
EXPORT_SYMBOL_GPL(ip_tunnel_change_mtu);
-static void ip_tunnel_dev_free(struct net_device *dev)
+static void ip_tunnel_destructor_free(struct net_device *dev)
{
struct ip_tunnel *tunnel = netdev_priv(dev);
gro_cells_destroy(&tunnel->gro_cells);
dst_cache_destroy(&tunnel->dst_cache);
free_percpu(dev->tstats);
+}
+
+static void ip_tunnel_dev_free(struct net_device *dev)
+{
+ ip_tunnel_destructor_free(dev);
free_netdev(dev);
}
@@ -1192,6 +1197,10 @@ void ip_tunnel_uninit(struct net_device *dev)
ip_tunnel_del(itn, netdev_priv(dev));
dst_cache_reset(&tunnel->dst_cache);
+
+ /* dev is not registered, perform the free instead of destructor */
+ if (dev->reg_state == NETREG_UNINITIALIZED)
+ ip_tunnel_destructor_free(dev);
}
EXPORT_SYMBOL_GPL(ip_tunnel_uninit);
--
1.9.1
^ permalink raw reply related
* [PATCH net v4 08/12] net: ip6_vti: Fix one possbile memleak when fail to register_netdevice
From: gfree.wind @ 2017-05-02 13:34 UTC (permalink / raw)
To: davem, jiri, mareklindner, sw, a, kuznet, jmorris, yoshfuji,
kaber, steffen.klassert, herbert, netdev
Cc: Gao Feng
In-Reply-To: <cover.1493699451.git.gfree.wind@foxmail.com>
From: Gao Feng <gfree.wind@foxmail.com>
The ip6_vti allocates some resources in its ndo_init func, and
free some of them in its destructor func. Then there is one memleak
that some errors happen after register_netdevice invokes the ndo_init
callback. Because only the ndo_uninit callback is invoked in the error
handler of register_netdevice, but destructor not.
Now create one new func vti6_destructor_free to free the mem in
the destructor, and ndo_uninit func also invokes it when fail to
register the vti6 device.
It's not only free all resources, but also follow the original desgin
that the resources are freed in the destructor normally after
register the device successfully.
Signed-off-by: Gao Feng <gfree.wind@foxmail.com>
---
net/ipv6/ip6_vti.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index 3d8a3b6..3b3f49a 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -177,9 +177,14 @@ struct vti6_net {
}
}
-static void vti6_dev_free(struct net_device *dev)
+static void vti6_destructor_free(struct net_device *dev)
{
free_percpu(dev->tstats);
+}
+
+static void vti6_dev_free(struct net_device *dev)
+{
+ vti6_destructor_free(dev);
free_netdev(dev);
}
@@ -296,6 +301,10 @@ static void vti6_dev_uninit(struct net_device *dev)
else
vti6_tnl_unlink(ip6n, t);
dev_put(dev);
+
+ /* dev is not registered, perform the free instead of destructor */
+ if (dev->reg_state == NETREG_UNINITIALIZED)
+ vti6_destructor_free(dev);
}
static int vti6_rcv(struct sk_buff *skb)
--
1.9.1
^ permalink raw reply related
* Re: [PATCH 0/3] net-SCTP: Fine-tuning for six function implementations
From: Neil Horman @ 2017-05-02 13:15 UTC (permalink / raw)
To: SF Markus Elfring
Cc: linux-sctp, netdev, David S. Miller, Vlad Yasevich, LKML,
kernel-janitors
In-Reply-To: <ea9206ca-2a2f-095e-d764-3f819610d5e8@users.sourceforge.net>
On Mon, May 01, 2017 at 03:30:25PM +0200, SF Markus Elfring wrote:
> From: Markus Elfring <elfring@users.sourceforge.net>
> Date: Mon, 1 May 2017 15:25:05 +0200
>
> A few update suggestions were taken into account
> from static source code analysis.
>
> Markus Elfring (3):
> Replace six seq_printf() calls by seq_putc()
> Combine two seq_printf() calls into one call in sctp_remaddr_seq_show()
> Replace four seq_printf() calls by seq_puts()
>
> net/sctp/proc.c | 39 ++++++++++++++++-----------------------
> 1 file changed, 16 insertions(+), 23 deletions(-)
>
> --
> 2.12.2
>
>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
^ permalink raw reply
* [PATCH 6/9] net: thunderx: Add support for XDP_DROP
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Adds support for XDP_DROP.
Also since in XDP mode there is just a single buffer per page,
made changes to recycle DMA mapping info as well along with pages.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 23 ++++++-
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 77 ++++++++++++++++------
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 4 +-
3 files changed, 79 insertions(+), 25 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index 9c48873..a58cc1e 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -18,6 +18,7 @@
#include <linux/irq.h>
#include <linux/iommu.h>
#include <linux/bpf.h>
+#include <linux/bpf_trace.h>
#include <linux/filter.h>
#include "nic_reg.h"
@@ -505,6 +506,7 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic,
struct cqe_rx_t *cqe_rx)
{
struct xdp_buff xdp;
+ struct page *page;
u32 action;
u16 len;
u64 dma_addr, cpu_addr;
@@ -527,12 +529,27 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic,
switch (action) {
case XDP_PASS:
case XDP_TX:
- case XDP_ABORTED:
- case XDP_DROP:
/* Pass on all packets to network stack */
return false;
default:
bpf_warn_invalid_xdp_action(action);
+ case XDP_ABORTED:
+ trace_xdp_exception(nic->netdev, prog, action);
+ case XDP_DROP:
+ page = virt_to_page(xdp.data);
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) == 1) {
+ dma_addr &= PAGE_MASK;
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr,
+ RCV_FRAG_LEN, DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ }
+ put_page(page);
+ return true;
}
return false;
}
@@ -645,7 +662,7 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx))
return;
- skb = nicvf_get_rcv_skb(snic, cqe_rx);
+ skb = nicvf_get_rcv_skb(snic, cqe_rx, nic->xdp_prog ? true : false);
if (!skb) {
netdev_dbg(nic->netdev, "Packet not received\n");
return;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 8c3c571..5009f49 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -117,6 +117,7 @@ static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
/* Save the page in page cache */
pgcache->page = page;
+ pgcache->dma_addr = 0;
rbdr->pgalloc++;
}
@@ -144,7 +145,7 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
/* Check if request can be accomodated in previous allocated page.
* But in XDP mode only one buffer per page is permitted.
*/
- if (!nic->pnicvf->xdp_prog && nic->rb_page &&
+ if (!rbdr->is_xdp && nic->rb_page &&
((nic->rb_page_offset + buf_len) <= PAGE_SIZE)) {
nic->rb_pageref++;
goto ret;
@@ -165,18 +166,24 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
if (pgcache)
nic->rb_page = pgcache->page;
ret:
- /* HW will ensure data coherency, CPU sync not required */
- *rbuf = (u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
- nic->rb_page_offset, buf_len,
- DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
- if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
- if (!nic->rb_page_offset)
- __free_pages(nic->rb_page, 0);
- nic->rb_page = NULL;
- return -ENOMEM;
+ if (rbdr->is_xdp && pgcache && pgcache->dma_addr) {
+ *rbuf = pgcache->dma_addr;
+ } else {
+ /* HW will ensure data coherency, CPU sync not required */
+ *rbuf = (u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
+ nic->rb_page_offset, buf_len,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
+ if (!nic->rb_page_offset)
+ __free_pages(nic->rb_page, 0);
+ nic->rb_page = NULL;
+ return -ENOMEM;
+ }
+ if (pgcache)
+ pgcache->dma_addr = *rbuf;
+ nic->rb_page_offset += buf_len;
}
- nic->rb_page_offset += buf_len;
return 0;
}
@@ -230,8 +237,16 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr,
* On embedded platforms i.e 81xx/83xx available memory itself
* is low and minimum ring size of RBDR is 8K, that takes away
* lots of memory.
+ *
+ * But for XDP it has to be a single buffer per page.
*/
- rbdr->pgcnt = ring_len / (PAGE_SIZE / buf_size);
+ if (!nic->pnicvf->xdp_prog) {
+ rbdr->pgcnt = ring_len / (PAGE_SIZE / buf_size);
+ rbdr->is_xdp = false;
+ } else {
+ rbdr->pgcnt = ring_len;
+ rbdr->is_xdp = true;
+ }
rbdr->pgcnt = roundup_pow_of_two(rbdr->pgcnt);
rbdr->pgcache = kzalloc(sizeof(*rbdr->pgcache) *
rbdr->pgcnt, GFP_KERNEL);
@@ -1454,8 +1469,31 @@ static inline unsigned frag_num(unsigned i)
#endif
}
+static void nicvf_unmap_rcv_buffer(struct nicvf *nic, u64 dma_addr,
+ u64 buf_addr, bool xdp)
+{
+ struct page *page = NULL;
+ int len = RCV_FRAG_LEN;
+
+ if (xdp) {
+ page = virt_to_page(phys_to_virt(buf_addr));
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) != 1)
+ return;
+ /* Receive buffers in XDP mode are mapped from page start */
+ dma_addr &= PAGE_MASK;
+ }
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr, len,
+ DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
+}
+
/* Returns SKB for a received packet */
-struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
+struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic,
+ struct cqe_rx_t *cqe_rx, bool xdp)
{
int frag;
int payload_len = 0;
@@ -1490,10 +1528,9 @@ struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
if (!frag) {
/* First fragment */
- dma_unmap_page_attrs(&nic->pdev->dev,
- *rb_ptrs - cqe_rx->align_pad,
- RCV_FRAG_LEN, DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
+ nicvf_unmap_rcv_buffer(nic,
+ *rb_ptrs - cqe_rx->align_pad,
+ phys_addr, xdp);
skb = nicvf_rb_ptr_to_skb(nic,
phys_addr - cqe_rx->align_pad,
payload_len);
@@ -1503,9 +1540,7 @@ struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
skb_put(skb, payload_len);
} else {
/* Add fragments */
- dma_unmap_page_attrs(&nic->pdev->dev, *rb_ptrs,
- RCV_FRAG_LEN, DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
+ nicvf_unmap_rcv_buffer(nic, *rb_ptrs, phys_addr, xdp);
page = virt_to_page(phys_to_virt(phys_addr));
offset = phys_to_virt(phys_addr) - page_address(page);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index 07136a2..db04c0e 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -228,6 +228,7 @@ struct rbdr {
u32 head;
u32 tail;
struct q_desc_mem dmem;
+ bool is_xdp;
/* For page recycling */
int pgidx;
@@ -339,7 +340,8 @@ void nicvf_sq_free_used_descs(struct net_device *netdev,
int nicvf_sq_append_skb(struct nicvf *nic, struct snd_queue *sq,
struct sk_buff *skb, u8 sq_num);
-struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx);
+struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic,
+ struct cqe_rx_t *cqe_rx, bool xdp);
void nicvf_rbdr_task(unsigned long data);
void nicvf_rbdr_work(struct work_struct *work);
--
2.7.4
^ permalink raw reply related
* [PATCH 3/9] net: thunderx: Optimize CQE_TX handling
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Optimized CQE handling with below changes
- Feeing descriptors back to SQ in bulk i.e once per NAPI
instance instead for every CQE_TX, this will reduce number
of atomic updates to 'sq->free_cnt'.
- Checking errors in CQE_TX and CQE_RX before calling appropriate
fn()s to update error stats i.e reduce branching.
Also removed debug messages in packet handling path which otherwise
causes issues if DEBUG is enabled.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 44 +++++++++++-----------
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 5 ---
2 files changed, 21 insertions(+), 28 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index 81a2fcb..0d79894 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -498,7 +498,7 @@ static int nicvf_init_resources(struct nicvf *nic)
static void nicvf_snd_pkt_handler(struct net_device *netdev,
struct cqe_send_t *cqe_tx,
- int cqe_type, int budget,
+ int budget, int *subdesc_cnt,
unsigned int *tx_pkts, unsigned int *tx_bytes)
{
struct sk_buff *skb = NULL;
@@ -513,12 +513,10 @@ static void nicvf_snd_pkt_handler(struct net_device *netdev,
if (hdr->subdesc_type != SQ_DESC_TYPE_HEADER)
return;
- netdev_dbg(nic->netdev,
- "%s Qset #%d SQ #%d SQ ptr #%d subdesc count %d\n",
- __func__, cqe_tx->sq_qs, cqe_tx->sq_idx,
- cqe_tx->sqe_ptr, hdr->subdesc_cnt);
+ /* Check for errors */
+ if (cqe_tx->send_status)
+ nicvf_check_cqe_tx_errs(nic->pnicvf, cqe_tx);
- nicvf_check_cqe_tx_errs(nic, cqe_tx);
skb = (struct sk_buff *)sq->skbuff[cqe_tx->sqe_ptr];
if (skb) {
/* Check for dummy descriptor used for HW TSO offload on 88xx */
@@ -528,12 +526,12 @@ static void nicvf_snd_pkt_handler(struct net_device *netdev,
(struct sq_hdr_subdesc *)GET_SQ_DESC(sq, hdr->rsvd2);
nicvf_unmap_sndq_buffers(nic, sq, hdr->rsvd2,
tso_sqe->subdesc_cnt);
- nicvf_put_sq_desc(sq, tso_sqe->subdesc_cnt + 1);
+ *subdesc_cnt += tso_sqe->subdesc_cnt + 1;
} else {
nicvf_unmap_sndq_buffers(nic, sq, cqe_tx->sqe_ptr,
hdr->subdesc_cnt);
}
- nicvf_put_sq_desc(sq, hdr->subdesc_cnt + 1);
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
prefetch(skb);
(*tx_pkts)++;
*tx_bytes += skb->len;
@@ -544,7 +542,7 @@ static void nicvf_snd_pkt_handler(struct net_device *netdev,
* a SKB attached, so just free SQEs here.
*/
if (!nic->hw_tso)
- nicvf_put_sq_desc(sq, hdr->subdesc_cnt + 1);
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
}
}
@@ -595,9 +593,11 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
}
/* Check for errors */
- err = nicvf_check_cqe_rx_errs(nic, cqe_rx);
- if (err && !cqe_rx->rb_cnt)
- return;
+ if (cqe_rx->err_level || cqe_rx->err_opcode) {
+ err = nicvf_check_cqe_rx_errs(nic, cqe_rx);
+ if (err && !cqe_rx->rb_cnt)
+ return;
+ }
skb = nicvf_get_rcv_skb(snic, cqe_rx);
if (!skb) {
@@ -646,6 +646,7 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
{
int processed_cqe, work_done = 0, tx_done = 0;
int cqe_count, cqe_head;
+ int subdesc_cnt = 0;
struct nicvf *nic = netdev_priv(netdev);
struct queue_set *qs = nic->qs;
struct cmp_queue *cq = &qs->cq[cq_idx];
@@ -667,8 +668,6 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
cqe_head = nicvf_queue_reg_read(nic, NIC_QSET_CQ_0_7_HEAD, cq_idx) >> 9;
cqe_head &= 0xFFFF;
- netdev_dbg(nic->netdev, "%s CQ%d cqe_count %d cqe_head %d\n",
- __func__, cq_idx, cqe_count, cqe_head);
while (processed_cqe < cqe_count) {
/* Get the CQ descriptor */
cq_desc = (struct cqe_rx_t *)GET_CQ_DESC(cq, cqe_head);
@@ -682,17 +681,15 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
break;
}
- netdev_dbg(nic->netdev, "CQ%d cq_desc->cqe_type %d\n",
- cq_idx, cq_desc->cqe_type);
switch (cq_desc->cqe_type) {
case CQE_TYPE_RX:
nicvf_rcv_pkt_handler(netdev, napi, cq_desc);
work_done++;
break;
case CQE_TYPE_SEND:
- nicvf_snd_pkt_handler(netdev,
- (void *)cq_desc, CQE_TYPE_SEND,
- budget, &tx_pkts, &tx_bytes);
+ nicvf_snd_pkt_handler(netdev, (void *)cq_desc,
+ budget, &subdesc_cnt,
+ &tx_pkts, &tx_bytes);
tx_done++;
break;
case CQE_TYPE_INVALID:
@@ -704,9 +701,6 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
}
processed_cqe++;
}
- netdev_dbg(nic->netdev,
- "%s CQ%d processed_cqe %d work_done %d budget %d\n",
- __func__, cq_idx, processed_cqe, work_done, budget);
/* Ring doorbell to inform H/W to reuse processed CQEs */
nicvf_queue_reg_write(nic, NIC_QSET_CQ_0_7_DOOR,
@@ -716,8 +710,12 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
goto loop;
done:
- /* Wakeup TXQ if its stopped earlier due to SQ full */
sq = &nic->qs->sq[cq_idx];
+ /* Update SQ's descriptor free count */
+ if (subdesc_cnt)
+ nicvf_put_sq_desc(sq, subdesc_cnt);
+
+ /* Wakeup TXQ if its stopped earlier due to SQ full */
if (tx_done ||
(atomic_read(&sq->free_cnt) >= MIN_SQ_DESC_PER_PKT_XMIT)) {
netdev = nic->pnicvf->netdev;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index dfc85a1..90c5bc7d 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -1640,9 +1640,6 @@ void nicvf_update_sq_stats(struct nicvf *nic, int sq_idx)
/* Check for errors in the receive cmp.queue entry */
int nicvf_check_cqe_rx_errs(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
{
- if (!cqe_rx->err_level && !cqe_rx->err_opcode)
- return 0;
-
if (netif_msg_rx_err(nic))
netdev_err(nic->netdev,
"%s: RX error CQE err_level 0x%x err_opcode 0x%x\n",
@@ -1731,8 +1728,6 @@ int nicvf_check_cqe_rx_errs(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
int nicvf_check_cqe_tx_errs(struct nicvf *nic, struct cqe_send_t *cqe_tx)
{
switch (cqe_tx->send_status) {
- case CQ_TX_ERROP_GOOD:
- return 0;
case CQ_TX_ERROP_DESC_FAULT:
this_cpu_inc(nic->drv_stats->tx_desc_fault);
break;
--
2.7.4
^ permalink raw reply related
* [PATCH 9/9] net: thunderx: Optimize page recycling for XDP
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Driver follows a method of taking one extra reference on the
page for recycling which is fine in usual packet path where
each 64KB page is segmented into multiple receive buffers.
But in XDP mode since there is just one receive buffer per
page taking extra page reference itself becomes big bottleneck
consuming ~50% of CPU cycles due to atomic operations.
This patch adds a internal ref count in pgcache for each
page and additional page references are taken in a batch
instead of just one at a time. Internal i.e 'pgcache->ref_count'
and page's i.e 'page->_refcount' counters are compared to check
page's recyclability.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 57 +++++++++++++++++++---
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 1 +
2 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 43428ce..2b18176 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -82,6 +82,8 @@ static void nicvf_free_q_desc_mem(struct nicvf *nic, struct q_desc_mem *dmem)
dmem->base = NULL;
}
+#define XDP_PAGE_REFCNT_REFILL 256
+
/* Allocate a new page or recycle one if possible
*
* We cannot optimize dma mapping here, since
@@ -90,9 +92,10 @@ static void nicvf_free_q_desc_mem(struct nicvf *nic, struct q_desc_mem *dmem)
* and not idx into RBDR ring, so can't refer to saved info.
* 3. There are multiple receive buffers per page
*/
-static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
- struct rbdr *rbdr, gfp_t gfp)
+static inline struct pgcache *nicvf_alloc_page(struct nicvf *nic,
+ struct rbdr *rbdr, gfp_t gfp)
{
+ int ref_count;
struct page *page = NULL;
struct pgcache *pgcache, *next;
@@ -100,8 +103,23 @@ static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
pgcache = &rbdr->pgcache[rbdr->pgidx];
page = pgcache->page;
/* Check if page can be recycled */
- if (page && (page_ref_count(page) != 1))
- page = NULL;
+ if (page) {
+ ref_count = page_ref_count(page);
+ /* Check if this page has been used once i.e 'put_page'
+ * called after packet transmission i.e internal ref_count
+ * and page's ref_count are equal i.e page can be recycled.
+ */
+ if (rbdr->is_xdp && (ref_count == pgcache->ref_count))
+ pgcache->ref_count--;
+ else
+ page = NULL;
+
+ /* In non-XDP mode, page's ref_count needs to be '1' for it
+ * to be recycled.
+ */
+ if (!rbdr->is_xdp && (ref_count != 1))
+ page = NULL;
+ }
if (!page) {
page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN, 0);
@@ -120,11 +138,30 @@ static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
/* Save the page in page cache */
pgcache->page = page;
pgcache->dma_addr = 0;
+ pgcache->ref_count = 0;
rbdr->pgalloc++;
}
- /* Take extra page reference for recycling */
- page_ref_add(page, 1);
+ /* Take additional page references for recycling */
+ if (rbdr->is_xdp) {
+ /* Since there is single RBDR (i.e single core doing
+ * page recycling) per 8 Rx queues, in XDP mode adjusting
+ * page references atomically is the biggest bottleneck, so
+ * take bunch of references at a time.
+ *
+ * So here, below reference counts defer by '1'.
+ */
+ if (!pgcache->ref_count) {
+ pgcache->ref_count = XDP_PAGE_REFCNT_REFILL;
+ page_ref_add(page, XDP_PAGE_REFCNT_REFILL);
+ }
+ } else {
+ /* In non-XDP case, single 64K page is divided across multiple
+ * receive buffers, so cost of recycling is less anyway.
+ * So we can do with just one extra reference.
+ */
+ page_ref_add(page, 1);
+ }
rbdr->pgidx++;
rbdr->pgidx &= (rbdr->pgcnt - 1);
@@ -327,8 +364,14 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr)
head = 0;
while (head < rbdr->pgcnt) {
pgcache = &rbdr->pgcache[head];
- if (pgcache->page && page_ref_count(pgcache->page) != 0)
+ if (pgcache->page && page_ref_count(pgcache->page) != 0) {
+ if (!rbdr->is_xdp) {
+ put_page(pgcache->page);
+ continue;
+ }
+ page_ref_sub(pgcache->page, pgcache->ref_count - 1);
put_page(pgcache->page);
+ }
head++;
}
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index a07d5b4..5785852 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -216,6 +216,7 @@ struct q_desc_mem {
struct pgcache {
struct page *page;
+ int ref_count;
u64 dma_addr;
};
--
2.7.4
^ permalink raw reply related
* [PATCH 8/9] net: thunderx: Support for XDP header adjustment
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
When in XDP mode reserve XDP_PACKET_HEADROOM bytes at the start
of receive buffer for XDP program to modify headers and adjust
packet start. Additional code changes done to handle such packets.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 63 ++++++++++++++++------
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 9 +++-
2 files changed, 55 insertions(+), 17 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index bb13dee..d6477af 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -502,13 +502,15 @@ static int nicvf_init_resources(struct nicvf *nic)
}
static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
- struct cqe_rx_t *cqe_rx, struct snd_queue *sq)
+ struct cqe_rx_t *cqe_rx, struct snd_queue *sq,
+ struct sk_buff **skb)
{
struct xdp_buff xdp;
struct page *page;
u32 action;
- u16 len;
+ u16 len, offset = 0;
u64 dma_addr, cpu_addr;
+ void *orig_data;
/* Retrieve packet buffer's DMA address and length */
len = *((u16 *)((void *)cqe_rx + (3 * sizeof(u64))));
@@ -517,17 +519,47 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
cpu_addr = nicvf_iova_to_phys(nic, dma_addr);
if (!cpu_addr)
return false;
+ cpu_addr = (u64)phys_to_virt(cpu_addr);
+ page = virt_to_page((void *)cpu_addr);
- xdp.data = phys_to_virt(cpu_addr);
+ xdp.data_hard_start = page_address(page);
+ xdp.data = (void *)cpu_addr;
xdp.data_end = xdp.data + len;
+ orig_data = xdp.data;
rcu_read_lock();
action = bpf_prog_run_xdp(prog, &xdp);
rcu_read_unlock();
+ /* Check if XDP program has changed headers */
+ if (orig_data != xdp.data) {
+ len = xdp.data_end - xdp.data;
+ offset = orig_data - xdp.data;
+ dma_addr -= offset;
+ }
+
switch (action) {
case XDP_PASS:
- /* Pass on packet to network stack */
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) == 1) {
+ dma_addr &= PAGE_MASK;
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr,
+ RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ }
+
+ /* Build SKB and pass on packet to network stack */
+ *skb = build_skb(xdp.data,
+ RCV_FRAG_LEN - cqe_rx->align_pad + offset);
+ if (!*skb)
+ put_page(page);
+ else
+ skb_put(*skb, len);
return false;
case XDP_TX:
nicvf_xdp_sq_append_pkt(nic, sq, (u64)xdp.data, dma_addr, len);
@@ -537,7 +569,6 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
case XDP_ABORTED:
trace_xdp_exception(nic->netdev, prog, action);
case XDP_DROP:
- page = virt_to_page(xdp.data);
/* Check if it's a recycled page, if not
* unmap the DMA mapping.
*
@@ -546,7 +577,8 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
if (page_ref_count(page) == 1) {
dma_addr &= PAGE_MASK;
dma_unmap_page_attrs(&nic->pdev->dev, dma_addr,
- RCV_FRAG_LEN, DMA_FROM_DEVICE,
+ RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
+ DMA_FROM_DEVICE,
DMA_ATTR_SKIP_CPU_SYNC);
}
put_page(page);
@@ -654,7 +686,7 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
struct napi_struct *napi,
struct cqe_rx_t *cqe_rx, struct snd_queue *sq)
{
- struct sk_buff *skb;
+ struct sk_buff *skb = NULL;
struct nicvf *nic = netdev_priv(netdev);
struct nicvf *snic = nic;
int err = 0;
@@ -676,15 +708,17 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
}
/* For XDP, ignore pkts spanning multiple pages */
- if (nic->xdp_prog && (cqe_rx->rb_cnt == 1))
- if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx, sq))
+ if (nic->xdp_prog && (cqe_rx->rb_cnt == 1)) {
+ /* Packet consumed by XDP */
+ if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx, sq, &skb))
return;
+ } else {
+ skb = nicvf_get_rcv_skb(snic, cqe_rx,
+ nic->xdp_prog ? true : false);
+ }
- skb = nicvf_get_rcv_skb(snic, cqe_rx, nic->xdp_prog ? true : false);
- if (!skb) {
- netdev_dbg(nic->netdev, "Packet not received\n");
+ if (!skb)
return;
- }
if (netif_msg_pktdata(nic)) {
netdev_info(nic->netdev, "%s: skb 0x%p, len=%d\n", netdev->name,
@@ -1672,9 +1706,6 @@ static int nicvf_xdp_setup(struct nicvf *nic, struct bpf_prog *prog)
return -EOPNOTSUPP;
}
- if (prog && prog->xdp_adjust_head)
- return -EOPNOTSUPP;
-
/* ALL SQs attached to CQs i.e same as RQs, are treated as
* XDP Tx queues and more Tx queues are allocated for
* network stack to send pkts out.
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index ec234b6..43428ce 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -164,6 +164,11 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
}
nic->rb_page_offset = 0;
+
+ /* Reserve space for header modifications by BPF program */
+ if (rbdr->is_xdp)
+ buf_len += XDP_PACKET_HEADROOM;
+
/* Check if it's recycled */
if (pgcache)
nic->rb_page = pgcache->page;
@@ -183,7 +188,7 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
return -ENOMEM;
}
if (pgcache)
- pgcache->dma_addr = *rbuf;
+ pgcache->dma_addr = *rbuf + XDP_PACKET_HEADROOM;
nic->rb_page_offset += buf_len;
}
@@ -1575,6 +1580,8 @@ static void nicvf_unmap_rcv_buffer(struct nicvf *nic, u64 dma_addr,
*/
if (page_ref_count(page) != 1)
return;
+
+ len += XDP_PACKET_HEADROOM;
/* Receive buffers in XDP mode are mapped from page start */
dma_addr &= PAGE_MASK;
}
--
2.7.4
^ permalink raw reply related
* [PATCH 7/9] net: thunderx: Add support for XDP_TX
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Adds support for XDP_TX i.e transmits packet out of
the XDP TX queue mapped to the corresponding Rx queue
on which packet is received.
Since SQ for XDP TX will be used only on a single cpu i.e
SQ description creation and freeing, using atomic free count
is not necessary and will become a bottleneck. Hence added
a separate 'xdp_free_cnt' used for SQs designated for XDP
to track descriptor free count.
Changes also include
- A new entry 'xdp_page' is added to save transmitted packet's
page pointer for later cleanup.
- XDP Tx SQ's doorbell is ringed once per NAPI instance.
- Retrieving designated SQ for packets being sent out by stack
via 'nicvf_xmit'.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 63 ++++++++---
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 117 ++++++++++++++++++---
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 7 ++
3 files changed, 160 insertions(+), 27 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index a58cc1e..bb13dee 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -501,9 +501,8 @@ static int nicvf_init_resources(struct nicvf *nic)
return 0;
}
-static inline bool nicvf_xdp_rx(struct nicvf *nic,
- struct bpf_prog *prog,
- struct cqe_rx_t *cqe_rx)
+static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
+ struct cqe_rx_t *cqe_rx, struct snd_queue *sq)
{
struct xdp_buff xdp;
struct page *page;
@@ -528,9 +527,11 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic,
switch (action) {
case XDP_PASS:
- case XDP_TX:
- /* Pass on all packets to network stack */
+ /* Pass on packet to network stack */
return false;
+ case XDP_TX:
+ nicvf_xdp_sq_append_pkt(nic, sq, (u64)xdp.data, dma_addr, len);
+ return true;
default:
bpf_warn_invalid_xdp_action(action);
case XDP_ABORTED:
@@ -560,6 +561,7 @@ static void nicvf_snd_pkt_handler(struct net_device *netdev,
unsigned int *tx_pkts, unsigned int *tx_bytes)
{
struct sk_buff *skb = NULL;
+ struct page *page;
struct nicvf *nic = netdev_priv(netdev);
struct snd_queue *sq;
struct sq_hdr_subdesc *hdr;
@@ -575,6 +577,22 @@ static void nicvf_snd_pkt_handler(struct net_device *netdev,
if (cqe_tx->send_status)
nicvf_check_cqe_tx_errs(nic->pnicvf, cqe_tx);
+ /* Is this a XDP designated Tx queue */
+ if (sq->is_xdp) {
+ page = (struct page *)sq->xdp_page[cqe_tx->sqe_ptr];
+ /* Check if it's recycled page or else unmap DMA mapping */
+ if (page && (page_ref_count(page) == 1))
+ nicvf_unmap_sndq_buffers(nic, sq, cqe_tx->sqe_ptr,
+ hdr->subdesc_cnt);
+
+ /* Release page reference for recycling */
+ if (page)
+ put_page(page);
+ sq->xdp_page[cqe_tx->sqe_ptr] = (u64)NULL;
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
+ return;
+ }
+
skb = (struct sk_buff *)sq->skbuff[cqe_tx->sqe_ptr];
if (skb) {
/* Check for dummy descriptor used for HW TSO offload on 88xx */
@@ -634,7 +652,7 @@ static inline void nicvf_set_rxhash(struct net_device *netdev,
static void nicvf_rcv_pkt_handler(struct net_device *netdev,
struct napi_struct *napi,
- struct cqe_rx_t *cqe_rx)
+ struct cqe_rx_t *cqe_rx, struct snd_queue *sq)
{
struct sk_buff *skb;
struct nicvf *nic = netdev_priv(netdev);
@@ -659,7 +677,7 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
/* For XDP, ignore pkts spanning multiple pages */
if (nic->xdp_prog && (cqe_rx->rb_cnt == 1))
- if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx))
+ if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx, sq))
return;
skb = nicvf_get_rcv_skb(snic, cqe_rx, nic->xdp_prog ? true : false);
@@ -715,8 +733,8 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
struct cmp_queue *cq = &qs->cq[cq_idx];
struct cqe_rx_t *cq_desc;
struct netdev_queue *txq;
- struct snd_queue *sq;
- unsigned int tx_pkts = 0, tx_bytes = 0;
+ struct snd_queue *sq = &qs->sq[cq_idx];
+ unsigned int tx_pkts = 0, tx_bytes = 0, txq_idx;
spin_lock_bh(&cq->lock);
loop:
@@ -746,7 +764,7 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
switch (cq_desc->cqe_type) {
case CQE_TYPE_RX:
- nicvf_rcv_pkt_handler(netdev, napi, cq_desc);
+ nicvf_rcv_pkt_handler(netdev, napi, cq_desc, sq);
work_done++;
break;
case CQE_TYPE_SEND:
@@ -773,17 +791,26 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
goto loop;
done:
- sq = &nic->qs->sq[cq_idx];
/* Update SQ's descriptor free count */
if (subdesc_cnt)
nicvf_put_sq_desc(sq, subdesc_cnt);
+ txq_idx = nicvf_netdev_qidx(nic, cq_idx);
+ /* Handle XDP TX queues */
+ if (nic->pnicvf->xdp_prog) {
+ if (txq_idx < nic->pnicvf->xdp_tx_queues) {
+ nicvf_xdp_sq_doorbell(nic, sq, cq_idx);
+ goto out;
+ }
+ nic = nic->pnicvf;
+ txq_idx -= nic->pnicvf->xdp_tx_queues;
+ }
+
/* Wakeup TXQ if its stopped earlier due to SQ full */
if (tx_done ||
(atomic_read(&sq->free_cnt) >= MIN_SQ_DESC_PER_PKT_XMIT)) {
netdev = nic->pnicvf->netdev;
- txq = netdev_get_tx_queue(netdev,
- nicvf_netdev_qidx(nic, cq_idx));
+ txq = netdev_get_tx_queue(netdev, txq_idx);
if (tx_pkts)
netdev_tx_completed_queue(txq, tx_pkts, tx_bytes);
@@ -796,10 +823,11 @@ static int nicvf_cq_intr_handler(struct net_device *netdev, u8 cq_idx,
if (netif_msg_tx_err(nic))
netdev_warn(netdev,
"%s: Transmit queue wakeup SQ%d\n",
- netdev->name, cq_idx);
+ netdev->name, txq_idx);
}
}
+out:
spin_unlock_bh(&cq->lock);
return work_done;
}
@@ -1115,6 +1143,13 @@ static netdev_tx_t nicvf_xmit(struct sk_buff *skb, struct net_device *netdev)
return NETDEV_TX_OK;
}
+ /* In XDP case, initial HW tx queues are used for XDP,
+ * but stack's queue mapping starts at '0', so skip the
+ * Tx queues attached to Rx queues for XDP.
+ */
+ if (nic->xdp_prog)
+ qid += nic->xdp_tx_queues;
+
snic = nic;
/* Get secondary Qset's SQ structure */
if (qid >= MAX_SND_QUEUES_PER_QS) {
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 5009f49..ec234b6 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -19,6 +19,8 @@
#include "q_struct.h"
#include "nicvf_queues.h"
+static inline void nicvf_sq_add_gather_subdesc(struct snd_queue *sq, int qentry,
+ int size, u64 data);
static void nicvf_get_page(struct nicvf *nic)
{
if (!nic->rb_pageref || !nic->rb_page)
@@ -456,7 +458,7 @@ static void nicvf_free_cmp_queue(struct nicvf *nic, struct cmp_queue *cq)
/* Initialize transmit queue */
static int nicvf_init_snd_queue(struct nicvf *nic,
- struct snd_queue *sq, int q_len)
+ struct snd_queue *sq, int q_len, int qidx)
{
int err;
@@ -469,17 +471,38 @@ static int nicvf_init_snd_queue(struct nicvf *nic,
sq->skbuff = kcalloc(q_len, sizeof(u64), GFP_KERNEL);
if (!sq->skbuff)
return -ENOMEM;
+
sq->head = 0;
sq->tail = 0;
- atomic_set(&sq->free_cnt, q_len - 1);
sq->thresh = SND_QUEUE_THRESH;
- /* Preallocate memory for TSO segment's header */
- sq->tso_hdrs = dma_alloc_coherent(&nic->pdev->dev,
- q_len * TSO_HEADER_SIZE,
- &sq->tso_hdrs_phys, GFP_KERNEL);
- if (!sq->tso_hdrs)
- return -ENOMEM;
+ /* Check if this SQ is a XDP TX queue */
+ if (nic->sqs_mode)
+ qidx += ((nic->sqs_id + 1) * MAX_SND_QUEUES_PER_QS);
+ if (qidx < nic->pnicvf->xdp_tx_queues) {
+ /* Alloc memory to save page pointers for XDP_TX */
+ sq->xdp_page = kcalloc(q_len, sizeof(u64), GFP_KERNEL);
+ if (!sq->xdp_page)
+ return -ENOMEM;
+ sq->xdp_desc_cnt = 0;
+ sq->xdp_free_cnt = q_len - 1;
+ sq->is_xdp = true;
+ } else {
+ sq->xdp_page = NULL;
+ sq->xdp_desc_cnt = 0;
+ sq->xdp_free_cnt = 0;
+ sq->is_xdp = false;
+
+ atomic_set(&sq->free_cnt, q_len - 1);
+
+ /* Preallocate memory for TSO segment's header */
+ sq->tso_hdrs = dma_alloc_coherent(&nic->pdev->dev,
+ q_len * TSO_HEADER_SIZE,
+ &sq->tso_hdrs_phys,
+ GFP_KERNEL);
+ if (!sq->tso_hdrs)
+ return -ENOMEM;
+ }
return 0;
}
@@ -505,6 +528,7 @@ void nicvf_unmap_sndq_buffers(struct nicvf *nic, struct snd_queue *sq,
static void nicvf_free_snd_queue(struct nicvf *nic, struct snd_queue *sq)
{
struct sk_buff *skb;
+ struct page *page;
struct sq_hdr_subdesc *hdr;
struct sq_hdr_subdesc *tso_sqe;
@@ -522,8 +546,15 @@ static void nicvf_free_snd_queue(struct nicvf *nic, struct snd_queue *sq)
smp_rmb();
while (sq->head != sq->tail) {
skb = (struct sk_buff *)sq->skbuff[sq->head];
- if (!skb)
+ if (!skb || !sq->xdp_page)
+ goto next;
+
+ page = (struct page *)sq->xdp_page[sq->head];
+ if (!page)
goto next;
+ else
+ put_page(page);
+
hdr = (struct sq_hdr_subdesc *)GET_SQ_DESC(sq, sq->head);
/* Check for dummy descriptor used for HW TSO offload on 88xx */
if (hdr->dont_send) {
@@ -536,12 +567,14 @@ static void nicvf_free_snd_queue(struct nicvf *nic, struct snd_queue *sq)
nicvf_unmap_sndq_buffers(nic, sq, sq->head,
hdr->subdesc_cnt);
}
- dev_kfree_skb_any(skb);
+ if (skb)
+ dev_kfree_skb_any(skb);
next:
sq->head++;
sq->head &= (sq->dmem.q_len - 1);
}
kfree(sq->skbuff);
+ kfree(sq->xdp_page);
nicvf_free_q_desc_mem(nic, &sq->dmem);
}
@@ -932,7 +965,7 @@ static int nicvf_alloc_resources(struct nicvf *nic)
/* Alloc send queue */
for (qidx = 0; qidx < qs->sq_cnt; qidx++) {
- if (nicvf_init_snd_queue(nic, &qs->sq[qidx], qs->sq_len))
+ if (nicvf_init_snd_queue(nic, &qs->sq[qidx], qs->sq_len, qidx))
goto alloc_fail;
}
@@ -1035,7 +1068,10 @@ static inline int nicvf_get_sq_desc(struct snd_queue *sq, int desc_cnt)
int qentry;
qentry = sq->tail;
- atomic_sub(desc_cnt, &sq->free_cnt);
+ if (!sq->is_xdp)
+ atomic_sub(desc_cnt, &sq->free_cnt);
+ else
+ sq->xdp_free_cnt -= desc_cnt;
sq->tail += desc_cnt;
sq->tail &= (sq->dmem.q_len - 1);
@@ -1053,7 +1089,10 @@ static inline void nicvf_rollback_sq_desc(struct snd_queue *sq,
/* Free descriptor back to SQ for future use */
void nicvf_put_sq_desc(struct snd_queue *sq, int desc_cnt)
{
- atomic_add(desc_cnt, &sq->free_cnt);
+ if (!sq->is_xdp)
+ atomic_add(desc_cnt, &sq->free_cnt);
+ else
+ sq->xdp_free_cnt += desc_cnt;
sq->head += desc_cnt;
sq->head &= (sq->dmem.q_len - 1);
}
@@ -1111,6 +1150,58 @@ void nicvf_sq_free_used_descs(struct net_device *netdev, struct snd_queue *sq,
}
}
+/* XDP Transmit APIs */
+void nicvf_xdp_sq_doorbell(struct nicvf *nic,
+ struct snd_queue *sq, int sq_num)
+{
+ if (!sq->xdp_desc_cnt)
+ return;
+
+ /* make sure all memory stores are done before ringing doorbell */
+ wmb();
+
+ /* Inform HW to xmit all TSO segments */
+ nicvf_queue_reg_write(nic, NIC_QSET_SQ_0_7_DOOR,
+ sq_num, sq->xdp_desc_cnt);
+ sq->xdp_desc_cnt = 0;
+}
+
+static inline void
+nicvf_xdp_sq_add_hdr_subdesc(struct snd_queue *sq, int qentry,
+ int subdesc_cnt, u64 data, int len)
+{
+ struct sq_hdr_subdesc *hdr;
+
+ hdr = (struct sq_hdr_subdesc *)GET_SQ_DESC(sq, qentry);
+ memset(hdr, 0, SND_QUEUE_DESC_SIZE);
+ hdr->subdesc_type = SQ_DESC_TYPE_HEADER;
+ hdr->subdesc_cnt = subdesc_cnt;
+ hdr->tot_len = len;
+ hdr->post_cqe = 1;
+ sq->xdp_page[qentry] = (u64)virt_to_page((void *)data);
+}
+
+int nicvf_xdp_sq_append_pkt(struct nicvf *nic, struct snd_queue *sq,
+ u64 bufaddr, u64 dma_addr, u16 len)
+{
+ int subdesc_cnt = MIN_SQ_DESC_PER_PKT_XMIT;
+ int qentry;
+
+ if (subdesc_cnt > sq->xdp_free_cnt)
+ return 0;
+
+ qentry = nicvf_get_sq_desc(sq, subdesc_cnt);
+
+ nicvf_xdp_sq_add_hdr_subdesc(sq, qentry, subdesc_cnt - 1, bufaddr, len);
+
+ qentry = nicvf_get_nxt_sqentry(sq, qentry);
+ nicvf_sq_add_gather_subdesc(sq, qentry, len, dma_addr);
+
+ sq->xdp_desc_cnt += subdesc_cnt;
+
+ return 1;
+}
+
/* Calculate no of SQ subdescriptors needed to transmit all
* segments of this TSO packet.
* Taken from 'Tilera network driver' with a minor modification.
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index db04c0e..a07d5b4 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -271,6 +271,10 @@ struct snd_queue {
u32 tail;
u64 *skbuff;
void *desc;
+ u64 *xdp_page;
+ u16 xdp_desc_cnt;
+ u16 xdp_free_cnt;
+ bool is_xdp;
#define TSO_HEADER_SIZE 128
/* For TSO segment's header */
@@ -339,6 +343,9 @@ void nicvf_sq_free_used_descs(struct net_device *netdev,
struct snd_queue *sq, int qidx);
int nicvf_sq_append_skb(struct nicvf *nic, struct snd_queue *sq,
struct sk_buff *skb, u8 sq_num);
+int nicvf_xdp_sq_append_pkt(struct nicvf *nic, struct snd_queue *sq,
+ u64 bufaddr, u64 dma_addr, u16 len);
+void nicvf_xdp_sq_doorbell(struct nicvf *nic, struct snd_queue *sq, int sq_num);
struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic,
struct cqe_rx_t *cqe_rx, bool xdp);
--
2.7.4
^ permalink raw reply related
* [PATCH 5/9] net: thunderx: Add basic XDP support
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: Sunil Goutham, linux-kernel, linux-arm-kernel
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Adds basic XDP support i.e attaching a BPF program to an
interface. Also takes care of allocating separate Tx queues
for XDP path and for network stack packet transmission.
This patch doesn't support handling of any of the XDP actions,
all are treated as XDP_PASS i.e packets will be handed over to
the network stack.
Changes also involve allocating one receive buffer per page in XDP
mode and multiple in normal mode i.e when no BPF program is attached.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nic.h | 6 +-
.../net/ethernet/cavium/thunder/nicvf_ethtool.c | 26 +++-
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 162 ++++++++++++++++++++-
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 15 +-
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 9 ++
5 files changed, 199 insertions(+), 19 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h
index dca6aed..4a02e61 100644
--- a/drivers/net/ethernet/cavium/thunder/nic.h
+++ b/drivers/net/ethernet/cavium/thunder/nic.h
@@ -268,9 +268,9 @@ struct nicvf {
struct net_device *netdev;
struct pci_dev *pdev;
void __iomem *reg_base;
+ struct bpf_prog *xdp_prog;
#define MAX_QUEUES_PER_QSET 8
struct queue_set *qs;
- struct nicvf_cq_poll *napi[8];
void *iommu_domain;
u8 vf_id;
u8 sqs_id;
@@ -296,6 +296,7 @@ struct nicvf {
/* Queue count */
u8 rx_queues;
u8 tx_queues;
+ u8 xdp_tx_queues;
u8 max_queues;
u8 node;
@@ -320,6 +321,9 @@ struct nicvf {
struct nicvf_drv_stats __percpu *drv_stats;
struct bgx_stats bgx_stats;
+ /* Napi */
+ struct nicvf_cq_poll *napi[8];
+
/* MSI-X */
u8 num_vec;
char irq_name[NIC_VF_MSIX_VECTORS][IFNAMSIZ + 15];
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
index a89db5f..b9ece9c 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
@@ -721,7 +721,7 @@ static int nicvf_set_channels(struct net_device *dev,
struct nicvf *nic = netdev_priv(dev);
int err = 0;
bool if_up = netif_running(dev);
- int cqcount;
+ u8 cqcount, txq_count;
if (!channel->rx_count || !channel->tx_count)
return -EINVAL;
@@ -730,10 +730,26 @@ static int nicvf_set_channels(struct net_device *dev,
if (channel->tx_count > nic->max_queues)
return -EINVAL;
+ if (nic->xdp_prog &&
+ ((channel->tx_count + channel->rx_count) > nic->max_queues)) {
+ netdev_err(nic->netdev,
+ "XDP mode, RXQs + TXQs > Max %d\n",
+ nic->max_queues);
+ return -EINVAL;
+ }
+
if (if_up)
nicvf_stop(dev);
- cqcount = max(channel->rx_count, channel->tx_count);
+ nic->rx_queues = channel->rx_count;
+ nic->tx_queues = channel->tx_count;
+ if (!nic->xdp_prog)
+ nic->xdp_tx_queues = 0;
+ else
+ nic->xdp_tx_queues = channel->rx_count;
+
+ txq_count = nic->xdp_tx_queues + nic->tx_queues;
+ cqcount = max(nic->rx_queues, txq_count);
if (cqcount > MAX_CMP_QUEUES_PER_QS) {
nic->sqs_count = roundup(cqcount, MAX_CMP_QUEUES_PER_QS);
@@ -742,12 +758,10 @@ static int nicvf_set_channels(struct net_device *dev,
nic->sqs_count = 0;
}
- nic->qs->rq_cnt = min_t(u32, channel->rx_count, MAX_RCV_QUEUES_PER_QS);
- nic->qs->sq_cnt = min_t(u32, channel->tx_count, MAX_SND_QUEUES_PER_QS);
+ nic->qs->rq_cnt = min_t(u8, nic->rx_queues, MAX_RCV_QUEUES_PER_QS);
+ nic->qs->sq_cnt = min_t(u8, txq_count, MAX_SND_QUEUES_PER_QS);
nic->qs->cq_cnt = max(nic->qs->rq_cnt, nic->qs->sq_cnt);
- nic->rx_queues = channel->rx_count;
- nic->tx_queues = channel->tx_count;
err = nicvf_set_real_num_queues(dev, nic->tx_queues, nic->rx_queues);
if (err)
return err;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index 0d79894..9c48873 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -17,6 +17,8 @@
#include <linux/prefetch.h>
#include <linux/irq.h>
#include <linux/iommu.h>
+#include <linux/bpf.h>
+#include <linux/filter.h>
#include "nic_reg.h"
#include "nic.h"
@@ -397,8 +399,10 @@ static void nicvf_request_sqs(struct nicvf *nic)
if (nic->rx_queues > MAX_RCV_QUEUES_PER_QS)
rx_queues = nic->rx_queues - MAX_RCV_QUEUES_PER_QS;
- if (nic->tx_queues > MAX_SND_QUEUES_PER_QS)
- tx_queues = nic->tx_queues - MAX_SND_QUEUES_PER_QS;
+
+ tx_queues = nic->tx_queues + nic->xdp_tx_queues;
+ if (tx_queues > MAX_SND_QUEUES_PER_QS)
+ tx_queues = tx_queues - MAX_SND_QUEUES_PER_QS;
/* Set no of Rx/Tx queues in each of the SQsets */
for (sqs = 0; sqs < nic->sqs_count; sqs++) {
@@ -496,6 +500,43 @@ static int nicvf_init_resources(struct nicvf *nic)
return 0;
}
+static inline bool nicvf_xdp_rx(struct nicvf *nic,
+ struct bpf_prog *prog,
+ struct cqe_rx_t *cqe_rx)
+{
+ struct xdp_buff xdp;
+ u32 action;
+ u16 len;
+ u64 dma_addr, cpu_addr;
+
+ /* Retrieve packet buffer's DMA address and length */
+ len = *((u16 *)((void *)cqe_rx + (3 * sizeof(u64))));
+ dma_addr = *((u64 *)((void *)cqe_rx + (7 * sizeof(u64))));
+
+ cpu_addr = nicvf_iova_to_phys(nic, dma_addr);
+ if (!cpu_addr)
+ return false;
+
+ xdp.data = phys_to_virt(cpu_addr);
+ xdp.data_end = xdp.data + len;
+
+ rcu_read_lock();
+ action = bpf_prog_run_xdp(prog, &xdp);
+ rcu_read_unlock();
+
+ switch (action) {
+ case XDP_PASS:
+ case XDP_TX:
+ case XDP_ABORTED:
+ case XDP_DROP:
+ /* Pass on all packets to network stack */
+ return false;
+ default:
+ bpf_warn_invalid_xdp_action(action);
+ }
+ return false;
+}
+
static void nicvf_snd_pkt_handler(struct net_device *netdev,
struct cqe_send_t *cqe_tx,
int budget, int *subdesc_cnt,
@@ -599,6 +640,11 @@ static void nicvf_rcv_pkt_handler(struct net_device *netdev,
return;
}
+ /* For XDP, ignore pkts spanning multiple pages */
+ if (nic->xdp_prog && (cqe_rx->rb_cnt == 1))
+ if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx))
+ return;
+
skb = nicvf_get_rcv_skb(snic, cqe_rx);
if (!skb) {
netdev_dbg(nic->netdev, "Packet not received\n");
@@ -1529,6 +1575,117 @@ static int nicvf_set_features(struct net_device *netdev,
return 0;
}
+static void nicvf_set_xdp_queues(struct nicvf *nic, bool bpf_attached)
+{
+ u8 cq_count, txq_count;
+
+ /* Set XDP Tx queue count same as Rx queue count */
+ if (!bpf_attached)
+ nic->xdp_tx_queues = 0;
+ else
+ nic->xdp_tx_queues = nic->rx_queues;
+
+ /* If queue count > MAX_CMP_QUEUES_PER_QS, then additional qsets
+ * needs to be allocated, check how many.
+ */
+ txq_count = nic->xdp_tx_queues + nic->tx_queues;
+ cq_count = max(nic->rx_queues, txq_count);
+ if (cq_count > MAX_CMP_QUEUES_PER_QS) {
+ nic->sqs_count = roundup(cq_count, MAX_CMP_QUEUES_PER_QS);
+ nic->sqs_count = (nic->sqs_count / MAX_CMP_QUEUES_PER_QS) - 1;
+ } else {
+ nic->sqs_count = 0;
+ }
+
+ /* Set primary Qset's resources */
+ nic->qs->rq_cnt = min_t(u8, nic->rx_queues, MAX_RCV_QUEUES_PER_QS);
+ nic->qs->sq_cnt = min_t(u8, txq_count, MAX_SND_QUEUES_PER_QS);
+ nic->qs->cq_cnt = max_t(u8, nic->qs->rq_cnt, nic->qs->sq_cnt);
+
+ /* Update stack */
+ nicvf_set_real_num_queues(nic->netdev, nic->tx_queues, nic->rx_queues);
+}
+
+static int nicvf_xdp_setup(struct nicvf *nic, struct bpf_prog *prog)
+{
+ struct net_device *dev = nic->netdev;
+ bool if_up = netif_running(nic->netdev);
+ struct bpf_prog *old_prog;
+ bool bpf_attached = false;
+
+ /* For now just support only the usual MTU sized frames */
+ if (prog && (dev->mtu > 1500)) {
+ netdev_warn(dev, "Jumbo frames not yet supported with XDP, current MTU %d.\n",
+ dev->mtu);
+ return -EOPNOTSUPP;
+ }
+
+ if (prog && prog->xdp_adjust_head)
+ return -EOPNOTSUPP;
+
+ /* ALL SQs attached to CQs i.e same as RQs, are treated as
+ * XDP Tx queues and more Tx queues are allocated for
+ * network stack to send pkts out.
+ *
+ * No of Tx queues are either same as Rx queues or whatever
+ * is left in max no of queues possible.
+ */
+ if ((nic->rx_queues + nic->tx_queues) > nic->max_queues) {
+ netdev_warn(dev,
+ "Failed to attach BPF prog, RXQs + TXQs > Max %d\n",
+ nic->max_queues);
+ return -ENOMEM;
+ }
+
+ if (if_up)
+ nicvf_stop(nic->netdev);
+
+ old_prog = xchg(&nic->xdp_prog, prog);
+ /* Detach old prog, if any */
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ if (nic->xdp_prog) {
+ /* Attach BPF program */
+ nic->xdp_prog = bpf_prog_add(nic->xdp_prog, nic->rx_queues - 1);
+ if (!IS_ERR(nic->xdp_prog))
+ bpf_attached = true;
+ }
+
+ /* Calculate Tx queues needed for XDP and network stack */
+ nicvf_set_xdp_queues(nic, bpf_attached);
+
+ if (if_up) {
+ /* Reinitialize interface, clean slate */
+ nicvf_open(nic->netdev);
+ netif_trans_update(nic->netdev);
+ }
+
+ return 0;
+}
+
+static int nicvf_xdp(struct net_device *netdev, struct netdev_xdp *xdp)
+{
+ struct nicvf *nic = netdev_priv(netdev);
+
+ /* To avoid checks while retrieving buffer address from CQE_RX,
+ * do not support XDP for T88 pass1.x silicons which are anyway
+ * not in use widely.
+ */
+ if (pass1_silicon(nic->pdev))
+ return -EOPNOTSUPP;
+
+ switch (xdp->command) {
+ case XDP_SETUP_PROG:
+ return nicvf_xdp_setup(nic, xdp->prog);
+ case XDP_QUERY_PROG:
+ xdp->prog_attached = !!nic->xdp_prog;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
static const struct net_device_ops nicvf_netdev_ops = {
.ndo_open = nicvf_open,
.ndo_stop = nicvf_stop,
@@ -1539,6 +1696,7 @@ static const struct net_device_ops nicvf_netdev_ops = {
.ndo_tx_timeout = nicvf_tx_timeout,
.ndo_fix_features = nicvf_fix_features,
.ndo_set_features = nicvf_set_features,
+ .ndo_xdp = nicvf_xdp,
};
static int nicvf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index e4a02a9..8c3c571 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -19,14 +19,6 @@
#include "q_struct.h"
#include "nicvf_queues.h"
-static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr)
-{
- /* Translation is installed only when IOMMU is present */
- if (nic->iommu_domain)
- return iommu_iova_to_phys(nic->iommu_domain, dma_addr);
- return dma_addr;
-}
-
static void nicvf_get_page(struct nicvf *nic)
{
if (!nic->rb_pageref || !nic->rb_page)
@@ -149,8 +141,10 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
{
struct pgcache *pgcache = NULL;
- /* Check if request can be accomodated in previous allocated page */
- if (nic->rb_page &&
+ /* Check if request can be accomodated in previous allocated page.
+ * But in XDP mode only one buffer per page is permitted.
+ */
+ if (!nic->pnicvf->xdp_prog && nic->rb_page &&
((nic->rb_page_offset + buf_len) <= PAGE_SIZE)) {
nic->rb_pageref++;
goto ret;
@@ -961,6 +955,7 @@ int nicvf_set_qset_resources(struct nicvf *nic)
nic->rx_queues = qs->rq_cnt;
nic->tx_queues = qs->sq_cnt;
+ nic->xdp_tx_queues = 0;
return 0;
}
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index da48366..07136a2 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -10,6 +10,7 @@
#define NICVF_QUEUES_H
#include <linux/netdevice.h>
+#include <linux/iommu.h>
#include "q_struct.h"
#define MAX_QUEUE_SET 128
@@ -312,6 +313,14 @@ struct queue_set {
#define CQ_ERR_MASK (CQ_WR_FULL | CQ_WR_DISABLE | CQ_WR_FAULT)
+static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr)
+{
+ /* Translation is installed only when IOMMU is present */
+ if (nic->iommu_domain)
+ return iommu_iova_to_phys(nic->iommu_domain, dma_addr);
+ return dma_addr;
+}
+
void nicvf_unmap_sndq_buffers(struct nicvf *nic, struct snd_queue *sq,
int hdr_sqe, u8 subdesc_cnt);
void nicvf_config_vlan_stripping(struct nicvf *nic,
--
2.7.4
^ permalink raw reply related
* [PATCH 4/9] net: thunderx: Cleanup receive buffer allocation
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Get rid of unnecessary double pointer references and type casting
in receive buffer allocation code.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 90c5bc7d..e4a02a9 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -145,7 +145,7 @@ static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
/* Allocate buffer for packet reception */
static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
- gfp_t gfp, u32 buf_len, u64 **rbuf)
+ gfp_t gfp, u32 buf_len, u64 *rbuf)
{
struct pgcache *pgcache = NULL;
@@ -172,10 +172,10 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
nic->rb_page = pgcache->page;
ret:
/* HW will ensure data coherency, CPU sync not required */
- *rbuf = (u64 *)((u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
- nic->rb_page_offset, buf_len,
- DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC));
+ *rbuf = (u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
+ nic->rb_page_offset, buf_len,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
if (!nic->rb_page_offset)
__free_pages(nic->rb_page, 0);
@@ -212,7 +212,7 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr,
int ring_len, int buf_size)
{
int idx;
- u64 *rbuf;
+ u64 rbuf;
struct rbdr_entry_t *desc;
int err;
@@ -257,7 +257,7 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr,
}
desc = GET_RBDR_DESC(rbdr, idx);
- desc->buf_addr = (u64)rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
+ desc->buf_addr = rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
}
nicvf_get_page(nic);
@@ -330,7 +330,7 @@ static void nicvf_refill_rbdr(struct nicvf *nic, gfp_t gfp)
int refill_rb_cnt;
struct rbdr *rbdr;
struct rbdr_entry_t *desc;
- u64 *rbuf;
+ u64 rbuf;
int new_rb = 0;
refill:
@@ -364,7 +364,7 @@ static void nicvf_refill_rbdr(struct nicvf *nic, gfp_t gfp)
break;
desc = GET_RBDR_DESC(rbdr, tail);
- desc->buf_addr = (u64)rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
+ desc->buf_addr = rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
refill_rb_cnt--;
new_rb++;
}
--
2.7.4
^ permalink raw reply related
* [PATCH 2/9] net: thunderx: Optimize RBDR descriptor handling
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: Sunil Goutham, linux-kernel, linux-arm-kernel
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Receive buffer's physical address or iova will anyway not
go beyond 49bits, since it is the max supported HW address.
As per perf, updating bitfields i.e buf_addr:42 in RBDR
descriptor entry consumes lots of cpu cycles, hence changed
it to a 64bit field with alignment requirements taken care of.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 8 ++++----
drivers/net/ethernet/cavium/thunder/q_struct.h | 10 +---------
2 files changed, 5 insertions(+), 13 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 12f9709..dfc85a1 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -257,7 +257,7 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr,
}
desc = GET_RBDR_DESC(rbdr, idx);
- desc->buf_addr = (u64)rbuf >> NICVF_RCV_BUF_ALIGN;
+ desc->buf_addr = (u64)rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
}
nicvf_get_page(nic);
@@ -286,7 +286,7 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr)
/* Release page references */
while (head != tail) {
desc = GET_RBDR_DESC(rbdr, head);
- buf_addr = ((u64)desc->buf_addr) << NICVF_RCV_BUF_ALIGN;
+ buf_addr = desc->buf_addr;
phys_addr = nicvf_iova_to_phys(nic, buf_addr);
dma_unmap_page_attrs(&nic->pdev->dev, buf_addr, RCV_FRAG_LEN,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
@@ -297,7 +297,7 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr)
}
/* Release buffer of tail desc */
desc = GET_RBDR_DESC(rbdr, tail);
- buf_addr = ((u64)desc->buf_addr) << NICVF_RCV_BUF_ALIGN;
+ buf_addr = desc->buf_addr;
phys_addr = nicvf_iova_to_phys(nic, buf_addr);
dma_unmap_page_attrs(&nic->pdev->dev, buf_addr, RCV_FRAG_LEN,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
@@ -364,7 +364,7 @@ static void nicvf_refill_rbdr(struct nicvf *nic, gfp_t gfp)
break;
desc = GET_RBDR_DESC(rbdr, tail);
- desc->buf_addr = (u64)rbuf >> NICVF_RCV_BUF_ALIGN;
+ desc->buf_addr = (u64)rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
refill_rb_cnt--;
new_rb++;
}
diff --git a/drivers/net/ethernet/cavium/thunder/q_struct.h b/drivers/net/ethernet/cavium/thunder/q_struct.h
index f363472..e47205a 100644
--- a/drivers/net/ethernet/cavium/thunder/q_struct.h
+++ b/drivers/net/ethernet/cavium/thunder/q_struct.h
@@ -359,15 +359,7 @@ union cq_desc_t {
};
struct rbdr_entry_t {
-#if defined(__BIG_ENDIAN_BITFIELD)
- u64 rsvd0:15;
- u64 buf_addr:42;
- u64 cache_align:7;
-#elif defined(__LITTLE_ENDIAN_BITFIELD)
- u64 cache_align:7;
- u64 buf_addr:42;
- u64 rsvd0:15;
-#endif
+ u64 buf_addr;
};
/* TCP reassembly context */
--
2.7.4
^ permalink raw reply related
* [PATCH 1/9] net: thunderx: Support for page recycling
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com>
From: Sunil Goutham <sgoutham@cavium.com>
Adds support for page recycling for allocating receive buffers
to reduce cost of refilling RBDR ring. Also got rid of using
compound pages when pagesize is 4K, only order-0 pages now.
Only page is recycled, DMA mappings still needs to be done for
every receive buffer allocated due to following constraints
- Cannot have just one receive buffer per 64KB page.
- There is just one buffer ring shared across 8 Rx queues, so
buffers of same page can go to any Rx queue.
- HW gives buffer address where packet has been DMA'ed and not
the index into buffer ring.
This makes it not possible to resue DMA mapping info. So unfortunately
have to go through costly mapping route for every buffer.
Signed-off-by: Sunil Goutham <sgoutham@cavium.com>
---
drivers/net/ethernet/cavium/thunder/nic.h | 4 +-
.../net/ethernet/cavium/thunder/nicvf_ethtool.c | 3 +-
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 121 ++++++++++++++++++---
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 11 ++
4 files changed, 119 insertions(+), 20 deletions(-)
diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h
index 6fb4421..dca6aed 100644
--- a/drivers/net/ethernet/cavium/thunder/nic.h
+++ b/drivers/net/ethernet/cavium/thunder/nic.h
@@ -252,12 +252,14 @@ struct nicvf_drv_stats {
u64 tx_csum_overflow;
/* driver debug stats */
- u64 rcv_buffer_alloc_failures;
u64 tx_tso;
u64 tx_timeout;
u64 txq_stop;
u64 txq_wake;
+ u64 rcv_buffer_alloc_failures;
+ u64 page_alloc;
+
struct u64_stats_sync syncp;
};
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
index 02a986c..a89db5f 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
@@ -100,11 +100,12 @@ static const struct nicvf_stat nicvf_drv_stats[] = {
NICVF_DRV_STAT(tx_csum_overlap),
NICVF_DRV_STAT(tx_csum_overflow),
- NICVF_DRV_STAT(rcv_buffer_alloc_failures),
NICVF_DRV_STAT(tx_tso),
NICVF_DRV_STAT(tx_timeout),
NICVF_DRV_STAT(txq_stop),
NICVF_DRV_STAT(txq_wake),
+ NICVF_DRV_STAT(rcv_buffer_alloc_failures),
+ NICVF_DRV_STAT(page_alloc),
};
static const struct nicvf_stat nicvf_queue_stats[] = {
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 7b0fd8d..12f9709 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -19,8 +19,6 @@
#include "q_struct.h"
#include "nicvf_queues.h"
-#define NICVF_PAGE_ORDER ((PAGE_SIZE <= 4096) ? PAGE_ALLOC_COSTLY_ORDER : 0)
-
static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr)
{
/* Translation is installed only when IOMMU is present */
@@ -90,33 +88,88 @@ static void nicvf_free_q_desc_mem(struct nicvf *nic, struct q_desc_mem *dmem)
dmem->base = NULL;
}
-/* Allocate buffer for packet reception
- * HW returns memory address where packet is DMA'ed but not a pointer
- * into RBDR ring, so save buffer address at the start of fragment and
- * align the start address to a cache aligned address
+/* Allocate a new page or recycle one if possible
+ *
+ * We cannot optimize dma mapping here, since
+ * 1. It's only one RBDR ring for 8 Rx queues.
+ * 2. CQE_RX gives address of the buffer where pkt has been DMA'ed
+ * and not idx into RBDR ring, so can't refer to saved info.
+ * 3. There are multiple receive buffers per page
*/
-static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp,
- u32 buf_len, u64 **rbuf)
+static struct pgcache *nicvf_alloc_page(struct nicvf *nic,
+ struct rbdr *rbdr, gfp_t gfp)
{
- int order = NICVF_PAGE_ORDER;
+ struct page *page = NULL;
+ struct pgcache *pgcache, *next;
+
+ /* Check if page is already allocated */
+ pgcache = &rbdr->pgcache[rbdr->pgidx];
+ page = pgcache->page;
+ /* Check if page can be recycled */
+ if (page && (page_ref_count(page) != 1))
+ page = NULL;
+
+ if (!page) {
+ page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN, 0);
+ if (!page)
+ return NULL;
+
+ this_cpu_inc(nic->pnicvf->drv_stats->page_alloc);
+
+ /* Check for space */
+ if (rbdr->pgalloc >= rbdr->pgcnt) {
+ /* Page can still be used */
+ nic->rb_page = page;
+ return NULL;
+ }
+
+ /* Save the page in page cache */
+ pgcache->page = page;
+ rbdr->pgalloc++;
+ }
+
+ /* Take extra page reference for recycling */
+ page_ref_add(page, 1);
+
+ rbdr->pgidx++;
+ rbdr->pgidx &= (rbdr->pgcnt - 1);
+
+ /* Prefetch refcount of next page in page cache */
+ next = &rbdr->pgcache[rbdr->pgidx];
+ page = next->page;
+ if (page)
+ prefetch(&page->_refcount);
+
+ return pgcache;
+}
+
+/* Allocate buffer for packet reception */
+static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
+ gfp_t gfp, u32 buf_len, u64 **rbuf)
+{
+ struct pgcache *pgcache = NULL;
/* Check if request can be accomodated in previous allocated page */
if (nic->rb_page &&
- ((nic->rb_page_offset + buf_len) < (PAGE_SIZE << order))) {
+ ((nic->rb_page_offset + buf_len) <= PAGE_SIZE)) {
nic->rb_pageref++;
goto ret;
}
nicvf_get_page(nic);
+ nic->rb_page = NULL;
- /* Allocate a new page */
- nic->rb_page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN,
- order);
- if (!nic->rb_page) {
+ /* Get new page, either recycled or new one */
+ pgcache = nicvf_alloc_page(nic, rbdr, gfp);
+ if (!pgcache && !nic->rb_page) {
this_cpu_inc(nic->pnicvf->drv_stats->rcv_buffer_alloc_failures);
return -ENOMEM;
}
+
nic->rb_page_offset = 0;
+ /* Check if it's recycled */
+ if (pgcache)
+ nic->rb_page = pgcache->page;
ret:
/* HW will ensure data coherency, CPU sync not required */
*rbuf = (u64 *)((u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
@@ -125,7 +178,7 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp,
DMA_ATTR_SKIP_CPU_SYNC));
if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
if (!nic->rb_page_offset)
- __free_pages(nic->rb_page, order);
+ __free_pages(nic->rb_page, 0);
nic->rb_page = NULL;
return -ENOMEM;
}
@@ -177,10 +230,26 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr,
rbdr->head = 0;
rbdr->tail = 0;
+ /* Initialize page recycling stuff.
+ *
+ * Can't use single buffer per page especially with 64K pages.
+ * On embedded platforms i.e 81xx/83xx available memory itself
+ * is low and minimum ring size of RBDR is 8K, that takes away
+ * lots of memory.
+ */
+ rbdr->pgcnt = ring_len / (PAGE_SIZE / buf_size);
+ rbdr->pgcnt = roundup_pow_of_two(rbdr->pgcnt);
+ rbdr->pgcache = kzalloc(sizeof(*rbdr->pgcache) *
+ rbdr->pgcnt, GFP_KERNEL);
+ if (!rbdr->pgcache)
+ return -ENOMEM;
+ rbdr->pgidx = 0;
+ rbdr->pgalloc = 0;
+
nic->rb_page = NULL;
for (idx = 0; idx < ring_len; idx++) {
- err = nicvf_alloc_rcv_buffer(nic, GFP_KERNEL, RCV_FRAG_LEN,
- &rbuf);
+ err = nicvf_alloc_rcv_buffer(nic, rbdr, GFP_KERNEL,
+ RCV_FRAG_LEN, &rbuf);
if (err) {
/* To free already allocated and mapped ones */
rbdr->tail = idx - 1;
@@ -201,6 +270,7 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr)
{
int head, tail;
u64 buf_addr, phys_addr;
+ struct pgcache *pgcache;
struct rbdr_entry_t *desc;
if (!rbdr)
@@ -234,6 +304,18 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr)
if (phys_addr)
put_page(virt_to_page(phys_to_virt(phys_addr)));
+ /* Sync page cache info */
+ smp_rmb();
+
+ /* Release additional page references held for recycling */
+ head = 0;
+ while (head < rbdr->pgcnt) {
+ pgcache = &rbdr->pgcache[head];
+ if (pgcache->page && page_ref_count(pgcache->page) != 0)
+ put_page(pgcache->page);
+ head++;
+ }
+
/* Free RBDR ring */
nicvf_free_q_desc_mem(nic, &rbdr->dmem);
}
@@ -269,13 +351,16 @@ static void nicvf_refill_rbdr(struct nicvf *nic, gfp_t gfp)
else
refill_rb_cnt = qs->rbdr_len - qcount - 1;
+ /* Sync page cache info */
+ smp_rmb();
+
/* Start filling descs from tail */
tail = nicvf_queue_reg_read(nic, NIC_QSET_RBDR_0_1_TAIL, rbdr_idx) >> 3;
while (refill_rb_cnt) {
tail++;
tail &= (rbdr->dmem.q_len - 1);
- if (nicvf_alloc_rcv_buffer(nic, gfp, RCV_FRAG_LEN, &rbuf))
+ if (nicvf_alloc_rcv_buffer(nic, rbdr, gfp, RCV_FRAG_LEN, &rbuf))
break;
desc = GET_RBDR_DESC(rbdr, tail);
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index 10cb4b8..da48366 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -213,6 +213,11 @@ struct q_desc_mem {
void *unalign_base;
};
+struct pgcache {
+ struct page *page;
+ u64 dma_addr;
+};
+
struct rbdr {
bool enable;
u32 dma_size;
@@ -222,6 +227,12 @@ struct rbdr {
u32 head;
u32 tail;
struct q_desc_mem dmem;
+
+ /* For page recycling */
+ int pgidx;
+ int pgcnt;
+ int pgalloc;
+ struct pgcache *pgcache;
} ____cacheline_aligned_in_smp;
struct rcv_queue {
--
2.7.4
^ permalink raw reply related
* [PATCH 0/9] net: thunderx: Adds XDP support
From: sunil.kovvuri @ 2017-05-02 13:06 UTC (permalink / raw)
To: netdev; +Cc: linux-kernel, linux-arm-kernel, Sunil Goutham
From: Sunil Goutham <sgoutham@cavium.com>
This patch series adds support for XDP to ThunderX NIC driver
which is used on CN88xx, CN81xx and CN83xx platforms.
Patches 1-4 are performance improvement and cleanup patches
which are done keeping XDP performance bottlenecks in view.
Rest of the patches adds actual XDP support.
Sunil Goutham (9):
net: thunderx: Support for page recycling
net: thunderx: Optimize RBDR descriptor handling
net: thunderx: Optimize CQE_TX handling
net: thunderx: Cleanup receive buffer allocation
net: thunderx: Add basic XDP support
net: thunderx: Add support for XDP_DROP
net: thunderx: Add support for XDP_TX
net: thunderx: Support for XDP header adjustment
net: thunderx: Optimize page recycling for XDP
drivers/net/ethernet/cavium/thunder/nic.h | 10 +-
.../net/ethernet/cavium/thunder/nicvf_ethtool.c | 29 +-
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 313 +++++++++++++++--
drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 387 +++++++++++++++++----
drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 32 +-
drivers/net/ethernet/cavium/thunder/q_struct.h | 10 +-
6 files changed, 657 insertions(+), 124 deletions(-)
--
2.7.4
^ permalink raw reply
* Re: [PATCH v2 net-next 0/7] Extend socket timestamping API
From: Miroslav Lichvar @ 2017-05-02 12:55 UTC (permalink / raw)
To: netdev; +Cc: Richard Cochran, Willem de Bruijn
In-Reply-To: <20170502101103.30444-1-mlichvar@redhat.com>
Hm, I see that net-next was closed. I missed the annoucement. Sorry
for the spam.
On Tue, May 02, 2017 at 02:46:02PM +0200, Miroslav Lichvar wrote:
> Changes v1->v2:
> - added separate patch for new NAPI functions
> - split code from __sock_recv_timestamp() for better readability
> - fixed RCU locking
> - fixed compiler warning (missing case in switch in first patch)
> - inline sw_tx_timestamp() in its only user
--
Miroslav Lichvar
^ permalink raw reply
* Re: net/smc and the RDMA core
From: Ursula Braun @ 2017-05-02 12:41 UTC (permalink / raw)
To: Parav Pandit, Bart Van Assche, hch-jcswGhMUV9g@public.gmane.org,
davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org
Cc: netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <HE1PR0502MB30048AFD086C4B0D535BFC52D1140-692Kmc8YnlL9PhveBwpv4cDSnupUy6xnnBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
On 05/01/2017 07:55 PM, Parav Pandit wrote:
> Hi Bart, Ursula, Dave,
>
> I am particularly concerned about SMC as address family.
> It should not be treated as address family, but rather an additional protocol similar for socket type SOCK_STREAM.
We tried to avoid changes of the kernel TCP code. A new address family
seemed to be a feasible way to achieve this.
> While doing performance benchmarking last month and while porting few database application,
>
> I encountered a major hurdle where user space library heavily depend on AF_INET and AF_INET6 family through get_addrinfo and other friend functions.
> Adding or treating AF_SMC as AF_INET just doesn't sound right.
>
> Most user space code doesn't care for the protocol field, but do handle domain field.
>
> I personally believe it's not too late to modify SMC to drop expose AF_SMC and have it exposed through new protocol that can be exposed through socket() API.
>
> Parav
>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
>> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Bart Van Assche
>> Sent: Monday, May 1, 2017 12:30 PM
>> To: hch-jcswGhMUV9g@public.gmane.org; davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org; ubraun-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org
>> Cc: netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: net/smc and the RDMA core
>>
>> On Mon, 2017-05-01 at 18:33 +0200, Christoph Hellwig wrote:
>>> Hi Ursual, hi netdev reviewers,
>>>
>>> how did the smc protocol manage to get merged without any review on
>>> linux-rdma at all? As the results it seems it's very substandard in
>>> terms of RDMA API usage, e.g. it neither uses the proper CQ API nor
>>> the RDMA R/W API, and other will probably find additional issues as
>>> well.
>>
>> Hello Dave and Ursula,
>>
>> It seems very rude to me to have merged the SMC protocol driver without
>> having involved the linux-rdma community. Anyway, I have the following
>> questions for Dave and Ursula:
>> * Since the Linux kernel is standards based: where can we find the standard
>> that defines the SMC wire protocol? If this protocol has not been
>> standardized yet: in what file (other than *.[ch]) in the Linux kernel
>> tree has this protocol been documented?
>> * What are the differences between the SMC protocol, the SDP protocol and
>> the rsockets protocol? How do existing implementations for these protocols
>> compare to each other from a performance point of view? If no performance
>> comparison between these protocols is available, shouldn't the performance
>> of these protocols have been compared with each other before a review of
>> the SMC driver even started?
>> * What are the reasons why the SDP driver was never accepted upstream? Do
>> the arguments why SDP was not accepted upstream also apply to the SMC
>> driver (SDP = Sockets Direct Protocol)?
>> * Since SMC has to be selected by specifying AF_SMC, how are users expected
>> to specify whether AF_INET, AF_INET6 or yet another address family should
>> be used to set up a connection between SMC endpoints?
>> * Is the SMC driver limited to RoCE? Are you aware that the rsockets library
>> supports multiple transport layers (RoCE, IB and iWARP)?
>> * Since functionality that is similar what the SMC driver provides already
>> exists in user space (rsockets), why has this functionality been
>> reimplemented as a kernel driver (SMC)?
>>
>> Bart.--
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body
>> of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: net/smc and the RDMA core
From: Ursula Braun @ 2017-05-02 12:34 UTC (permalink / raw)
To: Bart Van Assche, hch-jcswGhMUV9g@public.gmane.org,
davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org
Cc: netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1493659776.2665.7.camel-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
On 05/01/2017 07:29 PM, Bart Van Assche wrote:
> On Mon, 2017-05-01 at 18:33 +0200, Christoph Hellwig wrote:
>> Hi Ursual, hi netdev reviewers,
>>
>> how did the smc protocol manage to get merged without any review
>> on linux-rdma at all? As the results it seems it's very substandard
>> in terms of RDMA API usage, e.g. it neither uses the proper CQ API
>> nor the RDMA R/W API, and other will probably find additional issues
>> as well.
>
> Hello Dave and Ursula,
>
> It seems very rude to me to have merged the SMC protocol driver without
> having involved the linux-rdma community. Anyway, I have the following
> questions for Dave and Ursula:
> * Since the Linux kernel is standards based: where can we find the standard
> that defines the SMC wire protocol? If this protocol has not been
> standardized yet: in what file (other than *.[ch]) in the Linux kernel
> tree has this protocol been documented?
Hello Bart,
The protocol is standardized, see: http://www.rfc-editor.org/info/rfc7609.
I described this and some more protocol details in my patch series
overview mail, see for instance:
http://marc.info/?l=linux-s390&m=148397751211964&w=2
This description explains the reasons to come up with SMC-R.
> * What are the differences between the SMC protocol, the SDP protocol and
> the rsockets protocol? How do existing implementations for these protocols
> compare to each other from a performance point of view? If no performance
> comparison between these protocols is available, shouldn't the performance
> of these protocols have been compared with each other before a review of
> the SMC driver even started?
> * What are the reasons why the SDP driver was never accepted upstream? Do
> the arguments why SDP was not accepted upstream also apply to the SMC
> driver (SDP = Sockets Direct Protocol)?
> * Since SMC has to be selected by specifying AF_SMC, how are users expected
> to specify whether AF_INET, AF_INET6 or yet another address family should
> be used to set up a connection between SMC
> endpoints?
The IPv6 support in SMC-R is on our todo-list.
> * Is the SMC driver limited to RoCE? Are you aware that the rsockets library
> supports multiple transport layers (RoCE, IB and iWARP)?
For now, only RoCE is supported. Other transports might be added in the future.
> * Since functionality that is similar what the SMC driver provides already
> exists in user space (rsockets), why has this functionality been
> reimplemented as a kernel driver (SMC)?
>
> Bart.
>
Regards, Ursula
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* IPsec PFP support on linux
From: Sowmini Varadhan @ 2017-05-02 12:32 UTC (permalink / raw)
To: netdev, herbert, linux-crypto, swan, steffen.klassert, borisp,
ilant
Cc: sowmini.varadhan
I have a question about linux support for IPsec PFP (as defined in
rfc 4301). I am assuming this exists, and is accessible from uspace,
in which case I need some hints on how to set it up.
Assuming I have a server listening at port 5001 that I want to
secure via ipsec. Suppose I want to make sure that each TCP/UDP 5-tuple
sending packets to port 5001 gets its own SA.
RFC4301 has this:
- SPD-S: For traffic that is to be protected using IPsec, the
entry consists of the values of the selectors that apply to the
traffic to be protected via AH or ESP, controls on how to
create SAs based on these selectors, ...
and further down
If IPsec processing is specified for
an entry, a "populate from packet" (PFP) flag may be asserted for
one or more of the selectors in the SPD entry (Local IP address;
Remote IP address; Next Layer Protocol; and, depending on Next
Layer Protocol, Local port and Remote port, or ICMP type/code, or
Mobility Header type). If asserted for a given selector X, the
flag indicates that the SA to be created should take its value for
X from the value in the packet. Otherwise, the SA should take its
value(s) for X from the value(s) in the SPD entry.
A google search produces a discarded patch
http://marc.info/?l=linux-netdev&m=119746758904140
but its not clear to me how to set this up (if PFP works fine,
as suggested by Herbert's response above)
I tried experimenting with IP_XFRM_POLICY from a simple udp client but
(a) that seems to require a SPI and reqid to set up the SPD
(b) I see the SADB_ACQUIRE upcall being triggered after the local port
is bound (and SADB entry is set up for the lport). But ike phase2
does not converge for the lport specific sadb added
by the bind (even in quick mode)
My understanding is that pluto shoud be generating spi's to make sure
they are sufficiently unique/random etc. so (a) makes me think I'm
either not setting this up or not using this correctly.
Any hints/sample code/RTFMs would be helpful (documentation for
IP_XFRM_POLICY seems scant, afaict). I'd be happy to share my
udp client program, if it can provide more context to my question.
--Sowmini
^ permalink raw reply
* [net-next PATCH 4/4] samples/bpf: export map_data[] for more info on maps
From: Jesper Dangaard Brouer @ 2017-05-02 12:32 UTC (permalink / raw)
To: kafai
Cc: netdev, eric, Daniel Borkmann, Alexei Starovoitov,
Jesper Dangaard Brouer
In-Reply-To: <149372826543.22268.3617359219409721129.stgit@firesoul>
Giving *_user.c side tools access to map_data[] provides easier
access to information on the maps being loaded. Still provide
the guarantee that the order maps are being defined in inside the
_kern.c file corresponds with the order in the array. Now user
tools are not blind, but can inspect and verify the maps that got
loaded from the ELF binary.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/bpf_load.h | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/samples/bpf/bpf_load.h b/samples/bpf/bpf_load.h
index 4d4fd4678a64..ca0563d04744 100644
--- a/samples/bpf/bpf_load.h
+++ b/samples/bpf/bpf_load.h
@@ -24,12 +24,18 @@ struct bpf_map_data {
typedef void (*fixup_map_cb)(struct bpf_map_data *map, int idx);
-extern int map_fd[MAX_MAPS];
extern int prog_fd[MAX_PROGS];
extern int event_fd[MAX_PROGS];
extern char bpf_log_buf[BPF_LOG_BUF_SIZE];
extern int prog_cnt;
+/* There is a one-to-one mapping between map_fd[] and map_data[].
+ * The map_data[] just contains more rich info on the given map.
+ */
+extern int map_fd[MAX_MAPS];
+extern struct bpf_map_data map_data[MAX_MAPS];
+extern int map_data_count;
+
/* parses elf file compiled by llvm .c->.o
* . parses 'maps' section and creates maps via BPF syscall
* . parses 'license' section and passes it to syscall
^ permalink raw reply related
* [net-next PATCH 3/4] samples/bpf: load_bpf.c make callback fixup more flexible
From: Jesper Dangaard Brouer @ 2017-05-02 12:32 UTC (permalink / raw)
To: kafai
Cc: netdev, eric, Daniel Borkmann, Alexei Starovoitov,
Jesper Dangaard Brouer
In-Reply-To: <149372826543.22268.3617359219409721129.stgit@firesoul>
Do this change before others start to use this callback.
Change map_perf_test_user.c which seems to be the only user.
This patch extends capabilities of commit 9fd63d05f3e8 ("bpf:
Allow bpf sample programs (*_user.c) to change bpf_map_def").
Give fixup callback access to struct bpf_map_data, instead of
only stuct bpf_map_def. This add flexibility to allow userspace
to reassign the map file descriptor. This is very useful when
wanting to share maps between several bpf programs.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/bpf_load.c | 17 ++++++++---------
samples/bpf/bpf_load.h | 10 ++++++++--
samples/bpf/map_perf_test_user.c | 14 +++++++-------
3 files changed, 23 insertions(+), 18 deletions(-)
diff --git a/samples/bpf/bpf_load.c b/samples/bpf/bpf_load.c
index fedec29c7817..74456b3eb89a 100644
--- a/samples/bpf/bpf_load.c
+++ b/samples/bpf/bpf_load.c
@@ -39,13 +39,6 @@ int event_fd[MAX_PROGS];
int prog_cnt;
int prog_array_fd = -1;
-/* Keeping relevant info on maps */
-struct bpf_map_data {
- int fd;
- char *name;
- size_t elf_offset;
- struct bpf_map_def def;
-};
struct bpf_map_data map_data[MAX_MAPS];
int map_data_count = 0;
@@ -202,8 +195,14 @@ static int load_maps(struct bpf_map_data *maps, int nr_maps,
int i;
for (i = 0; i < nr_maps; i++) {
- if (fixup_map)
- fixup_map(&maps[i].def, maps[i].name, i);
+ if (fixup_map) {
+ fixup_map(&maps[i], i);
+ /* Allow userspace to assign map FD prior to creation */
+ if (maps[i].fd != -1) {
+ map_fd[i] = maps[i].fd;
+ continue;
+ }
+ }
if (maps[i].def.type == BPF_MAP_TYPE_ARRAY_OF_MAPS ||
maps[i].def.type == BPF_MAP_TYPE_HASH_OF_MAPS) {
diff --git a/samples/bpf/bpf_load.h b/samples/bpf/bpf_load.h
index 05822f83173a..4d4fd4678a64 100644
--- a/samples/bpf/bpf_load.h
+++ b/samples/bpf/bpf_load.h
@@ -15,8 +15,14 @@ struct bpf_map_def {
unsigned int inner_map_idx;
};
-typedef void (*fixup_map_cb)(struct bpf_map_def *map, const char *map_name,
- int idx);
+struct bpf_map_data {
+ int fd;
+ char *name;
+ size_t elf_offset;
+ struct bpf_map_def def;
+};
+
+typedef void (*fixup_map_cb)(struct bpf_map_data *map, int idx);
extern int map_fd[MAX_MAPS];
extern int prog_fd[MAX_PROGS];
diff --git a/samples/bpf/map_perf_test_user.c b/samples/bpf/map_perf_test_user.c
index 6ac778153315..1a8894b5ac51 100644
--- a/samples/bpf/map_perf_test_user.c
+++ b/samples/bpf/map_perf_test_user.c
@@ -320,21 +320,21 @@ static void fill_lpm_trie(void)
assert(!r);
}
-static void fixup_map(struct bpf_map_def *map, const char *name, int idx)
+static void fixup_map(struct bpf_map_data *map, int idx)
{
int i;
- if (!strcmp("inner_lru_hash_map", name)) {
+ if (!strcmp("inner_lru_hash_map", map->name)) {
inner_lru_hash_idx = idx;
- inner_lru_hash_size = map->max_entries;
+ inner_lru_hash_size = map->def.max_entries;
}
- if (!strcmp("array_of_lru_hashs", name)) {
+ if (!strcmp("array_of_lru_hashs", map->name)) {
if (inner_lru_hash_idx == -1) {
printf("inner_lru_hash_map must be defined before array_of_lru_hashs\n");
exit(1);
}
- map->inner_map_idx = inner_lru_hash_idx;
+ map->def.inner_map_idx = inner_lru_hash_idx;
array_of_lru_hashs_idx = idx;
}
@@ -345,9 +345,9 @@ static void fixup_map(struct bpf_map_def *map, const char *name, int idx)
/* Only change the max_entries for the enabled test(s) */
for (i = 0; i < NR_TESTS; i++) {
- if (!strcmp(test_map_names[i], name) &&
+ if (!strcmp(test_map_names[i], map->name) &&
(check_test_flags(i))) {
- map->max_entries = num_map_entries;
+ map->def.max_entries = num_map_entries;
}
}
}
^ permalink raw reply related
* [net-next PATCH 0/4] Improve bpf ELF-loader under samples/bpf
From: Jesper Dangaard Brouer @ 2017-05-02 12:31 UTC (permalink / raw)
To: kafai
Cc: netdev, eric, Daniel Borkmann, Alexei Starovoitov,
Jesper Dangaard Brouer
This series improves and fixes bpf ELF loader and programs under
samples/bpf. The bpf_load.c created some hard to debug issues when
the struct (bpf_map_def) used in the ELF maps section format changed
in commit fb30d4b71214 ("bpf: Add tests for map-in-map").
This was hotfixed in commit 409526bea3c3 ("samples/bpf: bpf_load.c
detect and abort if ELF maps section size is wrong") by detecting the
issue and aborting the program.
In most situations the bpf-loader should be able to handle these kind
of changes to the struct size. This patch series aim to do proper
backward and forward compabilility handling when loading ELF files.
This series also adjust the callback that was introduced in commit
9fd63d05f3e8 ("bpf: Allow bpf sample programs (*_user.c) to change
bpf_map_def") to use the new bpf_map_data structure, before more users
start to use this callback.
Hoping these changes can make the merge window, as above mentioned
commits have not been merged yet, and it would be good to avoid users
hitting these issues.
---
Jesper Dangaard Brouer (4):
samples/bpf: adjust rlimit RLIMIT_MEMLOCK for traceex2, tracex3 and tracex4
samples/bpf: make bpf_load.c code compatible with ELF maps section changes
samples/bpf: load_bpf.c make callback fixup more flexible
samples/bpf: export map_data[] for more info on maps
samples/bpf/bpf_load.c | 229 ++++++++++++++++++++++++++------------
samples/bpf/bpf_load.h | 18 ++-
samples/bpf/map_perf_test_user.c | 14 +-
samples/bpf/tracex2_user.c | 7 +
samples/bpf/tracex3_user.c | 7 +
samples/bpf/tracex4_user.c | 8 +
6 files changed, 201 insertions(+), 82 deletions(-)
^ permalink raw reply
* [net-next PATCH 1/4] samples/bpf: adjust rlimit RLIMIT_MEMLOCK for traceex2, tracex3 and tracex4
From: Jesper Dangaard Brouer @ 2017-05-02 12:31 UTC (permalink / raw)
To: kafai
Cc: netdev, eric, Daniel Borkmann, Alexei Starovoitov,
Jesper Dangaard Brouer
In-Reply-To: <149372826543.22268.3617359219409721129.stgit@firesoul>
Needed to adjust max locked memory RLIMIT_MEMLOCK for testing these bpf samples
as these are using more and larger maps than can fit in distro default 64Kbytes limit.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/tracex2_user.c | 7 +++++++
samples/bpf/tracex3_user.c | 7 +++++++
samples/bpf/tracex4_user.c | 8 ++++++++
3 files changed, 22 insertions(+)
diff --git a/samples/bpf/tracex2_user.c b/samples/bpf/tracex2_user.c
index ded9804c5034..7fee0f1ba9a3 100644
--- a/samples/bpf/tracex2_user.c
+++ b/samples/bpf/tracex2_user.c
@@ -4,6 +4,7 @@
#include <signal.h>
#include <linux/bpf.h>
#include <string.h>
+#include <sys/resource.h>
#include "libbpf.h"
#include "bpf_load.h"
@@ -112,6 +113,7 @@ static void int_exit(int sig)
int main(int ac, char **argv)
{
+ struct rlimit r = {1024*1024, RLIM_INFINITY};
char filename[256];
long key, next_key, value;
FILE *f;
@@ -119,6 +121,11 @@ int main(int ac, char **argv)
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK)");
+ return 1;
+ }
+
signal(SIGINT, int_exit);
/* start 'ping' in the background to have some kfree_skb events */
diff --git a/samples/bpf/tracex3_user.c b/samples/bpf/tracex3_user.c
index 8f7d199d5945..fe372239d505 100644
--- a/samples/bpf/tracex3_user.c
+++ b/samples/bpf/tracex3_user.c
@@ -11,6 +11,7 @@
#include <stdbool.h>
#include <string.h>
#include <linux/bpf.h>
+#include <sys/resource.h>
#include "libbpf.h"
#include "bpf_load.h"
@@ -112,11 +113,17 @@ static void print_hist(int fd)
int main(int ac, char **argv)
{
+ struct rlimit r = {1024*1024, RLIM_INFINITY};
char filename[256];
int i;
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK)");
+ return 1;
+ }
+
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
return 1;
diff --git a/samples/bpf/tracex4_user.c b/samples/bpf/tracex4_user.c
index 03449f773cb1..22c644f1f4c3 100644
--- a/samples/bpf/tracex4_user.c
+++ b/samples/bpf/tracex4_user.c
@@ -12,6 +12,8 @@
#include <string.h>
#include <time.h>
#include <linux/bpf.h>
+#include <sys/resource.h>
+
#include "libbpf.h"
#include "bpf_load.h"
@@ -50,11 +52,17 @@ static void print_old_objects(int fd)
int main(int ac, char **argv)
{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
char filename[256];
int i;
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK, RLIM_INFINITY)");
+ return 1;
+ }
+
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
return 1;
^ permalink raw reply related
* [net-next PATCH 2/4] samples/bpf: make bpf_load.c code compatible with ELF maps section changes
From: Jesper Dangaard Brouer @ 2017-05-02 12:31 UTC (permalink / raw)
To: kafai
Cc: netdev, eric, Daniel Borkmann, Alexei Starovoitov,
Jesper Dangaard Brouer
In-Reply-To: <149372826543.22268.3617359219409721129.stgit@firesoul>
This patch does proper parsing of the ELF "maps" section, in-order to
be both backwards and forwards compatible with changes to the map
definition struct bpf_map_def, which gets compiled into the ELF file.
The assumption is that new features with value zero, means that they
are not in-use. For backward compatibility where loading an ELF file
with a smaller struct bpf_map_def, only copy objects ELF size, leaving
rest of loaders struct zero. For forward compatibility where ELF file
have a larger struct bpf_map_def, only copy loaders own struct size
and verify that rest of the larger struct is zero, assuming this means
the newer feature was not activated, thus it should be safe for this
older loader to load this newer ELF file.
Fixes: fb30d4b71214 ("bpf: Add tests for map-in-map")
Fixes: 409526bea3c3 ("samples/bpf: bpf_load.c detect and abort if ELF maps section size is wrong")
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/bpf/bpf_load.c | 224 +++++++++++++++++++++++++++++++++---------------
1 file changed, 155 insertions(+), 69 deletions(-)
diff --git a/samples/bpf/bpf_load.c b/samples/bpf/bpf_load.c
index 4221dc359453..fedec29c7817 100644
--- a/samples/bpf/bpf_load.c
+++ b/samples/bpf/bpf_load.c
@@ -39,6 +39,16 @@ int event_fd[MAX_PROGS];
int prog_cnt;
int prog_array_fd = -1;
+/* Keeping relevant info on maps */
+struct bpf_map_data {
+ int fd;
+ char *name;
+ size_t elf_offset;
+ struct bpf_map_def def;
+};
+struct bpf_map_data map_data[MAX_MAPS];
+int map_data_count = 0;
+
static int populate_prog_array(const char *event, int prog_fd)
{
int ind = atoi(event), err;
@@ -186,42 +196,39 @@ static int load_and_attach(const char *event, struct bpf_insn *prog, int size)
return 0;
}
-static int load_maps(struct bpf_map_def *maps, int nr_maps,
- const char **map_names, fixup_map_cb fixup_map)
+static int load_maps(struct bpf_map_data *maps, int nr_maps,
+ fixup_map_cb fixup_map)
{
int i;
- /*
- * Warning: Using "maps" pointing to ELF data_maps->d_buf as
- * an array of struct bpf_map_def is a wrong assumption about
- * the ELF maps section format.
- */
+
for (i = 0; i < nr_maps; i++) {
if (fixup_map)
- fixup_map(&maps[i], map_names[i], i);
+ fixup_map(&maps[i].def, maps[i].name, i);
- if (maps[i].type == BPF_MAP_TYPE_ARRAY_OF_MAPS ||
- maps[i].type == BPF_MAP_TYPE_HASH_OF_MAPS) {
- int inner_map_fd = map_fd[maps[i].inner_map_idx];
+ if (maps[i].def.type == BPF_MAP_TYPE_ARRAY_OF_MAPS ||
+ maps[i].def.type == BPF_MAP_TYPE_HASH_OF_MAPS) {
+ int inner_map_fd = map_fd[maps[i].def.inner_map_idx];
- map_fd[i] = bpf_create_map_in_map(maps[i].type,
- maps[i].key_size,
- inner_map_fd,
- maps[i].max_entries,
- maps[i].map_flags);
+ map_fd[i] = bpf_create_map_in_map(maps[i].def.type,
+ maps[i].def.key_size,
+ inner_map_fd,
+ maps[i].def.max_entries,
+ maps[i].def.map_flags);
} else {
- map_fd[i] = bpf_create_map(maps[i].type,
- maps[i].key_size,
- maps[i].value_size,
- maps[i].max_entries,
- maps[i].map_flags);
+ map_fd[i] = bpf_create_map(maps[i].def.type,
+ maps[i].def.key_size,
+ maps[i].def.value_size,
+ maps[i].def.max_entries,
+ maps[i].def.map_flags);
}
if (map_fd[i] < 0) {
printf("failed to create a map: %d %s\n",
errno, strerror(errno));
return 1;
}
+ maps[i].fd = map_fd[i];
- if (maps[i].type == BPF_MAP_TYPE_PROG_ARRAY)
+ if (maps[i].def.type == BPF_MAP_TYPE_PROG_ARRAY)
prog_array_fd = map_fd[i];
}
return 0;
@@ -251,7 +258,8 @@ static int get_sec(Elf *elf, int i, GElf_Ehdr *ehdr, char **shname,
}
static int parse_relo_and_apply(Elf_Data *data, Elf_Data *symbols,
- GElf_Shdr *shdr, struct bpf_insn *insn)
+ GElf_Shdr *shdr, struct bpf_insn *insn,
+ struct bpf_map_data *maps, int nr_maps)
{
int i, nrels;
@@ -261,6 +269,8 @@ static int parse_relo_and_apply(Elf_Data *data, Elf_Data *symbols,
GElf_Sym sym;
GElf_Rel rel;
unsigned int insn_idx;
+ bool match = false;
+ int j, map_idx;
gelf_getrel(data, i, &rel);
@@ -274,11 +284,21 @@ static int parse_relo_and_apply(Elf_Data *data, Elf_Data *symbols,
return 1;
}
insn[insn_idx].src_reg = BPF_PSEUDO_MAP_FD;
- /*
- * Warning: Using sizeof(struct bpf_map_def) here is a
- * wrong assumption about ELF maps section format
- */
- insn[insn_idx].imm = map_fd[sym.st_value / sizeof(struct bpf_map_def)];
+
+ /* Match FD relocation against recorded map_data[] offset */
+ for (map_idx = 0; map_idx < nr_maps; map_idx++) {
+ if (maps[map_idx].elf_offset == sym.st_value) {
+ match = true;
+ break;
+ }
+ }
+ if (match) {
+ insn[insn_idx].imm = maps[map_idx].fd;
+ } else {
+ printf("invalid relo for insn[%d] no map_data match\n",
+ insn_idx);
+ return 1;
+ }
}
return 0;
@@ -297,40 +317,112 @@ static int cmp_symbols(const void *l, const void *r)
return 0;
}
-static int get_sorted_map_names(Elf *elf, Elf_Data *symbols, int maps_shndx,
- int strtabidx, char **map_names)
+static int load_elf_maps_section(struct bpf_map_data *maps, int maps_shndx,
+ Elf *elf, Elf_Data *symbols, int strtabidx)
{
- GElf_Sym map_symbols[MAX_MAPS];
- int i, nr_maps = 0;
+ int map_sz_elf, map_sz_copy;
+ bool validate_zero = false;
+ Elf_Data *data_maps;
+ int i, nr_maps;
+ GElf_Sym *sym;
+ Elf_Scn *scn;
+ int copy_sz;
+
+ if (maps_shndx < 0)
+ return -EINVAL;
+ if (!symbols)
+ return -EINVAL;
+
+ /* Get data for maps section via elf index */
+ scn = elf_getscn(elf, maps_shndx);
+ if (scn)
+ data_maps = elf_getdata(scn, NULL);
+ if (!scn || !data_maps) {
+ printf("Failed to get Elf_Data from maps section %d\n",
+ maps_shndx);
+ return -EINVAL;
+ }
- for (i = 0; i < symbols->d_size / sizeof(GElf_Sym); i++) {
- assert(nr_maps < MAX_MAPS);
- if (!gelf_getsym(symbols, i, &map_symbols[nr_maps]))
+ /* For each map get corrosponding symbol table entry */
+ sym = calloc(MAX_MAPS+1, sizeof(GElf_Sym));
+ for (i = 0, nr_maps = 0; i < symbols->d_size / sizeof(GElf_Sym); i++) {
+ assert(nr_maps < MAX_MAPS+1);
+ if (!gelf_getsym(symbols, i, &sym[nr_maps]))
continue;
- if (map_symbols[nr_maps].st_shndx != maps_shndx)
+ if (sym[nr_maps].st_shndx != maps_shndx)
continue;
+ /* Only increment iif maps section */
nr_maps++;
}
- qsort(map_symbols, nr_maps, sizeof(GElf_Sym), cmp_symbols);
+ /* Align to map_fd[] order, via sort on offset in sym.st_value */
+ qsort(sym, nr_maps, sizeof(GElf_Sym), cmp_symbols);
+
+ /* Keeping compatible with ELF maps section changes
+ * ------------------------------------------------
+ * The program size of struct bpf_map_def is known by loader
+ * code, but struct stored in ELF file can be different.
+ *
+ * Unfortunately sym[i].st_size is zero. To calculate the
+ * struct size stored in the ELF file, assume all struct have
+ * the same size, and simply divide with number of map
+ * symbols.
+ */
+ map_sz_elf = data_maps->d_size / nr_maps;
+ map_sz_copy = sizeof(struct bpf_map_def);
+ if (map_sz_elf < map_sz_copy) {
+ /*
+ * Backward compat, loading older ELF file with
+ * smaller struct, keeping remaining bytes zero.
+ */
+ map_sz_copy = map_sz_elf;
+ } else if (map_sz_elf > map_sz_copy) {
+ /*
+ * Forward compat, loading newer ELF file with larger
+ * struct with unknown features. Assume zero means
+ * feature not used. Thus, validate rest of struct
+ * data is zero.
+ */
+ validate_zero = true;
+ }
+ /* Memcpy relevant part of ELF maps data to loader maps */
for (i = 0; i < nr_maps; i++) {
- char *map_name;
-
- map_name = elf_strptr(elf, strtabidx, map_symbols[i].st_name);
- if (!map_name) {
- printf("cannot get map symbol\n");
- return -1;
- }
-
- map_names[i] = strdup(map_name);
- if (!map_names[i]) {
+ unsigned char *addr, *end;
+ struct bpf_map_def *def;
+ const char *map_name;
+ size_t offset;
+
+ map_name = elf_strptr(elf, strtabidx, sym[i].st_name);
+ maps[i].name = strdup(map_name);
+ if (!maps[i].name) {
printf("strdup(%s): %s(%d)\n", map_name,
strerror(errno), errno);
- return -1;
+ free(sym);
+ return -errno;
+ }
+
+ /* Symbol value is offset into ELF maps section data area */
+ offset = sym[i].st_value;
+ def = (struct bpf_map_def *)(data_maps->d_buf + offset);
+ maps[i].elf_offset = offset;
+ memset(&maps[i].def, 0, sizeof(struct bpf_map_def));
+ memcpy(&maps[i].def, def, map_sz_copy);
+
+ /* Verify no newer features were requested */
+ if (validate_zero) {
+ addr = (unsigned char*) def + map_sz_copy;
+ end = (unsigned char*) def + map_sz_elf;
+ for (; addr < end; addr++) {
+ if (*addr != 0) {
+ free(sym);
+ return -EFBIG;
+ }
+ }
}
}
+ free(sym);
return nr_maps;
}
@@ -341,7 +433,8 @@ static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
GElf_Ehdr ehdr;
GElf_Shdr shdr, shdr_prog;
Elf_Data *data, *data_prog, *data_maps = NULL, *symbols = NULL;
- char *shname, *shname_prog, *map_names[MAX_MAPS] = { NULL };
+ char *shname, *shname_prog;
+ int nr_maps = 0;
/* reset global variables */
kern_version = 0;
@@ -389,8 +482,12 @@ static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
}
memcpy(&kern_version, data->d_buf, sizeof(int));
} else if (strcmp(shname, "maps") == 0) {
+ int j;
+
maps_shndx = i;
data_maps = data;
+ for (j = 0; j < MAX_MAPS; j++)
+ map_data[j].fd = -1;
} else if (shdr.sh_type == SHT_SYMTAB) {
strtabidx = shdr.sh_link;
symbols = data;
@@ -405,27 +502,17 @@ static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
}
if (data_maps) {
- int nr_maps;
- int prog_elf_map_sz;
-
- nr_maps = get_sorted_map_names(elf, symbols, maps_shndx,
- strtabidx, map_names);
- if (nr_maps < 0)
- goto done;
-
- /* Deduce map struct size stored in ELF maps section */
- prog_elf_map_sz = data_maps->d_size / nr_maps;
- if (prog_elf_map_sz != sizeof(struct bpf_map_def)) {
- printf("Error: ELF maps sec wrong size (%d/%lu),"
- " old kern.o file?\n",
- prog_elf_map_sz, sizeof(struct bpf_map_def));
+ nr_maps = load_elf_maps_section(map_data, maps_shndx,
+ elf, symbols, strtabidx);
+ if (nr_maps < 0) {
+ printf("Error: Failed loading ELF maps (errno:%d):%s\n",
+ nr_maps, strerror(-nr_maps));
ret = 1;
goto done;
}
-
- if (load_maps(data_maps->d_buf, nr_maps,
- (const char **)map_names, fixup_map))
+ if (load_maps(map_data, nr_maps, fixup_map))
goto done;
+ map_data_count = nr_maps;
processed_sec[maps_shndx] = true;
}
@@ -453,7 +540,8 @@ static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
processed_sec[shdr.sh_info] = true;
processed_sec[i] = true;
- if (parse_relo_and_apply(data, symbols, &shdr, insns))
+ if (parse_relo_and_apply(data, symbols, &shdr, insns,
+ map_data, nr_maps))
continue;
if (memcmp(shname_prog, "kprobe/", 7) == 0 ||
@@ -488,8 +576,6 @@ static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
ret = 0;
done:
- for (i = 0; i < MAX_MAPS; i++)
- free(map_names[i]);
close(fd);
return ret;
}
^ permalink raw reply related
* Re: net/smc and the RDMA core
From: Ursula Braun @ 2017-05-02 12:25 UTC (permalink / raw)
To: Christoph Hellwig, David S. Miller
Cc: netdev-u79uwXL29TY76Z2rM5mHXA, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20170501163311.GA22209-jcswGhMUV9g@public.gmane.org>
On 05/01/2017 06:33 PM, Christoph Hellwig wrote:
> Hi Ursual, hi netdev reviewers,
>
> how did the smc protocol manage to get merged without any review
> on linux-rdma at all? As the results it seems it's very substandard
> in terms of RDMA API usage, e.g. it neither uses the proper CQ API
> nor the RDMA R/W API, and other will probably find additional issues
> as well.
>
Hi Christoph,
We have been posting SMC-R patches on netdev since 2015, there was never
any secrecy about it. Still sorry for omitting linux-rdma, will include
with future postings from now on.
Of course, we are open to any further code reviews, so if you can point
out specific issues, we will be happy to work with you to get them
addressed!
Regards, Ursula
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch net-next repost] net: sched: add helpers to handle extended actions
From: Jiri Pirko @ 2017-05-02 12:07 UTC (permalink / raw)
To: Jamal Hadi Salim; +Cc: netdev, davem, xiyou.wangcong, mlxsw
In-Reply-To: <35328c1b-4571-f115-50c6-5f0c46ced3cc@mojatatu.com>
Tue, May 02, 2017 at 01:59:20PM CEST, jhs@mojatatu.com wrote:
>On 17-05-02 04:12 AM, Jiri Pirko wrote:
>> From: Jiri Pirko <jiri@mellanox.com>
>>
>> Jump is now the only one using value action opcode. This is going to
>> change soon. So introduce helpers to work with this. Convert TC_ACT_JUMP.
>>
>> This also fixes the TC_ACT_JUMP check, which is incorrectly done as a
>> bit check, not a value check.
>>
>> Fixes: e0ee84ded796 ("net sched actions: Complete the JUMPX opcode")
>> Signed-off-by: Jiri Pirko <jiri@mellanox.com>
>> ---
>> Dave, I'm sending this for -net-next although I know it is closed. But
>> the mentioned commit is not yet in -net. Feel free to take this either
>> to -net-next or -net, whatever suits you better. Thanks.
>
>I think you are pushing the boundary a little calling it a bug fix
Well, it is a bugfix. Otherwise we could not use bit 1 for anything else
then jump in the future. This is also UAPI. That's why I'm pushing it as
a fix.
>and this could go with your patch series instead.
>The name TC_ACT_EXT_CMP should be TC_ACT_EXT_CMP_OPCODE
I was thinking about it as well, I would like to keep it shorter. The
current name is quite appropriate and it is clear what the macro does.
>Other than that:
>
>Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Thanks.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox