* [PATCH 1/4] igb: prepare for AF_XDP zero-copy support
2023-07-04 9:59 [PATCH 0/4] igb: Add support for AF_XDP zero-copy Sriram Yagnaraman
@ 2023-07-04 9:59 ` Sriram Yagnaraman
2023-07-04 9:59 ` [PATCH 2/4] igb: Introduce txrx ring enable/disable functions Sriram Yagnaraman
` (3 subsequent siblings)
4 siblings, 0 replies; 11+ messages in thread
From: Sriram Yagnaraman @ 2023-07-04 9:59 UTC (permalink / raw)
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Sriram Yagnaraman
Remove static qualifiers on the following functions to be able to call
from XSK specific file that is added in the later patches
- igb_xdp_tx_queue_mapping
- igb_xdp_ring_update_tail
- igb_run_xdp
- igb_process_skb_fields
Introduce igb_xdp_is_enabled() to check if an XDP program is assigned to
the device.
Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
---
drivers/net/ethernet/intel/igb/igb.h | 13 +++++++++++++
drivers/net/ethernet/intel/igb/igb_main.c | 23 ++++++++++++-----------
2 files changed, 25 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index 015b78144114..94440af6cf4b 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -718,6 +718,8 @@ extern char igb_driver_name[];
int igb_xmit_xdp_ring(struct igb_adapter *adapter,
struct igb_ring *ring,
struct xdp_frame *xdpf);
+struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter);
+void igb_xdp_ring_update_tail(struct igb_ring *ring);
int igb_open(struct net_device *netdev);
int igb_close(struct net_device *netdev);
int igb_up(struct igb_adapter *);
@@ -737,6 +739,12 @@ void igb_setup_tctl(struct igb_adapter *);
void igb_setup_rctl(struct igb_adapter *);
void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *);
netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *);
+struct sk_buff *igb_run_xdp(struct igb_adapter *adapter,
+ struct igb_ring *rx_ring,
+ struct xdp_buff *xdp);
+void igb_process_skb_fields(struct igb_ring *rx_ring,
+ union e1000_adv_rx_desc *rx_desc,
+ struct sk_buff *skb);
void igb_alloc_rx_buffers(struct igb_ring *, u16);
void igb_update_stats(struct igb_adapter *);
bool igb_has_link(struct igb_adapter *adapter);
@@ -797,6 +805,11 @@ static inline struct netdev_queue *txring_txq(const struct igb_ring *tx_ring)
return netdev_get_tx_queue(tx_ring->netdev, tx_ring->queue_index);
}
+static inline bool igb_xdp_is_enabled(struct igb_adapter *adapter)
+{
+ return !!adapter->xdp_prog;
+}
+
int igb_add_filter(struct igb_adapter *adapter,
struct igb_nfc_filter *input);
int igb_erase_filter(struct igb_adapter *adapter,
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index 9a2561409b06..dadc3d423cfd 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -2939,7 +2939,8 @@ static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp)
}
}
-static void igb_xdp_ring_update_tail(struct igb_ring *ring)
+/* This function assumes __netif_tx_lock is held by the caller. */
+void igb_xdp_ring_update_tail(struct igb_ring *ring)
{
/* Force memory writes to complete before letting h/w know there
* are new descriptors to fetch.
@@ -2948,7 +2949,7 @@ static void igb_xdp_ring_update_tail(struct igb_ring *ring)
writel(ring->next_to_use, ring->tail);
}
-static struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter)
+struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter)
{
unsigned int r_idx = smp_processor_id();
@@ -3025,11 +3026,11 @@ static int igb_xdp_xmit(struct net_device *dev, int n,
nxmit++;
}
- __netif_tx_unlock(nq);
-
if (unlikely(flags & XDP_XMIT_FLUSH))
igb_xdp_ring_update_tail(tx_ring);
+ __netif_tx_unlock(nq);
+
return nxmit;
}
@@ -6631,7 +6632,7 @@ static int igb_change_mtu(struct net_device *netdev, int new_mtu)
struct igb_adapter *adapter = netdev_priv(netdev);
int max_frame = new_mtu + IGB_ETH_PKT_HDR_PAD;
- if (adapter->xdp_prog) {
+ if (igb_xdp_is_enabled(adapter)) {
int i;
for (i = 0; i < adapter->num_rx_queues; i++) {
@@ -8600,9 +8601,9 @@ static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring,
return skb;
}
-static struct sk_buff *igb_run_xdp(struct igb_adapter *adapter,
- struct igb_ring *rx_ring,
- struct xdp_buff *xdp)
+struct sk_buff *igb_run_xdp(struct igb_adapter *adapter,
+ struct igb_ring *rx_ring,
+ struct xdp_buff *xdp)
{
int err, result = IGB_XDP_PASS;
struct bpf_prog *xdp_prog;
@@ -8798,9 +8799,9 @@ static bool igb_cleanup_headers(struct igb_ring *rx_ring,
* order to populate the hash, checksum, VLAN, timestamp, protocol, and
* other fields within the skb.
**/
-static void igb_process_skb_fields(struct igb_ring *rx_ring,
- union e1000_adv_rx_desc *rx_desc,
- struct sk_buff *skb)
+void igb_process_skb_fields(struct igb_ring *rx_ring,
+ union e1000_adv_rx_desc *rx_desc,
+ struct sk_buff *skb)
{
struct net_device *dev = rx_ring->netdev;
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 2/4] igb: Introduce txrx ring enable/disable functions
2023-07-04 9:59 [PATCH 0/4] igb: Add support for AF_XDP zero-copy Sriram Yagnaraman
2023-07-04 9:59 ` [PATCH 1/4] igb: prepare for AF_XDP zero-copy support Sriram Yagnaraman
@ 2023-07-04 9:59 ` Sriram Yagnaraman
2023-07-04 15:38 ` Maciej Fijalkowski
2023-07-04 9:59 ` [PATCH 3/4] igb: add AF_XDP zero-copy Rx support Sriram Yagnaraman
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Sriram Yagnaraman @ 2023-07-04 9:59 UTC (permalink / raw)
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Sriram Yagnaraman
Add enable/disable functions for TX and RX rings, will be used in later
patches when AF_XDP zero-copy support is added.
Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
---
drivers/net/ethernet/intel/igb/igb.h | 5 ++-
drivers/net/ethernet/intel/igb/igb_main.c | 41 +++++++++++++++++++++++
2 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index 94440af6cf4b..5fa011c6ef2f 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -384,7 +384,8 @@ enum e1000_ring_flags_t {
IGB_RING_FLAG_RX_SCTP_CSUM,
IGB_RING_FLAG_RX_LB_VLAN_BSWAP,
IGB_RING_FLAG_TX_CTX_IDX,
- IGB_RING_FLAG_TX_DETECT_HANG
+ IGB_RING_FLAG_TX_DETECT_HANG,
+ IGB_RING_FLAG_TX_DISABLED
};
#define ring_uses_large_buffer(ring) \
@@ -735,6 +736,8 @@ void igb_free_tx_resources(struct igb_ring *);
void igb_free_rx_resources(struct igb_ring *);
void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *);
void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *);
+void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid);
+void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid);
void igb_setup_tctl(struct igb_adapter *);
void igb_setup_rctl(struct igb_adapter *);
void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *);
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index dadc3d423cfd..391c0eb136d9 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -4856,6 +4856,47 @@ static void igb_configure_rx(struct igb_adapter *adapter)
}
}
+void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ struct igb_ring *tx_ring = adapter->tx_ring[qid];
+ struct igb_ring *rx_ring = adapter->rx_ring[qid];
+
+ set_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags);
+
+ wr32(E1000_TXDCTL(tx_ring->reg_idx), 0);
+ wr32(E1000_RXDCTL(rx_ring->reg_idx), 0);
+
+ /* Rx/Tx share the same napi context. */
+ napi_disable(&rx_ring->q_vector->napi);
+
+ igb_clean_tx_ring(tx_ring);
+ igb_clean_rx_ring(rx_ring);
+
+ memset(&rx_ring->rx_stats, 0, sizeof(rx_ring->rx_stats));
+ memset(&tx_ring->tx_stats, 0, sizeof(tx_ring->tx_stats));
+}
+
+void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid)
+{
+ struct igb_ring *tx_ring = adapter->tx_ring[qid];
+ struct igb_ring *rx_ring = adapter->rx_ring[qid];
+
+ /* Rx/Tx share the same napi context. */
+ napi_enable(&rx_ring->q_vector->napi);
+
+ igb_configure_tx_ring(adapter, tx_ring);
+ igb_configure_rx_ring(adapter, rx_ring);
+
+ /* call igb_desc_unused which always leaves
+ * at least 1 descriptor unused to make sure
+ * next_to_use != next_to_clean
+ */
+ igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
+
+ clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags);
+}
+
/**
* igb_free_tx_resources - Free Tx Resources per Queue
* @tx_ring: Tx descriptor ring for a specific queue
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 2/4] igb: Introduce txrx ring enable/disable functions
2023-07-04 9:59 ` [PATCH 2/4] igb: Introduce txrx ring enable/disable functions Sriram Yagnaraman
@ 2023-07-04 15:38 ` Maciej Fijalkowski
0 siblings, 0 replies; 11+ messages in thread
From: Maciej Fijalkowski @ 2023-07-04 15:38 UTC (permalink / raw)
To: Sriram Yagnaraman
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend
On Tue, Jul 04, 2023 at 11:59:13AM +0200, Sriram Yagnaraman wrote:
> Add enable/disable functions for TX and RX rings, will be used in later
> patches when AF_XDP zero-copy support is added.
>
> Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
> ---
> drivers/net/ethernet/intel/igb/igb.h | 5 ++-
> drivers/net/ethernet/intel/igb/igb_main.c | 41 +++++++++++++++++++++++
> 2 files changed, 45 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
> index 94440af6cf4b..5fa011c6ef2f 100644
> --- a/drivers/net/ethernet/intel/igb/igb.h
> +++ b/drivers/net/ethernet/intel/igb/igb.h
> @@ -384,7 +384,8 @@ enum e1000_ring_flags_t {
> IGB_RING_FLAG_RX_SCTP_CSUM,
> IGB_RING_FLAG_RX_LB_VLAN_BSWAP,
> IGB_RING_FLAG_TX_CTX_IDX,
> - IGB_RING_FLAG_TX_DETECT_HANG
> + IGB_RING_FLAG_TX_DETECT_HANG,
> + IGB_RING_FLAG_TX_DISABLED
> };
>
> #define ring_uses_large_buffer(ring) \
> @@ -735,6 +736,8 @@ void igb_free_tx_resources(struct igb_ring *);
> void igb_free_rx_resources(struct igb_ring *);
> void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *);
> void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *);
> +void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid);
> +void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid);
> void igb_setup_tctl(struct igb_adapter *);
> void igb_setup_rctl(struct igb_adapter *);
> void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *);
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index dadc3d423cfd..391c0eb136d9 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -4856,6 +4856,47 @@ static void igb_configure_rx(struct igb_adapter *adapter)
> }
> }
>
> +void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid)
they could be static funcs defined in igb_xsk.c i believe? I'll review the
rest after you address the things I have requested on cover letter
response.
> +{
> + struct e1000_hw *hw = &adapter->hw;
> + struct igb_ring *tx_ring = adapter->tx_ring[qid];
> + struct igb_ring *rx_ring = adapter->rx_ring[qid];
> +
> + set_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags);
> +
> + wr32(E1000_TXDCTL(tx_ring->reg_idx), 0);
> + wr32(E1000_RXDCTL(rx_ring->reg_idx), 0);
> +
> + /* Rx/Tx share the same napi context. */
> + napi_disable(&rx_ring->q_vector->napi);
> +
> + igb_clean_tx_ring(tx_ring);
> + igb_clean_rx_ring(rx_ring);
> +
> + memset(&rx_ring->rx_stats, 0, sizeof(rx_ring->rx_stats));
> + memset(&tx_ring->tx_stats, 0, sizeof(tx_ring->tx_stats));
> +}
> +
> +void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid)
> +{
> + struct igb_ring *tx_ring = adapter->tx_ring[qid];
> + struct igb_ring *rx_ring = adapter->rx_ring[qid];
> +
> + /* Rx/Tx share the same napi context. */
> + napi_enable(&rx_ring->q_vector->napi);
> +
> + igb_configure_tx_ring(adapter, tx_ring);
> + igb_configure_rx_ring(adapter, rx_ring);
> +
> + /* call igb_desc_unused which always leaves
> + * at least 1 descriptor unused to make sure
> + * next_to_use != next_to_clean
> + */
> + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
> +
> + clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags);
> +}
> +
> /**
> * igb_free_tx_resources - Free Tx Resources per Queue
> * @tx_ring: Tx descriptor ring for a specific queue
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 3/4] igb: add AF_XDP zero-copy Rx support
2023-07-04 9:59 [PATCH 0/4] igb: Add support for AF_XDP zero-copy Sriram Yagnaraman
2023-07-04 9:59 ` [PATCH 1/4] igb: prepare for AF_XDP zero-copy support Sriram Yagnaraman
2023-07-04 9:59 ` [PATCH 2/4] igb: Introduce txrx ring enable/disable functions Sriram Yagnaraman
@ 2023-07-04 9:59 ` Sriram Yagnaraman
2023-07-04 19:30 ` Simon Horman
2023-07-05 17:16 ` kernel test robot
2023-07-04 9:59 ` [PATCH 4/4] igb: add AF_XDP zero-copy Tx support Sriram Yagnaraman
2023-07-04 15:37 ` [PATCH 0/4] igb: Add support for AF_XDP zero-copy Maciej Fijalkowski
4 siblings, 2 replies; 11+ messages in thread
From: Sriram Yagnaraman @ 2023-07-04 9:59 UTC (permalink / raw)
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Sriram Yagnaraman
Add support for AF_XDP zero-copy receive path.
Add IGB_RING_FLAG_AF_XDP_ZC ring flag to indicate that a ring has AF_XDP
zero-copy support. This flag is used in igb_configure_rx_ring to
register XSK_BUFF_POOL (if zero-copy is enabled) memory model or the
default PAGE_SHARED model otherwise.
When AF_XDP zero-copy is enabled, the rx buffers are allocated from the
xsk buff pool using igb_alloc_rx_buffers_zc.
Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
---
drivers/net/ethernet/intel/igb/Makefile | 2 +-
drivers/net/ethernet/intel/igb/igb.h | 34 +-
drivers/net/ethernet/intel/igb/igb_main.c | 72 ++++-
drivers/net/ethernet/intel/igb/igb_xsk.c | 373 ++++++++++++++++++++++
4 files changed, 463 insertions(+), 18 deletions(-)
create mode 100644 drivers/net/ethernet/intel/igb/igb_xsk.c
diff --git a/drivers/net/ethernet/intel/igb/Makefile b/drivers/net/ethernet/intel/igb/Makefile
index 394c1e0656b9..86d25dba507d 100644
--- a/drivers/net/ethernet/intel/igb/Makefile
+++ b/drivers/net/ethernet/intel/igb/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_IGB) += igb.o
igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \
e1000_mac.o e1000_nvm.o e1000_phy.o e1000_mbx.o \
- e1000_i210.o igb_ptp.o igb_hwmon.o
+ e1000_i210.o igb_ptp.o igb_hwmon.o igb_xsk.o
diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index 5fa011c6ef2f..4f474d7338b5 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -20,6 +20,7 @@
#include <linux/mdio.h>
#include <net/xdp.h>
+#include <net/xdp_sock_drv.h>
struct igb_adapter;
@@ -86,6 +87,7 @@ struct igb_adapter;
#define IGB_XDP_CONSUMED BIT(0)
#define IGB_XDP_TX BIT(1)
#define IGB_XDP_REDIR BIT(2)
+#define IGB_XDP_EXIT BIT(3)
struct vf_data_storage {
unsigned char vf_mac_addresses[ETH_ALEN];
@@ -278,14 +280,23 @@ struct igb_tx_buffer {
};
struct igb_rx_buffer {
- dma_addr_t dma;
- struct page *page;
+ union {
+ struct {
+ struct sk_buff *skb;
+ dma_addr_t dma;
+ struct page *page;
#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
- __u32 page_offset;
+ __u32 page_offset;
#else
- __u16 page_offset;
+ __u16 page_offset;
#endif
- __u16 pagecnt_bias;
+ __u16 pagecnt_bias;
+ };
+ struct {
+ struct xdp_buff *xdp;
+ };
+ };
+
};
struct igb_tx_queue_stats {
@@ -357,6 +368,7 @@ struct igb_ring {
};
};
struct xdp_rxq_info xdp_rxq;
+ struct xsk_buff_pool *xsk_pool;
} ____cacheline_internodealigned_in_smp;
struct igb_q_vector {
@@ -385,7 +397,8 @@ enum e1000_ring_flags_t {
IGB_RING_FLAG_RX_LB_VLAN_BSWAP,
IGB_RING_FLAG_TX_CTX_IDX,
IGB_RING_FLAG_TX_DETECT_HANG,
- IGB_RING_FLAG_TX_DISABLED
+ IGB_RING_FLAG_TX_DISABLED,
+ IGB_RING_FLAG_AF_XDP_ZC
};
#define ring_uses_large_buffer(ring) \
@@ -823,4 +836,13 @@ int igb_add_mac_steering_filter(struct igb_adapter *adapter,
int igb_del_mac_steering_filter(struct igb_adapter *adapter,
const u8 *addr, u8 queue, u8 flags);
+struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter,
+ struct igb_ring *ring);
+int igb_xsk_pool_setup(struct igb_adapter *adapter,
+ struct xsk_buff_pool *pool,
+ u16 qid);
+bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count);
+int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget);
+int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags);
+
#endif /* _IGB_H_ */
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index 391c0eb136d9..f4dbb75d6eac 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -2013,7 +2013,9 @@ static void igb_configure(struct igb_adapter *adapter)
*/
for (i = 0; i < adapter->num_rx_queues; i++) {
struct igb_ring *ring = adapter->rx_ring[i];
- igb_alloc_rx_buffers(ring, igb_desc_unused(ring));
+ ring->xsk_pool ?
+ igb_alloc_rx_buffers_zc(ring, igb_desc_unused(ring)) :
+ igb_alloc_rx_buffers(ring, igb_desc_unused(ring));
}
}
@@ -2931,9 +2933,14 @@ static int igb_xdp_setup(struct net_device *dev, struct netdev_bpf *bpf)
static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp)
{
+ struct igb_adapter *adapter = netdev_priv(dev);
+
switch (xdp->command) {
case XDP_SETUP_PROG:
return igb_xdp_setup(dev, xdp);
+ case XDP_SETUP_XSK_POOL:
+ return igb_xsk_pool_setup(adapter, xdp->xsk.pool,
+ xdp->xsk.queue_id);
default:
return -EINVAL;
}
@@ -3010,6 +3017,9 @@ static int igb_xdp_xmit(struct net_device *dev, int n,
if (unlikely(!tx_ring))
return -ENXIO;
+ if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags)))
+ return -ENXIO;
+
nq = txring_txq(tx_ring);
__netif_tx_lock(nq, cpu);
@@ -3060,6 +3070,7 @@ static const struct net_device_ops igb_netdev_ops = {
.ndo_setup_tc = igb_setup_tc,
.ndo_bpf = igb_xdp,
.ndo_xdp_xmit = igb_xdp_xmit,
+ .ndo_xsk_wakeup = igb_xsk_wakeup,
};
/**
@@ -3356,7 +3367,8 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
netdev->priv_flags |= IFF_SUPP_NOFCS;
netdev->priv_flags |= IFF_UNICAST_FLT;
- netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT;
+ netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT |
+ NETDEV_XDP_ACT_XSK_ZEROCOPY;
/* MTU range: 68 - 9216 */
netdev->min_mtu = ETH_MIN_MTU;
@@ -4734,12 +4746,17 @@ void igb_setup_srrctl(struct igb_adapter *adapter, struct igb_ring *ring)
struct e1000_hw *hw = &adapter->hw;
int reg_idx = ring->reg_idx;
u32 srrctl = 0;
+ u32 buf_size;
- srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT;
- if (ring_uses_large_buffer(ring))
- srrctl |= IGB_RXBUFFER_3072 >> E1000_SRRCTL_BSIZEPKT_SHIFT;
+ if (ring->xsk_pool)
+ buf_size = xsk_pool_get_rx_frame_size(ring->xsk_pool);
+ else if (ring_uses_large_buffer(ring))
+ buf_size = IGB_RXBUFFER_3072;
else
- srrctl |= IGB_RXBUFFER_2048 >> E1000_SRRCTL_BSIZEPKT_SHIFT;
+ buf_size = IGB_RXBUFFER_2048;
+
+ srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT;
+ srrctl |= buf_size >> E1000_SRRCTL_BSIZEPKT_SHIFT;
srrctl |= E1000_SRRCTL_DESCTYPE_ADV_ONEBUF;
if (hw->mac.type >= e1000_82580)
srrctl |= E1000_SRRCTL_TIMESTAMP;
@@ -4771,8 +4788,16 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
u32 rxdctl = 0;
xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
- WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
- MEM_TYPE_PAGE_SHARED, NULL));
+ ring->xsk_pool = igb_xsk_pool(adapter, ring);
+ if (ring->xsk_pool) {
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_XSK_BUFF_POOL,
+ NULL));
+ xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq);
+ } else {
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_PAGE_SHARED, NULL));
+ }
/* disable the queue */
wr32(E1000_RXDCTL(reg_idx), 0);
@@ -4892,7 +4917,9 @@ void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid)
* at least 1 descriptor unused to make sure
* next_to_use != next_to_clean
*/
- igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
+ rx_ring->xsk_pool ?
+ igb_alloc_rx_buffers_zc(rx_ring, igb_desc_unused(rx_ring)) :
+ igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags);
}
@@ -5069,6 +5096,13 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring)
while (i != rx_ring->next_to_alloc) {
struct igb_rx_buffer *buffer_info = &rx_ring->rx_buffer_info[i];
+ if (rx_ring->xsk_pool) {
+ if (buffer_info->xdp)
+ xsk_buff_free(buffer_info->xdp);
+ buffer_info->xdp = NULL;
+ goto skip_for_xsk;
+ }
+
/* Invalidate cache lines that may have been written to by
* device so that we avoid corrupting memory.
*/
@@ -5087,6 +5121,7 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring)
__page_frag_cache_drain(buffer_info->page,
buffer_info->pagecnt_bias);
+skip_for_xsk:
i++;
if (i == rx_ring->count)
i = 0;
@@ -6518,6 +6553,9 @@ netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb,
return NETDEV_TX_BUSY;
}
+ if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags)))
+ return NETDEV_TX_BUSY;
+
/* record the location of the first descriptor for this packet */
first = &tx_ring->tx_buffer_info[tx_ring->next_to_use];
first->type = IGB_TYPE_SKB;
@@ -8260,7 +8298,9 @@ static int igb_poll(struct napi_struct *napi, int budget)
clean_complete = igb_clean_tx_irq(q_vector, budget);
if (q_vector->rx.ring) {
- int cleaned = igb_clean_rx_irq(q_vector, budget);
+ int cleaned = q_vector->rx.ring->xsk_pool ?
+ igb_clean_rx_irq_zc(q_vector, budget) :
+ igb_clean_rx_irq(q_vector, budget);
work_done += cleaned;
if (cleaned >= budget)
@@ -8668,8 +8708,13 @@ struct sk_buff *igb_run_xdp(struct igb_adapter *adapter,
break;
case XDP_REDIRECT:
err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog);
- if (err)
+ if (err) {
+ if (xsk_uses_need_wakeup(rx_ring->xsk_pool) && err == -ENOBUFS)
+ result = IGB_XDP_EXIT;
+ else
+ result = IGB_XDP_CONSUMED;
goto out_failure;
+ }
result = IGB_XDP_REDIR;
break;
default:
@@ -8929,6 +8974,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
struct igb_adapter *adapter = q_vector->adapter;
struct igb_ring *rx_ring = q_vector->rx.ring;
struct sk_buff *skb = rx_ring->skb;
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
unsigned int total_bytes = 0, total_packets = 0;
u16 cleaned_count = igb_desc_unused(rx_ring);
unsigned int xdp_xmit = 0;
@@ -9059,7 +9106,10 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
if (xdp_xmit & IGB_XDP_TX) {
struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter);
+ nq = txring_txq(tx_ring);
+ __netif_tx_lock(nq, cpu);
igb_xdp_ring_update_tail(tx_ring);
+ __netif_tx_unlock(nq);
}
u64_stats_update_begin(&rx_ring->rx_syncp);
diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c
new file mode 100644
index 000000000000..eae616e7608c
--- /dev/null
+++ b/drivers/net/ethernet/intel/igb/igb_xsk.c
@@ -0,0 +1,373 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2018 Intel Corporation. */
+
+#include <linux/bpf_trace.h>
+#include <net/xdp_sock_drv.h>
+#include <net/xdp.h>
+
+#include "e1000_hw.h"
+#include "igb.h"
+
+struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter,
+ struct igb_ring *ring)
+{
+ int qid = ring->queue_index;
+
+ if (!igb_xdp_is_enabled(adapter) ||
+ !test_bit(IGB_RING_FLAG_AF_XDP_ZC, &ring->flags))
+ return NULL;
+
+ return xsk_get_pool_from_qid(adapter->netdev, qid);
+}
+
+static int igb_xsk_pool_enable(struct igb_adapter *adapter,
+ struct xsk_buff_pool *pool,
+ u16 qid)
+{
+ struct net_device *netdev = adapter->netdev;
+ struct igb_ring *rx_ring;
+ bool if_running;
+ int err;
+
+ if (qid >= adapter->num_rx_queues)
+ return -EINVAL;
+
+ if (qid >= netdev->real_num_rx_queues)
+ return -EINVAL;
+
+ err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IGB_RX_DMA_ATTR);
+ if (err)
+ return err;
+
+ rx_ring = adapter->rx_ring[qid];
+ if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter);
+ if (if_running)
+ igb_txrx_ring_disable(adapter, qid);
+
+ set_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
+
+ if (if_running) {
+ igb_txrx_ring_enable(adapter, qid);
+
+ /* Kick start the NAPI context so that receiving will start */
+ err = igb_xsk_wakeup(adapter->netdev, qid, XDP_WAKEUP_RX);
+ if (err) {
+ clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
+ xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR);
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid)
+{
+ struct igb_ring *rx_ring;
+ struct xsk_buff_pool *pool;
+ bool if_running;
+
+ pool = xsk_get_pool_from_qid(adapter->netdev, qid);
+ if (!pool)
+ return -EINVAL;
+
+ rx_ring = adapter->rx_ring[qid];
+ if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter);
+ if (if_running)
+ igb_txrx_ring_disable(adapter, qid);
+
+ xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR);
+ clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
+
+ if (if_running)
+ igb_txrx_ring_enable(adapter, qid);
+
+ return 0;
+}
+
+int igb_xsk_pool_setup(struct igb_adapter *adapter,
+ struct xsk_buff_pool *pool,
+ u16 qid)
+{
+ return pool ? igb_xsk_pool_enable(adapter, pool, qid) :
+ igb_xsk_pool_disable(adapter, qid);
+}
+
+bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count)
+{
+ union e1000_adv_rx_desc *rx_desc;
+ struct igb_rx_buffer *buffer_info;
+ u16 i = rx_ring->next_to_use;
+ dma_addr_t dma;
+ bool ok = true;
+
+ /* nothing to do */
+ if (!count)
+ return true;
+
+ rx_desc = IGB_RX_DESC(rx_ring, i);
+ buffer_info = &rx_ring->rx_buffer_info[i];
+ i -= rx_ring->count;
+
+ do {
+ buffer_info->xdp = xsk_buff_alloc(rx_ring->xsk_pool);
+ if (!buffer_info->xdp) {
+ ok = false;
+ break;
+ }
+
+ dma = xsk_buff_xdp_get_dma(buffer_info->xdp);
+
+ /* Refresh the desc even if buffer_addrs didn't change
+ * because each write-back erases this info.
+ */
+ rx_desc->read.pkt_addr = cpu_to_le64(dma);
+
+ rx_desc++;
+ buffer_info++;
+ i++;
+ if (unlikely(!i)) {
+ rx_desc = IGB_RX_DESC(rx_ring, 0);
+ buffer_info = rx_ring->rx_buffer_info;
+ i -= rx_ring->count;
+ }
+
+ /* clear the length for the next_to_use descriptor */
+ rx_desc->wb.upper.length = 0;
+
+ count--;
+ } while (count);
+
+ i += rx_ring->count;
+
+ if (rx_ring->next_to_use != i) {
+ rx_ring->next_to_use = i;
+
+ /* Force memory writes to complete before letting h/w
+ * know there are new descriptors to fetch. (Only
+ * applicable for weak-ordered memory model archs,
+ * such as IA-64).
+ */
+ wmb();
+ writel(i, rx_ring->tail);
+ }
+
+ return ok;
+}
+
+static struct sk_buff *igb_construct_skb_zc(struct igb_ring *rx_ring,
+ struct xdp_buff *xdp,
+ ktime_t timestamp)
+{
+ unsigned int totalsize = xdp->data_end - xdp->data_meta;
+ unsigned int metasize = xdp->data - xdp->data_meta;
+ struct sk_buff *skb;
+
+ net_prefetch(xdp->data_meta);
+
+ /* allocate a skb to store the frags */
+ skb = __napi_alloc_skb(&rx_ring->q_vector->napi, totalsize,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!skb))
+ return NULL;
+
+ if (timestamp)
+ skb_hwtstamps(skb)->hwtstamp = timestamp;
+
+ memcpy(__skb_put(skb, totalsize), xdp->data_meta,
+ ALIGN(totalsize, sizeof(long)));
+
+ if (metasize) {
+ skb_metadata_set(skb, metasize);
+ __skb_pull(skb, metasize);
+ }
+
+ return skb;
+}
+
+static void igb_update_ntc(struct igb_ring *rx_ring)
+{
+ u32 ntc = rx_ring->next_to_clean + 1;
+
+ /* fetch, update, and store next to clean */
+ ntc = (ntc < rx_ring->count) ? ntc : 0;
+ rx_ring->next_to_clean = ntc;
+
+ prefetch(IGB_RX_DESC(rx_ring, ntc));
+}
+
+int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget)
+{
+ struct igb_adapter *adapter = q_vector->adapter;
+ struct igb_ring *rx_ring = q_vector->rx.ring;
+ struct sk_buff *skb;
+ int cpu = smp_processor_id();
+ struct netdev_queue *nq;
+ unsigned int total_bytes = 0, total_packets = 0;
+ u16 cleaned_count = igb_desc_unused(rx_ring);
+ unsigned int xdp_xmit = 0;
+ bool failure = false;
+
+ while (likely(total_packets < budget)) {
+ union e1000_adv_rx_desc *rx_desc;
+ struct igb_rx_buffer *rx_buffer;
+ ktime_t timestamp = 0;
+ unsigned int size;
+
+ /* return some buffers to hardware, one at a time is too slow */
+ if (cleaned_count >= IGB_RX_BUFFER_WRITE) {
+ igb_alloc_rx_buffers_zc(rx_ring, cleaned_count);
+ cleaned_count = 0;
+ }
+
+ rx_desc = IGB_RX_DESC(rx_ring, rx_ring->next_to_clean);
+ size = le16_to_cpu(rx_desc->wb.upper.length);
+ if (!size)
+ break;
+
+ /* This memory barrier is needed to keep us from reading
+ * any other fields out of the rx_desc until we know the
+ * descriptor has been written back
+ */
+ dma_rmb();
+
+ rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
+ rx_buffer->xdp->data_end = rx_buffer->xdp->data + size;
+ xsk_buff_dma_sync_for_cpu(rx_buffer->xdp, rx_ring->xsk_pool);
+
+ /* pull rx packet timestamp if available and valid */
+ if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) {
+ int ts_hdr_len;
+
+ ts_hdr_len = igb_ptp_rx_pktstamp(rx_ring->q_vector,
+ rx_buffer->xdp->data,
+ ×tamp);
+
+ rx_buffer->xdp->data += ts_hdr_len;
+ rx_buffer->xdp->data_meta += ts_hdr_len;
+ size -= ts_hdr_len;
+ }
+
+ skb = igb_run_xdp(adapter, rx_ring, rx_buffer->xdp);
+
+ if (IS_ERR(skb)) {
+ unsigned int xdp_res = -PTR_ERR(skb);
+
+ if (likely(xdp_res & (IGB_XDP_TX | IGB_XDP_REDIR))) {
+ xdp_xmit |= xdp_res;
+ } else if (xdp_res == IGB_XDP_EXIT) {
+ failure = true;
+ break;
+ } else if (xdp_res == IGB_XDP_CONSUMED) {
+ xsk_buff_free(rx_buffer->xdp);
+ }
+
+ total_packets++;
+ total_bytes += size;
+
+ rx_buffer->xdp = NULL;
+ cleaned_count++;
+ igb_update_ntc(rx_ring);
+ continue;
+ }
+
+ skb = igb_construct_skb_zc(rx_ring, rx_buffer->xdp, timestamp);
+
+ /* exit if we failed to retrieve a buffer */
+ if (!skb) {
+ rx_ring->rx_stats.alloc_failed++;
+ break;
+ }
+
+ xsk_buff_free(rx_buffer->xdp);
+ rx_buffer->xdp = NULL;
+ cleaned_count++;
+ igb_update_ntc(rx_ring);
+
+ if (eth_skb_pad(skb))
+ continue;
+
+ /* probably a little skewed due to removing CRC */
+ total_bytes += skb->len;
+
+ /* populate checksum, timestamp, VLAN, and protocol */
+ igb_process_skb_fields(rx_ring, rx_desc, skb);
+
+ napi_gro_receive(&q_vector->napi, skb);
+
+ /* reset skb pointer */
+ skb = NULL;
+
+ /* update budget accounting */
+ total_packets++;
+ }
+
+ if (xdp_xmit & IGB_XDP_REDIR)
+ xdp_do_flush();
+
+ if (xdp_xmit & IGB_XDP_TX) {
+ struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter);
+
+ nq = txring_txq(tx_ring);
+ __netif_tx_lock(nq, cpu);
+ igb_xdp_ring_update_tail(tx_ring);
+ __netif_tx_unlock(nq);
+ }
+
+ u64_stats_update_begin(&rx_ring->rx_syncp);
+ rx_ring->rx_stats.packets += total_packets;
+ rx_ring->rx_stats.bytes += total_bytes;
+ u64_stats_update_end(&rx_ring->rx_syncp);
+ q_vector->rx.total_packets += total_packets;
+ q_vector->rx.total_bytes += total_bytes;
+
+ if (cleaned_count)
+ igb_alloc_rx_buffers_zc(rx_ring, cleaned_count);
+
+ if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
+ if (failure || rx_ring->next_to_clean == rx_ring->next_to_use)
+ xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
+ else
+ xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
+
+ return (int)total_packets;
+ }
+ return failure ? budget : (int)total_packets;
+}
+
+int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags)
+{
+ struct igb_adapter *adapter = netdev_priv(dev);
+ struct igb_ring *ring;
+ struct e1000_hw *hw = &adapter->hw;
+ u32 eics = 0;
+
+ if (test_bit(__IGB_DOWN, &adapter->state))
+ return -ENETDOWN;
+
+ if (!igb_xdp_is_enabled(adapter))
+ return -EINVAL;
+
+ if (qid >= adapter->num_tx_queues)
+ return -EINVAL;
+
+ ring = adapter->tx_ring[qid];
+
+ if (test_bit(IGB_RING_FLAG_TX_DISABLED, &ring->flags))
+ return -ENETDOWN;
+
+ if (!ring->xsk_pool)
+ return -EINVAL;
+
+ if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) {
+ /* Cause software interrupt to ensure Rx ring is cleaned */
+ if (adapter->flags & IGB_FLAG_HAS_MSIX) {
+ eics |= ring->q_vector->eims_value;
+ wr32(E1000_EICS, eics);
+ } else {
+ wr32(E1000_ICS, E1000_ICS_RXDMT0);
+ }
+ }
+
+ return 0;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 3/4] igb: add AF_XDP zero-copy Rx support
2023-07-04 9:59 ` [PATCH 3/4] igb: add AF_XDP zero-copy Rx support Sriram Yagnaraman
@ 2023-07-04 19:30 ` Simon Horman
2023-07-05 17:16 ` kernel test robot
1 sibling, 0 replies; 11+ messages in thread
From: Simon Horman @ 2023-07-04 19:30 UTC (permalink / raw)
To: Sriram Yagnaraman
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend
On Tue, Jul 04, 2023 at 11:59:14AM +0200, Sriram Yagnaraman wrote:
> Add support for AF_XDP zero-copy receive path.
>
> Add IGB_RING_FLAG_AF_XDP_ZC ring flag to indicate that a ring has AF_XDP
> zero-copy support. This flag is used in igb_configure_rx_ring to
> register XSK_BUFF_POOL (if zero-copy is enabled) memory model or the
> default PAGE_SHARED model otherwise.
>
> When AF_XDP zero-copy is enabled, the rx buffers are allocated from the
> xsk buff pool using igb_alloc_rx_buffers_zc.
>
> Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
...
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index 391c0eb136d9..f4dbb75d6eac 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -2013,7 +2013,9 @@ static void igb_configure(struct igb_adapter *adapter)
> */
> for (i = 0; i < adapter->num_rx_queues; i++) {
> struct igb_ring *ring = adapter->rx_ring[i];
> - igb_alloc_rx_buffers(ring, igb_desc_unused(ring));
> + ring->xsk_pool ?
> + igb_alloc_rx_buffers_zc(ring, igb_desc_unused(ring)) :
> + igb_alloc_rx_buffers(ring, igb_desc_unused(ring));
This construction seems a little unusual (to me) as the result of
the ternary operator is not assigned. I wonder if it it would
be sensible to follow a simple if/else pattern here.o
Flagged by Sparse as:
.../igb_main.c:2016:32: error: incompatible types in conditional expression (different base types):
.../igb_main.c:2016:32: bool
.../igb_main.c:2016:32: void
...
> @@ -4892,7 +4917,9 @@ void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid)
> * at least 1 descriptor unused to make sure
> * next_to_use != next_to_clean
> */
> - igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
> + rx_ring->xsk_pool ?
> + igb_alloc_rx_buffers_zc(rx_ring, igb_desc_unused(rx_ring)) :
> + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring));
Ditto.
...
> +static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid)
> +{
> + struct igb_ring *rx_ring;
> + struct xsk_buff_pool *pool;
> + bool if_running;
Please use reverse xmas tree - longest line to shortest - for
new Networking code. Likewise elsewhere in this patch.
You can use the tool at the link below to help.
https://github.com/ecree-solarflare/xmastree
...
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH 3/4] igb: add AF_XDP zero-copy Rx support
2023-07-04 9:59 ` [PATCH 3/4] igb: add AF_XDP zero-copy Rx support Sriram Yagnaraman
2023-07-04 19:30 ` Simon Horman
@ 2023-07-05 17:16 ` kernel test robot
1 sibling, 0 replies; 11+ messages in thread
From: kernel test robot @ 2023-07-05 17:16 UTC (permalink / raw)
To: Sriram Yagnaraman
Cc: oe-kbuild-all, intel-wired-lan, bpf, netdev, Jesse Brandeburg,
Tony Nguyen, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
Jesper Dangaard Brouer, John Fastabend, Sriram Yagnaraman
Hi Sriram,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tnguy-next-queue/dev-queue]
[also build test WARNING on tnguy-net-queue/dev-queue horms-ipvs/master linus/master v6.4 next-20230705]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Sriram-Yagnaraman/igb-prepare-for-AF_XDP-zero-copy-support/20230704-180613
base: https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue.git dev-queue
patch link: https://lore.kernel.org/r/20230704095915.9750-4-sriram.yagnaraman%40est.tech
patch subject: [PATCH 3/4] igb: add AF_XDP zero-copy Rx support
config: openrisc-randconfig-r093-20230705 (https://download.01.org/0day-ci/archive/20230706/202307060154.LjqekYIL-lkp@intel.com/config)
compiler: or1k-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230706/202307060154.LjqekYIL-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307060154.LjqekYIL-lkp@intel.com/
sparse warnings: (new ones prefixed by >>)
>> drivers/net/ethernet/intel/igb/igb_main.c:2016:32: sparse: sparse: incompatible types in conditional expression (different base types):
>> drivers/net/ethernet/intel/igb/igb_main.c:2016:32: sparse: bool
>> drivers/net/ethernet/intel/igb/igb_main.c:2016:32: sparse: void
drivers/net/ethernet/intel/igb/igb_main.c:4920:27: sparse: sparse: incompatible types in conditional expression (different base types):
drivers/net/ethernet/intel/igb/igb_main.c:4920:27: sparse: bool
drivers/net/ethernet/intel/igb/igb_main.c:4920:27: sparse: void
vim +2016 drivers/net/ethernet/intel/igb/igb_main.c
1984
1985 /**
1986 * igb_configure - configure the hardware for RX and TX
1987 * @adapter: private board structure
1988 **/
1989 static void igb_configure(struct igb_adapter *adapter)
1990 {
1991 struct net_device *netdev = adapter->netdev;
1992 int i;
1993
1994 igb_get_hw_control(adapter);
1995 igb_set_rx_mode(netdev);
1996 igb_setup_tx_mode(adapter);
1997
1998 igb_restore_vlan(adapter);
1999
2000 igb_setup_tctl(adapter);
2001 igb_setup_mrqc(adapter);
2002 igb_setup_rctl(adapter);
2003
2004 igb_nfc_filter_restore(adapter);
2005 igb_configure_tx(adapter);
2006 igb_configure_rx(adapter);
2007
2008 igb_rx_fifo_flush_82575(&adapter->hw);
2009
2010 /* call igb_desc_unused which always leaves
2011 * at least 1 descriptor unused to make sure
2012 * next_to_use != next_to_clean
2013 */
2014 for (i = 0; i < adapter->num_rx_queues; i++) {
2015 struct igb_ring *ring = adapter->rx_ring[i];
> 2016 ring->xsk_pool ?
2017 igb_alloc_rx_buffers_zc(ring, igb_desc_unused(ring)) :
2018 igb_alloc_rx_buffers(ring, igb_desc_unused(ring));
2019 }
2020 }
2021
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 4/4] igb: add AF_XDP zero-copy Tx support
2023-07-04 9:59 [PATCH 0/4] igb: Add support for AF_XDP zero-copy Sriram Yagnaraman
` (2 preceding siblings ...)
2023-07-04 9:59 ` [PATCH 3/4] igb: add AF_XDP zero-copy Rx support Sriram Yagnaraman
@ 2023-07-04 9:59 ` Sriram Yagnaraman
2023-07-04 15:37 ` [PATCH 0/4] igb: Add support for AF_XDP zero-copy Maciej Fijalkowski
4 siblings, 0 replies; 11+ messages in thread
From: Sriram Yagnaraman @ 2023-07-04 9:59 UTC (permalink / raw)
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, Sriram Yagnaraman
Add support for AF_XDP zero-copy transmit path.
A new TX buffer type IGB_TYPE_XSK is introduced to indicate that the Tx
frame was allocated from the xsk buff pool, so igb_clean_tx_ring and
igb_clean_tx_irq can clean the buffers correctly based on type.
igb_xmit_zc performs the actual packet transmit when AF_XDP zero-copy is
enabled. We share the TX ring between slow path, XDP and AF_XDP
zero-copy, so we use the netdev queue lock to ensure mutual exclusion.
Signed-off-by: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
---
drivers/net/ethernet/intel/igb/igb.h | 2 +
drivers/net/ethernet/intel/igb/igb_main.c | 44 +++++++++++++--
drivers/net/ethernet/intel/igb/igb_xsk.c | 67 ++++++++++++++++++++++-
3 files changed, 105 insertions(+), 8 deletions(-)
diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index 4f474d7338b5..564706ab0646 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -257,6 +257,7 @@ enum igb_tx_flags {
enum igb_tx_buf_type {
IGB_TYPE_SKB = 0,
IGB_TYPE_XDP,
+ IGB_TYPE_XSK
};
/* wrapper around a pointer to a socket buffer,
@@ -843,6 +844,7 @@ int igb_xsk_pool_setup(struct igb_adapter *adapter,
u16 qid);
bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count);
int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget);
+bool igb_xmit_zc(struct igb_ring *tx_ring, unsigned int budget);
int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags);
#endif /* _IGB_H_ */
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index f4dbb75d6eac..c6bb5b1944c8 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -4390,6 +4390,8 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
u64 tdba = ring->dma;
int reg_idx = ring->reg_idx;
+ ring->xsk_pool = igb_xsk_pool(adapter, ring);
+
wr32(E1000_TDLEN(reg_idx),
ring->count * sizeof(union e1000_adv_tx_desc));
wr32(E1000_TDBAL(reg_idx),
@@ -4970,15 +4972,20 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring)
{
u16 i = tx_ring->next_to_clean;
struct igb_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
+ u32 xsk_frames = 0;
while (i != tx_ring->next_to_use) {
union e1000_adv_tx_desc *eop_desc, *tx_desc;
/* Free all the Tx ring sk_buffs or xdp frames */
- if (tx_buffer->type == IGB_TYPE_SKB)
+ if (tx_buffer->type == IGB_TYPE_SKB) {
dev_kfree_skb_any(tx_buffer->skb);
- else
+ } else if (tx_buffer->type == IGB_TYPE_XDP) {
xdp_return_frame(tx_buffer->xdpf);
+ } else if (tx_buffer->type == IGB_TYPE_XSK) {
+ xsk_frames++;
+ goto skip_for_xsk;
+ }
/* unmap skb header data */
dma_unmap_single(tx_ring->dev,
@@ -5009,6 +5016,7 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring)
DMA_TO_DEVICE);
}
+skip_for_xsk:
tx_buffer->next_to_watch = NULL;
/* move us one more past the eop_desc for start of next pkt */
@@ -5023,6 +5031,9 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring)
/* reset BQL for queue */
netdev_tx_reset_queue(txring_txq(tx_ring));
+ if (tx_ring->xsk_pool && xsk_frames)
+ xsk_tx_completed(tx_ring->xsk_pool, xsk_frames);
+
/* reset next_to_use and next_to_clean */
tx_ring->next_to_use = 0;
tx_ring->next_to_clean = 0;
@@ -8330,12 +8341,16 @@ static int igb_poll(struct napi_struct *napi, int budget)
static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget)
{
struct igb_adapter *adapter = q_vector->adapter;
+ int cpu = smp_processor_id();
struct igb_ring *tx_ring = q_vector->tx.ring;
struct igb_tx_buffer *tx_buffer;
union e1000_adv_tx_desc *tx_desc;
+ struct netdev_queue *nq;
unsigned int total_bytes = 0, total_packets = 0;
unsigned int budget = q_vector->tx.work_limit;
unsigned int i = tx_ring->next_to_clean;
+ u32 xsk_frames = 0;
+ bool xsk_xmit_done = true;
if (test_bit(__IGB_DOWN, &adapter->state))
return true;
@@ -8366,10 +8381,14 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget)
total_packets += tx_buffer->gso_segs;
/* free the skb */
- if (tx_buffer->type == IGB_TYPE_SKB)
+ if (tx_buffer->type == IGB_TYPE_SKB) {
napi_consume_skb(tx_buffer->skb, napi_budget);
- else
+ } else if (tx_buffer->type == IGB_TYPE_XDP) {
xdp_return_frame(tx_buffer->xdpf);
+ } else if (tx_buffer->type == IGB_TYPE_XSK) {
+ xsk_frames++;
+ goto skip_for_xsk;
+ }
/* unmap skb header data */
dma_unmap_single(tx_ring->dev,
@@ -8401,6 +8420,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget)
}
}
+skip_for_xsk:
/* move us one more past the eop_desc for start of next pkt */
tx_buffer++;
tx_desc++;
@@ -8429,6 +8449,20 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget)
q_vector->tx.total_bytes += total_bytes;
q_vector->tx.total_packets += total_packets;
+ if (tx_ring->xsk_pool) {
+ if (xsk_frames)
+ xsk_tx_completed(tx_ring->xsk_pool, xsk_frames);
+ if (xsk_uses_need_wakeup(tx_ring->xsk_pool))
+ xsk_set_tx_need_wakeup(tx_ring->xsk_pool);
+
+ nq = txring_txq(tx_ring);
+ __netif_tx_lock(nq, cpu);
+ /* Avoid transmit queue timeout since we share it with the slow path */
+ txq_trans_cond_update(nq);
+ xsk_xmit_done = igb_xmit_zc(tx_ring, q_vector->tx.work_limit);
+ __netif_tx_unlock(nq);
+ }
+
if (test_bit(IGB_RING_FLAG_TX_DETECT_HANG, &tx_ring->flags)) {
struct e1000_hw *hw = &adapter->hw;
@@ -8491,7 +8525,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget)
}
}
- return !!budget;
+ return !!budget && xsk_xmit_done;
}
/**
diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c
index eae616e7608c..8b60285ec242 100644
--- a/drivers/net/ethernet/intel/igb/igb_xsk.c
+++ b/drivers/net/ethernet/intel/igb/igb_xsk.c
@@ -25,25 +25,28 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter,
u16 qid)
{
struct net_device *netdev = adapter->netdev;
- struct igb_ring *rx_ring;
+ struct igb_ring *tx_ring, *rx_ring;
bool if_running;
int err;
if (qid >= adapter->num_rx_queues)
return -EINVAL;
- if (qid >= netdev->real_num_rx_queues)
+ if (qid >= netdev->real_num_rx_queues ||
+ qid >= netdev->real_num_tx_queues)
return -EINVAL;
err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IGB_RX_DMA_ATTR);
if (err)
return err;
+ tx_ring = adapter->tx_ring[qid];
rx_ring = adapter->rx_ring[qid];
if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter);
if (if_running)
igb_txrx_ring_disable(adapter, qid);
+ set_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags);
set_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
if (if_running) {
@@ -52,6 +55,7 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter,
/* Kick start the NAPI context so that receiving will start */
err = igb_xsk_wakeup(adapter->netdev, qid, XDP_WAKEUP_RX);
if (err) {
+ clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags);
clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR);
return err;
@@ -63,7 +67,7 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter,
static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid)
{
- struct igb_ring *rx_ring;
+ struct igb_ring *tx_ring, *rx_ring;
struct xsk_buff_pool *pool;
bool if_running;
@@ -71,12 +75,14 @@ static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid)
if (!pool)
return -EINVAL;
+ tx_ring = adapter->tx_ring[qid];
rx_ring = adapter->rx_ring[qid];
if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter);
if (if_running)
igb_txrx_ring_disable(adapter, qid);
xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR);
+ clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags);
clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags);
if (if_running)
@@ -335,6 +341,61 @@ int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget)
return failure ? budget : (int)total_packets;
}
+bool igb_xmit_zc(struct igb_ring *tx_ring, unsigned int budget)
+{
+ struct xsk_buff_pool *pool = tx_ring->xsk_pool;
+ union e1000_adv_tx_desc *tx_desc = NULL;
+ struct igb_tx_buffer *tx_bi;
+ bool work_done = true;
+ struct xdp_desc desc;
+ dma_addr_t dma;
+ u32 cmd_type;
+
+ while (budget-- > 0) {
+ if (unlikely(!igb_desc_unused(tx_ring))) {
+ work_done = false;
+ break;
+ }
+
+ if (!netif_carrier_ok(tx_ring->netdev))
+ break;
+
+ if (!xsk_tx_peek_desc(pool, &desc))
+ break;
+
+ dma = xsk_buff_raw_get_dma(pool, desc.addr);
+ xsk_buff_raw_dma_sync_for_device(pool, dma, desc.len);
+
+ tx_bi = &tx_ring->tx_buffer_info[tx_ring->next_to_use];
+ tx_bi->bytecount = desc.len;
+ tx_bi->type = IGB_TYPE_XSK;
+ tx_bi->xdpf = NULL;
+ tx_bi->gso_segs = 1;
+
+ tx_desc = IGB_TX_DESC(tx_ring, tx_ring->next_to_use);
+ tx_desc->read.buffer_addr = cpu_to_le64(dma);
+
+ /* put descriptor type bits */
+ cmd_type = E1000_ADVTXD_DTYP_DATA | E1000_ADVTXD_DCMD_DEXT |
+ E1000_ADVTXD_DCMD_IFCS;
+
+ cmd_type |= desc.len | IGB_TXD_DCMD;
+ tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
+ tx_desc->read.olinfo_status = 0;
+
+ tx_ring->next_to_use++;
+ if (tx_ring->next_to_use == tx_ring->count)
+ tx_ring->next_to_use = 0;
+ }
+
+ if (tx_desc) {
+ igb_xdp_ring_update_tail(tx_ring);
+ xsk_tx_release(pool);
+ }
+
+ return !!budget && work_done;
+}
+
int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags)
{
struct igb_adapter *adapter = netdev_priv(dev);
--
2.34.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 0/4] igb: Add support for AF_XDP zero-copy
2023-07-04 9:59 [PATCH 0/4] igb: Add support for AF_XDP zero-copy Sriram Yagnaraman
` (3 preceding siblings ...)
2023-07-04 9:59 ` [PATCH 4/4] igb: add AF_XDP zero-copy Tx support Sriram Yagnaraman
@ 2023-07-04 15:37 ` Maciej Fijalkowski
2023-07-04 18:48 ` Sriram Yagnaraman
4 siblings, 1 reply; 11+ messages in thread
From: Maciej Fijalkowski @ 2023-07-04 15:37 UTC (permalink / raw)
To: Sriram Yagnaraman
Cc: intel-wired-lan, bpf, netdev, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend
On Tue, Jul 04, 2023 at 11:59:11AM +0200, Sriram Yagnaraman wrote:
Hi Sriram,
> Disclaimer: My first patches to Intel drivers, implemented AF_XDP
> zero-copy feature which seemed to be missing for igb. Not sure if it was
> a conscious choice to not spend time implementing this for older
> devices, nevertheless I send them to the list for review.
>
> The first couple of patches adds helper funcctions to prepare for AF_XDP
> zero-copy support which comes in the last couple of patches, one each
> for Rx and TX paths.
please include perf numbers in cover letter, CC AF_XDP maintainers and use
batch XSK APIs: xsk_buff_alloc_batch(), xsk_tx_peek_release_desc_batch().
Thanks!
>
> Sriram Yagnaraman (4):
> igb: prepare for AF_XDP zero-copy support
> igb: Introduce txrx ring enable/disable functions
> igb: add AF_XDP zero-copy Rx support
> igb: add AF_XDP zero-copy Tx support
>
> drivers/net/ethernet/intel/igb/Makefile | 2 +-
> drivers/net/ethernet/intel/igb/igb.h | 52 ++-
> drivers/net/ethernet/intel/igb/igb_main.c | 178 +++++++--
> drivers/net/ethernet/intel/igb/igb_xsk.c | 434 ++++++++++++++++++++++
> 4 files changed, 633 insertions(+), 33 deletions(-)
> create mode 100644 drivers/net/ethernet/intel/igb/igb_xsk.c
>
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread* RE: [PATCH 0/4] igb: Add support for AF_XDP zero-copy
2023-07-04 15:37 ` [PATCH 0/4] igb: Add support for AF_XDP zero-copy Maciej Fijalkowski
@ 2023-07-04 18:48 ` Sriram Yagnaraman
0 siblings, 0 replies; 11+ messages in thread
From: Sriram Yagnaraman @ 2023-07-04 18:48 UTC (permalink / raw)
To: Maciej Fijalkowski
Cc: intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org,
netdev@vger.kernel.org, Jesse Brandeburg, Tony Nguyen,
David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend
> -----Original Message-----
> From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
> Sent: Tuesday, 4 July 2023 17:37
> To: Sriram Yagnaraman <sriram.yagnaraman@est.tech>
> Cc: intel-wired-lan@lists.osuosl.org; bpf@vger.kernel.org;
> netdev@vger.kernel.org; Jesse Brandeburg <jesse.brandeburg@intel.com>;
> Tony Nguyen <anthony.l.nguyen@intel.com>; David S . Miller
> <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub
> Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Alexei
> Starovoitov <ast@kernel.org>; Daniel Borkmann <daniel@iogearbox.net>;
> Jesper Dangaard Brouer <hawk@kernel.org>; John Fastabend
> <john.fastabend@gmail.com>
> Subject: Re: [PATCH 0/4] igb: Add support for AF_XDP zero-copy
>
> On Tue, Jul 04, 2023 at 11:59:11AM +0200, Sriram Yagnaraman wrote:
>
> Hi Sriram,
>
> > Disclaimer: My first patches to Intel drivers, implemented AF_XDP
> > zero-copy feature which seemed to be missing for igb. Not sure if it
> > was a conscious choice to not spend time implementing this for older
> > devices, nevertheless I send them to the list for review.
> >
> > The first couple of patches adds helper funcctions to prepare for
> > AF_XDP zero-copy support which comes in the last couple of patches,
> > one each for Rx and TX paths.
>
> please include perf numbers in cover letter, CC AF_XDP maintainers and use
> batch XSK APIs: xsk_buff_alloc_batch(), xsk_tx_peek_release_desc_batch().
>
> Thanks!
>
Thank you so much for the quick reply. I will respin addressing your comments.
For the perf numbers, I must confess, I only used the newly delivered IGB device emulation in qemu. I don't have access to a real NIC to provide realistic numbers. But of course, I can provide a comparison between XSK_COPY and XSK_ZEROCOPY using the emulator.
> >
> > Sriram Yagnaraman (4):
> > igb: prepare for AF_XDP zero-copy support
> > igb: Introduce txrx ring enable/disable functions
> > igb: add AF_XDP zero-copy Rx support
> > igb: add AF_XDP zero-copy Tx support
> >
> > drivers/net/ethernet/intel/igb/Makefile | 2 +-
> > drivers/net/ethernet/intel/igb/igb.h | 52 ++-
> > drivers/net/ethernet/intel/igb/igb_main.c | 178 +++++++--
> > drivers/net/ethernet/intel/igb/igb_xsk.c | 434 ++++++++++++++++++++++
> > 4 files changed, 633 insertions(+), 33 deletions(-) create mode
> > 100644 drivers/net/ethernet/intel/igb/igb_xsk.c
> >
> > --
> > 2.34.1
> >
> >
^ permalink raw reply [flat|nested] 11+ messages in thread