[Intel-wired-lan] [PATCH 0/9] igc: Add XDP support

[Intel-wired-lan] [PATCH 0/9] igc: Add XDP support

[Andre Guedes]

Hi all,

This patch series adds XDP support for the igc driver. The approach implemented
by this series follows the same approach implemented in other Intel drivers as
much as possible for the sake of consistency.

The series is organized in two parts. In the first part, i.e. patches from 1 to
5, igc_main.c and igc_ptp.c code is refactored in preparation for landing the
XDP support, which is introduced in the second part (patches from 6 to 9).

As far as code organization is concerned, XDP-related helpers are defined in a
new file, igc_xdp.c, and are called by igc_main.c.

The features added by this series have been tested with the samples provided in
samples/bpf/: xdp1, xdp2, xdp_redirect_cpu, and xdp_redirect_map.

Upcoming series will add support of UMEM and zero-copy features from AF_XDP.

Best regards,
Andre

Andre Guedes (9):
  igc: Fix igc_ptp_rx_pktstamp()
  igc: Remove unused argument from igc_tx_cmd_type()
  igc: Introduce igc_rx_buffer_flip() helper
  igc: Introduce igc_get_rx_frame_truesize() helper
  igc: Refactor rx timestamp handling
  igc: Add Initial XDP support
  igc: Add support for XDP_TX action
  igc: Add support for XDP_REDIRECT action
  igc: Implement ndo_xdp_xmit callback

 drivers/net/ethernet/intel/igc/Makefile   |   2 +-
 drivers/net/ethernet/intel/igc/igc.h      |  18 +-
 drivers/net/ethernet/intel/igc/igc_main.c | 280 +++++++++++++++++-----
 drivers/net/ethernet/intel/igc/igc_ptp.c  |  87 ++++---
 drivers/net/ethernet/intel/igc/igc_xdp.c  | 202 ++++++++++++++++
 drivers/net/ethernet/intel/igc/igc_xdp.h  |  27 +++
 6 files changed, 516 insertions(+), 100 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.c
 create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.h

-- 
2.26.2

[Intel-wired-lan] [PATCH 1/9] igc: Fix igc_ptp_rx_pktstamp()

[Andre Guedes]

The comment describing the timestamps layout in the packet buffer is
wrong and the code is actually retrieving the timestamp in Timer 1
reference instead of Timer 0. This hasn't been a big issue so far
because hardware is configured to report both timestamps using Timer 0
(see IGC_SRRCTL register configuration in igc_ptp_enable_rx_timestamp()
helper). This patch fixes the comment and the code so we retrieve the
timestamp in Timer 0 reference as expected.

This patch also takes the opportunity to get rid of the hw.mac.type check
since it is not required.

Fixes: 81b055205e8ba ("igc: Add support for RX timestamping")
Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc.h     |  2 +-
 drivers/net/ethernet/intel/igc/igc_ptp.c | 72 +++++++++++++-----------
 2 files changed, 41 insertions(+), 33 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
index 35baae900c1f..b9b924794b22 100644
--- a/drivers/net/ethernet/intel/igc/igc.h
+++ b/drivers/net/ethernet/intel/igc/igc.h
@@ -545,7 +545,7 @@ void igc_ptp_init(struct igc_adapter *adapter);
 void igc_ptp_reset(struct igc_adapter *adapter);
 void igc_ptp_suspend(struct igc_adapter *adapter);
 void igc_ptp_stop(struct igc_adapter *adapter);
-void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, void *va,
+void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
 			 struct sk_buff *skb);
 int igc_ptp_set_ts_config(struct net_device *netdev, struct ifreq *ifr);
 int igc_ptp_get_ts_config(struct net_device *netdev, struct ifreq *ifr);
diff --git a/drivers/net/ethernet/intel/igc/igc_ptp.c b/drivers/net/ethernet/intel/igc/igc_ptp.c
index ac0b9c85da7c..916721e2cd49 100644
--- a/drivers/net/ethernet/intel/igc/igc_ptp.c
+++ b/drivers/net/ethernet/intel/igc/igc_ptp.c
@@ -152,46 +152,54 @@ static void igc_ptp_systim_to_hwtstamp(struct igc_adapter *adapter,
 }
 
 /**
- * igc_ptp_rx_pktstamp - retrieve Rx per packet timestamp
+ * igc_ptp_rx_pktstamp - Retrieve timestamp from rx packet buffer
  * @q_vector: Pointer to interrupt specific structure
  * @va: Pointer to address containing Rx buffer
  * @skb: Buffer containing timestamp and packet
  *
- * This function is meant to retrieve the first timestamp from the
- * first buffer of an incoming frame. The value is stored in little
- * endian format starting on byte 0. There's a second timestamp
- * starting on byte 8.
- **/
-void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, void *va,
+ * This function retrieves the timestamp saved in the beginning of packet
+ * buffer. While two timestamps are available, one in timer0 reference and the
+ * other in timer1 reference, this function considers only the timestamp in
+ * timer0 reference.
+ */
+void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
 			 struct sk_buff *skb)
 {
 	struct igc_adapter *adapter = q_vector->adapter;
-	__le64 *regval = (__le64 *)va;
-	int adjust = 0;
-
-	/* The timestamp is recorded in little endian format.
-	 * DWORD: | 0          | 1           | 2          | 3
-	 * Field: | Timer0 Low | Timer0 High | Timer1 Low | Timer1 High
+	u64 regval;
+	int adjust;
+
+	/* Timestamps are saved in little endian at the beginning of the packet
+	 * buffer following the layout:
+	 *
+	 * | 0              | 1              | 2              | 3              |
+	 * | Timer1 SYSTIML | Timer1 SYSTIMH | Timer0 SYSTIML | Timer0 SYSTIMH |
+	 *
+	 * SYSTIML holds the nanoseconds part while SYSTIMH holds the seconds
+	 * part of the timestamp.
 	 */
-	igc_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb),
-				   le64_to_cpu(regval[0]));
-
-	/* adjust timestamp for the RX latency based on link speed */
-	if (adapter->hw.mac.type == igc_i225) {
-		switch (adapter->link_speed) {
-		case SPEED_10:
-			adjust = IGC_I225_RX_LATENCY_10;
-			break;
-		case SPEED_100:
-			adjust = IGC_I225_RX_LATENCY_100;
-			break;
-		case SPEED_1000:
-			adjust = IGC_I225_RX_LATENCY_1000;
-			break;
-		case SPEED_2500:
-			adjust = IGC_I225_RX_LATENCY_2500;
-			break;
-		}
+	regval = le32_to_cpu(va[2]);
+	regval |= (u64)le32_to_cpu(va[3]) << 32;
+	igc_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb), regval);
+
+	/* Adjust timestamp for the RX latency based on link speed */
+	switch (adapter->link_speed) {
+	case SPEED_10:
+		adjust = IGC_I225_RX_LATENCY_10;
+		break;
+	case SPEED_100:
+		adjust = IGC_I225_RX_LATENCY_100;
+		break;
+	case SPEED_1000:
+		adjust = IGC_I225_RX_LATENCY_1000;
+		break;
+	case SPEED_2500:
+		adjust = IGC_I225_RX_LATENCY_2500;
+		break;
+	default:
+		adjust = 0;
+		netdev_warn_once(adapter->netdev, "Imprecise timestamp\n");
+		break;
 	}
 	skb_hwtstamps(skb)->hwtstamp =
 		ktime_sub_ns(skb_hwtstamps(skb)->hwtstamp, adjust);
-- 
2.26.2

[Intel-wired-lan] [PATCH 2/9] igc: Remove unused argument from igc_tx_cmd_type()

[Andre Guedes]

The 'skb' argument from igc_tx_cmd_type() is not used so this patch
removes it.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 897e2d5514a1..63a0d7a2213c 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -1040,7 +1040,7 @@ static inline int igc_maybe_stop_tx(struct igc_ring *tx_ring, const u16 size)
 	 ((u32)((_input) & (_flag)) * ((_result) / (_flag))) :	\
 	 ((u32)((_input) & (_flag)) / ((_flag) / (_result))))
 
-static u32 igc_tx_cmd_type(struct sk_buff *skb, u32 tx_flags)
+static u32 igc_tx_cmd_type(u32 tx_flags)
 {
 	/* set type for advanced descriptor with frame checksum insertion */
 	u32 cmd_type = IGC_ADVTXD_DTYP_DATA |
@@ -1089,7 +1089,7 @@ static int igc_tx_map(struct igc_ring *tx_ring,
 	u16 i = tx_ring->next_to_use;
 	unsigned int data_len, size;
 	dma_addr_t dma;
-	u32 cmd_type = igc_tx_cmd_type(skb, tx_flags);
+	u32 cmd_type = igc_tx_cmd_type(tx_flags);
 
 	tx_desc = IGC_TX_DESC(tx_ring, i);
 
-- 
2.26.2

[Intel-wired-lan] [PATCH 3/9] igc: Introduce igc_rx_buffer_flip() helper

[Andre Guedes]

The igc driver implements the same page recycling scheme from other
Intel drivers which reuses the page by flipping the buffer. The code
to handle buffer flips is duplicated in many locations so this patch
introduces the igc_rx_buffer_flip() helper and uses it where applicable.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc_main.c | 42 +++++++++++------------
 1 file changed, 20 insertions(+), 22 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 63a0d7a2213c..5015c39a82ba 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -1511,6 +1511,16 @@ static struct igc_rx_buffer *igc_get_rx_buffer(struct igc_ring *rx_ring,
 	return rx_buffer;
 }
 
+static void igc_rx_buffer_flip(struct igc_rx_buffer *buffer,
+			       unsigned int truesize)
+{
+#if (PAGE_SIZE < 8192)
+	buffer->page_offset ^= truesize;
+#else
+	buffer->page_offset += truesize;
+#endif
+}
+
 /**
  * igc_add_rx_frag - Add contents of Rx buffer to sk_buff
  * @rx_ring: rx descriptor ring to transact packets on
@@ -1525,20 +1535,18 @@ static void igc_add_rx_frag(struct igc_ring *rx_ring,
 			    struct sk_buff *skb,
 			    unsigned int size)
 {
+	unsigned int truesize;
 #if (PAGE_SIZE < 8192)
-	unsigned int truesize = igc_rx_pg_size(rx_ring) / 2;
-
-	skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
-			rx_buffer->page_offset, size, truesize);
-	rx_buffer->page_offset ^= truesize;
+	truesize = igc_rx_pg_size(rx_ring) / 2;
 #else
-	unsigned int truesize = ring_uses_build_skb(rx_ring) ?
-				SKB_DATA_ALIGN(IGC_SKB_PAD + size) :
-				SKB_DATA_ALIGN(size);
+	truesize = ring_uses_build_skb(rx_ring) ?
+		   SKB_DATA_ALIGN(IGC_SKB_PAD + size) :
+		   SKB_DATA_ALIGN(size);
+#endif
 	skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
 			rx_buffer->page_offset, size, truesize);
-	rx_buffer->page_offset += truesize;
-#endif
+
+	igc_rx_buffer_flip(rx_buffer, truesize);
 }
 
 static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
@@ -1567,13 +1575,7 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
 	skb_reserve(skb, IGC_SKB_PAD);
 	__skb_put(skb, size);
 
-	/* update buffer offset */
-#if (PAGE_SIZE < 8192)
-	rx_buffer->page_offset ^= truesize;
-#else
-	rx_buffer->page_offset += truesize;
-#endif
-
+	igc_rx_buffer_flip(rx_buffer, truesize);
 	return skb;
 }
 
@@ -1619,11 +1621,7 @@ static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
 		skb_add_rx_frag(skb, 0, rx_buffer->page,
 				(va + headlen) - page_address(rx_buffer->page),
 				size, truesize);
-#if (PAGE_SIZE < 8192)
-		rx_buffer->page_offset ^= truesize;
-#else
-		rx_buffer->page_offset += truesize;
-#endif
+		igc_rx_buffer_flip(rx_buffer, truesize);
 	} else {
 		rx_buffer->pagecnt_bias++;
 	}
-- 
2.26.2

[Intel-wired-lan] [PATCH 4/9] igc: Introduce igc_get_rx_frame_truesize() helper

[Andre Guedes]

The RX frame truesize calculation is scattered throughout the RX code.
This patch creates the helper function igc_get_rx_frame_truesize() and
uses it where applicable.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc_main.c | 29 ++++++++++++++---------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 5015c39a82ba..850e2bbd9773 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -1521,6 +1521,22 @@ static void igc_rx_buffer_flip(struct igc_rx_buffer *buffer,
 #endif
 }
 
+static unsigned int igc_get_rx_frame_truesize(struct igc_ring *ring,
+					      unsigned int size)
+{
+	unsigned int truesize;
+
+#if (PAGE_SIZE < 8192)
+	truesize = igc_rx_pg_size(ring) / 2;
+#else
+	truesize = ring_uses_build_skb(ring) ?
+		   SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
+		   SKB_DATA_ALIGN(IGC_SKB_PAD + size) :
+		   SKB_DATA_ALIGN(size);
+#endif
+	return truesize;
+}
+
 /**
  * igc_add_rx_frag - Add contents of Rx buffer to sk_buff
  * @rx_ring: rx descriptor ring to transact packets on
@@ -1555,12 +1571,7 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
 				     unsigned int size)
 {
 	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
-#if (PAGE_SIZE < 8192)
-	unsigned int truesize = igc_rx_pg_size(rx_ring) / 2;
-#else
-	unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
-				SKB_DATA_ALIGN(IGC_SKB_PAD + size);
-#endif
+	unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
 	struct sk_buff *skb;
 
 	/* prefetch first cache line of first page */
@@ -1585,11 +1596,7 @@ static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
 					 unsigned int size)
 {
 	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
-#if (PAGE_SIZE < 8192)
-	unsigned int truesize = igc_rx_pg_size(rx_ring) / 2;
-#else
-	unsigned int truesize = SKB_DATA_ALIGN(size);
-#endif
+	unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
 	unsigned int headlen;
 	struct sk_buff *skb;
 
-- 
2.26.2

[Intel-wired-lan] [PATCH 5/9] igc: Refactor rx timestamp handling

[Andre Guedes]

This patch refactors the rx timestamp handling in preparation to land
XDP support.

RX timestamps are put in the rx buffer by hardware, before the packet
data. When creating the xdp buffer, we will need to check the rx
descriptor to determine if the buffer contains timestamp information
and consider the offset when setting xdp.data.

The rx descriptor check is already done in igc_construct_skb(). To
avoid code duplication, this patch moves the timestamp handling to
igc_clean_rx_irq() so both skb and xdp paths can reuse it.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc.h      |  3 +--
 drivers/net/ethernet/intel/igc/igc_main.c | 30 +++++++++++++++--------
 drivers/net/ethernet/intel/igc/igc_ptp.c  | 25 ++++++++++---------
 3 files changed, 34 insertions(+), 24 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
index b9b924794b22..b9d15e6f22f6 100644
--- a/drivers/net/ethernet/intel/igc/igc.h
+++ b/drivers/net/ethernet/intel/igc/igc.h
@@ -545,8 +545,7 @@ void igc_ptp_init(struct igc_adapter *adapter);
 void igc_ptp_reset(struct igc_adapter *adapter);
 void igc_ptp_suspend(struct igc_adapter *adapter);
 void igc_ptp_stop(struct igc_adapter *adapter);
-void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
-			 struct sk_buff *skb);
+ktime_t igc_ptp_rx_pktstamp(struct igc_adapter *adapter, u32 *buf);
 int igc_ptp_set_ts_config(struct net_device *netdev, struct ifreq *ifr);
 int igc_ptp_get_ts_config(struct net_device *netdev, struct ifreq *ifr);
 void igc_ptp_tx_hang(struct igc_adapter *adapter);
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 850e2bbd9773..d026609eec4d 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -1592,10 +1592,11 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
 
 static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
 					 struct igc_rx_buffer *rx_buffer,
-					 union igc_adv_rx_desc *rx_desc,
-					 unsigned int size)
+					 unsigned int size, int pkt_offset,
+					 ktime_t timestamp)
 {
-	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset +
+		   pkt_offset;
 	unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
 	unsigned int headlen;
 	struct sk_buff *skb;
@@ -1608,11 +1609,8 @@ static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
 	if (unlikely(!skb))
 		return NULL;
 
-	if (unlikely(igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP))) {
-		igc_ptp_rx_pktstamp(rx_ring->q_vector, va, skb);
-		va += IGC_TS_HDR_LEN;
-		size -= IGC_TS_HDR_LEN;
-	}
+	if (timestamp)
+		skb_hwtstamps(skb)->hwtstamp = timestamp;
 
 	/* Determine available headroom for copy */
 	headlen = size;
@@ -1911,6 +1909,8 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 	while (likely(total_packets < budget)) {
 		union igc_adv_rx_desc *rx_desc;
 		struct igc_rx_buffer *rx_buffer;
+		ktime_t timestamp = 0;
+		int pkt_offset = 0;
 		unsigned int size;
 
 		/* return some buffers to hardware, one@a time is too slow */
@@ -1932,14 +1932,24 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 
 		rx_buffer = igc_get_rx_buffer(rx_ring, size);
 
+		if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
+			void *pktbuf = page_address(rx_buffer->page) +
+				       rx_buffer->page_offset;
+
+			timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
+							pktbuf);
+			pkt_offset = IGC_TS_HDR_LEN;
+			size -= IGC_TS_HDR_LEN;
+		}
+
 		/* retrieve a buffer from the ring */
 		if (skb)
 			igc_add_rx_frag(rx_ring, rx_buffer, skb, size);
 		else if (ring_uses_build_skb(rx_ring))
 			skb = igc_build_skb(rx_ring, rx_buffer, rx_desc, size);
 		else
-			skb = igc_construct_skb(rx_ring, rx_buffer,
-						rx_desc, size);
+			skb = igc_construct_skb(rx_ring, rx_buffer, size,
+						pkt_offset, timestamp);
 
 		/* exit if we failed to retrieve a buffer */
 		if (!skb) {
diff --git a/drivers/net/ethernet/intel/igc/igc_ptp.c b/drivers/net/ethernet/intel/igc/igc_ptp.c
index 916721e2cd49..bf95ff5b0932 100644
--- a/drivers/net/ethernet/intel/igc/igc_ptp.c
+++ b/drivers/net/ethernet/intel/igc/igc_ptp.c
@@ -153,20 +153,20 @@ static void igc_ptp_systim_to_hwtstamp(struct igc_adapter *adapter,
 
 /**
  * igc_ptp_rx_pktstamp - Retrieve timestamp from rx packet buffer
- * @q_vector: Pointer to interrupt specific structure
- * @va: Pointer to address containing Rx buffer
- * @skb: Buffer containing timestamp and packet
+ * @adapter: Pointer to adapter the packet buffer belongs to
+ * @buf: Pointer to packet buffer
  *
  * This function retrieves the timestamp saved in the beginning of packet
  * buffer. While two timestamps are available, one in timer0 reference and the
  * other in timer1 reference, this function considers only the timestamp in
  * timer0 reference.
+ *
+ * Returns: Timestamp value.
  */
-void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
-			 struct sk_buff *skb)
+ktime_t igc_ptp_rx_pktstamp(struct igc_adapter *adapter, u32 *buf)
 {
-	struct igc_adapter *adapter = q_vector->adapter;
-	u64 regval;
+	ktime_t timestamp;
+	u32 secs, nsecs;
 	int adjust;
 
 	/* Timestamps are saved in little endian at the beginning of the packet
@@ -178,9 +178,10 @@ void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
 	 * SYSTIML holds the nanoseconds part while SYSTIMH holds the seconds
 	 * part of the timestamp.
 	 */
-	regval = le32_to_cpu(va[2]);
-	regval |= (u64)le32_to_cpu(va[3]) << 32;
-	igc_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb), regval);
+	nsecs = le32_to_cpu(buf[2]);
+	secs = le32_to_cpu(buf[3]);
+
+	timestamp = ktime_set(secs, nsecs);
 
 	/* Adjust timestamp for the RX latency based on link speed */
 	switch (adapter->link_speed) {
@@ -201,8 +202,8 @@ void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
 		netdev_warn_once(adapter->netdev, "Imprecise timestamp\n");
 		break;
 	}
-	skb_hwtstamps(skb)->hwtstamp =
-		ktime_sub_ns(skb_hwtstamps(skb)->hwtstamp, adjust);
+
+	return ktime_sub_ns(timestamp, adjust);
 }
 
 static void igc_ptp_disable_rx_timestamp(struct igc_adapter *adapter)
-- 
2.26.2

[Intel-wired-lan] [PATCH 6/9] igc: Add Initial XDP support

[Andre Guedes]

This patch adds the initial XDP support to the igc driver. For now,
only XDP_PASS, XDP_DROP, XDP_ABORT actions are supported. Upcoming
patches will add support for the remaining XDP actions.

XDP-specific helpers are defined in a new file, igc_xdp.c. These
helpers are utilized in igc_main.c to implement the ndo_bpf callback
and to handle XDP programs execution in igc_clean_rx_irq(). The driver
doesn't support XDP functionality with frames that span over multiple
buffers so jumbo frames are not allowed for now.

The approach implemented by this patch follows the approach implemented
in other Intel drivers as much as possible for the sake of consistency
across the drivers.

Quick comment regarding igc_build_skb(): this patch doesn't touch it
because the function is never called. It seems its support is
incomplete/in progress. The function was added by commit 0507ef8a0372b
("igc: Add transmit and receive fastpath and interrupt handlers") but
ring_ uses_build_skb() always return False since the IGC_RING_FLAG_RX_
BUILD_SKB_ENABLED isn't set anywhere in the driver code.

This patch has been tested with the sample app "xdp1" located in
samples/bpf/ dir.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/Makefile   |  2 +-
 drivers/net/ethernet/intel/igc/igc.h      |  2 +
 drivers/net/ethernet/intel/igc/igc_main.c | 58 +++++++++++++++---
 drivers/net/ethernet/intel/igc/igc_xdp.c  | 75 +++++++++++++++++++++++
 drivers/net/ethernet/intel/igc/igc_xdp.h  | 18 ++++++
 5 files changed, 144 insertions(+), 11 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.c
 create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.h

diff --git a/drivers/net/ethernet/intel/igc/Makefile b/drivers/net/ethernet/intel/igc/Makefile
index 1c3051db9085..95d1e8c490a4 100644
--- a/drivers/net/ethernet/intel/igc/Makefile
+++ b/drivers/net/ethernet/intel/igc/Makefile
@@ -8,4 +8,4 @@
 obj-$(CONFIG_IGC) += igc.o
 
 igc-objs := igc_main.o igc_mac.o igc_i225.o igc_base.o igc_nvm.o igc_phy.o \
-igc_diag.o igc_ethtool.o igc_ptp.o igc_dump.o igc_tsn.o
+igc_diag.o igc_ethtool.o igc_ptp.o igc_dump.o igc_tsn.o igc_xdp.o
diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
index b9d15e6f22f6..e013ac65001b 100644
--- a/drivers/net/ethernet/intel/igc/igc.h
+++ b/drivers/net/ethernet/intel/igc/igc.h
@@ -217,6 +217,8 @@ struct igc_adapter {
 	struct timecounter tc;
 	struct timespec64 prev_ptp_time; /* Pre-reset PTP clock */
 	ktime_t ptp_reset_start; /* Reset time in clock mono */
+
+	struct bpf_prog *xdp_prog;
 };
 
 void igc_up(struct igc_adapter *adapter);
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index d026609eec4d..8cdd3a16e637 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -16,6 +16,7 @@
 #include "igc.h"
 #include "igc_hw.h"
 #include "igc_tsn.h"
+#include "igc_xdp.h"
 
 #define DRV_SUMMARY	"Intel(R) 2.5G Ethernet Linux Driver"
 
@@ -1592,11 +1593,11 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
 
 static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
 					 struct igc_rx_buffer *rx_buffer,
-					 unsigned int size, int pkt_offset,
+					 struct xdp_buff *xdp,
 					 ktime_t timestamp)
 {
-	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset +
-		   pkt_offset;
+	void *va = xdp->data;
+	unsigned int size = xdp->data_end - xdp->data;
 	unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
 	unsigned int headlen;
 	struct sk_buff *skb;
@@ -1746,6 +1747,10 @@ static bool igc_cleanup_headers(struct igc_ring *rx_ring,
 				union igc_adv_rx_desc *rx_desc,
 				struct sk_buff *skb)
 {
+	/* XDP packets use error pointer so abort at this point */
+	if (IS_ERR(skb))
+		return true;
+
 	if (unlikely(igc_test_staterr(rx_desc, IGC_RXDEXT_STATERR_RXE))) {
 		struct net_device *netdev = rx_ring->netdev;
 
@@ -1911,7 +1916,9 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 		struct igc_rx_buffer *rx_buffer;
 		ktime_t timestamp = 0;
 		int pkt_offset = 0;
+		struct xdp_buff xdp;
 		unsigned int size;
+		void *pktbuf;
 
 		/* return some buffers to hardware, one at a time is too slow */
 		if (cleaned_count >= IGC_RX_BUFFER_WRITE) {
@@ -1932,24 +1939,37 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 
 		rx_buffer = igc_get_rx_buffer(rx_ring, size);
 
-		if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
-			void *pktbuf = page_address(rx_buffer->page) +
-				       rx_buffer->page_offset;
+		pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
 
+		if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
 			timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
 							pktbuf);
 			pkt_offset = IGC_TS_HDR_LEN;
 			size -= IGC_TS_HDR_LEN;
 		}
 
-		/* retrieve a buffer from the ring */
-		if (skb)
+		if (!skb) {
+			struct igc_adapter *adapter = q_vector->adapter;
+
+			xdp.data = pktbuf + pkt_offset;
+			xdp.data_end = xdp.data + size;
+			xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
+			xdp_set_data_meta_invalid(&xdp);
+
+			skb = igc_xdp_run_prog(adapter, &xdp);
+		}
+
+		if (IS_ERR(skb)) {
+			rx_buffer->pagecnt_bias++;
+			total_packets++;
+			total_bytes += size;
+		} else if (skb)
 			igc_add_rx_frag(rx_ring, rx_buffer, skb, size);
 		else if (ring_uses_build_skb(rx_ring))
 			skb = igc_build_skb(rx_ring, rx_buffer, rx_desc, size);
 		else
-			skb = igc_construct_skb(rx_ring, rx_buffer, size,
-						pkt_offset, timestamp);
+			skb = igc_construct_skb(rx_ring, rx_buffer, &xdp,
+						timestamp);
 
 		/* exit if we failed to retrieve a buffer */
 		if (!skb) {
@@ -3881,6 +3901,11 @@ static int igc_change_mtu(struct net_device *netdev, int new_mtu)
 	int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
 	struct igc_adapter *adapter = netdev_priv(netdev);
 
+	if (igc_xdp_is_enabled(adapter) && new_mtu > ETH_DATA_LEN) {
+		netdev_dbg(netdev, "Jumbo frames not supported with XDP");
+		return -EINVAL;
+	}
+
 	/* adjust max frame to be at least the size of a standard frame */
 	if (max_frame < (ETH_FRAME_LEN + ETH_FCS_LEN))
 		max_frame = ETH_FRAME_LEN + ETH_FCS_LEN;
@@ -4867,6 +4892,18 @@ static int igc_setup_tc(struct net_device *dev, enum tc_setup_type type,
 	}
 }
 
+static int igc_bpf(struct net_device *dev, struct netdev_bpf *bpf)
+{
+	struct igc_adapter *adapter = netdev_priv(dev);
+
+	switch (bpf->command) {
+	case XDP_SETUP_PROG:
+		return igc_xdp_set_prog(adapter, bpf->prog, bpf->extack);
+	default:
+		return -EOPNOTSUPP;
+	}
+}
+
 static const struct net_device_ops igc_netdev_ops = {
 	.ndo_open		= igc_open,
 	.ndo_stop		= igc_close,
@@ -4880,6 +4917,7 @@ static const struct net_device_ops igc_netdev_ops = {
 	.ndo_features_check	= igc_features_check,
 	.ndo_do_ioctl		= igc_ioctl,
 	.ndo_setup_tc		= igc_setup_tc,
+	.ndo_bpf		= igc_bpf,
 };
 
 /* PCIe configuration access */
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
new file mode 100644
index 000000000000..0085d8b85466
--- /dev/null
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2020, Intel Corporation. */
+
+#include <linux/bpf_trace.h>
+
+#include "igc.h"
+#include "igc_xdp.h"
+
+int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
+		     struct netlink_ext_ack *extack)
+{
+	struct net_device *dev = adapter->netdev;
+	bool if_running = netif_running(dev);
+	struct bpf_prog *old_prog;
+
+	if (dev->mtu > ETH_DATA_LEN) {
+		/* For now, the driver doesn't support XDP functionality with
+		 * jumbo frames so we return error.
+		 */
+		NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported");
+		return -EOPNOTSUPP;
+	}
+
+	if (if_running)
+		igc_close(dev);
+
+	old_prog = xchg(&adapter->xdp_prog, prog);
+	if (old_prog)
+		bpf_prog_put(old_prog);
+
+	if (if_running)
+		igc_open(dev);
+
+	netdev_dbg(dev, "XDP program set successfully");
+	return 0;
+}
+
+struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
+				 struct xdp_buff *xdp)
+{
+	struct bpf_prog *prog;
+	int res;
+	u32 act;
+
+	rcu_read_lock();
+
+	prog = READ_ONCE(adapter->xdp_prog);
+	if (!prog)
+		goto unlock;
+
+	act = bpf_prog_run_xdp(prog, xdp);
+	switch (act) {
+	case XDP_PASS:
+		res = IGC_XDP_PASS;
+		break;
+	default:
+		bpf_warn_invalid_xdp_action(act);
+		fallthrough;
+	case XDP_ABORTED:
+		trace_xdp_exception(adapter->netdev, prog, act);
+		fallthrough;
+	case XDP_DROP:
+		res = IGC_XDP_CONSUMED;
+		break;
+	}
+
+unlock:
+	rcu_read_unlock();
+	return ERR_PTR(-res);
+}
+
+bool igc_xdp_is_enabled(struct igc_adapter *adapter)
+{
+	return !!adapter->xdp_prog;
+}
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
new file mode 100644
index 000000000000..ab36ab4d48fd
--- /dev/null
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2020, Intel Corporation. */
+
+#ifndef _IGC_XDP_H_
+#define _IGC_XDP_H_
+
+#define IGC_XDP_PASS		0
+#define IGC_XDP_CONSUMED	BIT(0)
+
+int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
+		     struct netlink_ext_ack *extack);
+
+struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
+				 struct xdp_buff *xdp);
+
+bool igc_xdp_is_enabled(struct igc_adapter *adapter);
+
+#endif /* _IGC_XDP_H_ */
-- 
2.26.2

[Intel-wired-lan] [PATCH 7/9] igc: Add support for XDP_TX action

[Andre Guedes]

This patch adds support for XDP_TX action which enables XDP programs to
transmit back receiving frames.

The main changes in igc_xdp.c are related to the transmission of the
packet (e.g. select the transmission queue, prepare buffer, prepare
descriptor) while the changes in igc_main.c are related to tx_buffer
management and XDP resources setup required to perform the XDP_TX
action.

By default, the driver uses rx buffers with 2 KB size. When XDP is
enabled, it uses larger buffers so we have enough space to accommodate
the headroom and tailroom required by XDP infrastructure. While the
driver had already code to handle larger buffers (IGC_RING_FLAG_RX_3K_
BUFFER flag), it was missing the set/clear helpers that enable/disable
that feature so this patch also introduces these missing helpers.

I225 controller has only 4 tx hardware queues. Since XDP programs may
not even issue an XDP_TX action, this patch doesn't reserve dedicated
queues just for XDP like other Intel drivers do. Instead, the queues
are shared between the network stack and XDP. The netdev queue lock is
used to ensure mutual exclusion.

Since frames can now be transmitted via XDP_TX, the igc_tx_buffer
structure is modified so we are able to save a reference to the xdp
frame for later clean up once the packet is transmitted. The tx_buffer
is mapped to either a skb or a xdpf so we use a union to save the skb
or xdpf pointer and have a bit in tx_flags to indicate which field to
use.

This patch has been tested with the sample app "xdp2" located in
samples/bpf/ dir.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc.h      |  13 ++-
 drivers/net/ethernet/intel/igc/igc_main.c |  87 +++++++++++++--
 drivers/net/ethernet/intel/igc/igc_xdp.c  | 122 ++++++++++++++++++++++
 drivers/net/ethernet/intel/igc/igc_xdp.h  |   6 ++
 4 files changed, 217 insertions(+), 11 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
index e013ac65001b..cbf85dea8b2e 100644
--- a/drivers/net/ethernet/intel/igc/igc.h
+++ b/drivers/net/ethernet/intel/igc/igc.h
@@ -111,6 +111,8 @@ struct igc_ring {
 			struct sk_buff *skb;
 		};
 	};
+
+	struct xdp_rxq_info xdp_rxq;
 } ____cacheline_internodealigned_in_smp;
 
 /* Board specific private data structure */
@@ -373,6 +375,8 @@ enum igc_tx_flags {
 	/* olinfo flags */
 	IGC_TX_FLAGS_IPV4	= 0x10,
 	IGC_TX_FLAGS_CSUM	= 0x20,
+
+	IGC_TX_FLAGS_XDP	= 0x100,
 };
 
 enum igc_boards {
@@ -395,7 +399,10 @@ enum igc_boards {
 struct igc_tx_buffer {
 	union igc_adv_tx_desc *next_to_watch;
 	unsigned long time_stamp;
-	struct sk_buff *skb;
+	union {
+		struct sk_buff *skb;
+		struct xdp_frame *xdpf;
+	};
 	unsigned int bytecount;
 	u16 gso_segs;
 	__be16 protocol;
@@ -504,6 +511,10 @@ enum igc_ring_flags_t {
 
 #define ring_uses_large_buffer(ring) \
 	test_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
+#define set_ring_uses_large_buffer(ring) \
+	set_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
+#define clear_ring_uses_large_buffer(ring) \
+	clear_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
 
 #define ring_uses_build_skb(ring) \
 	test_bit(IGC_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags)
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 8cdd3a16e637..7005e6720261 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -176,8 +176,10 @@ static void igc_clean_tx_ring(struct igc_ring *tx_ring)
 	while (i != tx_ring->next_to_use) {
 		union igc_adv_tx_desc *eop_desc, *tx_desc;
 
-		/* Free all the Tx ring sk_buffs */
-		dev_kfree_skb_any(tx_buffer->skb);
+		if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
+			xdp_return_frame(tx_buffer->xdpf);
+		else
+			dev_kfree_skb_any(tx_buffer->skb);
 
 		/* unmap skb header data */
 		dma_unmap_single(tx_ring->dev,
@@ -345,6 +347,8 @@ static void igc_clean_rx_ring(struct igc_ring *rx_ring)
 {
 	u16 i = rx_ring->next_to_clean;
 
+	clear_ring_uses_large_buffer(rx_ring);
+
 	dev_kfree_skb(rx_ring->skb);
 	rx_ring->skb = NULL;
 
@@ -403,6 +407,8 @@ void igc_free_rx_resources(struct igc_ring *rx_ring)
 {
 	igc_clean_rx_ring(rx_ring);
 
+	igc_xdp_unregister_rxq_info(rx_ring);
+
 	vfree(rx_ring->rx_buffer_info);
 	rx_ring->rx_buffer_info = NULL;
 
@@ -440,7 +446,11 @@ int igc_setup_rx_resources(struct igc_ring *rx_ring)
 {
 	struct net_device *ndev = rx_ring->netdev;
 	struct device *dev = rx_ring->dev;
-	int size, desc_len;
+	int size, desc_len, res;
+
+	res = igc_xdp_register_rxq_info(rx_ring);
+	if (res < 0)
+		return res;
 
 	size = sizeof(struct igc_rx_buffer) * rx_ring->count;
 	rx_ring->rx_buffer_info = vzalloc(size);
@@ -466,6 +476,7 @@ int igc_setup_rx_resources(struct igc_ring *rx_ring)
 	return 0;
 
 err:
+	igc_xdp_unregister_rxq_info(rx_ring);
 	vfree(rx_ring->rx_buffer_info);
 	rx_ring->rx_buffer_info = NULL;
 	netdev_err(ndev, "Unable to allocate memory for Rx descriptor ring\n");
@@ -513,6 +524,9 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter,
 	u32 srrctl = 0, rxdctl = 0;
 	u64 rdba = ring->dma;
 
+	if (igc_xdp_is_enabled(adapter))
+		set_ring_uses_large_buffer(ring);
+
 	/* disable the queue */
 	wr32(IGC_RXDCTL(reg_idx), 0);
 
@@ -1790,7 +1804,14 @@ static void igc_put_rx_buffer(struct igc_ring *rx_ring,
 
 static inline unsigned int igc_rx_offset(struct igc_ring *rx_ring)
 {
-	return ring_uses_build_skb(rx_ring) ? IGC_SKB_PAD : 0;
+	struct igc_adapter *adapter = rx_ring->q_vector->adapter;
+
+	if (ring_uses_build_skb(rx_ring))
+		return IGC_SKB_PAD;
+	if (igc_xdp_is_enabled(adapter))
+		return XDP_PACKET_HEADROOM;
+
+	return 0;
 }
 
 static bool igc_alloc_mapped_page(struct igc_ring *rx_ring,
@@ -1904,12 +1925,41 @@ static void igc_alloc_rx_buffers(struct igc_ring *rx_ring, u16 cleaned_count)
 	}
 }
 
+/* This function assumes __netif_tx_lock is held by the caller. */
+static void igc_flush_tx_descriptors(struct igc_ring *ring)
+{
+	/* Once tail pointer is updated, hardware can fetch the descriptors
+	 * any time so we issue a write membar here to ensure all memory
+	 * writes are complete before the tail pointer is updated.
+	 */
+	wmb();
+	writel(ring->next_to_use, ring->tail);
+}
+
+static void igc_finalize_xdp(struct igc_adapter *adapter, int status)
+{
+	if (status & IGC_XDP_TX) {
+		int cpu = smp_processor_id();
+		struct netdev_queue *nq;
+		struct igc_ring *ring;
+
+		ring = igc_xdp_get_tx_ring(adapter, cpu);
+		nq = txring_txq(ring);
+
+		__netif_tx_lock(nq, cpu);
+		igc_flush_tx_descriptors(ring);
+		__netif_tx_unlock(nq);
+	}
+}
+
 static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 {
 	unsigned int total_bytes = 0, total_packets = 0;
+	struct igc_adapter *adapter = q_vector->adapter;
 	struct igc_ring *rx_ring = q_vector->rx.ring;
 	struct sk_buff *skb = rx_ring->skb;
 	u16 cleaned_count = igc_desc_unused(rx_ring);
+	int xdp_status = 0;
 
 	while (likely(total_packets < budget)) {
 		union igc_adv_rx_desc *rx_desc;
@@ -1917,7 +1967,7 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 		ktime_t timestamp = 0;
 		int pkt_offset = 0;
 		struct xdp_buff xdp;
-		unsigned int size;
+		unsigned int size, truesize;
 		void *pktbuf;
 
 		/* return some buffers to hardware, one@a time is too slow */
@@ -1938,6 +1988,7 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 		dma_rmb();
 
 		rx_buffer = igc_get_rx_buffer(rx_ring, size);
+		truesize = igc_get_rx_frame_truesize(rx_ring, size);
 
 		pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
 
@@ -1949,18 +2000,29 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 		}
 
 		if (!skb) {
-			struct igc_adapter *adapter = q_vector->adapter;
-
 			xdp.data = pktbuf + pkt_offset;
 			xdp.data_end = xdp.data + size;
 			xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
 			xdp_set_data_meta_invalid(&xdp);
+			xdp.frame_sz = truesize;
+			xdp.rxq = &rx_ring->xdp_rxq;
 
 			skb = igc_xdp_run_prog(adapter, &xdp);
 		}
 
 		if (IS_ERR(skb)) {
-			rx_buffer->pagecnt_bias++;
+			unsigned int xdp_res = -PTR_ERR(skb);
+
+			switch (xdp_res) {
+			case IGC_XDP_CONSUMED:
+				rx_buffer->pagecnt_bias++;
+				break;
+			case IGC_XDP_TX:
+				igc_rx_buffer_flip(rx_buffer, truesize);
+				xdp_status |= xdp_res;
+				break;
+			}
+
 			total_packets++;
 			total_bytes += size;
 		} else if (skb)
@@ -2006,6 +2068,9 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 		total_packets++;
 	}
 
+	if (xdp_status)
+		igc_finalize_xdp(adapter, xdp_status);
+
 	/* place incomplete frames back on ring for completion */
 	rx_ring->skb = skb;
 
@@ -2067,8 +2132,10 @@ static bool igc_clean_tx_irq(struct igc_q_vector *q_vector, int napi_budget)
 		total_bytes += tx_buffer->bytecount;
 		total_packets += tx_buffer->gso_segs;
 
-		/* free the skb */
-		napi_consume_skb(tx_buffer->skb, napi_budget);
+		if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
+			xdp_return_frame(tx_buffer->xdpf);
+		else
+			napi_consume_skb(tx_buffer->skb, napi_budget);
 
 		/* unmap skb header data */
 		dma_unmap_single(tx_ring->dev,
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
index 0085d8b85466..9f4c58436170 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.c
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
@@ -35,6 +35,94 @@ int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
 	return 0;
 }
 
+static int igc_xdp_init_tx_buffer(struct igc_tx_buffer *buffer,
+				  struct xdp_frame *xdpf,
+				  struct igc_ring *ring)
+{
+	dma_addr_t dma;
+
+	dma = dma_map_single(ring->dev, xdpf->data, xdpf->len, DMA_TO_DEVICE);
+	if (dma_mapping_error(ring->dev, dma)) {
+		netdev_err_once(ring->netdev, "Failed to map DMA for TX\n");
+		return -ENOMEM;
+	}
+
+	buffer->xdpf = xdpf;
+	buffer->tx_flags = IGC_TX_FLAGS_XDP;
+	buffer->protocol = 0;
+	buffer->bytecount = xdpf->len;
+	buffer->gso_segs = 1;
+	buffer->time_stamp = jiffies;
+	dma_unmap_len_set(buffer, len, xdpf->len);
+	dma_unmap_addr_set(buffer, dma, dma);
+	return 0;
+}
+
+/* This function requires __netif_tx_lock is held by the caller. */
+static int igc_xdp_init_tx_descriptor(struct igc_ring *ring,
+				      struct xdp_frame *xdpf)
+{
+	struct igc_tx_buffer *buffer;
+	union igc_adv_tx_desc *desc;
+	u32 cmd_type, olinfo_status;
+	int err;
+
+	if (!igc_desc_unused(ring))
+		return -EBUSY;
+
+	buffer = &ring->tx_buffer_info[ring->next_to_use];
+	err = igc_xdp_init_tx_buffer(buffer, xdpf, ring);
+	if (err)
+		return err;
+
+	cmd_type = IGC_ADVTXD_DTYP_DATA | IGC_ADVTXD_DCMD_DEXT |
+		   IGC_ADVTXD_DCMD_IFCS | IGC_TXD_DCMD |
+		   buffer->bytecount;
+	olinfo_status = buffer->bytecount << IGC_ADVTXD_PAYLEN_SHIFT;
+
+	desc = IGC_TX_DESC(ring, ring->next_to_use);
+	desc->read.cmd_type_len = cpu_to_le32(cmd_type);
+	desc->read.olinfo_status = cpu_to_le32(olinfo_status);
+	desc->read.buffer_addr = cpu_to_le64(dma_unmap_addr(buffer, dma));
+
+	netdev_tx_sent_queue(txring_txq(ring), buffer->bytecount);
+
+	buffer->next_to_watch = desc;
+
+	ring->next_to_use = (ring->next_to_use + 1) % ring->count;
+	return 0;
+}
+
+struct igc_ring *igc_xdp_get_tx_ring(struct igc_adapter *adapter, int cpu)
+{
+	struct igc_ring *ring;
+	int index;
+
+	index = cpu % adapter->num_tx_queues;
+	ring = adapter->tx_ring[index];
+	return ring;
+}
+
+static int igc_xdp_xmit_back(struct igc_adapter *adapter, struct xdp_buff *xdp)
+{
+	struct xdp_frame *xdpf = xdp_convert_buff_to_frame(xdp);
+	int cpu = smp_processor_id();
+	struct netdev_queue *nq;
+	struct igc_ring *ring;
+	int res;
+
+	if (unlikely(!xdpf))
+		return -EFAULT;
+
+	ring = igc_xdp_get_tx_ring(adapter, cpu);
+	nq = txring_txq(ring);
+
+	__netif_tx_lock(nq, cpu);
+	res = igc_xdp_init_tx_descriptor(ring, xdpf);
+	__netif_tx_unlock(nq);
+	return res;
+}
+
 struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
 				 struct xdp_buff *xdp)
 {
@@ -53,6 +141,12 @@ struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
 	case XDP_PASS:
 		res = IGC_XDP_PASS;
 		break;
+	case XDP_TX:
+		if (igc_xdp_xmit_back(adapter, xdp) < 0)
+			res = IGC_XDP_CONSUMED;
+		else
+			res = IGC_XDP_TX;
+		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
 		fallthrough;
@@ -73,3 +167,31 @@ bool igc_xdp_is_enabled(struct igc_adapter *adapter)
 {
 	return !!adapter->xdp_prog;
 }
+
+int igc_xdp_register_rxq_info(struct igc_ring *ring)
+{
+	struct net_device *dev = ring->netdev;
+	int err;
+
+	err = xdp_rxq_info_reg(&ring->xdp_rxq, dev, ring->queue_index);
+	if (err) {
+		netdev_err(dev, "Failed to register xdp rxq info\n");
+		return err;
+	}
+
+	err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, MEM_TYPE_PAGE_SHARED,
+					 NULL);
+	if (err) {
+		netdev_err(dev, "Failed to register xdp rxq mem model\n");
+		xdp_rxq_info_unreg(&ring->xdp_rxq);
+		return err;
+	}
+
+	return 0;
+}
+
+void igc_xdp_unregister_rxq_info(struct igc_ring *ring)
+{
+	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
+	xdp_rxq_info_unreg(&ring->xdp_rxq);
+}
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
index ab36ab4d48fd..fdb31f40dbe5 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.h
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
@@ -6,6 +6,7 @@
 
 #define IGC_XDP_PASS		0
 #define IGC_XDP_CONSUMED	BIT(0)
+#define IGC_XDP_TX		BIT(1)
 
 int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
 		     struct netlink_ext_ack *extack);
@@ -15,4 +16,9 @@ struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
 
 bool igc_xdp_is_enabled(struct igc_adapter *adapter);
 
+int igc_xdp_register_rxq_info(struct igc_ring *ring);
+void igc_xdp_unregister_rxq_info(struct igc_ring *ring);
+
+struct igc_ring *igc_xdp_get_tx_ring(struct igc_adapter *adapter, int cpu);
+
 #endif /* _IGC_XDP_H_ */
-- 
2.26.2

[Intel-wired-lan] [PATCH 8/9] igc: Add support for XDP_REDIRECT action

[Andre Guedes]

This patch adds support for the XDP_REDIRECT action which enables XDP
programs to redirect packets arriving at I225 NIC.

The patch also tweaks the driver's page counting scheme (as described
in '8ce29c679a6e i40e: tweak page counting for XDP_REDIRECT' and
implemented by other Intel drivers) in order to properly support
XDP_REDIRECT action.

This patch has been tested with the sample apps "xdp_redirect_cpu" and
"xdp_redirect_map" located in samples/bpf/.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc_main.c | 11 ++++++++---
 drivers/net/ethernet/intel/igc/igc_xdp.c  |  6 ++++++
 drivers/net/ethernet/intel/igc/igc_xdp.h  |  1 +
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 7005e6720261..7cdfce6c0d63 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -1708,8 +1708,8 @@ static bool igc_can_reuse_rx_page(struct igc_rx_buffer *rx_buffer)
 	 * the pagecnt_bias and page count so that we fully restock the
 	 * number of references the driver holds.
 	 */
-	if (unlikely(!pagecnt_bias)) {
-		page_ref_add(page, USHRT_MAX);
+	if (unlikely(pagecnt_bias == 1)) {
+		page_ref_add(page, USHRT_MAX - 1);
 		rx_buffer->pagecnt_bias = USHRT_MAX;
 	}
 
@@ -1850,7 +1850,8 @@ static bool igc_alloc_mapped_page(struct igc_ring *rx_ring,
 	bi->dma = dma;
 	bi->page = page;
 	bi->page_offset = igc_rx_offset(rx_ring);
-	bi->pagecnt_bias = 1;
+	page_ref_add(page, USHRT_MAX - 1);
+	bi->pagecnt_bias = USHRT_MAX;
 
 	return true;
 }
@@ -1950,6 +1951,9 @@ static void igc_finalize_xdp(struct igc_adapter *adapter, int status)
 		igc_flush_tx_descriptors(ring);
 		__netif_tx_unlock(nq);
 	}
+
+	if (status & IGC_XDP_REDIRECT)
+		xdp_do_flush();
 }
 
 static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
@@ -2018,6 +2022,7 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
 				rx_buffer->pagecnt_bias++;
 				break;
 			case IGC_XDP_TX:
+			case IGC_XDP_REDIRECT:
 				igc_rx_buffer_flip(rx_buffer, truesize);
 				xdp_status |= xdp_res;
 				break;
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
index 9f4c58436170..e9a81975324f 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.c
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
@@ -147,6 +147,12 @@ struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
 		else
 			res = IGC_XDP_TX;
 		break;
+	case XDP_REDIRECT:
+		if (xdp_do_redirect(adapter->netdev, xdp, prog) < 0)
+			res = IGC_XDP_CONSUMED;
+		else
+			res = IGC_XDP_REDIRECT;
+		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
 		fallthrough;
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
index fdb31f40dbe5..1c38a80c3aa0 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.h
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
@@ -7,6 +7,7 @@
 #define IGC_XDP_PASS		0
 #define IGC_XDP_CONSUMED	BIT(0)
 #define IGC_XDP_TX		BIT(1)
+#define IGC_XDP_REDIRECT	BIT(2)
 
 int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
 		     struct netlink_ext_ack *extack);
-- 
2.26.2

[Intel-wired-lan] [PATCH 9/9] igc: Implement ndo_xdp_xmit callback

[Andre Guedes]

This patch introduces the helper igc_xdp_xmit() which implements the
ndo_xdp_xmit ops, enabling the igc driver to transmit packets forwarded
to it by xdp programs running on other interfaces.

This patch has been tested with the sample app "xdp_redirect_map"
located in samples/bpf/.

Signed-off-by: Andre Guedes <andre.guedes@intel.com>
---
 drivers/net/ethernet/intel/igc/igc_main.c | 41 +++++++++++++++++++++++
 drivers/net/ethernet/intel/igc/igc_xdp.c  |  3 +-
 drivers/net/ethernet/intel/igc/igc_xdp.h  |  2 ++
 3 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 7cdfce6c0d63..907238d1094a 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -4976,6 +4976,46 @@ static int igc_bpf(struct net_device *dev, struct netdev_bpf *bpf)
 	}
 }
 
+static int igc_xdp_xmit(struct net_device *dev, int num_frames,
+			struct xdp_frame **frames, u32 flags)
+{
+	struct igc_adapter *adapter = netdev_priv(dev);
+	int cpu = smp_processor_id();
+	struct netdev_queue *nq;
+	struct igc_ring *ring;
+	int i, drops;
+
+	if (unlikely(test_bit(__IGC_DOWN, &adapter->state)))
+		return -ENETDOWN;
+
+	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+		return -EINVAL;
+
+	ring = igc_xdp_get_tx_ring(adapter, cpu);
+	nq = txring_txq(ring);
+
+	__netif_tx_lock(nq, cpu);
+
+	drops = 0;
+	for (i = 0; i < num_frames; i++) {
+		int err;
+		struct xdp_frame *xdpf = frames[i];
+
+		err = igc_xdp_init_tx_descriptor(ring, xdpf);
+		if (err) {
+			xdp_return_frame_rx_napi(xdpf);
+			drops++;
+		}
+	}
+
+	if (flags & XDP_XMIT_FLUSH)
+		igc_flush_tx_descriptors(ring);
+
+	__netif_tx_unlock(nq);
+
+	return num_frames - drops;
+}
+
 static const struct net_device_ops igc_netdev_ops = {
 	.ndo_open		= igc_open,
 	.ndo_stop		= igc_close,
@@ -4990,6 +5030,7 @@ static const struct net_device_ops igc_netdev_ops = {
 	.ndo_do_ioctl		= igc_ioctl,
 	.ndo_setup_tc		= igc_setup_tc,
 	.ndo_bpf		= igc_bpf,
+	.ndo_xdp_xmit		= igc_xdp_xmit,
 };
 
 /* PCIe configuration access */
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
index e9a81975324f..bc473161899c 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.c
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
@@ -59,8 +59,7 @@ static int igc_xdp_init_tx_buffer(struct igc_tx_buffer *buffer,
 }
 
 /* This function requires __netif_tx_lock is held by the caller. */
-static int igc_xdp_init_tx_descriptor(struct igc_ring *ring,
-				      struct xdp_frame *xdpf)
+int igc_xdp_init_tx_descriptor(struct igc_ring *ring, struct xdp_frame *xdpf)
 {
 	struct igc_tx_buffer *buffer;
 	union igc_adv_tx_desc *desc;
diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
index 1c38a80c3aa0..801fb27fbb85 100644
--- a/drivers/net/ethernet/intel/igc/igc_xdp.h
+++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
@@ -22,4 +22,6 @@ void igc_xdp_unregister_rxq_info(struct igc_ring *ring);
 
 struct igc_ring *igc_xdp_get_tx_ring(struct igc_adapter *adapter, int cpu);
 
+int igc_xdp_init_tx_descriptor(struct igc_ring *ring, struct xdp_frame *xdpf);
+
 #endif /* _IGC_XDP_H_ */
-- 
2.26.2

[Intel-wired-lan] [PATCH 1/9] igc: Fix igc_ptp_rx_pktstamp()

[Maciej Fijalkowski]

On Thu, Oct 08, 2020 at 07:53:41PM -0700, Andre Guedes wrote:
> The comment describing the timestamps layout in the packet buffer is
> wrong and the code is actually retrieving the timestamp in Timer 1
> reference instead of Timer 0. This hasn't been a big issue so far
> because hardware is configured to report both timestamps using Timer 0
> (see IGC_SRRCTL register configuration in igc_ptp_enable_rx_timestamp()
> helper). This patch fixes the comment and the code so we retrieve the
> timestamp in Timer 0 reference as expected.
> 
> This patch also takes the opportunity to get rid of the hw.mac.type check
> since it is not required.

How is this related to adding XDP support?

> 
> Fixes: 81b055205e8ba ("igc: Add support for RX timestamping")
> Signed-off-by: Andre Guedes <andre.guedes@intel.com>
> ---
>  drivers/net/ethernet/intel/igc/igc.h     |  2 +-
>  drivers/net/ethernet/intel/igc/igc_ptp.c | 72 +++++++++++++-----------
>  2 files changed, 41 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
> index 35baae900c1f..b9b924794b22 100644
> --- a/drivers/net/ethernet/intel/igc/igc.h
> +++ b/drivers/net/ethernet/intel/igc/igc.h
> @@ -545,7 +545,7 @@ void igc_ptp_init(struct igc_adapter *adapter);
>  void igc_ptp_reset(struct igc_adapter *adapter);
>  void igc_ptp_suspend(struct igc_adapter *adapter);
>  void igc_ptp_stop(struct igc_adapter *adapter);
> -void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, void *va,
> +void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
>  			 struct sk_buff *skb);
>  int igc_ptp_set_ts_config(struct net_device *netdev, struct ifreq *ifr);
>  int igc_ptp_get_ts_config(struct net_device *netdev, struct ifreq *ifr);
> diff --git a/drivers/net/ethernet/intel/igc/igc_ptp.c b/drivers/net/ethernet/intel/igc/igc_ptp.c
> index ac0b9c85da7c..916721e2cd49 100644
> --- a/drivers/net/ethernet/intel/igc/igc_ptp.c
> +++ b/drivers/net/ethernet/intel/igc/igc_ptp.c
> @@ -152,46 +152,54 @@ static void igc_ptp_systim_to_hwtstamp(struct igc_adapter *adapter,
>  }
>  
>  /**
> - * igc_ptp_rx_pktstamp - retrieve Rx per packet timestamp
> + * igc_ptp_rx_pktstamp - Retrieve timestamp from rx packet buffer
>   * @q_vector: Pointer to interrupt specific structure
>   * @va: Pointer to address containing Rx buffer
>   * @skb: Buffer containing timestamp and packet
>   *
> - * This function is meant to retrieve the first timestamp from the
> - * first buffer of an incoming frame. The value is stored in little
> - * endian format starting on byte 0. There's a second timestamp
> - * starting on byte 8.
> - **/
> -void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, void *va,
> + * This function retrieves the timestamp saved in the beginning of packet
> + * buffer. While two timestamps are available, one in timer0 reference and the
> + * other in timer1 reference, this function considers only the timestamp in
> + * timer0 reference.
> + */
> +void igc_ptp_rx_pktstamp(struct igc_q_vector *q_vector, u32 *va,
>  			 struct sk_buff *skb)
>  {
>  	struct igc_adapter *adapter = q_vector->adapter;
> -	__le64 *regval = (__le64 *)va;
> -	int adjust = 0;
> -
> -	/* The timestamp is recorded in little endian format.
> -	 * DWORD: | 0          | 1           | 2          | 3
> -	 * Field: | Timer0 Low | Timer0 High | Timer1 Low | Timer1 High
> +	u64 regval;
> +	int adjust;
> +
> +	/* Timestamps are saved in little endian at the beginning of the packet
> +	 * buffer following the layout:
> +	 *
> +	 * | 0              | 1              | 2              | 3              |
> +	 * | Timer1 SYSTIML | Timer1 SYSTIMH | Timer0 SYSTIML | Timer0 SYSTIMH |
> +	 *
> +	 * SYSTIML holds the nanoseconds part while SYSTIMH holds the seconds
> +	 * part of the timestamp.
>  	 */
> -	igc_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb),
> -				   le64_to_cpu(regval[0]));
> -
> -	/* adjust timestamp for the RX latency based on link speed */
> -	if (adapter->hw.mac.type == igc_i225) {
> -		switch (adapter->link_speed) {
> -		case SPEED_10:
> -			adjust = IGC_I225_RX_LATENCY_10;
> -			break;
> -		case SPEED_100:
> -			adjust = IGC_I225_RX_LATENCY_100;
> -			break;
> -		case SPEED_1000:
> -			adjust = IGC_I225_RX_LATENCY_1000;
> -			break;
> -		case SPEED_2500:
> -			adjust = IGC_I225_RX_LATENCY_2500;
> -			break;
> -		}
> +	regval = le32_to_cpu(va[2]);
> +	regval |= (u64)le32_to_cpu(va[3]) << 32;
> +	igc_ptp_systim_to_hwtstamp(adapter, skb_hwtstamps(skb), regval);
> +
> +	/* Adjust timestamp for the RX latency based on link speed */
> +	switch (adapter->link_speed) {
> +	case SPEED_10:
> +		adjust = IGC_I225_RX_LATENCY_10;
> +		break;
> +	case SPEED_100:
> +		adjust = IGC_I225_RX_LATENCY_100;
> +		break;
> +	case SPEED_1000:
> +		adjust = IGC_I225_RX_LATENCY_1000;
> +		break;
> +	case SPEED_2500:
> +		adjust = IGC_I225_RX_LATENCY_2500;
> +		break;
> +	default:
> +		adjust = 0;
> +		netdev_warn_once(adapter->netdev, "Imprecise timestamp\n");
> +		break;
>  	}
>  	skb_hwtstamps(skb)->hwtstamp =
>  		ktime_sub_ns(skb_hwtstamps(skb)->hwtstamp, adjust);
> -- 
> 2.26.2
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan at osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

[Intel-wired-lan] [PATCH 6/9] igc: Add Initial XDP support

[Maciej Fijalkowski]

On Thu, Oct 08, 2020 at 07:53:46PM -0700, Andre Guedes wrote:
> This patch adds the initial XDP support to the igc driver. For now,
> only XDP_PASS, XDP_DROP, XDP_ABORT actions are supported. Upcoming

s/XDP_ABORT/XDP_ABORTED

> patches will add support for the remaining XDP actions.
> 
> XDP-specific helpers are defined in a new file, igc_xdp.c. These
> helpers are utilized in igc_main.c to implement the ndo_bpf callback
> and to handle XDP programs execution in igc_clean_rx_irq(). The driver
> doesn't support XDP functionality with frames that span over multiple
> buffers so jumbo frames are not allowed for now.
> 
> The approach implemented by this patch follows the approach implemented
> in other Intel drivers as much as possible for the sake of consistency
> across the drivers.
> 
> Quick comment regarding igc_build_skb(): this patch doesn't touch it
> because the function is never called. It seems its support is
> incomplete/in progress. The function was added by commit 0507ef8a0372b
> ("igc: Add transmit and receive fastpath and interrupt handlers") but
> ring_ uses_build_skb() always return False since the IGC_RING_FLAG_RX_
> BUILD_SKB_ENABLED isn't set anywhere in the driver code.
> 
> This patch has been tested with the sample app "xdp1" located in
> samples/bpf/ dir.
> 
> Signed-off-by: Andre Guedes <andre.guedes@intel.com>
> ---
>  drivers/net/ethernet/intel/igc/Makefile   |  2 +-
>  drivers/net/ethernet/intel/igc/igc.h      |  2 +
>  drivers/net/ethernet/intel/igc/igc_main.c | 58 +++++++++++++++---
>  drivers/net/ethernet/intel/igc/igc_xdp.c  | 75 +++++++++++++++++++++++
>  drivers/net/ethernet/intel/igc/igc_xdp.h  | 18 ++++++
>  5 files changed, 144 insertions(+), 11 deletions(-)
>  create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.c
>  create mode 100644 drivers/net/ethernet/intel/igc/igc_xdp.h
> 
> diff --git a/drivers/net/ethernet/intel/igc/Makefile b/drivers/net/ethernet/intel/igc/Makefile
> index 1c3051db9085..95d1e8c490a4 100644
> --- a/drivers/net/ethernet/intel/igc/Makefile
> +++ b/drivers/net/ethernet/intel/igc/Makefile
> @@ -8,4 +8,4 @@
>  obj-$(CONFIG_IGC) += igc.o
>  
>  igc-objs := igc_main.o igc_mac.o igc_i225.o igc_base.o igc_nvm.o igc_phy.o \
> -igc_diag.o igc_ethtool.o igc_ptp.o igc_dump.o igc_tsn.o
> +igc_diag.o igc_ethtool.o igc_ptp.o igc_dump.o igc_tsn.o igc_xdp.o
> diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
> index b9d15e6f22f6..e013ac65001b 100644
> --- a/drivers/net/ethernet/intel/igc/igc.h
> +++ b/drivers/net/ethernet/intel/igc/igc.h
> @@ -217,6 +217,8 @@ struct igc_adapter {
>  	struct timecounter tc;
>  	struct timespec64 prev_ptp_time; /* Pre-reset PTP clock */
>  	ktime_t ptp_reset_start; /* Reset time in clock mono */
> +
> +	struct bpf_prog *xdp_prog;
>  };
>  
>  void igc_up(struct igc_adapter *adapter);
> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> index d026609eec4d..8cdd3a16e637 100644
> --- a/drivers/net/ethernet/intel/igc/igc_main.c
> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> @@ -16,6 +16,7 @@
>  #include "igc.h"
>  #include "igc_hw.h"
>  #include "igc_tsn.h"
> +#include "igc_xdp.h"
>  
>  #define DRV_SUMMARY	"Intel(R) 2.5G Ethernet Linux Driver"
>  
> @@ -1592,11 +1593,11 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
>  
>  static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
>  					 struct igc_rx_buffer *rx_buffer,
> -					 unsigned int size, int pkt_offset,
> +					 struct xdp_buff *xdp,
>  					 ktime_t timestamp)
>  {
> -	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset +
> -		   pkt_offset;
> +	void *va = xdp->data;
> +	unsigned int size = xdp->data_end - xdp->data;
>  	unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
>  	unsigned int headlen;
>  	struct sk_buff *skb;

RCT

> @@ -1746,6 +1747,10 @@ static bool igc_cleanup_headers(struct igc_ring *rx_ring,
>  				union igc_adv_rx_desc *rx_desc,
>  				struct sk_buff *skb)
>  {
> +	/* XDP packets use error pointer so abort at this point */
> +	if (IS_ERR(skb))
> +		return true;
> +
>  	if (unlikely(igc_test_staterr(rx_desc, IGC_RXDEXT_STATERR_RXE))) {
>  		struct net_device *netdev = rx_ring->netdev;
>  
> @@ -1911,7 +1916,9 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  		struct igc_rx_buffer *rx_buffer;
>  		ktime_t timestamp = 0;
>  		int pkt_offset = 0;
> +		struct xdp_buff xdp;
>  		unsigned int size;
> +		void *pktbuf;
>  
>  		/* return some buffers to hardware, one at a time is too slow */
>  		if (cleaned_count >= IGC_RX_BUFFER_WRITE) {
> @@ -1932,24 +1939,37 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  
>  		rx_buffer = igc_get_rx_buffer(rx_ring, size);
>  
> -		if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> -			void *pktbuf = page_address(rx_buffer->page) +
> -				       rx_buffer->page_offset;
> +		pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
>  
> +		if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
>  			timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
>  							pktbuf);
>  			pkt_offset = IGC_TS_HDR_LEN;
>  			size -= IGC_TS_HDR_LEN;
>  		}
>  
> -		/* retrieve a buffer from the ring */
> -		if (skb)
> +		if (!skb) {
> +			struct igc_adapter *adapter = q_vector->adapter;
> +
> +			xdp.data = pktbuf + pkt_offset;
> +			xdp.data_end = xdp.data + size;
> +			xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);

This needs some fixing. You're saying that build_skb() can't be currently
used which means that the headroom given for XDP will always be zero.

You need to either teach igc_rx_offset to return XDP_PACKET_HEADROOM for
case when ring is not using build_skb() or make build_skb() usable.

> +			xdp_set_data_meta_invalid(&xdp);

What about xdp.frame_sz?

> +
> +			skb = igc_xdp_run_prog(adapter, &xdp);
> +		}
> +
> +		if (IS_ERR(skb)) {
> +			rx_buffer->pagecnt_bias++;
> +			total_packets++;
> +			total_bytes += size;
> +		} else if (skb)
>  			igc_add_rx_frag(rx_ring, rx_buffer, skb, size);
>  		else if (ring_uses_build_skb(rx_ring))
>  			skb = igc_build_skb(rx_ring, rx_buffer, rx_desc, size);
>  		else
> -			skb = igc_construct_skb(rx_ring, rx_buffer, size,
> -						pkt_offset, timestamp);
> +			skb = igc_construct_skb(rx_ring, rx_buffer, &xdp,
> +						timestamp);
>  
>  		/* exit if we failed to retrieve a buffer */
>  		if (!skb) {
> @@ -3881,6 +3901,11 @@ static int igc_change_mtu(struct net_device *netdev, int new_mtu)
>  	int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;

I think I was mentioning it with igb xdp review, shouldn't we take double
vlan header into account?

>  	struct igc_adapter *adapter = netdev_priv(netdev);
>  
> +	if (igc_xdp_is_enabled(adapter) && new_mtu > ETH_DATA_LEN) {
> +		netdev_dbg(netdev, "Jumbo frames not supported with XDP");
> +		return -EINVAL;
> +	}
> +
>  	/* adjust max frame to be at least the size of a standard frame */
>  	if (max_frame < (ETH_FRAME_LEN + ETH_FCS_LEN))
>  		max_frame = ETH_FRAME_LEN + ETH_FCS_LEN;
> @@ -4867,6 +4892,18 @@ static int igc_setup_tc(struct net_device *dev, enum tc_setup_type type,
>  	}
>  }
>  
> +static int igc_bpf(struct net_device *dev, struct netdev_bpf *bpf)
> +{
> +	struct igc_adapter *adapter = netdev_priv(dev);
> +
> +	switch (bpf->command) {
> +	case XDP_SETUP_PROG:
> +		return igc_xdp_set_prog(adapter, bpf->prog, bpf->extack);
> +	default:
> +		return -EOPNOTSUPP;
> +	}
> +}
> +
>  static const struct net_device_ops igc_netdev_ops = {
>  	.ndo_open		= igc_open,
>  	.ndo_stop		= igc_close,
> @@ -4880,6 +4917,7 @@ static const struct net_device_ops igc_netdev_ops = {
>  	.ndo_features_check	= igc_features_check,
>  	.ndo_do_ioctl		= igc_ioctl,
>  	.ndo_setup_tc		= igc_setup_tc,
> +	.ndo_bpf		= igc_bpf,
>  };
>  
>  /* PCIe configuration access */
> diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
> new file mode 100644
> index 000000000000..0085d8b85466
> --- /dev/null
> +++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
> @@ -0,0 +1,75 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2020, Intel Corporation. */
> +
> +#include <linux/bpf_trace.h>
> +
> +#include "igc.h"
> +#include "igc_xdp.h"
> +
> +int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
> +		     struct netlink_ext_ack *extack)
> +{
> +	struct net_device *dev = adapter->netdev;
> +	bool if_running = netif_running(dev);
> +	struct bpf_prog *old_prog;
> +
> +	if (dev->mtu > ETH_DATA_LEN) {
> +		/* For now, the driver doesn't support XDP functionality with
> +		 * jumbo frames so we return error.
> +		 */
> +		NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported");
> +		return -EOPNOTSUPP;
> +	}
> +
> +	if (if_running)
> +		igc_close(dev);
> +
> +	old_prog = xchg(&adapter->xdp_prog, prog);
> +	if (old_prog)
> +		bpf_prog_put(old_prog);
> +
> +	if (if_running)
> +		igc_open(dev);
> +
> +	netdev_dbg(dev, "XDP program set successfully");

This can be dropped

> +	return 0;
> +}
> +
> +struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
> +				 struct xdp_buff *xdp)
> +{
> +	struct bpf_prog *prog;
> +	int res;
> +	u32 act;
> +
> +	rcu_read_lock();
> +
> +	prog = READ_ONCE(adapter->xdp_prog);
> +	if (!prog)
> +		goto unlock;

res will be uninitialized if you hit this goto.

> +
> +	act = bpf_prog_run_xdp(prog, xdp);
> +	switch (act) {
> +	case XDP_PASS:
> +		res = IGC_XDP_PASS;
> +		break;
> +	default:
> +		bpf_warn_invalid_xdp_action(act);
> +		fallthrough;
> +	case XDP_ABORTED:
> +		trace_xdp_exception(adapter->netdev, prog, act);
> +		fallthrough;
> +	case XDP_DROP:
> +		res = IGC_XDP_CONSUMED;
> +		break;
> +	}
> +
> +unlock:
> +	rcu_read_unlock();
> +	return ERR_PTR(-res);
> +}
> +
> +bool igc_xdp_is_enabled(struct igc_adapter *adapter)
> +{
> +	return !!adapter->xdp_prog;
> +}
> diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
> new file mode 100644
> index 000000000000..ab36ab4d48fd
> --- /dev/null
> +++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
> @@ -0,0 +1,18 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (c) 2020, Intel Corporation. */
> +
> +#ifndef _IGC_XDP_H_
> +#define _IGC_XDP_H_
> +
> +#define IGC_XDP_PASS		0
> +#define IGC_XDP_CONSUMED	BIT(0)
> +
> +int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
> +		     struct netlink_ext_ack *extack);
> +
> +struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
> +				 struct xdp_buff *xdp);
> +
> +bool igc_xdp_is_enabled(struct igc_adapter *adapter);
> +
> +#endif /* _IGC_XDP_H_ */
> -- 
> 2.26.2
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan at osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

[Intel-wired-lan] [PATCH 7/9] igc: Add support for XDP_TX action

[Maciej Fijalkowski]

On Thu, Oct 08, 2020 at 07:53:47PM -0700, Andre Guedes wrote:
> This patch adds support for XDP_TX action which enables XDP programs to
> transmit back receiving frames.
> 
> The main changes in igc_xdp.c are related to the transmission of the
> packet (e.g. select the transmission queue, prepare buffer, prepare
> descriptor) while the changes in igc_main.c are related to tx_buffer
> management and XDP resources setup required to perform the XDP_TX
> action.
> 
> By default, the driver uses rx buffers with 2 KB size. When XDP is
> enabled, it uses larger buffers so we have enough space to accommodate
> the headroom and tailroom required by XDP infrastructure. While the
> driver had already code to handle larger buffers (IGC_RING_FLAG_RX_3K_
> BUFFER flag), it was missing the set/clear helpers that enable/disable
> that feature so this patch also introduces these missing helpers.

Ugh. This belongs to the previous patch.

> 
> I225 controller has only 4 tx hardware queues. Since XDP programs may
> not even issue an XDP_TX action, this patch doesn't reserve dedicated
> queues just for XDP like other Intel drivers do. Instead, the queues
> are shared between the network stack and XDP. The netdev queue lock is
> used to ensure mutual exclusion.
> 
> Since frames can now be transmitted via XDP_TX, the igc_tx_buffer
> structure is modified so we are able to save a reference to the xdp
> frame for later clean up once the packet is transmitted. The tx_buffer
> is mapped to either a skb or a xdpf so we use a union to save the skb
> or xdpf pointer and have a bit in tx_flags to indicate which field to
> use.
> 
> This patch has been tested with the sample app "xdp2" located in
> samples/bpf/ dir.
> 
> Signed-off-by: Andre Guedes <andre.guedes@intel.com>
> ---
>  drivers/net/ethernet/intel/igc/igc.h      |  13 ++-
>  drivers/net/ethernet/intel/igc/igc_main.c |  87 +++++++++++++--
>  drivers/net/ethernet/intel/igc/igc_xdp.c  | 122 ++++++++++++++++++++++
>  drivers/net/ethernet/intel/igc/igc_xdp.h  |   6 ++
>  4 files changed, 217 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h
> index e013ac65001b..cbf85dea8b2e 100644
> --- a/drivers/net/ethernet/intel/igc/igc.h
> +++ b/drivers/net/ethernet/intel/igc/igc.h
> @@ -111,6 +111,8 @@ struct igc_ring {
>  			struct sk_buff *skb;
>  		};
>  	};
> +
> +	struct xdp_rxq_info xdp_rxq;
>  } ____cacheline_internodealigned_in_smp;
>  
>  /* Board specific private data structure */
> @@ -373,6 +375,8 @@ enum igc_tx_flags {
>  	/* olinfo flags */
>  	IGC_TX_FLAGS_IPV4	= 0x10,
>  	IGC_TX_FLAGS_CSUM	= 0x20,
> +
> +	IGC_TX_FLAGS_XDP	= 0x100,
>  };
>  
>  enum igc_boards {
> @@ -395,7 +399,10 @@ enum igc_boards {
>  struct igc_tx_buffer {
>  	union igc_adv_tx_desc *next_to_watch;
>  	unsigned long time_stamp;
> -	struct sk_buff *skb;
> +	union {
> +		struct sk_buff *skb;
> +		struct xdp_frame *xdpf;
> +	};
>  	unsigned int bytecount;
>  	u16 gso_segs;
>  	__be16 protocol;
> @@ -504,6 +511,10 @@ enum igc_ring_flags_t {
>  
>  #define ring_uses_large_buffer(ring) \
>  	test_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
> +#define set_ring_uses_large_buffer(ring) \
> +	set_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
> +#define clear_ring_uses_large_buffer(ring) \
> +	clear_bit(IGC_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
>  
>  #define ring_uses_build_skb(ring) \
>  	test_bit(IGC_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags)
> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
> index 8cdd3a16e637..7005e6720261 100644
> --- a/drivers/net/ethernet/intel/igc/igc_main.c
> +++ b/drivers/net/ethernet/intel/igc/igc_main.c
> @@ -176,8 +176,10 @@ static void igc_clean_tx_ring(struct igc_ring *tx_ring)
>  	while (i != tx_ring->next_to_use) {
>  		union igc_adv_tx_desc *eop_desc, *tx_desc;
>  
> -		/* Free all the Tx ring sk_buffs */
> -		dev_kfree_skb_any(tx_buffer->skb);
> +		if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
> +			xdp_return_frame(tx_buffer->xdpf);
> +		else
> +			dev_kfree_skb_any(tx_buffer->skb);
>  
>  		/* unmap skb header data */
>  		dma_unmap_single(tx_ring->dev,
> @@ -345,6 +347,8 @@ static void igc_clean_rx_ring(struct igc_ring *rx_ring)
>  {
>  	u16 i = rx_ring->next_to_clean;
>  
> +	clear_ring_uses_large_buffer(rx_ring);
> +
>  	dev_kfree_skb(rx_ring->skb);
>  	rx_ring->skb = NULL;
>  
> @@ -403,6 +407,8 @@ void igc_free_rx_resources(struct igc_ring *rx_ring)
>  {
>  	igc_clean_rx_ring(rx_ring);
>  
> +	igc_xdp_unregister_rxq_info(rx_ring);
> +
>  	vfree(rx_ring->rx_buffer_info);
>  	rx_ring->rx_buffer_info = NULL;
>  
> @@ -440,7 +446,11 @@ int igc_setup_rx_resources(struct igc_ring *rx_ring)
>  {
>  	struct net_device *ndev = rx_ring->netdev;
>  	struct device *dev = rx_ring->dev;
> -	int size, desc_len;
> +	int size, desc_len, res;
> +
> +	res = igc_xdp_register_rxq_info(rx_ring);
> +	if (res < 0)
> +		return res;
>  
>  	size = sizeof(struct igc_rx_buffer) * rx_ring->count;
>  	rx_ring->rx_buffer_info = vzalloc(size);
> @@ -466,6 +476,7 @@ int igc_setup_rx_resources(struct igc_ring *rx_ring)
>  	return 0;
>  
>  err:
> +	igc_xdp_unregister_rxq_info(rx_ring);
>  	vfree(rx_ring->rx_buffer_info);
>  	rx_ring->rx_buffer_info = NULL;
>  	netdev_err(ndev, "Unable to allocate memory for Rx descriptor ring\n");
> @@ -513,6 +524,9 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter,
>  	u32 srrctl = 0, rxdctl = 0;
>  	u64 rdba = ring->dma;
>  
> +	if (igc_xdp_is_enabled(adapter))
> +		set_ring_uses_large_buffer(ring);
> +
>  	/* disable the queue */
>  	wr32(IGC_RXDCTL(reg_idx), 0);
>  
> @@ -1790,7 +1804,14 @@ static void igc_put_rx_buffer(struct igc_ring *rx_ring,
>  
>  static inline unsigned int igc_rx_offset(struct igc_ring *rx_ring)
>  {
> -	return ring_uses_build_skb(rx_ring) ? IGC_SKB_PAD : 0;
> +	struct igc_adapter *adapter = rx_ring->q_vector->adapter;
> +
> +	if (ring_uses_build_skb(rx_ring))
> +		return IGC_SKB_PAD;
> +	if (igc_xdp_is_enabled(adapter))
> +		return XDP_PACKET_HEADROOM;

This looks correct. We normally propagate the bpf_prog pointer down to
ring structure so if you would do this as well you wouldn't need to
retrieve the adapter.

> +
> +	return 0;
>  }
>  
>  static bool igc_alloc_mapped_page(struct igc_ring *rx_ring,
> @@ -1904,12 +1925,41 @@ static void igc_alloc_rx_buffers(struct igc_ring *rx_ring, u16 cleaned_count)
>  	}
>  }
>  
> +/* This function assumes __netif_tx_lock is held by the caller. */
> +static void igc_flush_tx_descriptors(struct igc_ring *ring)
> +{
> +	/* Once tail pointer is updated, hardware can fetch the descriptors
> +	 * any time so we issue a write membar here to ensure all memory
> +	 * writes are complete before the tail pointer is updated.
> +	 */
> +	wmb();
> +	writel(ring->next_to_use, ring->tail);
> +}
> +
> +static void igc_finalize_xdp(struct igc_adapter *adapter, int status)
> +{
> +	if (status & IGC_XDP_TX) {
> +		int cpu = smp_processor_id();
> +		struct netdev_queue *nq;
> +		struct igc_ring *ring;

move the variable declarations to the beginning of the functions.

> +
> +		ring = igc_xdp_get_tx_ring(adapter, cpu);
> +		nq = txring_txq(ring);
> +
> +		__netif_tx_lock(nq, cpu);
> +		igc_flush_tx_descriptors(ring);
> +		__netif_tx_unlock(nq);
> +	}
> +}
> +
>  static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  {
>  	unsigned int total_bytes = 0, total_packets = 0;
> +	struct igc_adapter *adapter = q_vector->adapter;
>  	struct igc_ring *rx_ring = q_vector->rx.ring;
>  	struct sk_buff *skb = rx_ring->skb;
>  	u16 cleaned_count = igc_desc_unused(rx_ring);
> +	int xdp_status = 0;
>  
>  	while (likely(total_packets < budget)) {
>  		union igc_adv_rx_desc *rx_desc;
> @@ -1917,7 +1967,7 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  		ktime_t timestamp = 0;
>  		int pkt_offset = 0;
>  		struct xdp_buff xdp;
> -		unsigned int size;
> +		unsigned int size, truesize;
>  		void *pktbuf;
>  
>  		/* return some buffers to hardware, one at a time is too slow */
> @@ -1938,6 +1988,7 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  		dma_rmb();
>  
>  		rx_buffer = igc_get_rx_buffer(rx_ring, size);
> +		truesize = igc_get_rx_frame_truesize(rx_ring, size);
>  
>  		pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
>  
> @@ -1949,18 +2000,29 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  		}
>  
>  		if (!skb) {
> -			struct igc_adapter *adapter = q_vector->adapter;
> -
>  			xdp.data = pktbuf + pkt_offset;
>  			xdp.data_end = xdp.data + size;
>  			xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
>  			xdp_set_data_meta_invalid(&xdp);
> +			xdp.frame_sz = truesize;
> +			xdp.rxq = &rx_ring->xdp_rxq;
>  
>  			skb = igc_xdp_run_prog(adapter, &xdp);
>  		}
>  
>  		if (IS_ERR(skb)) {
> -			rx_buffer->pagecnt_bias++;
> +			unsigned int xdp_res = -PTR_ERR(skb);
> +
> +			switch (xdp_res) {
> +			case IGC_XDP_CONSUMED:
> +				rx_buffer->pagecnt_bias++;
> +				break;
> +			case IGC_XDP_TX:
> +				igc_rx_buffer_flip(rx_buffer, truesize);
> +				xdp_status |= xdp_res;
> +				break;
> +			}
> +
>  			total_packets++;
>  			total_bytes += size;
>  		} else if (skb)
> @@ -2006,6 +2068,9 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
>  		total_packets++;
>  	}
>  
> +	if (xdp_status)
> +		igc_finalize_xdp(adapter, xdp_status);
> +
>  	/* place incomplete frames back on ring for completion */
>  	rx_ring->skb = skb;
>  
> @@ -2067,8 +2132,10 @@ static bool igc_clean_tx_irq(struct igc_q_vector *q_vector, int napi_budget)
>  		total_bytes += tx_buffer->bytecount;
>  		total_packets += tx_buffer->gso_segs;
>  
> -		/* free the skb */
> -		napi_consume_skb(tx_buffer->skb, napi_budget);
> +		if (tx_buffer->tx_flags & IGC_TX_FLAGS_XDP)
> +			xdp_return_frame(tx_buffer->xdpf);
> +		else
> +			napi_consume_skb(tx_buffer->skb, napi_budget);
>  
>  		/* unmap skb header data */
>  		dma_unmap_single(tx_ring->dev,
> diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.c b/drivers/net/ethernet/intel/igc/igc_xdp.c
> index 0085d8b85466..9f4c58436170 100644
> --- a/drivers/net/ethernet/intel/igc/igc_xdp.c
> +++ b/drivers/net/ethernet/intel/igc/igc_xdp.c
> @@ -35,6 +35,94 @@ int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
>  	return 0;
>  }
>  
> +static int igc_xdp_init_tx_buffer(struct igc_tx_buffer *buffer,
> +				  struct xdp_frame *xdpf,
> +				  struct igc_ring *ring)
> +{
> +	dma_addr_t dma;
> +
> +	dma = dma_map_single(ring->dev, xdpf->data, xdpf->len, DMA_TO_DEVICE);
> +	if (dma_mapping_error(ring->dev, dma)) {
> +		netdev_err_once(ring->netdev, "Failed to map DMA for TX\n");
> +		return -ENOMEM;
> +	}
> +
> +	buffer->xdpf = xdpf;
> +	buffer->tx_flags = IGC_TX_FLAGS_XDP;
> +	buffer->protocol = 0;
> +	buffer->bytecount = xdpf->len;
> +	buffer->gso_segs = 1;
> +	buffer->time_stamp = jiffies;
> +	dma_unmap_len_set(buffer, len, xdpf->len);
> +	dma_unmap_addr_set(buffer, dma, dma);
> +	return 0;
> +}
> +
> +/* This function requires __netif_tx_lock is held by the caller. */
> +static int igc_xdp_init_tx_descriptor(struct igc_ring *ring,
> +				      struct xdp_frame *xdpf)
> +{
> +	struct igc_tx_buffer *buffer;
> +	union igc_adv_tx_desc *desc;
> +	u32 cmd_type, olinfo_status;
> +	int err;
> +
> +	if (!igc_desc_unused(ring))
> +		return -EBUSY;
> +
> +	buffer = &ring->tx_buffer_info[ring->next_to_use];
> +	err = igc_xdp_init_tx_buffer(buffer, xdpf, ring);
> +	if (err)
> +		return err;
> +
> +	cmd_type = IGC_ADVTXD_DTYP_DATA | IGC_ADVTXD_DCMD_DEXT |
> +		   IGC_ADVTXD_DCMD_IFCS | IGC_TXD_DCMD |
> +		   buffer->bytecount;
> +	olinfo_status = buffer->bytecount << IGC_ADVTXD_PAYLEN_SHIFT;
> +
> +	desc = IGC_TX_DESC(ring, ring->next_to_use);
> +	desc->read.cmd_type_len = cpu_to_le32(cmd_type);
> +	desc->read.olinfo_status = cpu_to_le32(olinfo_status);
> +	desc->read.buffer_addr = cpu_to_le64(dma_unmap_addr(buffer, dma));
> +
> +	netdev_tx_sent_queue(txring_txq(ring), buffer->bytecount);

So you're letting stack know that you sent a frame but that's not what is
happening because you wait with bumping the tail until you are done with
napi rx loop.

this probably should be called during the tx irq cleanup.

> +
> +	buffer->next_to_watch = desc;
> +
> +	ring->next_to_use = (ring->next_to_use + 1) % ring->count;

Please avoid modulos.

> +	return 0;
> +}
> +
> +struct igc_ring *igc_xdp_get_tx_ring(struct igc_adapter *adapter, int cpu)
> +{
> +	struct igc_ring *ring;
> +	int index;
> +
> +	index = cpu % adapter->num_tx_queues;

ditto

> +	ring = adapter->tx_ring[index];
> +	return ring;
> +}
> +
> +static int igc_xdp_xmit_back(struct igc_adapter *adapter, struct xdp_buff *xdp)
> +{
> +	struct xdp_frame *xdpf = xdp_convert_buff_to_frame(xdp);
> +	int cpu = smp_processor_id();
> +	struct netdev_queue *nq;
> +	struct igc_ring *ring;
> +	int res;
> +
> +	if (unlikely(!xdpf))
> +		return -EFAULT;
> +
> +	ring = igc_xdp_get_tx_ring(adapter, cpu);
> +	nq = txring_txq(ring);
> +
> +	__netif_tx_lock(nq, cpu);
> +	res = igc_xdp_init_tx_descriptor(ring, xdpf);
> +	__netif_tx_unlock(nq);
> +	return res;
> +}
> +
>  struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
>  				 struct xdp_buff *xdp)
>  {
> @@ -53,6 +141,12 @@ struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
>  	case XDP_PASS:
>  		res = IGC_XDP_PASS;
>  		break;
> +	case XDP_TX:
> +		if (igc_xdp_xmit_back(adapter, xdp) < 0)
> +			res = IGC_XDP_CONSUMED;
> +		else
> +			res = IGC_XDP_TX;
> +		break;
>  	default:
>  		bpf_warn_invalid_xdp_action(act);
>  		fallthrough;
> @@ -73,3 +167,31 @@ bool igc_xdp_is_enabled(struct igc_adapter *adapter)
>  {
>  	return !!adapter->xdp_prog;
>  }
> +
> +int igc_xdp_register_rxq_info(struct igc_ring *ring)
> +{
> +	struct net_device *dev = ring->netdev;
> +	int err;
> +
> +	err = xdp_rxq_info_reg(&ring->xdp_rxq, dev, ring->queue_index);
> +	if (err) {
> +		netdev_err(dev, "Failed to register xdp rxq info\n");
> +		return err;
> +	}
> +
> +	err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, MEM_TYPE_PAGE_SHARED,
> +					 NULL);
> +	if (err) {
> +		netdev_err(dev, "Failed to register xdp rxq mem model\n");
> +		xdp_rxq_info_unreg(&ring->xdp_rxq);
> +		return err;
> +	}
> +
> +	return 0;
> +}
> +
> +void igc_xdp_unregister_rxq_info(struct igc_ring *ring)
> +{
> +	xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);

xdp_rxq_info_unreg will call internally the xdp_rxq_info_unreg_mem_model
so I think the call above is redundant.

> +	xdp_rxq_info_unreg(&ring->xdp_rxq);
> +}
> diff --git a/drivers/net/ethernet/intel/igc/igc_xdp.h b/drivers/net/ethernet/intel/igc/igc_xdp.h
> index ab36ab4d48fd..fdb31f40dbe5 100644
> --- a/drivers/net/ethernet/intel/igc/igc_xdp.h
> +++ b/drivers/net/ethernet/intel/igc/igc_xdp.h
> @@ -6,6 +6,7 @@
>  
>  #define IGC_XDP_PASS		0
>  #define IGC_XDP_CONSUMED	BIT(0)
> +#define IGC_XDP_TX		BIT(1)
>  
>  int igc_xdp_set_prog(struct igc_adapter *adapter, struct bpf_prog *prog,
>  		     struct netlink_ext_ack *extack);
> @@ -15,4 +16,9 @@ struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
>  
>  bool igc_xdp_is_enabled(struct igc_adapter *adapter);
>  
> +int igc_xdp_register_rxq_info(struct igc_ring *ring);
> +void igc_xdp_unregister_rxq_info(struct igc_ring *ring);
> +
> +struct igc_ring *igc_xdp_get_tx_ring(struct igc_adapter *adapter, int cpu);
> +
>  #endif /* _IGC_XDP_H_ */
> -- 
> 2.26.2
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan at osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

[Intel-wired-lan] [PATCH 1/9] igc: Fix igc_ptp_rx_pktstamp()

[Andre Guedes]

Hi Maciej,

Quoting Maciej Fijalkowski (2020-10-21 06:50:05)
> On Thu, Oct 08, 2020 at 07:53:41PM -0700, Andre Guedes wrote:
> > The comment describing the timestamps layout in the packet buffer is
> > wrong and the code is actually retrieving the timestamp in Timer 1
> > reference instead of Timer 0. This hasn't been a big issue so far
> > because hardware is configured to report both timestamps using Timer 0
> > (see IGC_SRRCTL register configuration in igc_ptp_enable_rx_timestamp()
> > helper). This patch fixes the comment and the code so we retrieve the
> > timestamp in Timer 0 reference as expected.
> > 
> > This patch also takes the opportunity to get rid of the hw.mac.type check
> > since it is not required.
> 
> How is this related to adding XDP support?

It is not directly related, however patches in this series depends on this patch
to apply (see "igc: Refactor rx timestamp handling"). That's why I put it as the
first patch in the series.

- Andre

[Intel-wired-lan] [PATCH 6/9] igc: Add Initial XDP support

[Andre Guedes]

Hi Maciej,

Thank you for your review.

Quoting Maciej Fijalkowski (2020-10-21 07:23:49)
> On Thu, Oct 08, 2020 at 07:53:46PM -0700, Andre Guedes wrote:
> > This patch adds the initial XDP support to the igc driver. For now,
> > only XDP_PASS, XDP_DROP, XDP_ABORT actions are supported. Upcoming
> 
> s/XDP_ABORT/XDP_ABORTED

Good catch, I'll fix it.

> > @@ -1592,11 +1593,11 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
> >  
> >  static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
> >                                        struct igc_rx_buffer *rx_buffer,
> > -                                      unsigned int size, int pkt_offset,
> > +                                      struct xdp_buff *xdp,
> >                                        ktime_t timestamp)
> >  {
> > -     void *va = page_address(rx_buffer->page) + rx_buffer->page_offset +
> > -                pkt_offset;
> > +     void *va = xdp->data;
> > +     unsigned int size = xdp->data_end - xdp->data;
> >       unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
> >       unsigned int headlen;
> >       struct sk_buff *skb;
> 
> RCT

We can move 'va' declaration around, but note that 'size' is utilized to
calculate 'truesize' so the RCT won't be perfect still.

> > @@ -1932,24 +1939,37 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
> >  
> >               rx_buffer = igc_get_rx_buffer(rx_ring, size);
> >  
> > -             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> > -                     void *pktbuf = page_address(rx_buffer->page) +
> > -                                    rx_buffer->page_offset;
> > +             pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
> >  
> > +             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> >                       timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
> >                                                       pktbuf);
> >                       pkt_offset = IGC_TS_HDR_LEN;
> >                       size -= IGC_TS_HDR_LEN;
> >               }
> >  
> > -             /* retrieve a buffer from the ring */
> > -             if (skb)
> > +             if (!skb) {
> > +                     struct igc_adapter *adapter = q_vector->adapter;
> > +
> > +                     xdp.data = pktbuf + pkt_offset;
> > +                     xdp.data_end = xdp.data + size;
> > +                     xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
> 
> This needs some fixing. You're saying that build_skb() can't be currently
> used which means that the headroom given for XDP will always be zero.
> 
> You need to either teach igc_rx_offset to return XDP_PACKET_HEADROOM for
> case when ring is not using build_skb() or make build_skb() usable.

The xdp headroom doesn't seem to be required to support XDP_PASS, XDP_DROP and
XDP_ABORTED actions, which is the goal of this patch. That was the rationale to
not change igc_rx_offset() in this patch.

The igc_rx_offset() helper is updated in the next patch which adds support for
XDP_TX action since it requires xdp headroom.

> > +                     xdp_set_data_meta_invalid(&xdp);
> 
> What about xdp.frame_sz?

Same as above.

> > @@ -3881,6 +3901,11 @@ static int igc_change_mtu(struct net_device *netdev, int new_mtu)
> >       int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
> 
> I think I was mentioning it with igb xdp review, shouldn't we take double
> vlan header into account?

Good question. It seems like we should take it into consideration when updating
adapter->max_frame_size later in the igc_change_mtu(). However, note that in
the code added below, we check 'new_mtu', not 'max_frame', so with or without
double tagging it should do the right thing.

> > +     if (igc_xdp_is_enabled(adapter) && new_mtu > ETH_DATA_LEN) {
> > +             netdev_dbg(netdev, "Jumbo frames not supported with XDP");
> > +             return -EINVAL;
> > +     }
> > +

[...]

> > +struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
> > +                              struct xdp_buff *xdp)
> > +{
> > +     struct bpf_prog *prog;
> > +     int res;
> > +     u32 act;
> > +
> > +     rcu_read_lock();
> > +
> > +     prog = READ_ONCE(adapter->xdp_prog);
> > +     if (!prog)
> > +             goto unlock;
> 
> res will be uninitialized if you hit this goto.

Good catch, I'll fix it.

Best,
Andre

[Intel-wired-lan] [PATCH 7/9] igc: Add support for XDP_TX action

[Andre Guedes]

Hi Maciej,

Quoting Maciej Fijalkowski (2020-10-21 07:57:42)
> On Thu, Oct 08, 2020 at 07:53:47PM -0700, Andre Guedes wrote:
> > This patch adds support for XDP_TX action which enables XDP programs to
> > transmit back receiving frames.
> > 
> > The main changes in igc_xdp.c are related to the transmission of the
> > packet (e.g. select the transmission queue, prepare buffer, prepare
> > descriptor) while the changes in igc_main.c are related to tx_buffer
> > management and XDP resources setup required to perform the XDP_TX
> > action.
> > 
> > By default, the driver uses rx buffers with 2 KB size. When XDP is
> > enabled, it uses larger buffers so we have enough space to accommodate
> > the headroom and tailroom required by XDP infrastructure. While the
> > driver had already code to handle larger buffers (IGC_RING_FLAG_RX_3K_
> > BUFFER flag), it was missing the set/clear helpers that enable/disable
> > that feature so this patch also introduces these missing helpers.
> 
> Ugh. This belongs to the previous patch.

Hope the reply on the previous patch clarifies why this is done in this patch
instead.

> > +/* This function requires __netif_tx_lock is held by the caller. */
> > +static int igc_xdp_init_tx_descriptor(struct igc_ring *ring,
> > +                                   struct xdp_frame *xdpf)
> > +{
> > +     struct igc_tx_buffer *buffer;
> > +     union igc_adv_tx_desc *desc;
> > +     u32 cmd_type, olinfo_status;
> > +     int err;
> > +
> > +     if (!igc_desc_unused(ring))
> > +             return -EBUSY;
> > +
> > +     buffer = &ring->tx_buffer_info[ring->next_to_use];
> > +     err = igc_xdp_init_tx_buffer(buffer, xdpf, ring);
> > +     if (err)
> > +             return err;
> > +
> > +     cmd_type = IGC_ADVTXD_DTYP_DATA | IGC_ADVTXD_DCMD_DEXT |
> > +                IGC_ADVTXD_DCMD_IFCS | IGC_TXD_DCMD |
> > +                buffer->bytecount;
> > +     olinfo_status = buffer->bytecount << IGC_ADVTXD_PAYLEN_SHIFT;
> > +
> > +     desc = IGC_TX_DESC(ring, ring->next_to_use);
> > +     desc->read.cmd_type_len = cpu_to_le32(cmd_type);
> > +     desc->read.olinfo_status = cpu_to_le32(olinfo_status);
> > +     desc->read.buffer_addr = cpu_to_le64(dma_unmap_addr(buffer, dma));
> > +
> > +     netdev_tx_sent_queue(txring_txq(ring), buffer->bytecount);
> 
> So you're letting stack know that you sent a frame but that's not what is
> happening because you wait with bumping the tail until you are done with
> napi rx loop.
> 
> this probably should be called during the tx irq cleanup.

I see your point, and I think calling it during tx irq cleanup should work too.

The rationale to put it here was to keep it similar to ndo_start_xmit's flow
i.e. the stack is notified before packets are actually transmitted by
hardware. Also, if we do it in tx irq cleanup, we probably need to acquire
__netif_tx_lock to ensure we don't have a race conditions with ndo_start_xmit.

Let me know your thoughts here.

> > +
> > +     buffer->next_to_watch = desc;
> > +
> > +     ring->next_to_use = (ring->next_to_use + 1) % ring->count;
> 
> Please avoid modulos.

Sure, I'll do it. Just out of curiosity, why modulo operation should be
avoided?

> > +void igc_xdp_unregister_rxq_info(struct igc_ring *ring)
> > +{
> > +     xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
> 
> xdp_rxq_info_unreg will call internally the xdp_rxq_info_unreg_mem_model
> so I think the call above is redundant.

Yep, you're right, this is redundant. I'll remove it.

Thanks,
Andre

[Intel-wired-lan] [PATCH 6/9] igc: Add Initial XDP support

[Maciej Fijalkowski]

On Wed, Oct 21, 2020 at 03:16:51PM -0700, Andre Guedes wrote:
> Hi Maciej,
> 
> Thank you for your review.
> 
> Quoting Maciej Fijalkowski (2020-10-21 07:23:49)
> > On Thu, Oct 08, 2020 at 07:53:46PM -0700, Andre Guedes wrote:
> > > This patch adds the initial XDP support to the igc driver. For now,
> > > only XDP_PASS, XDP_DROP, XDP_ABORT actions are supported. Upcoming
> > 
> > s/XDP_ABORT/XDP_ABORTED
> 
> Good catch, I'll fix it.
> 
> > > @@ -1592,11 +1593,11 @@ static struct sk_buff *igc_build_skb(struct igc_ring *rx_ring,
> > >  
> > >  static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring,
> > >                                        struct igc_rx_buffer *rx_buffer,
> > > -                                      unsigned int size, int pkt_offset,
> > > +                                      struct xdp_buff *xdp,
> > >                                        ktime_t timestamp)
> > >  {
> > > -     void *va = page_address(rx_buffer->page) + rx_buffer->page_offset +
> > > -                pkt_offset;
> > > +     void *va = xdp->data;
> > > +     unsigned int size = xdp->data_end - xdp->data;
> > >       unsigned int truesize = igc_get_rx_frame_truesize(rx_ring, size);
> > >       unsigned int headlen;
> > >       struct sk_buff *skb;
> > 
> > RCT
> 
> We can move 'va' declaration around, but note that 'size' is utilized to
> calculate 'truesize' so the RCT won't be perfect still.

Right, but va/size misorder was the thing that caught my attention.

> 
> > > @@ -1932,24 +1939,37 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
> > >  
> > >               rx_buffer = igc_get_rx_buffer(rx_ring, size);
> > >  
> > > -             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> > > -                     void *pktbuf = page_address(rx_buffer->page) +
> > > -                                    rx_buffer->page_offset;
> > > +             pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
> > >  
> > > +             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> > >                       timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
> > >                                                       pktbuf);
> > >                       pkt_offset = IGC_TS_HDR_LEN;
> > >                       size -= IGC_TS_HDR_LEN;
> > >               }
> > >  
> > > -             /* retrieve a buffer from the ring */
> > > -             if (skb)
> > > +             if (!skb) {
> > > +                     struct igc_adapter *adapter = q_vector->adapter;
> > > +
> > > +                     xdp.data = pktbuf + pkt_offset;
> > > +                     xdp.data_end = xdp.data + size;
> > > +                     xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
> > 
> > This needs some fixing. You're saying that build_skb() can't be currently
> > used which means that the headroom given for XDP will always be zero.
> > 
> > You need to either teach igc_rx_offset to return XDP_PACKET_HEADROOM for
> > case when ring is not using build_skb() or make build_skb() usable.
> 
> The xdp headroom doesn't seem to be required to support XDP_PASS, XDP_DROP and
> XDP_ABORTED actions, which is the goal of this patch. That was the rationale to
> not change igc_rx_offset() in this patch.
> 
> The igc_rx_offset() helper is updated in the next patch which adds support for
> XDP_TX action since it requires xdp headroom.

Sorry when I wrote that I haven't looked at next patch.
The thing is that you can have a bpf program that will do the
encapsulation and return XDP_PASS for example. For encapsulation you need
to provide a headroom.

> 
> > > +                     xdp_set_data_meta_invalid(&xdp);
> > 
> > What about xdp.frame_sz?
> 
> Same as above.
> 
> > > @@ -3881,6 +3901,11 @@ static int igc_change_mtu(struct net_device *netdev, int new_mtu)
> > >       int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
> > 
> > I think I was mentioning it with igb xdp review, shouldn't we take double
> > vlan header into account?
> 
> Good question. It seems like we should take it into consideration when updating
> adapter->max_frame_size later in the igc_change_mtu(). However, note that in
> the code added below, we check 'new_mtu', not 'max_frame', so with or without
> double tagging it should do the right thing.

Hm yeah but I think it should be fixed anyway even without xdp being in
picture?

> 
> > > +     if (igc_xdp_is_enabled(adapter) && new_mtu > ETH_DATA_LEN) {
> > > +             netdev_dbg(netdev, "Jumbo frames not supported with XDP");
> > > +             return -EINVAL;
> > > +     }
> > > +
> 
> [...]
> 
> > > +struct sk_buff *igc_xdp_run_prog(struct igc_adapter *adapter,
> > > +                              struct xdp_buff *xdp)
> > > +{
> > > +     struct bpf_prog *prog;
> > > +     int res;
> > > +     u32 act;
> > > +
> > > +     rcu_read_lock();
> > > +
> > > +     prog = READ_ONCE(adapter->xdp_prog);
> > > +     if (!prog)
> > > +             goto unlock;
> > 
> > res will be uninitialized if you hit this goto.
> 
> Good catch, I'll fix it.
> 
> Best,
> Andre

[Intel-wired-lan] [PATCH 7/9] igc: Add support for XDP_TX action

[Maciej Fijalkowski]

On Wed, Oct 21, 2020 at 03:17:04PM -0700, Andre Guedes wrote:
> Hi Maciej,
> 
> Quoting Maciej Fijalkowski (2020-10-21 07:57:42)
> > On Thu, Oct 08, 2020 at 07:53:47PM -0700, Andre Guedes wrote:
> > > This patch adds support for XDP_TX action which enables XDP programs to
> > > transmit back receiving frames.
> > > 
> > > The main changes in igc_xdp.c are related to the transmission of the
> > > packet (e.g. select the transmission queue, prepare buffer, prepare
> > > descriptor) while the changes in igc_main.c are related to tx_buffer
> > > management and XDP resources setup required to perform the XDP_TX
> > > action.
> > > 
> > > By default, the driver uses rx buffers with 2 KB size. When XDP is
> > > enabled, it uses larger buffers so we have enough space to accommodate
> > > the headroom and tailroom required by XDP infrastructure. While the
> > > driver had already code to handle larger buffers (IGC_RING_FLAG_RX_3K_
> > > BUFFER flag), it was missing the set/clear helpers that enable/disable
> > > that feature so this patch also introduces these missing helpers.
> > 
> > Ugh. This belongs to the previous patch.
> 
> Hope the reply on the previous patch clarifies why this is done in this patch
> instead.
> 
> > > +/* This function requires __netif_tx_lock is held by the caller. */
> > > +static int igc_xdp_init_tx_descriptor(struct igc_ring *ring,
> > > +                                   struct xdp_frame *xdpf)
> > > +{
> > > +     struct igc_tx_buffer *buffer;
> > > +     union igc_adv_tx_desc *desc;
> > > +     u32 cmd_type, olinfo_status;
> > > +     int err;
> > > +
> > > +     if (!igc_desc_unused(ring))
> > > +             return -EBUSY;
> > > +
> > > +     buffer = &ring->tx_buffer_info[ring->next_to_use];
> > > +     err = igc_xdp_init_tx_buffer(buffer, xdpf, ring);
> > > +     if (err)
> > > +             return err;
> > > +
> > > +     cmd_type = IGC_ADVTXD_DTYP_DATA | IGC_ADVTXD_DCMD_DEXT |
> > > +                IGC_ADVTXD_DCMD_IFCS | IGC_TXD_DCMD |
> > > +                buffer->bytecount;
> > > +     olinfo_status = buffer->bytecount << IGC_ADVTXD_PAYLEN_SHIFT;
> > > +
> > > +     desc = IGC_TX_DESC(ring, ring->next_to_use);
> > > +     desc->read.cmd_type_len = cpu_to_le32(cmd_type);
> > > +     desc->read.olinfo_status = cpu_to_le32(olinfo_status);
> > > +     desc->read.buffer_addr = cpu_to_le64(dma_unmap_addr(buffer, dma));
> > > +
> > > +     netdev_tx_sent_queue(txring_txq(ring), buffer->bytecount);
> > 
> > So you're letting stack know that you sent a frame but that's not what is
> > happening because you wait with bumping the tail until you are done with
> > napi rx loop.
> > 
> > this probably should be called during the tx irq cleanup.
> 
> I see your point, and I think calling it during tx irq cleanup should work too.
> 
> The rationale to put it here was to keep it similar to ndo_start_xmit's flow
> i.e. the stack is notified before packets are actually transmitted by
> hardware. Also, if we do it in tx irq cleanup, we probably need to acquire
> __netif_tx_lock to ensure we don't have a race conditions with ndo_start_xmit.
> 
> Let me know your thoughts here.

Sorry, ignore my previous comment. It's fine.

> 
> > > +
> > > +     buffer->next_to_watch = desc;
> > > +
> > > +     ring->next_to_use = (ring->next_to_use + 1) % ring->count;
> > 
> > Please avoid modulos.
> 
> Sure, I'll do it. Just out of curiosity, why modulo operation should be
> avoided?

Just because it's slow and you're in fast path :p but you can take it as a
suggestion.

> 
> > > +void igc_xdp_unregister_rxq_info(struct igc_ring *ring)
> > > +{
> > > +     xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
> > 
> > xdp_rxq_info_unreg will call internally the xdp_rxq_info_unreg_mem_model
> > so I think the call above is redundant.
> 
> Yep, you're right, this is redundant. I'll remove it.
> 
> Thanks,
> Andre

[Intel-wired-lan] [PATCH 6/9] igc: Add Initial XDP support

[Andre Guedes]

Hi Maciej,

Quoting Maciej Fijalkowski (2020-10-22 02:53:10)
> > > > @@ -1932,24 +1939,37 @@ static int igc_clean_rx_irq(struct igc_q_vector *q_vector, const int budget)
> > > >  
> > > >               rx_buffer = igc_get_rx_buffer(rx_ring, size);
> > > >  
> > > > -             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> > > > -                     void *pktbuf = page_address(rx_buffer->page) +
> > > > -                                    rx_buffer->page_offset;
> > > > +             pktbuf = page_address(rx_buffer->page) + rx_buffer->page_offset;
> > > >  
> > > > +             if (igc_test_staterr(rx_desc, IGC_RXDADV_STAT_TSIP)) {
> > > >                       timestamp = igc_ptp_rx_pktstamp(q_vector->adapter,
> > > >                                                       pktbuf);
> > > >                       pkt_offset = IGC_TS_HDR_LEN;
> > > >                       size -= IGC_TS_HDR_LEN;
> > > >               }
> > > >  
> > > > -             /* retrieve a buffer from the ring */
> > > > -             if (skb)
> > > > +             if (!skb) {
> > > > +                     struct igc_adapter *adapter = q_vector->adapter;
> > > > +
> > > > +                     xdp.data = pktbuf + pkt_offset;
> > > > +                     xdp.data_end = xdp.data + size;
> > > > +                     xdp.data_hard_start = pktbuf - igc_rx_offset(rx_ring);
> > > 
> > > This needs some fixing. You're saying that build_skb() can't be currently
> > > used which means that the headroom given for XDP will always be zero.
> > > 
> > > You need to either teach igc_rx_offset to return XDP_PACKET_HEADROOM for
> > > case when ring is not using build_skb() or make build_skb() usable.
> > 
> > The xdp headroom doesn't seem to be required to support XDP_PASS, XDP_DROP and
> > XDP_ABORTED actions, which is the goal of this patch. That was the rationale to
> > not change igc_rx_offset() in this patch.
> > 
> > The igc_rx_offset() helper is updated in the next patch which adds support for
> > XDP_TX action since it requires xdp headroom.
> 
> Sorry when I wrote that I haven't looked at next patch.
> The thing is that you can have a bpf program that will do the
> encapsulation and return XDP_PASS for example. For encapsulation you need
> to provide a headroom.

Understood, thanks for the example. I was not aware of the encapsulation +
XDP_PASS case. I'll fix this in the v2.

> > > > @@ -3881,6 +3901,11 @@ static int igc_change_mtu(struct net_device *netdev, int new_mtu)
> > > >       int max_frame = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
> > > 
> > > I think I was mentioning it with igb xdp review, shouldn't we take double
> > > vlan header into account?
> > 
> > Good question. It seems like we should take it into consideration when updating
> > adapter->max_frame_size later in the igc_change_mtu(). However, note that in
> > the code added below, we check 'new_mtu', not 'max_frame', so with or without
> > double tagging it should do the right thing.
> 
> Hm yeah but I think it should be fixed anyway even without xdp being in
> picture?

Agreed. I can provide a separate patch soon.

Cheers,
Andre

[Intel-wired-lan] [PATCH 7/9] igc: Add support for XDP_TX action

[Andre Guedes]

Hi Maciej,

Quoting Maciej Fijalkowski (2020-10-22 03:10:03)
> > > > +
> > > > +     buffer->next_to_watch = desc;
> > > > +
> > > > +     ring->next_to_use = (ring->next_to_use + 1) % ring->count;
> > > 
> > > Please avoid modulos.
> > 
> > Sure, I'll do it. Just out of curiosity, why modulo operation should be
> > avoided?
> 
> Just because it's slow and you're in fast path :p but you can take it as a
> suggestion.

Thanks, it makes sense. Also, looking at other Intel drivers, they handle
wrapping with an if-clause. I'll do the same here.

Cheers,
Andre