[PATCH net-next 0/5] net: tso: expand to UDP support

[PATCH net-next 0/5] net: tso: expand to UDP support

[Eric Dumazet]

With QUIC getting more attention these days, it is worth
implementing UDP direct segmentation, the same we did for TCP.

Drivers will need to advertize NETIF_F_GSO_UDP_L4 so that
GSO stack does not do the (more expensive) segmentation.

Eric Dumazet (5):
  net: tso: double TSO_HEADER_SIZE value
  net: tso: shrink struct tso_t
  net: tso: constify tso_count_descs() and friends
  net: tso: cache transport header length
  net: tso: add UDP segmentation support

 .../ethernet/cavium/thunder/nicvf_queues.c    |  5 ++-
 drivers/net/ethernet/freescale/fec_main.c     |  5 +--
 drivers/net/ethernet/marvell/mv643xx_eth.c    |  5 +--
 drivers/net/ethernet/marvell/mvneta.c         |  5 +--
 .../net/ethernet/marvell/mvpp2/mvpp2_main.c   |  6 +--
 .../marvell/octeontx2/nic/otx2_txrx.c         |  6 +--
 include/net/tso.h                             | 23 +++++-----
 net/core/tso.c                                | 44 ++++++++++++-------
 8 files changed, 54 insertions(+), 45 deletions(-)

-- 
2.27.0.290.gba653c62da-goog

[PATCH net-next 1/5] net: tso: double TSO_HEADER_SIZE value

[Eric Dumazet]

Transport header size could be 60 bytes, and network header
size can also be 60 bytes. Add the Ethernet header and we
are above 128 bytes.

Since drivers using net/core/tso.c usually allocates
one DMA coherent piece of memory per TX queue, this patch
might cause issues if a driver was using too many slots.

For 1024 slots, we would need 256 KB of physically
contiguous memory instead of 128 KB.

Alternative fix would be to add checks in the fast path,
but this involves more work in all drivers using net/core/tso.c.

Fixes: f9cbe9a556af ("net: define the TSO header size in net/tso.h")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Antoine Tenart <antoine.tenart@bootlin.com>
---
Note : please wait at least one cycle before backporting to stable versions.

 include/net/tso.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/net/tso.h b/include/net/tso.h
index 7e166a5703497fadf4662acc474f827b2754da78..c33dd00c161f7a6aa65f586b0ceede46af2e8730 100644
--- a/include/net/tso.h
+++ b/include/net/tso.h
@@ -4,7 +4,7 @@
 
 #include <net/ip.h>
 
-#define TSO_HEADER_SIZE		128
+#define TSO_HEADER_SIZE		256
 
 struct tso_t {
 	int next_frag_idx;
-- 
2.27.0.290.gba653c62da-goog

Re: [PATCH net-next 1/5] net: tso: double TSO_HEADER_SIZE value

[Jakub Kicinski]

On Wed, 17 Jun 2020 11:48:15 -0700 Eric Dumazet wrote:
> Transport header size could be 60 bytes, and network header
> size can also be 60 bytes. Add the Ethernet header and we
> are above 128 bytes.
> 
> Since drivers using net/core/tso.c usually allocates
> one DMA coherent piece of memory per TX queue, this patch
> might cause issues if a driver was using too many slots.
> 
> For 1024 slots, we would need 256 KB of physically
> contiguous memory instead of 128 KB.
> 
> Alternative fix would be to add checks in the fast path,
> but this involves more work in all drivers using net/core/tso.c.
> 
> Fixes: f9cbe9a556af ("net: define the TSO header size in net/tso.h")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Antoine Tenart <antoine.tenart@bootlin.com>

Some warnings popping up in this series with W=1 C=1:

drivers/net/ethernet/marvell/octeontx2/af/common.h:65:26: warning: cast truncates bits from constant value (100 becomes 0)

Re: [PATCH net-next 1/5] net: tso: double TSO_HEADER_SIZE value

[Eric Dumazet]

On 6/17/20 5:02 PM, Jakub Kicinski wrote:
> On Wed, 17 Jun 2020 11:48:15 -0700 Eric Dumazet wrote:
>> Transport header size could be 60 bytes, and network header
>> size can also be 60 bytes. Add the Ethernet header and we
>> are above 128 bytes.
>>
>> Since drivers using net/core/tso.c usually allocates
>> one DMA coherent piece of memory per TX queue, this patch
>> might cause issues if a driver was using too many slots.
>>
>> For 1024 slots, we would need 256 KB of physically
>> contiguous memory instead of 128 KB.
>>
>> Alternative fix would be to add checks in the fast path,
>> but this involves more work in all drivers using net/core/tso.c.
>>
>> Fixes: f9cbe9a556af ("net: define the TSO header size in net/tso.h")
>> Signed-off-by: Eric Dumazet <edumazet@google.com>
>> Cc: Antoine Tenart <antoine.tenart@bootlin.com>
> 
> Some warnings popping up in this series with W=1 C=1:
> 
> drivers/net/ethernet/marvell/octeontx2/af/common.h:65:26: warning: cast truncates bits from constant value (100 becomes 0)
>

Nice, thanks I will take a look.

[PATCH net-next 2/5] net: tso: shrink struct tso_t

[Eric Dumazet]

size field can be an int, no need for size_t

Removes a 32bit hole on 64bit kernels.

And align fields for better readability.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 include/net/tso.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/net/tso.h b/include/net/tso.h
index c33dd00c161f7a6aa65f586b0ceede46af2e8730..d9b0a14b2a57b388ae4518fc63497ffd600b8887 100644
--- a/include/net/tso.h
+++ b/include/net/tso.h
@@ -7,12 +7,12 @@
 #define TSO_HEADER_SIZE		256
 
 struct tso_t {
-	int next_frag_idx;
-	void *data;
-	size_t size;
-	u16 ip_id;
-	bool ipv6;
-	u32 tcp_seq;
+	int	next_frag_idx;
+	int	size;
+	void	*data;
+	u16	ip_id;
+	bool	ipv6;
+	u32	tcp_seq;
 };
 
 int tso_count_descs(struct sk_buff *skb);
-- 
2.27.0.290.gba653c62da-goog

[PATCH net-next 3/5] net: tso: constify tso_count_descs() and friends

[Eric Dumazet]

skb argument of tso_count_descs(), tso_build_hdr() and tso_build_data() can be const.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 include/net/tso.h | 6 +++---
 net/core/tso.c    | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/net/tso.h b/include/net/tso.h
index d9b0a14b2a57b388ae4518fc63497ffd600b8887..32d9272ade6af0e3dd1272ecafa948f1535ea61f 100644
--- a/include/net/tso.h
+++ b/include/net/tso.h
@@ -15,10 +15,10 @@ struct tso_t {
 	u32	tcp_seq;
 };
 
-int tso_count_descs(struct sk_buff *skb);
-void tso_build_hdr(struct sk_buff *skb, char *hdr, struct tso_t *tso,
+int tso_count_descs(const struct sk_buff *skb);
+void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 		   int size, bool is_last);
-void tso_build_data(struct sk_buff *skb, struct tso_t *tso, int size);
+void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size);
 void tso_start(struct sk_buff *skb, struct tso_t *tso);
 
 #endif	/* _TSO_H */
diff --git a/net/core/tso.c b/net/core/tso.c
index d4d5c077ad7293aa71c3a64f67629e1079060227..56487e3bb26dc01b65f73f96fd0157bec73ea0d0 100644
--- a/net/core/tso.c
+++ b/net/core/tso.c
@@ -6,14 +6,14 @@
 #include <asm/unaligned.h>
 
 /* Calculate expected number of TX descriptors */
-int tso_count_descs(struct sk_buff *skb)
+int tso_count_descs(const struct sk_buff *skb)
 {
 	/* The Marvell Way */
 	return skb_shinfo(skb)->gso_segs * 2 + skb_shinfo(skb)->nr_frags;
 }
 EXPORT_SYMBOL(tso_count_descs);
 
-void tso_build_hdr(struct sk_buff *skb, char *hdr, struct tso_t *tso,
+void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 		   int size, bool is_last)
 {
 	struct tcphdr *tcph;
@@ -44,7 +44,7 @@ void tso_build_hdr(struct sk_buff *skb, char *hdr, struct tso_t *tso,
 }
 EXPORT_SYMBOL(tso_build_hdr);
 
-void tso_build_data(struct sk_buff *skb, struct tso_t *tso, int size)
+void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size)
 {
 	tso->tcp_seq += size;
 	tso->size -= size;
-- 
2.27.0.290.gba653c62da-goog

[PATCH net-next 4/5] net: tso: cache transport header length

[Eric Dumazet]

Add tlen field into struct tso_t, and change tso_start()
to return skb_transport_offset(skb) + tso->tlen

This removes from callers the need to use tcp_hdrlen(skb) and
will ease UDP segmentation offload addition.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 drivers/net/ethernet/cavium/thunder/nicvf_queues.c    |  5 +++--
 drivers/net/ethernet/freescale/fec_main.c             |  5 ++---
 drivers/net/ethernet/marvell/mv643xx_eth.c            |  5 ++---
 drivers/net/ethernet/marvell/mvneta.c                 |  5 ++---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c       |  6 +++---
 .../net/ethernet/marvell/octeontx2/nic/otx2_txrx.c    |  6 +++---
 include/net/tso.h                                     |  3 ++-
 net/core/tso.c                                        | 11 +++++++----
 8 files changed, 24 insertions(+), 22 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index 069e7413f1ef57ef401adfa6c7efdaad8005aba0..a45223f0cca5813324dd6543095f5375ffe5f27e 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -1489,9 +1489,10 @@ static int nicvf_sq_append_tso(struct nicvf *nic, struct snd_queue *sq,
 	int seg_subdescs = 0, desc_cnt = 0;
 	int seg_len, total_len, data_left;
 	int hdr_qentry = qentry;
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+	int hdr_len;
+
+	hdr_len = tso_start(skb, &tso);
 
-	tso_start(skb, &tso);
 	total_len = skb->len - hdr_len;
 	while (total_len > 0) {
 		char *hdr;
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index 2d0d313ee7c5a193a805f858b9fcd83f98a4ebea..9f80a33c5b16b07b30aa8ebfbc4dc8338a6ae056 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -710,8 +710,7 @@ static int fec_enet_txq_submit_tso(struct fec_enet_priv_tx_q *txq,
 				   struct net_device *ndev)
 {
 	struct fec_enet_private *fep = netdev_priv(ndev);
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
-	int total_len, data_left;
+	int hdr_len, total_len, data_left;
 	struct bufdesc *bdp = txq->bd.cur;
 	struct tso_t tso;
 	unsigned int index = 0;
@@ -731,7 +730,7 @@ static int fec_enet_txq_submit_tso(struct fec_enet_priv_tx_q *txq,
 	}
 
 	/* Initialize the TSO handler, and prepare the first payload */
-	tso_start(skb, &tso);
+	hdr_len = tso_start(skb, &tso);
 
 	total_len = skb->len - hdr_len;
 	while (total_len > 0) {
diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c b/drivers/net/ethernet/marvell/mv643xx_eth.c
index 4d4b6243318a59ac01190446766d2864f016564e..90e6111ce534dba7f16de983549ade748890124c 100644
--- a/drivers/net/ethernet/marvell/mv643xx_eth.c
+++ b/drivers/net/ethernet/marvell/mv643xx_eth.c
@@ -816,10 +816,9 @@ static int txq_submit_tso(struct tx_queue *txq, struct sk_buff *skb,
 			  struct net_device *dev)
 {
 	struct mv643xx_eth_private *mp = txq_to_mp(txq);
-	int total_len, data_left, ret;
+	int hdr_len, total_len, data_left, ret;
 	int desc_count = 0;
 	struct tso_t tso;
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
 	struct tx_desc *first_tx_desc;
 	u32 first_cmd_sts = 0;
 
@@ -832,7 +831,7 @@ static int txq_submit_tso(struct tx_queue *txq, struct sk_buff *skb,
 	first_tx_desc = &txq->tx_desc_area[txq->tx_curr_desc];
 
 	/* Initialize the TSO handler, and prepare the first payload */
-	tso_start(skb, &tso);
+	hdr_len = tso_start(skb, &tso);
 
 	total_len = skb->len - hdr_len;
 	while (total_len > 0) {
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 946925bbcb2de93629f45c3b9eecbfaf7338e8d1..95b447c14411479a026cf5a0375e79b345152d7c 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -2604,11 +2604,10 @@ mvneta_tso_put_data(struct net_device *dev, struct mvneta_tx_queue *txq,
 static int mvneta_tx_tso(struct sk_buff *skb, struct net_device *dev,
 			 struct mvneta_tx_queue *txq)
 {
-	int total_len, data_left;
+	int hdr_len, total_len, data_left;
 	int desc_count = 0;
 	struct mvneta_port *pp = netdev_priv(dev);
 	struct tso_t tso;
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
 	int i;
 
 	/* Count needed descriptors */
@@ -2621,7 +2620,7 @@ static int mvneta_tx_tso(struct sk_buff *skb, struct net_device *dev,
 	}
 
 	/* Initialize the TSO handler, and prepare the first payload */
-	tso_start(skb, &tso);
+	hdr_len = tso_start(skb, &tso);
 
 	total_len = skb->len - hdr_len;
 	while (total_len > 0) {
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 24f4d8e0da989daaca10333ec9c5a3a22ec66c48..e9f28756802628cc0a77dd80451a36ffe4b7f19c 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -3160,9 +3160,8 @@ static int mvpp2_tx_tso(struct sk_buff *skb, struct net_device *dev,
 			struct mvpp2_txq_pcpu *txq_pcpu)
 {
 	struct mvpp2_port *port = netdev_priv(dev);
+	int hdr_sz, i, len, descs = 0;
 	struct tso_t tso;
-	int hdr_sz = skb_transport_offset(skb) + tcp_hdrlen(skb);
-	int i, len, descs = 0;
 
 	/* Check number of available descriptors */
 	if (mvpp2_aggr_desc_num_check(port, aggr_txq, tso_count_descs(skb)) ||
@@ -3170,7 +3169,8 @@ static int mvpp2_tx_tso(struct sk_buff *skb, struct net_device *dev,
 					     tso_count_descs(skb)))
 		return 0;
 
-	tso_start(skb, &tso);
+	hdr_sz = tso_start(skb, &tso);
+
 	len = skb->len - hdr_sz;
 	while (len > 0) {
 		int left = min_t(int, skb_shinfo(skb)->gso_size, len);
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c
index b04f5429d72d911da145ee13b99506d84a9d7925..e54df6ed6daaa8ba90798904942e0155a90ea889 100644
--- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c
@@ -619,8 +619,7 @@ static void otx2_sq_append_tso(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
 			       struct sk_buff *skb, u16 qidx)
 {
 	struct netdev_queue *txq = netdev_get_tx_queue(pfvf->netdev, qidx);
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
-	int tcp_data, seg_len, pkt_len, offset;
+	int hdr_len, tcp_data, seg_len, pkt_len, offset;
 	struct nix_sqe_hdr_s *sqe_hdr;
 	int first_sqe = sq->head;
 	struct sg_list list;
@@ -636,7 +635,8 @@ static void otx2_sq_append_tso(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
 
 	netdev_tx_sent_queue(txq, skb->len);
 
-	tso_start(skb, &tso);
+	hdr_len = tso_start(skb, &tso);
+
 	tcp_data = skb->len - hdr_len;
 	while (tcp_data > 0) {
 		char *hdr;
diff --git a/include/net/tso.h b/include/net/tso.h
index 32d9272ade6af0e3dd1272ecafa948f1535ea61f..62c98a9c60f15d7b4869f1ab523dfeb83fb18ba6 100644
--- a/include/net/tso.h
+++ b/include/net/tso.h
@@ -11,6 +11,7 @@ struct tso_t {
 	int	size;
 	void	*data;
 	u16	ip_id;
+	u8	tlen; /* transport header len */
 	bool	ipv6;
 	u32	tcp_seq;
 };
@@ -19,6 +20,6 @@ int tso_count_descs(const struct sk_buff *skb);
 void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 		   int size, bool is_last);
 void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size);
-void tso_start(struct sk_buff *skb, struct tso_t *tso);
+int tso_start(struct sk_buff *skb, struct tso_t *tso);
 
 #endif	/* _TSO_H */
diff --git a/net/core/tso.c b/net/core/tso.c
index 56487e3bb26dc01b65f73f96fd0157bec73ea0d0..9f35518815bda275106d27bc5cc34b019429d254 100644
--- a/net/core/tso.c
+++ b/net/core/tso.c
@@ -17,7 +17,7 @@ void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 		   int size, bool is_last)
 {
 	struct tcphdr *tcph;
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+	int hdr_len = skb_transport_offset(skb) + tso->tlen;
 	int mac_hdr_len = skb_network_offset(skb);
 
 	memcpy(hdr, skb->data, hdr_len);
@@ -30,7 +30,7 @@ void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 	} else {
 		struct ipv6hdr *iph = (void *)(hdr + mac_hdr_len);
 
-		iph->payload_len = htons(size + tcp_hdrlen(skb));
+		iph->payload_len = htons(size + tso->tlen);
 	}
 	tcph = (struct tcphdr *)(hdr + skb_transport_offset(skb));
 	put_unaligned_be32(tso->tcp_seq, &tcph->seq);
@@ -62,10 +62,12 @@ void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size)
 }
 EXPORT_SYMBOL(tso_build_data);
 
-void tso_start(struct sk_buff *skb, struct tso_t *tso)
+int tso_start(struct sk_buff *skb, struct tso_t *tso)
 {
-	int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+	int tlen = tcp_hdrlen(skb);
+	int hdr_len = skb_transport_offset(skb) + tlen;
 
+	tso->tlen = tlen;
 	tso->ip_id = ntohs(ip_hdr(skb)->id);
 	tso->tcp_seq = ntohl(tcp_hdr(skb)->seq);
 	tso->next_frag_idx = 0;
@@ -83,5 +85,6 @@ void tso_start(struct sk_buff *skb, struct tso_t *tso)
 		tso->data = skb_frag_address(frag);
 		tso->next_frag_idx++;
 	}
+	return hdr_len;
 }
 EXPORT_SYMBOL(tso_start);
-- 
2.27.0.290.gba653c62da-goog

Re: [PATCH net-next 4/5] net: tso: cache transport header length

[Jakub Kicinski]

On Wed, 17 Jun 2020 11:48:18 -0700 Eric Dumazet wrote:
> Add tlen field into struct tso_t, and change tso_start()
> to return skb_transport_offset(skb) + tso->tlen
> 
> This removes from callers the need to use tcp_hdrlen(skb) and
> will ease UDP segmentation offload addition.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>

drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c: In function otx2_sq_append_tso:
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c:631:6: warning: hdr_len is used uninitialized in this function [-Wuninitialized]
 631 |  if (otx2_dma_map_tso_skb(pfvf, sq, skb, first_sqe, hdr_len)) {
     |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[PATCH net-next 5/5] net: tso: add UDP segmentation support

[Eric Dumazet]

Note that like TCP, we do not support additional encapsulations,
and that checksums must be offloaded to the NIC.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/core/tso.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/net/core/tso.c b/net/core/tso.c
index 9f35518815bda275106d27bc5cc34b019429d254..4148f6d48953e1e1ebd878c3953f3e41d47832d9 100644
--- a/net/core/tso.c
+++ b/net/core/tso.c
@@ -16,7 +16,6 @@ EXPORT_SYMBOL(tso_count_descs);
 void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 		   int size, bool is_last)
 {
-	struct tcphdr *tcph;
 	int hdr_len = skb_transport_offset(skb) + tso->tlen;
 	int mac_hdr_len = skb_network_offset(skb);
 
@@ -32,21 +31,29 @@ void tso_build_hdr(const struct sk_buff *skb, char *hdr, struct tso_t *tso,
 
 		iph->payload_len = htons(size + tso->tlen);
 	}
-	tcph = (struct tcphdr *)(hdr + skb_transport_offset(skb));
-	put_unaligned_be32(tso->tcp_seq, &tcph->seq);
+	hdr += skb_transport_offset(skb);
+	if (tso->tlen != sizeof(struct udphdr)) {
+		struct tcphdr *tcph = (struct tcphdr *)hdr;
 
-	if (!is_last) {
-		/* Clear all special flags for not last packet */
-		tcph->psh = 0;
-		tcph->fin = 0;
-		tcph->rst = 0;
+		put_unaligned_be32(tso->tcp_seq, &tcph->seq);
+
+		if (!is_last) {
+			/* Clear all special flags for not last packet */
+			tcph->psh = 0;
+			tcph->fin = 0;
+			tcph->rst = 0;
+		}
+	} else {
+		struct udphdr *uh = (struct udphdr *)hdr;
+
+		uh->len = htons(sizeof(*uh) + size);
 	}
 }
 EXPORT_SYMBOL(tso_build_hdr);
 
 void tso_build_data(const struct sk_buff *skb, struct tso_t *tso, int size)
 {
-	tso->tcp_seq += size;
+	tso->tcp_seq += size; /* not worth avoiding this operation for UDP */
 	tso->size -= size;
 	tso->data += size;
 
@@ -64,12 +71,12 @@ EXPORT_SYMBOL(tso_build_data);
 
 int tso_start(struct sk_buff *skb, struct tso_t *tso)
 {
-	int tlen = tcp_hdrlen(skb);
+	int tlen = skb_is_gso_tcp(skb) ? tcp_hdrlen(skb) : sizeof(struct udphdr);
 	int hdr_len = skb_transport_offset(skb) + tlen;
 
 	tso->tlen = tlen;
 	tso->ip_id = ntohs(ip_hdr(skb)->id);
-	tso->tcp_seq = ntohl(tcp_hdr(skb)->seq);
+	tso->tcp_seq = (tlen != sizeof(struct udphdr)) ? ntohl(tcp_hdr(skb)->seq) : 0;
 	tso->next_frag_idx = 0;
 	tso->ipv6 = vlan_get_protocol(skb) == htons(ETH_P_IPV6);
 
-- 
2.27.0.290.gba653c62da-goog