[PATCH ipsec v3 0/4] Support IPsec crypto offload for IPv6 ESP and IPv4 UDP-encapsulated ESP data paths

[PATCH ipsec v3 0/4] Support IPsec crypto offload for IPv6 ESP and IPv4 UDP-encapsulated ESP data paths

[Mike Yu]

Currently, IPsec crypto offload is enabled for GRO code path. However, there
are other code paths where the XFRM stack is involved; for example, IPv6 ESP
packets handled by xfrm6_esp_rcv() in ESP layer, and IPv4 UDP-encapsulated
ESP packets handled by udp_rcv() in UDP layer.

This patchset extends the crypto offload support to cover these two cases.
This is useful for devices with traffic accounting (e.g., Android), where GRO
can lead to inaccurate accounting on the underlying network. For example, VPN
traffic might not be counted on the wifi network interface wlan0 if the packets
are handled in GRO code path before entering the network stack for accounting.

Below is the RX data path scenario the crypto offload can be applied to.

  +-----------+   +-------+
  | HW Driver |-->| wlan0 |--------+
  +-----------+   +-------+        |
                                   v
                             +---------------+   +------+
                     +------>| Network Stack |-->| Apps |
                     |       +---------------+   +------+
                     |             |
                     |             v
                 +--------+   +------------+
                 | ipsec1 |<--| XFRM Stack |
                 +--------+   +------------+

v2 -> v3:
- Correct ESP seq in esp_xmit().
v1 -> v2:
- Fix comment style.

Mike Yu (4):
  xfrm: Support crypto offload for inbound IPv6 ESP packets not in GRO
    path
  xfrm: Allow UDP encapsulation in crypto offload control path
  xfrm: Support crypto offload for inbound IPv4 UDP-encapsulated ESP
    packet
  xfrm: Support crypto offload for outbound IPv4 UDP-encapsulated ESP
    packet

 net/ipv4/esp4.c         |  8 +++++++-
 net/ipv4/esp4_offload.c | 17 ++++++++++++++++-
 net/xfrm/xfrm_device.c  |  6 +++---
 net/xfrm/xfrm_input.c   |  3 ++-
 net/xfrm/xfrm_policy.c  |  5 ++++-
 5 files changed, 32 insertions(+), 7 deletions(-)

-- 
2.45.2.803.g4e1b14247a-goog

[PATCH ipsec v3 1/4] xfrm: Support crypto offload for inbound IPv6 ESP packets not in GRO path

[Mike Yu]

IPsec crypt offload supports outbound IPv6 ESP packets, but it doesn't
support inbound IPv6 ESP packets.

This change enables the crypto offload for inbound IPv6 ESP packets
that are not handled through GRO code path. If HW drivers add the
offload information to the skb, the packet will be handled in the
crypto offload rx code path.

Apart from the change in crypto offload rx code path, the change
in xfrm_policy_check is also needed.

Exampe of RX data path:

  +-----------+   +-------+
  | HW Driver |-->| wlan0 |--------+
  +-----------+   +-------+        |
                                   v
                             +---------------+   +------+
                     +------>| Network Stack |-->| Apps |
                     |       +---------------+   +------+
                     |             |
                     |             v
                 +--------+   +------------+
                 | ipsec1 |<--| XFRM Stack |
                 +--------+   +------------+

Test: Enabled both in/out IPsec crypto offload, and verified IPv6
      ESP packets on Android device on both wifi/cellular network
Signed-off-by: Mike Yu <yumike@google.com>
---
 net/xfrm/xfrm_input.c  | 2 +-
 net/xfrm/xfrm_policy.c | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c
index d2ea18dcb0cb..ba8deb0235ba 100644
--- a/net/xfrm/xfrm_input.c
+++ b/net/xfrm/xfrm_input.c
@@ -471,7 +471,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
 	struct xfrm_offload *xo = xfrm_offload(skb);
 	struct sec_path *sp;
 
-	if (encap_type < 0 || (xo && xo->flags & XFRM_GRO)) {
+	if (encap_type < 0 || (xo && (xo->flags & XFRM_GRO || encap_type == 0))) {
 		x = xfrm_input_state(skb);
 
 		if (unlikely(x->dir && x->dir != XFRM_SA_DIR_IN)) {
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 6603d3bd171f..2a9a31f2a9c1 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -3718,12 +3718,15 @@ int __xfrm_policy_check(struct sock *sk, int dir, struct sk_buff *skb,
 		pol = xfrm_in_fwd_icmp(skb, &fl, family, if_id);
 
 	if (!pol) {
+		const bool is_crypto_offload = sp &&
+			(xfrm_input_state(skb)->xso.type == XFRM_DEV_OFFLOAD_CRYPTO);
+
 		if (net->xfrm.policy_default[dir] == XFRM_USERPOLICY_BLOCK) {
 			XFRM_INC_STATS(net, LINUX_MIB_XFRMINNOPOLS);
 			return 0;
 		}
 
-		if (sp && secpath_has_nontransport(sp, 0, &xerr_idx)) {
+		if (sp && secpath_has_nontransport(sp, 0, &xerr_idx) && !is_crypto_offload) {
 			xfrm_secpath_reject(xerr_idx, skb, &fl);
 			XFRM_INC_STATS(net, LINUX_MIB_XFRMINNOPOLS);
 			return 0;
-- 
2.45.2.803.g4e1b14247a-goog

[PATCH ipsec v3 2/4] xfrm: Allow UDP encapsulation in crypto offload control path

[Mike Yu]

Unblock this limitation so that SAs with encapsulation specified
can be passed to HW drivers. HW drivers can still reject the SA
in their implementation of xdo_dev_state_add if the encapsulation
is not supported.

Test: Verified on Android device
Signed-off-by: Mike Yu <yumike@google.com>
---
 net/xfrm/xfrm_device.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c
index 2455a76a1cff..9a44d363ba62 100644
--- a/net/xfrm/xfrm_device.c
+++ b/net/xfrm/xfrm_device.c
@@ -261,9 +261,9 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x,
 
 	is_packet_offload = xuo->flags & XFRM_OFFLOAD_PACKET;
 
-	/* We don't yet support UDP encapsulation and TFC padding. */
-	if ((!is_packet_offload && x->encap) || x->tfcpad) {
-		NL_SET_ERR_MSG(extack, "Encapsulation and TFC padding can't be offloaded");
+	/* We don't yet support TFC padding. */
+	if (x->tfcpad) {
+		NL_SET_ERR_MSG(extack, "TFC padding can't be offloaded");
 		return -EINVAL;
 	}
 
-- 
2.45.2.803.g4e1b14247a-goog

[PATCH ipsec v3 3/4] xfrm: Support crypto offload for inbound IPv4 UDP-encapsulated ESP packet

[Mike Yu]

If xfrm_input() is called with UDP_ENCAP_ESPINUDP, the packet is
already processed in UDP layer that removes the UDP header.
Therefore, there should be no much difference to treat it as an
ESP packet in the XFRM stack.

Test: Enabled dir=in IPsec crypto offload, and verified IPv4
      UDP-encapsulated ESP packets on both wifi/cellular network
Signed-off-by: Mike Yu <yumike@google.com>
---
 net/xfrm/xfrm_input.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c
index ba8deb0235ba..7cee9c0a2cdc 100644
--- a/net/xfrm/xfrm_input.c
+++ b/net/xfrm/xfrm_input.c
@@ -471,7 +471,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
 	struct xfrm_offload *xo = xfrm_offload(skb);
 	struct sec_path *sp;
 
-	if (encap_type < 0 || (xo && (xo->flags & XFRM_GRO || encap_type == 0))) {
+	if (encap_type < 0 || (xo && (xo->flags & XFRM_GRO || encap_type == 0 ||
+				      encap_type == UDP_ENCAP_ESPINUDP))) {
 		x = xfrm_input_state(skb);
 
 		if (unlikely(x->dir && x->dir != XFRM_SA_DIR_IN)) {
-- 
2.45.2.803.g4e1b14247a-goog

[PATCH ipsec v3 4/4] xfrm: Support crypto offload for outbound IPv4 UDP-encapsulated ESP packet

[Mike Yu]

esp_xmit() is already able to handle UDP encapsulation through the call to
esp_output_head(). However, the ESP header and the outer IP header
are not correct and need to be corrected.

Test: Enabled both dir=in/out IPsec crypto offload, and verified IPv4
      UDP-encapsulated ESP packets on both wifi/cellular network
Signed-off-by: Mike Yu <yumike@google.com>
---
v2->v3: https://lore.kernel.org/all/20240709062326.939083-5-yumike@google.com
- Correct ESP seq in esp_xmit().
v1->v2: https://lore.kernel.org/all/20240702084452.2259237-5-yumike@google.com
- Fix comment style.
---
 net/ipv4/esp4.c         |  8 +++++++-
 net/ipv4/esp4_offload.c | 17 ++++++++++++++++-
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 3968d3f98e08..73981595f062 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -349,6 +349,7 @@ static struct ip_esp_hdr *esp_output_udp_encap(struct sk_buff *skb,
 {
 	struct udphdr *uh;
 	unsigned int len;
+	struct xfrm_offload *xo = xfrm_offload(skb);
 
 	len = skb->len + esp->tailen - skb_transport_offset(skb);
 	if (len + sizeof(struct iphdr) > IP_MAX_MTU)
@@ -360,7 +361,12 @@ static struct ip_esp_hdr *esp_output_udp_encap(struct sk_buff *skb,
 	uh->len = htons(len);
 	uh->check = 0;
 
-	*skb_mac_header(skb) = IPPROTO_UDP;
+	/* For IPv4 ESP with UDP encapsulation, if xo is not null, the skb is in the crypto offload
+	 * data path, which means that esp_output_udp_encap is called outside of the XFRM stack.
+	 * In this case, the mac header doesn't point to the IPv4 protocol field, so don't set it.
+	 */
+	if (!xo || encap_type != UDP_ENCAP_ESPINUDP)
+		*skb_mac_header(skb) = IPPROTO_UDP;
 
 	return (struct ip_esp_hdr *)(uh + 1);
 }
diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c
index b3271957ad9a..a37d18858c72 100644
--- a/net/ipv4/esp4_offload.c
+++ b/net/ipv4/esp4_offload.c
@@ -264,6 +264,7 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb,  netdev_features_
 	struct esp_info esp;
 	bool hw_offload = true;
 	__u32 seq;
+	int encap_type = 0;
 
 	esp.inplace = true;
 
@@ -296,8 +297,10 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb,  netdev_features_
 
 	esp.esph = ip_esp_hdr(skb);
 
+	if (x->encap)
+		encap_type = x->encap->encap_type;
 
-	if (!hw_offload || !skb_is_gso(skb)) {
+	if (!hw_offload || !skb_is_gso(skb) || (hw_offload && encap_type == UDP_ENCAP_ESPINUDP)) {
 		esp.nfrags = esp_output_head(x, skb, &esp);
 		if (esp.nfrags < 0)
 			return esp.nfrags;
@@ -324,6 +327,18 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb,  netdev_features_
 
 	esp.seqno = cpu_to_be64(seq + ((u64)xo->seq.hi << 32));
 
+	if (hw_offload && encap_type == UDP_ENCAP_ESPINUDP) {
+		/* In the XFRM stack, the encapsulation protocol is set to iphdr->protocol by
+		 * setting *skb_mac_header(skb) (see esp_output_udp_encap()) where skb->mac_header
+		 * points to iphdr->protocol (see xfrm4_tunnel_encap_add()).
+		 * However, in esp_xmit(), skb->mac_header doesn't point to iphdr->protocol.
+		 * Therefore, the protocol field needs to be corrected.
+		 */
+		ip_hdr(skb)->protocol = IPPROTO_UDP;
+
+		esph->seq_no = htonl(seq);
+	}
+
 	ip_hdr(skb)->tot_len = htons(skb->len);
 	ip_send_check(ip_hdr(skb));
 
-- 
2.45.2.803.g4e1b14247a-goog

Re: [PATCH ipsec v3 0/4] Support IPsec crypto offload for IPv6 ESP and IPv4 UDP-encapsulated ESP data paths

[Leon Romanovsky]

On Wed, Jul 10, 2024 at 07:16:50PM +0800, Mike Yu wrote:
> Currently, IPsec crypto offload is enabled for GRO code path. However, there
> are other code paths where the XFRM stack is involved; for example, IPv6 ESP
> packets handled by xfrm6_esp_rcv() in ESP layer, and IPv4 UDP-encapsulated
> ESP packets handled by udp_rcv() in UDP layer.
> 
> This patchset extends the crypto offload support to cover these two cases.
> This is useful for devices with traffic accounting (e.g., Android), where GRO
> can lead to inaccurate accounting on the underlying network. For example, VPN
> traffic might not be counted on the wifi network interface wlan0 if the packets
> are handled in GRO code path before entering the network stack for accounting.
> 
> Below is the RX data path scenario the crypto offload can be applied to.
> 
>   +-----------+   +-------+
>   | HW Driver |-->| wlan0 |--------+
>   +-----------+   +-------+        |
>                                    v
>                              +---------------+   +------+
>                      +------>| Network Stack |-->| Apps |
>                      |       +---------------+   +------+
>                      |             |
>                      |             v
>                  +--------+   +------------+
>                  | ipsec1 |<--| XFRM Stack |
>                  +--------+   +------------+
> 
> v2 -> v3:
> - Correct ESP seq in esp_xmit().
> v1 -> v2:
> - Fix comment style.
> 
> Mike Yu (4):
>   xfrm: Support crypto offload for inbound IPv6 ESP packets not in GRO
>     path
>   xfrm: Allow UDP encapsulation in crypto offload control path
>   xfrm: Support crypto offload for inbound IPv4 UDP-encapsulated ESP
>     packet
>   xfrm: Support crypto offload for outbound IPv4 UDP-encapsulated ESP
>     packet
> 
>  net/ipv4/esp4.c         |  8 +++++++-
>  net/ipv4/esp4_offload.c | 17 ++++++++++++++++-
>  net/xfrm/xfrm_device.c  |  6 +++---
>  net/xfrm/xfrm_input.c   |  3 ++-
>  net/xfrm/xfrm_policy.c  |  5 ++++-
>  5 files changed, 32 insertions(+), 7 deletions(-)

Steffen,

If it helps, we tested v2 version and it didn't break anything for us :).
But we didn't test this specific functionality.

Thanks

> 
> -- 
> 2.45.2.803.g4e1b14247a-goog
> 
> 

Re: [PATCH ipsec v3 0/4] Support IPsec crypto offload for IPv6 ESP and IPv4 UDP-encapsulated ESP data paths

[Steffen Klassert]

On Thu, Jul 11, 2024 at 12:52:08PM +0300, Leon Romanovsky wrote:
> On Wed, Jul 10, 2024 at 07:16:50PM +0800, Mike Yu wrote:
> > 
> > Mike Yu (4):
> >   xfrm: Support crypto offload for inbound IPv6 ESP packets not in GRO
> >     path
> >   xfrm: Allow UDP encapsulation in crypto offload control path
> >   xfrm: Support crypto offload for inbound IPv4 UDP-encapsulated ESP
> >     packet
> >   xfrm: Support crypto offload for outbound IPv4 UDP-encapsulated ESP
> >     packet
> > 
> >  net/ipv4/esp4.c         |  8 +++++++-
> >  net/ipv4/esp4_offload.c | 17 ++++++++++++++++-
> >  net/xfrm/xfrm_device.c  |  6 +++---
> >  net/xfrm/xfrm_input.c   |  3 ++-
> >  net/xfrm/xfrm_policy.c  |  5 ++++-
> >  5 files changed, 32 insertions(+), 7 deletions(-)
> 
> Steffen,
> 
> If it helps, we tested v2 version and it didn't break anything for us :).
> But we didn't test this specific functionality.
> 
> Thanks

Thanks for testing it Leon!

This is a new feature, so I don't want to apply it to the ipsec
tree. Mike, can you rebase on top of ipsec-next instead?

Thanks!

Re: [PATCH ipsec v3 0/4] Support IPsec crypto offload for IPv6 ESP and IPv4 UDP-encapsulated ESP data paths

[Mike Yu]

Sure. The patch series were already based on ipsec-next.
Sent v4 that targets ipsec-next tree.

Mike

Mike


On Thu, Jul 11, 2024 at 6:11 PM Steffen Klassert
<steffen.klassert@secunet.com> wrote:
>
> On Thu, Jul 11, 2024 at 12:52:08PM +0300, Leon Romanovsky wrote:
> > On Wed, Jul 10, 2024 at 07:16:50PM +0800, Mike Yu wrote:
> > >
> > > Mike Yu (4):
> > >   xfrm: Support crypto offload for inbound IPv6 ESP packets not in GRO
> > >     path
> > >   xfrm: Allow UDP encapsulation in crypto offload control path
> > >   xfrm: Support crypto offload for inbound IPv4 UDP-encapsulated ESP
> > >     packet
> > >   xfrm: Support crypto offload for outbound IPv4 UDP-encapsulated ESP
> > >     packet
> > >
> > >  net/ipv4/esp4.c         |  8 +++++++-
> > >  net/ipv4/esp4_offload.c | 17 ++++++++++++++++-
> > >  net/xfrm/xfrm_device.c  |  6 +++---
> > >  net/xfrm/xfrm_input.c   |  3 ++-
> > >  net/xfrm/xfrm_policy.c  |  5 ++++-
> > >  5 files changed, 32 insertions(+), 7 deletions(-)
> >
> > Steffen,
> >
> > If it helps, we tested v2 version and it didn't break anything for us :).
> > But we didn't test this specific functionality.
> >
> > Thanks
>
> Thanks for testing it Leon!
>
> This is a new feature, so I don't want to apply it to the ipsec
> tree. Mike, can you rebase on top of ipsec-next instead?
>
> Thanks!