* [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation
@ 2017-02-07 9:14 Steffen Klassert
2017-02-07 9:14 ` [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper Steffen Klassert
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw)
To: netdev
Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan,
Ilan Tayari
This patchset adds a software GRO codepath for IPsec ESP.
The ESP gro_receive callback functions decapsulate the
ESP packets at the GRO layer and reinject them back with
gro_cells_receive(). This saves a complete round through
the stack for IPsec ESP packets.
We also need this for ESP HW offload, because HW decrypt but
does not decapsulate the packet. We need to decapsulate before
the inbound policy check, otherwise this check will fail.
Patches 2 an 3 prepare the generic code for packet consuming
gro callbacks.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper. 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert @ 2017-02-07 9:14 ` Steffen Klassert 2017-02-08 18:18 ` David Miller 2017-02-07 9:14 ` [PATCH RFC ipsec-next 2/5] net: Add a skb_gro_flush_final helper Steffen Klassert ` (3 subsequent siblings) 4 siblings, 1 reply; 12+ messages in thread From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw) To: netdev Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan, Ilan Tayari Add a new helper to set the secpath to the skb. This avoids code duplication, as this is used in multiple places. Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- include/net/xfrm.h | 1 + net/ipv6/xfrm6_input.c | 15 +++------------ net/xfrm/xfrm_input.c | 35 +++++++++++++++++++++++------------ 3 files changed, 27 insertions(+), 24 deletions(-) diff --git a/include/net/xfrm.h b/include/net/xfrm.h index d9a81dc..25082e2 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -1009,6 +1009,7 @@ static inline int secpath_exists(struct sk_buff *skb) } struct sec_path *secpath_dup(struct sec_path *src); +int secpath_set(struct sk_buff *skb); static inline void secpath_reset(struct sk_buff *skb) diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c index b578956..662fb2c 100644 --- a/net/ipv6/xfrm6_input.c +++ b/net/ipv6/xfrm6_input.c @@ -69,18 +69,9 @@ int xfrm6_input_addr(struct sk_buff *skb, xfrm_address_t *daddr, struct xfrm_state *x = NULL; int i = 0; - /* Allocate new secpath or COW existing one. */ - if (!skb->sp || atomic_read(&skb->sp->refcnt) != 1) { - struct sec_path *sp; - - sp = secpath_dup(skb->sp); - if (!sp) { - XFRM_INC_STATS(net, LINUX_MIB_XFRMINERROR); - goto drop; - } - if (skb->sp) - secpath_put(skb->sp); - skb->sp = sp; + if (secpath_set(skb)) { + XFRM_INC_STATS(net, LINUX_MIB_XFRMINERROR); + goto drop; } if (1 + skb->sp->len == XFRM_MAX_DEPTH) { diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 3213fe8..58f5e44 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -126,6 +126,25 @@ struct sec_path *secpath_dup(struct sec_path *src) } EXPORT_SYMBOL(secpath_dup); +int secpath_set(struct sk_buff *skb) +{ + struct sec_path *sp; + + /* Allocate new secpath or COW existing one. */ + if (!skb->sp || atomic_read(&skb->sp->refcnt) != 1) { + + sp = secpath_dup(skb->sp); + if (!sp) { + return -ENOMEM; + } + if (skb->sp) + secpath_put(skb->sp); + skb->sp = sp; + } + return 0; +} +EXPORT_SYMBOL(secpath_set); + /* Fetch spi and seq from ipsec header */ int xfrm_parse_spi(struct sk_buff *skb, u8 nexthdr, __be32 *spi, __be32 *seq) @@ -221,18 +240,10 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) break; } - /* Allocate new secpath or COW existing one. */ - if (!skb->sp || atomic_read(&skb->sp->refcnt) != 1) { - struct sec_path *sp; - - sp = secpath_dup(skb->sp); - if (!sp) { - XFRM_INC_STATS(net, LINUX_MIB_XFRMINERROR); - goto drop; - } - if (skb->sp) - secpath_put(skb->sp); - skb->sp = sp; + err = secpath_set(skb); + if (err) { + XFRM_INC_STATS(net, LINUX_MIB_XFRMINERROR); + goto drop; } seq = 0; -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper. 2017-02-07 9:14 ` [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper Steffen Klassert @ 2017-02-08 18:18 ` David Miller 0 siblings, 0 replies; 12+ messages in thread From: David Miller @ 2017-02-08 18:18 UTC (permalink / raw) To: steffen.klassert; +Cc: netdev, eric.dumazet, sowmini.varadhan, ilant From: Steffen Klassert <steffen.klassert@secunet.com> Date: Tue, 7 Feb 2017 10:14:07 +0100 > +int secpath_set(struct sk_buff *skb) > +{ > + struct sec_path *sp; > + > + /* Allocate new secpath or COW existing one. */ > + if (!skb->sp || atomic_read(&skb->sp->refcnt) != 1) { > + > + sp = secpath_dup(skb->sp); Please remove this empty line. Thanks. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next 2/5] net: Add a skb_gro_flush_final helper. 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper Steffen Klassert @ 2017-02-07 9:14 ` Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 3/5] net: Prepare gro for packet consuming gro callbacks Steffen Klassert ` (2 subsequent siblings) 4 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw) To: netdev Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan, Ilan Tayari Add a skb_gro_flush_final helper to prepare for consuming skbs in call_gro_receive. We will extend this helper to not touch the skb if the skb is consumed by a gro callback with a followup patch. We need this to handle the upcomming IPsec ESP callbacks as they reinject the skb to the napi_gro_receive asynchronous. The handler is used in all gro_receive functions that can call the ESP gro handlers. Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- include/linux/netdevice.h | 5 +++++ net/ethernet/eth.c | 2 +- net/ipv4/af_inet.c | 2 +- net/ipv6/ip6_offload.c | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 58afbd1..f9da3ac 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2661,6 +2661,11 @@ static inline void skb_gro_remcsum_cleanup(struct sk_buff *skb, remcsum_unadjust((__sum16 *)ptr, grc->delta); } +static inline void skb_gro_flush_final(struct sk_buff *skb, struct sk_buff **pp, int flush) +{ + NAPI_GRO_CB(skb)->flush |= flush; +} + static inline int dev_hard_header(struct sk_buff *skb, struct net_device *dev, unsigned short type, const void *daddr, const void *saddr, diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c index efdaaab..c666ff0 100644 --- a/net/ethernet/eth.c +++ b/net/ethernet/eth.c @@ -474,7 +474,7 @@ struct sk_buff **eth_gro_receive(struct sk_buff **head, out_unlock: rcu_read_unlock(); out: - NAPI_GRO_CB(skb)->flush |= flush; + skb_gro_flush_final(skb, pp, flush); return pp; } diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 685ba53..602d40f 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1423,7 +1423,7 @@ struct sk_buff **inet_gro_receive(struct sk_buff **head, struct sk_buff *skb) rcu_read_unlock(); out: - NAPI_GRO_CB(skb)->flush |= flush; + skb_gro_flush_final(skb, pp, flush); return pp; } diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index fc7b401..0838e6d 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -253,7 +253,7 @@ static struct sk_buff **ipv6_gro_receive(struct sk_buff **head, rcu_read_unlock(); out: - NAPI_GRO_CB(skb)->flush |= flush; + skb_gro_flush_final(skb, pp, flush); return pp; } -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next 3/5] net: Prepare gro for packet consuming gro callbacks 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 2/5] net: Add a skb_gro_flush_final helper Steffen Klassert @ 2017-02-07 9:14 ` Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 4/5] xfrm: Export xfrm_parse_spi Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert 4 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw) To: netdev Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan, Ilan Tayari The upcomming IPsec ESP gro callbacks will consume the skb, so prepare for that. Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- include/linux/netdevice.h | 9 +++++++++ net/core/dev.c | 7 +++++++ net/xfrm/Kconfig | 4 ++++ 3 files changed, 20 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index f9da3ac..9e5d1cd 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -352,6 +352,7 @@ enum gro_result { GRO_HELD, GRO_NORMAL, GRO_DROP, + GRO_CONSUMED, }; typedef enum gro_result gro_result_t; @@ -2661,10 +2662,18 @@ static inline void skb_gro_remcsum_cleanup(struct sk_buff *skb, remcsum_unadjust((__sum16 *)ptr, grc->delta); } +#ifdef CONFIG_XFRM_OFFLOAD +static inline void skb_gro_flush_final(struct sk_buff *skb, struct sk_buff **pp, int flush) +{ + if (PTR_ERR(pp) != -EINPROGRESS) + NAPI_GRO_CB(skb)->flush |= flush; +} +#else static inline void skb_gro_flush_final(struct sk_buff *skb, struct sk_buff **pp, int flush) { NAPI_GRO_CB(skb)->flush |= flush; } +#endif static inline int dev_hard_header(struct sk_buff *skb, struct net_device *dev, unsigned short type, diff --git a/net/core/dev.c b/net/core/dev.c index 3e1a601..0f5e606 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4505,6 +4505,11 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff if (&ptype->list == head) goto normal; + if (PTR_ERR(pp) == -EINPROGRESS) { + ret = GRO_CONSUMED; + goto ok; + } + same_flow = NAPI_GRO_CB(skb)->same_flow; ret = NAPI_GRO_CB(skb)->free ? GRO_MERGED_FREE : GRO_MERGED; @@ -4609,6 +4614,7 @@ static gro_result_t napi_skb_finish(gro_result_t ret, struct sk_buff *skb) case GRO_HELD: case GRO_MERGED: + case GRO_CONSUMED: break; } @@ -4680,6 +4686,7 @@ static gro_result_t napi_frags_finish(struct napi_struct *napi, break; case GRO_MERGED: + case GRO_CONSUMED: break; } diff --git a/net/xfrm/Kconfig b/net/xfrm/Kconfig index bda1a13..a484451 100644 --- a/net/xfrm/Kconfig +++ b/net/xfrm/Kconfig @@ -5,6 +5,10 @@ config XFRM bool depends on NET +config XFRM_OFFLOAD + bool + depends on XFRM + config XFRM_ALGO tristate select XFRM -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next 4/5] xfrm: Export xfrm_parse_spi. 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert ` (2 preceding siblings ...) 2017-02-07 9:14 ` [PATCH RFC ipsec-next 3/5] net: Prepare gro for packet consuming gro callbacks Steffen Klassert @ 2017-02-07 9:14 ` Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert 4 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw) To: netdev Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan, Ilan Tayari We need it in the ESP offload handlers, so export it. Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- include/net/xfrm.h | 1 + net/xfrm/xfrm_input.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 25082e2..8acea1e 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -1522,6 +1522,7 @@ int xfrm4_rcv_encap(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type); int xfrm4_transport_finish(struct sk_buff *skb, int async); int xfrm4_rcv(struct sk_buff *skb); +int xfrm_parse_spi(struct sk_buff *skb, u8 nexthdr, __be32 *spi, __be32 *seq); static inline int xfrm4_rcv_spi(struct sk_buff *skb, int nexthdr, __be32 spi) { diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 58f5e44..538d338 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -180,6 +180,7 @@ int xfrm_parse_spi(struct sk_buff *skb, u8 nexthdr, __be32 *spi, __be32 *seq) *seq = *(__be32 *)(skb_transport_header(skb) + offset_seq); return 0; } +EXPORT_SYMBOL(xfrm_parse_spi); int xfrm_prepare_input(struct xfrm_state *x, struct sk_buff *skb) { -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert ` (3 preceding siblings ...) 2017-02-07 9:14 ` [PATCH RFC ipsec-next 4/5] xfrm: Export xfrm_parse_spi Steffen Klassert @ 2017-02-07 9:14 ` Steffen Klassert 2017-02-07 19:45 ` Eric Dumazet 2017-02-08 18:19 ` David Miller 4 siblings, 2 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-07 9:14 UTC (permalink / raw) To: netdev Cc: Steffen Klassert, David Miller, Eric Dumazet, Sowmini Varadhan, Ilan Tayari This patch adds GRO ifrastructure and callbacks for ESP on ipv4 and ipv6. In case the GRO layer detects an ESP packet, the esp{4,6}_gro_receive() function does a xfrm state lookup and calls the xfrm input layer if it finds a matching state. The packet will be decapsulated and reinjected it into layer 2. Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- include/net/xfrm.h | 18 +++++++- net/ipv4/Kconfig | 13 ++++++ net/ipv4/Makefile | 1 + net/ipv4/esp4_offload.c | 96 ++++++++++++++++++++++++++++++++++++++++ net/ipv4/xfrm4_input.c | 6 +++ net/ipv4/xfrm4_mode_transport.c | 3 +- net/ipv6/Kconfig | 13 ++++++ net/ipv6/Makefile | 1 + net/ipv6/esp6_offload.c | 98 +++++++++++++++++++++++++++++++++++++++++ net/ipv6/xfrm6_input.c | 5 +++ net/ipv6/xfrm6_mode_transport.c | 3 +- net/xfrm/xfrm_input.c | 32 +++++++++++--- 12 files changed, 280 insertions(+), 9 deletions(-) create mode 100644 net/ipv4/esp4_offload.c create mode 100644 net/ipv6/esp6_offload.c diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 8acea1e..8c94ef3 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -606,11 +606,25 @@ struct xfrm_mgr { int xfrm_register_km(struct xfrm_mgr *km); int xfrm_unregister_km(struct xfrm_mgr *km); -struct xfrm_tunnel_skb_cb { +/* + * This structure is used if we get the packet from the gro layer. + */ +struct xfrm_gro_skb_cb { union { struct inet_skb_parm h4; struct inet6_skb_parm h6; - } header; + + struct { + __be32 seq; + bool skb_is_gro; + } input; + } gro; +}; + +#define XFRM_GRO_SKB_CB(__skb) ((struct xfrm_gro_skb_cb *)&((__skb)->cb[0])) + +struct xfrm_tunnel_skb_cb { + struct xfrm_gro_skb_cb header; union { struct ip_tunnel *ip4; diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig index 6e7baaf..e0878c3 100644 --- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -360,6 +360,19 @@ config INET_ESP If unsure, say Y. +config INET_ESP_OFFLOAD + tristate "IP: ESP transformation offload" + depends on INET_ESP + select XFRM_OFFLOAD + default n + ---help--- + Support for ESP transformation offload. This makes sense + only if this system really does IPsec and want to do it + with high throughput. A typical desktop system does not + need it, even if it does IPsec. + + If unsure, say N. + config INET_IPCOMP tristate "IP: IPComp transformation" select INET_XFRM_TUNNEL diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile index 48af58a..c6d4238 100644 --- a/net/ipv4/Makefile +++ b/net/ipv4/Makefile @@ -29,6 +29,7 @@ obj-$(CONFIG_NET_IPVTI) += ip_vti.o obj-$(CONFIG_SYN_COOKIES) += syncookies.o obj-$(CONFIG_INET_AH) += ah4.o obj-$(CONFIG_INET_ESP) += esp4.o +obj-$(CONFIG_INET_ESP_OFFLOAD) += esp4_offload.o obj-$(CONFIG_INET_IPCOMP) += ipcomp.o obj-$(CONFIG_INET_XFRM_TUNNEL) += xfrm4_tunnel.o obj-$(CONFIG_INET_XFRM_MODE_BEET) += xfrm4_mode_beet.o diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c new file mode 100644 index 0000000..2f26085 --- /dev/null +++ b/net/ipv4/esp4_offload.c @@ -0,0 +1,96 @@ +/* + * IPV4 GSO/GRO offload support + * Linux INET implementation + * + * Copyright (C) 2016 secunet Security Networks AG + * Author: Steffen Klassert <steffen.klassert@secunet.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * ESP GRO support + */ + +#include <linux/skbuff.h> +#include <linux/init.h> +#include <net/protocol.h> +#include <crypto/aead.h> +#include <crypto/authenc.h> +#include <linux/err.h> +#include <linux/module.h> +#include <net/ip.h> +#include <net/xfrm.h> +#include <net/esp.h> +#include <linux/scatterlist.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <net/udp.h> + +static struct sk_buff **esp4_gro_receive(struct sk_buff **head, + struct sk_buff *skb) +{ + int err; + __be32 seq; + __be32 spi; + struct xfrm_state *x; + int offset = skb_gro_offset(skb); + + skb_pull(skb, offset); + + if ((err = xfrm_parse_spi(skb, IPPROTO_ESP, &spi, &seq)) != 0) + goto out; + + err = secpath_set(skb); + if (err) + goto out; + + if (skb->sp->len == XFRM_MAX_DEPTH) + goto out; + + x = xfrm_state_lookup(dev_net(skb->dev), skb->mark, + (xfrm_address_t *)&ip_hdr(skb)->daddr, + spi, IPPROTO_ESP, AF_INET); + if (!x) + goto out; + + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = NULL; + XFRM_SPI_SKB_CB(skb)->family = AF_INET; + XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr); + XFRM_GRO_SKB_CB(skb)->gro.input.seq = seq; + skb->sp->xvec[skb->sp->len++] = x; + + /* We don't need to handle errors from xfrm_input, it does all + * the error handling and frees the resources on error. */ + xfrm_input(skb, IPPROTO_ESP, spi, -2); + + return ERR_PTR(-EINPROGRESS); +out: + skb_push(skb, offset); + NAPI_GRO_CB(skb)->same_flow = 0; + NAPI_GRO_CB(skb)->flush = 1; + + return NULL; +} + +static const struct net_offload esp4_offload = { + .callbacks = { + .gro_receive = esp4_gro_receive, + }, +}; + +static int __init esp4_offload_init(void) +{ + return inet_add_offload(&esp4_offload, IPPROTO_ESP); +} + +static void __exit esp4_offload_exit(void) +{ + inet_del_offload(&esp4_offload, IPPROTO_ESP); +} + +module_init(esp4_offload_init); +module_exit(esp4_offload_exit); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Steffen Klassert <steffen.klassert@secunet.com>"); diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c index 62e1e72..7554c95 100644 --- a/net/ipv4/xfrm4_input.c +++ b/net/ipv4/xfrm4_input.c @@ -53,6 +53,12 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async) iph->tot_len = htons(skb->len); ip_send_check(iph); + + if (XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro) { + skb_mac_header_rebuild(skb); + return 0; + } + NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, dev_net(skb->dev), NULL, skb, skb->dev, NULL, xfrm4_rcv_encap_finish); diff --git a/net/ipv4/xfrm4_mode_transport.c b/net/ipv4/xfrm4_mode_transport.c index fd840c7..3827f83 100644 --- a/net/ipv4/xfrm4_mode_transport.c +++ b/net/ipv4/xfrm4_mode_transport.c @@ -50,7 +50,8 @@ static int xfrm4_transport_input(struct xfrm_state *x, struct sk_buff *skb) skb->network_header = skb->transport_header; } ip_hdr(skb)->tot_len = htons(skb->len + ihl); - skb_reset_transport_header(skb); + if (!(XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro)) + skb_reset_transport_header(skb); return 0; } diff --git a/net/ipv6/Kconfig b/net/ipv6/Kconfig index ec1267e..b2a85cc 100644 --- a/net/ipv6/Kconfig +++ b/net/ipv6/Kconfig @@ -75,6 +75,19 @@ config INET6_ESP If unsure, say Y. +config INET6_ESP_OFFLOAD + tristate "IPv6: ESP transformation offload" + depends on INET6_ESP + select XFRM_OFFLOAD + default n + ---help--- + Support for ESP transformation offload. This makes sense + only if this system really does IPsec and want to do it + with high throughput. A typical desktop system does not + need it, even if it does IPsec. + + If unsure, say N. + config INET6_IPCOMP tristate "IPv6: IPComp transformation" select INET6_XFRM_TUNNEL diff --git a/net/ipv6/Makefile b/net/ipv6/Makefile index a9e9fec..217e9ff 100644 --- a/net/ipv6/Makefile +++ b/net/ipv6/Makefile @@ -30,6 +30,7 @@ ipv6-objs += $(ipv6-y) obj-$(CONFIG_INET6_AH) += ah6.o obj-$(CONFIG_INET6_ESP) += esp6.o +obj-$(CONFIG_INET6_ESP_OFFLOAD) += esp6_offload.o obj-$(CONFIG_INET6_IPCOMP) += ipcomp6.o obj-$(CONFIG_INET6_XFRM_TUNNEL) += xfrm6_tunnel.o obj-$(CONFIG_INET6_TUNNEL) += tunnel6.o diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c new file mode 100644 index 0000000..f97d88b --- /dev/null +++ b/net/ipv6/esp6_offload.c @@ -0,0 +1,98 @@ +/* + * IPV6 GSO/GRO offload support + * Linux INET implementation + * + * Copyright (C) 2016 secunet Security Networks AG + * Author: Steffen Klassert <steffen.klassert@secunet.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * ESP GRO support + */ + +#include <linux/skbuff.h> +#include <linux/init.h> +#include <net/protocol.h> +#include <crypto/aead.h> +#include <crypto/authenc.h> +#include <linux/err.h> +#include <linux/module.h> +#include <net/ip.h> +#include <net/xfrm.h> +#include <net/esp.h> +#include <linux/scatterlist.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <net/ip6_route.h> +#include <net/ipv6.h> +#include <linux/icmpv6.h> + +static struct sk_buff **esp6_gro_receive(struct sk_buff **head, + struct sk_buff *skb) +{ + int err; + __be32 seq; + __be32 spi; + struct xfrm_state *x; + int offset = skb_gro_offset(skb); + + skb_pull(skb, offset); + + if ((err = xfrm_parse_spi(skb, IPPROTO_ESP, &spi, &seq)) != 0) + goto out; + + err = secpath_set(skb); + if (err) + goto out; + + if (skb->sp->len == XFRM_MAX_DEPTH) + goto out; + + x = xfrm_state_lookup(dev_net(skb->dev), skb->mark, + (xfrm_address_t *)&ipv6_hdr(skb)->daddr, + spi, IPPROTO_ESP, AF_INET6); + if (!x) + goto out; + + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6 = NULL; + XFRM_SPI_SKB_CB(skb)->family = AF_INET6; + XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct ipv6hdr, daddr); + XFRM_GRO_SKB_CB(skb)->gro.input.seq = seq; + skb->sp->xvec[skb->sp->len++] = x; + + /* We don't need to handle errors from xfrm_input, it does all + * the error handling and frees the resources on error. */ + xfrm_input(skb, IPPROTO_ESP, spi, -2); + + return ERR_PTR(-EINPROGRESS); +out: + skb_push(skb, offset); + NAPI_GRO_CB(skb)->same_flow = 0; + NAPI_GRO_CB(skb)->flush = 1; + + return NULL; +} + +static const struct net_offload esp6_offload = { + .callbacks = { + .gro_receive = esp6_gro_receive, + }, +}; + +static int __init esp6_offload_init(void) +{ + return inet6_add_offload(&esp6_offload, IPPROTO_ESP); +} + +static void __exit esp6_offload_exit(void) +{ + inet6_del_offload(&esp6_offload, IPPROTO_ESP); +} + +module_init(esp6_offload_init); +module_exit(esp6_offload_exit); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Steffen Klassert <steffen.klassert@secunet.com>"); diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c index 662fb2c..43373be 100644 --- a/net/ipv6/xfrm6_input.c +++ b/net/ipv6/xfrm6_input.c @@ -44,6 +44,11 @@ int xfrm6_transport_finish(struct sk_buff *skb, int async) ipv6_hdr(skb)->payload_len = htons(skb->len); __skb_push(skb, skb->data - skb_network_header(skb)); + if (XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro) { + skb_mac_header_rebuild(skb); + return -1; + } + NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING, dev_net(skb->dev), NULL, skb, skb->dev, NULL, ip6_rcv_finish); diff --git a/net/ipv6/xfrm6_mode_transport.c b/net/ipv6/xfrm6_mode_transport.c index 4e34410..11a6186 100644 --- a/net/ipv6/xfrm6_mode_transport.c +++ b/net/ipv6/xfrm6_mode_transport.c @@ -55,7 +55,8 @@ static int xfrm6_transport_input(struct xfrm_state *x, struct sk_buff *skb) } ipv6_hdr(skb)->payload_len = htons(skb->len + ihl - sizeof(struct ipv6hdr)); - skb_reset_transport_header(skb); + if (!(XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro)) + skb_reset_transport_header(skb); return 0; } diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 538d338..762547e 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -215,14 +215,24 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) unsigned int family; int decaps = 0; int async = 0; + bool xfrm_gro = false; - /* A negative encap_type indicates async resumption. */ if (encap_type < 0) { - async = 1; x = xfrm_input_state(skb); - seq = XFRM_SKB_CB(skb)->seq.input.low; family = x->outer_mode->afinfo->family; - goto resume; + + /* An encap_type of -1 indicates async resumption. */ + if (encap_type == -1) { + async = 1; + seq = XFRM_SKB_CB(skb)->seq.input.low; + xfrm_gro = XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro; + goto resume; + } + /* encap_type < -1 indicates a GRO call. */ + xfrm_gro = true; + encap_type = 0; + seq = XFRM_GRO_SKB_CB(skb)->gro.input.seq; + goto lock; } daddr = (xfrm_address_t *)(skb_network_header(skb) + @@ -268,6 +278,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) skb->sp->xvec[skb->sp->len++] = x; +lock: spin_lock(&x->lock); if (unlikely(x->km.state != XFRM_STATE_VALID)) { @@ -305,6 +316,10 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) XFRM_SKB_CB(skb)->seq.input.low = seq; XFRM_SKB_CB(skb)->seq.input.hi = seq_hi; + XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro = false; + + if (xfrm_gro) + XFRM_GRO_SKB_CB(skb)->gro.input.skb_is_gro = true; skb_dst_force(skb); dev_hold(skb->dev); @@ -389,7 +404,14 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) gro_cells_receive(&gro_cells, skb); return 0; } else { - return x->inner_mode->afinfo->transport_finish(skb, async); + err = x->inner_mode->afinfo->transport_finish(skb, async); + if (xfrm_gro) { + skb_dst_drop(skb); + gro_cells_receive(&gro_cells, skb); + return err; + } + + return err; } drop_unlock: -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-02-07 9:14 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert @ 2017-02-07 19:45 ` Eric Dumazet 2017-02-08 11:43 ` Steffen Klassert 2017-02-08 18:19 ` David Miller 1 sibling, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2017-02-07 19:45 UTC (permalink / raw) To: Steffen Klassert; +Cc: netdev, David Miller, Sowmini Varadhan, Ilan Tayari On Tue, 2017-02-07 at 10:14 +0100, Steffen Klassert wrote: > This patch adds GRO ifrastructure and callbacks for ESP on > ipv4 and ipv6. > I am a bit confused. > > -struct xfrm_tunnel_skb_cb { > +/* > + * This structure is used if we get the packet from the gro layer. > + */ > +struct xfrm_gro_skb_cb { > union { > struct inet_skb_parm h4; > struct inet6_skb_parm h6; > - } header; > + > + struct { > + __be32 seq; > + bool skb_is_gro; > + } input; > + } gro; > +}; > + > +#define XFRM_GRO_SKB_CB(__skb) ((struct xfrm_gro_skb_cb *)&((__skb)->cb[0])) > + Then : > + > + x = xfrm_state_lookup(dev_net(skb->dev), skb->mark, > + (xfrm_address_t *)&ip_hdr(skb)->daddr, > + spi, IPPROTO_ESP, AF_INET); > + if (!x) > + goto out; > + > + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = NULL; > + XFRM_SPI_SKB_CB(skb)->family = AF_INET; > + XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr); > + XFRM_GRO_SKB_CB(skb)->gro.input.seq = seq; > + skb->sp->xvec[skb->sp->len++] = x; > + > + /* We don't need to handle errors from xfrm_input, it does all > + * the error handling and frees the resources on error. */ > + xfrm_input(skb, IPPROTO_ESP, spi, -2); > + > + return ERR_PTR(-EINPROGRESS); > +out: > + skb_push(skb, offset); > + NAPI_GRO_CB(skb)->same_flow = 0; > + NAPI_GRO_CB(skb)->flush = 1; > + How can you mix XFRM_SPI_SKB_CB(), XFRM_GRO_SKB_CB() and NAPI_GRO_CB() at the same time on one skb ? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-02-07 19:45 ` Eric Dumazet @ 2017-02-08 11:43 ` Steffen Klassert 0 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-08 11:43 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, David Miller, Sowmini Varadhan, Ilan Tayari On Tue, Feb 07, 2017 at 11:45:06AM -0800, Eric Dumazet wrote: > On Tue, 2017-02-07 at 10:14 +0100, Steffen Klassert wrote: > > This patch adds GRO ifrastructure and callbacks for ESP on > > ipv4 and ipv6. > > > > > I am a bit confused. > > > > > -struct xfrm_tunnel_skb_cb { > > +/* > > + * This structure is used if we get the packet from the gro layer. > > + */ > > +struct xfrm_gro_skb_cb { > > union { > > struct inet_skb_parm h4; > > struct inet6_skb_parm h6; > > - } header; > > + > > + struct { > > + __be32 seq; > > + bool skb_is_gro; > > + } input; > > + } gro; > > +}; > > + > > +#define XFRM_GRO_SKB_CB(__skb) ((struct xfrm_gro_skb_cb *)&((__skb)->cb[0])) > > + > > Then : > > > + > > + x = xfrm_state_lookup(dev_net(skb->dev), skb->mark, > > + (xfrm_address_t *)&ip_hdr(skb)->daddr, > > + spi, IPPROTO_ESP, AF_INET); > > + if (!x) > > + goto out; > > + > > + XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = NULL; > > + XFRM_SPI_SKB_CB(skb)->family = AF_INET; > > + XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr); > > + XFRM_GRO_SKB_CB(skb)->gro.input.seq = seq; > > + skb->sp->xvec[skb->sp->len++] = x; > > + > > + /* We don't need to handle errors from xfrm_input, it does all > > + * the error handling and frees the resources on error. */ > > + xfrm_input(skb, IPPROTO_ESP, spi, -2); > > + > > + return ERR_PTR(-EINPROGRESS); > > +out: > > + skb_push(skb, offset); > > + NAPI_GRO_CB(skb)->same_flow = 0; > > + NAPI_GRO_CB(skb)->flush = 1; > > + > > > How can you mix XFRM_SPI_SKB_CB(), XFRM_GRO_SKB_CB() and NAPI_GRO_CB() > at the same time on one skb ? The fields of XFRM_SPI_SKB_CB start behind XFRM_GRO_SKB_CB, it is stacked. I hope not to mix NAPI_GRO_CB with XFRM_*_SKB_CB. If I don't find a xfrm_state, I use only NAPI_GRO_CB and return to the calling GRO handlers. If I find a xfrm_state, I use only the XFRM_*_SKB_CB and notify the calling GRO handlers that the packet is consumed. Patch 3 changed the potential callers to not touch the skb anymore in this case. Anyway, I think I can remove xfrm_gro_skb_cb, the sequence number fits on xfrm_spi_skb_cb, and the skb_is_gro flag can be added to the secpath. We have to extend the secpath anyway to carry HW offloading informations, so this flag could be there too. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-02-07 9:14 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert 2017-02-07 19:45 ` Eric Dumazet @ 2017-02-08 18:19 ` David Miller 2017-02-10 7:39 ` Steffen Klassert 1 sibling, 1 reply; 12+ messages in thread From: David Miller @ 2017-02-08 18:19 UTC (permalink / raw) To: steffen.klassert; +Cc: netdev, eric.dumazet, sowmini.varadhan, ilant From: Steffen Klassert <steffen.klassert@secunet.com> Date: Tue, 7 Feb 2017 10:14:11 +0100 > +static struct sk_buff **esp4_gro_receive(struct sk_buff **head, > + struct sk_buff *skb) > +{ > + int err; > + __be32 seq; > + __be32 spi; > + struct xfrm_state *x; > + int offset = skb_gro_offset(skb); Please order local variable declarations from longest to shortest. > +static struct sk_buff **esp6_gro_receive(struct sk_buff **head, > + struct sk_buff *skb) > +{ > + int err; > + __be32 seq; > + __be32 spi; > + struct xfrm_state *x; > + int offset = skb_gro_offset(skb); Likewise. Thank you. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-02-08 18:19 ` David Miller @ 2017-02-10 7:39 ` Steffen Klassert 0 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-02-10 7:39 UTC (permalink / raw) To: David Miller; +Cc: netdev, eric.dumazet, sowmini.varadhan, ilant On Wed, Feb 08, 2017 at 01:19:56PM -0500, David Miller wrote: > From: Steffen Klassert <steffen.klassert@secunet.com> > Date: Tue, 7 Feb 2017 10:14:11 +0100 > > > +static struct sk_buff **esp4_gro_receive(struct sk_buff **head, > > + struct sk_buff *skb) > > +{ > > + int err; > > + __be32 seq; > > + __be32 spi; > > + struct xfrm_state *x; > > + int offset = skb_gro_offset(skb); > > Please order local variable declarations from longest to shortest. No problem, will do. Thanks for looking over it! ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next] IPsec offload, part one @ 2017-01-04 8:23 Steffen Klassert 2017-01-04 8:23 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert 0 siblings, 1 reply; 12+ messages in thread From: Steffen Klassert @ 2017-01-04 8:23 UTC (permalink / raw) To: David Miller, netdev; +Cc: Steffen Klassert, Sowmini Varadhan, Ilan Tayari This is the first part of the IPsec offload work we talked at the IPsec workshop at the last netdev conference. I plan to apply this to ipsec-next after this round of review. Patch 1 and 2 try to avoid skb linearization in the ESP layer. Patch 3 introduces a hepler to seup the esp trailer. Patch 4 prepares the generic network code for IPsec GRO. The main reason why we need this, is that we need to reinject the decrypted inner packet back to the GRO layer. Patch 5 introduces GRO handlers for ESP, GRO can enabled with a IPsec offload config option. This config option will also be used for the upcomming hardware offload. David, patch 3 touches generic networking code. Is it ok to integrate such a generic preparation patch into an IPsec pull request, or do you prefer to get it as a separate patch? ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath 2017-01-04 8:23 [PATCH RFC ipsec-next] IPsec offload, part one Steffen Klassert @ 2017-01-04 8:23 ` Steffen Klassert 0 siblings, 0 replies; 12+ messages in thread From: Steffen Klassert @ 2017-01-04 8:23 UTC (permalink / raw) To: David Miller, netdev; +Cc: Steffen Klassert, Sowmini Varadhan, Ilan Tayari This patch adds GRO callbacks for ESP on ipv4 and ipv6. In case the GRO layer detects an ESP packet, the esp{4,6}_gro_receive() function calls the xfrm input layer which decapsulates the packet and reinject it into layer 2 by calling netif_rx(). Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- net/ipv4/Kconfig | 6 +++ net/ipv4/Makefile | 1 + net/ipv4/esp4.c | 5 +++ net/ipv4/esp4_offload.c | 94 +++++++++++++++++++++++++++++++++++++++ net/ipv4/ip_vti.c | 4 ++ net/ipv4/xfrm4_input.c | 6 +++ net/ipv4/xfrm4_mode_transport.c | 3 +- net/ipv6/Kconfig | 6 +++ net/ipv6/Makefile | 1 + net/ipv6/esp6.c | 5 +++ net/ipv6/esp6_offload.c | 98 +++++++++++++++++++++++++++++++++++++++++ net/ipv6/xfrm6_input.c | 5 +++ net/xfrm/xfrm_input.c | 27 +++++++++--- 13 files changed, 253 insertions(+), 8 deletions(-) create mode 100644 net/ipv4/esp4_offload.c create mode 100644 net/ipv6/esp6_offload.c diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig index 6e7baaf..27df99f 100644 --- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -360,6 +360,12 @@ config INET_ESP If unsure, say Y. +config INET_ESP_OFFLOAD + tristate "IP: ESP transformation offload" + depends on INET_ESP + select XFRM_OFFLOAD + default n + config INET_IPCOMP tristate "IP: IPComp transformation" select INET_XFRM_TUNNEL diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile index 48af58a..c6d4238 100644 --- a/net/ipv4/Makefile +++ b/net/ipv4/Makefile @@ -29,6 +29,7 @@ obj-$(CONFIG_NET_IPVTI) += ip_vti.o obj-$(CONFIG_SYN_COOKIES) += syncookies.o obj-$(CONFIG_INET_AH) += ah4.o obj-$(CONFIG_INET_ESP) += esp4.o +obj-$(CONFIG_INET_ESP_OFFLOAD) += esp4_offload.o obj-$(CONFIG_INET_IPCOMP) += ipcomp.o obj-$(CONFIG_INET_XFRM_TUNNEL) += xfrm4_tunnel.o obj-$(CONFIG_INET_XFRM_MODE_BEET) += xfrm4_mode_beet.o diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c index b1e2444..4563aeb 100644 --- a/net/ipv4/esp4.c +++ b/net/ipv4/esp4.c @@ -629,6 +629,11 @@ static int esp_input(struct xfrm_state *x, struct sk_buff *skb) nfrags = 1; goto skip_cow; + } else if (skb_xfrm_gro(skb)) { + nfrags = skb_shinfo(skb)->nr_frags; + nfrags++; + + goto skip_cow; } else if (!skb_has_frag_list(skb)) { nfrags = skb_shinfo(skb)->nr_frags; nfrags++; diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c new file mode 100644 index 0000000..7277d15 --- /dev/null +++ b/net/ipv4/esp4_offload.c @@ -0,0 +1,94 @@ +/* + * IPV4 GSO/GRO offload support + * Linux INET implementation + * + * Copyright (C) 2016 secunet Security Networks AG + * Author: Steffen Klassert <steffen.klassert@secunet.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * ESP GRO support + */ + +#include <linux/skbuff.h> +#include <linux/init.h> +#include <net/protocol.h> +#include <crypto/aead.h> +#include <crypto/authenc.h> +#include <linux/err.h> +#include <linux/module.h> +#include <net/ip.h> +#include <net/xfrm.h> +#include <net/esp.h> +#include <linux/scatterlist.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <net/udp.h> + +static struct sk_buff **esp4_gro_receive(struct sk_buff **head, + struct sk_buff *skb) +{ + int err; + if (NAPI_GRO_CB(skb)->flush) + goto out; + + skb_pull(skb, skb_gro_offset(skb)); + skb->xfrm_gro = 1; + + err = xfrm4_rcv_encap(skb, IPPROTO_ESP, 0, -2); + if (err == -EOPNOTSUPP) { + skb_push(skb, skb_gro_offset(skb)); + NAPI_GRO_CB(skb)->same_flow = 0; + NAPI_GRO_CB(skb)->flush = 1; + skb->xfrm_gro = 0; + goto out; + } + + return ERR_PTR(-EINPROGRESS); +out: + return NULL; +} + +static int esp4_gro_complete(struct sk_buff *skb, int nhoff) +{ + struct xfrm_state *x = xfrm_input_state(skb); + struct crypto_aead *aead = x->data; + struct ip_esp_hdr *esph = (struct ip_esp_hdr *)(skb->data + nhoff); + struct packet_offload *ptype; + int err = -ENOENT; + __be16 type = skb->protocol; + + rcu_read_lock(); + ptype = gro_find_complete_by_type(type); + if (ptype != NULL) + err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(*esph) + crypto_aead_ivsize(aead)); + + rcu_read_unlock(); + + return err; +} + +static const struct net_offload esp4_offload = { + .callbacks = { + .gro_receive = esp4_gro_receive, + .gro_complete = esp4_gro_complete, + }, +}; + +static int __init esp4_offload_init(void) +{ + return inet_add_offload(&esp4_offload, IPPROTO_ESP); +} + +static void __exit esp4_offload_exit(void) +{ + inet_del_offload(&esp4_offload, IPPROTO_ESP); +} + +module_init(esp4_offload_init); +module_exit(esp4_offload_exit); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Steffen Klassert <steffen.klassert@secunet.com>"); diff --git a/net/ipv4/ip_vti.c b/net/ipv4/ip_vti.c index 8b14f14..f275feb 100644 --- a/net/ipv4/ip_vti.c +++ b/net/ipv4/ip_vti.c @@ -60,6 +60,10 @@ static int vti_input(struct sk_buff *skb, int nexthdr, __be32 spi, tunnel = ip_tunnel_lookup(itn, skb->dev->ifindex, TUNNEL_NO_KEY, iph->saddr, iph->daddr, 0); if (tunnel) { + /* encap_type < -1 indicates a GRO call, we don't support this. */ + if (encap_type < -1) + return -EOPNOTSUPP; + if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) goto drop; diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c index 62e1e72..6fe1a68 100644 --- a/net/ipv4/xfrm4_input.c +++ b/net/ipv4/xfrm4_input.c @@ -53,6 +53,12 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async) iph->tot_len = htons(skb->len); ip_send_check(iph); + + if (skb_xfrm_gro(skb)) { + skb_mac_header_rebuild(skb); + return 0; + } + NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, dev_net(skb->dev), NULL, skb, skb->dev, NULL, xfrm4_rcv_encap_finish); diff --git a/net/ipv4/xfrm4_mode_transport.c b/net/ipv4/xfrm4_mode_transport.c index fd840c7..7b447f3 100644 --- a/net/ipv4/xfrm4_mode_transport.c +++ b/net/ipv4/xfrm4_mode_transport.c @@ -50,7 +50,8 @@ static int xfrm4_transport_input(struct xfrm_state *x, struct sk_buff *skb) skb->network_header = skb->transport_header; } ip_hdr(skb)->tot_len = htons(skb->len + ihl); - skb_reset_transport_header(skb); + if (!skb_xfrm_gro(skb)) + skb_reset_transport_header(skb); return 0; } diff --git a/net/ipv6/Kconfig b/net/ipv6/Kconfig index ec1267e..8b0713b 100644 --- a/net/ipv6/Kconfig +++ b/net/ipv6/Kconfig @@ -75,6 +75,12 @@ config INET6_ESP If unsure, say Y. +config INET6_ESP_OFFLOAD + tristate "IPv6: ESP transformation offload" + depends on INET6_ESP + select XFRM_OFFLOAD + default n + config INET6_IPCOMP tristate "IPv6: IPComp transformation" select INET6_XFRM_TUNNEL diff --git a/net/ipv6/Makefile b/net/ipv6/Makefile index a9e9fec..217e9ff 100644 --- a/net/ipv6/Makefile +++ b/net/ipv6/Makefile @@ -30,6 +30,7 @@ ipv6-objs += $(ipv6-y) obj-$(CONFIG_INET6_AH) += ah6.o obj-$(CONFIG_INET6_ESP) += esp6.o +obj-$(CONFIG_INET6_ESP_OFFLOAD) += esp6_offload.o obj-$(CONFIG_INET6_IPCOMP) += ipcomp6.o obj-$(CONFIG_INET6_XFRM_TUNNEL) += xfrm6_tunnel.o obj-$(CONFIG_INET6_TUNNEL) += tunnel6.o diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c index ff54faa..1cca0f1 100644 --- a/net/ipv6/esp6.c +++ b/net/ipv6/esp6.c @@ -571,6 +571,11 @@ static int esp6_input(struct xfrm_state *x, struct sk_buff *skb) nfrags = 1; goto skip_cow; + } else if (skb_xfrm_gro(skb)) { + nfrags = skb_shinfo(skb)->nr_frags; + nfrags++; + + goto skip_cow; } else if (!skb_has_frag_list(skb)) { nfrags = skb_shinfo(skb)->nr_frags; nfrags++; diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c new file mode 100644 index 0000000..e0006cf --- /dev/null +++ b/net/ipv6/esp6_offload.c @@ -0,0 +1,98 @@ +/* + * IPV6 GSO/GRO offload support + * Linux INET implementation + * + * Copyright (C) 2016 secunet Security Networks AG + * Author: Steffen Klassert <steffen.klassert@secunet.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * ESP GRO support + */ + +#include <linux/skbuff.h> +#include <linux/init.h> +#include <net/protocol.h> +#include <crypto/aead.h> +#include <crypto/authenc.h> +#include <linux/err.h> +#include <linux/module.h> +#include <net/ip.h> +#include <net/xfrm.h> +#include <net/esp.h> +#include <linux/scatterlist.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <net/ip6_route.h> +#include <net/ipv6.h> +#include <linux/icmpv6.h> + +static struct sk_buff **esp6_gro_receive(struct sk_buff **head, + struct sk_buff *skb) +{ + int err; + if (NAPI_GRO_CB(skb)->flush) + goto out; + + skb_pull(skb, skb_gro_offset(skb)); + skb->xfrm_gro = 1; + + XFRM_SPI_SKB_CB(skb)->family = AF_INET6; + XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct ipv6hdr, daddr); + err = xfrm_input(skb, IPPROTO_ESP, 0, -2); + if (err == -EOPNOTSUPP) { + skb_push(skb, skb_gro_offset(skb)); + NAPI_GRO_CB(skb)->same_flow = 0; + NAPI_GRO_CB(skb)->flush = 1; + skb->xfrm_gro = 0; + goto out; + } + + return ERR_PTR(-EINPROGRESS); +out: + return NULL; +} + +static int esp6_gro_complete(struct sk_buff *skb, int nhoff) +{ + struct xfrm_state *x = xfrm_input_state(skb); + struct crypto_aead *aead = x->data; + struct ip_esp_hdr *esph = (struct ip_esp_hdr *)(skb->data + nhoff); + struct packet_offload *ptype; + int err = -ENOENT; + __be16 type = skb->protocol; + + rcu_read_lock(); + ptype = gro_find_complete_by_type(type); + if (ptype != NULL) + err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(*esph) + crypto_aead_ivsize(aead)); + + rcu_read_unlock(); + + return err; +} + +static const struct net_offload esp6_offload = { + .callbacks = { + .gro_receive = esp6_gro_receive, + .gro_complete = esp6_gro_complete, + }, +}; + +static int __init esp6_offload_init(void) +{ + return inet6_add_offload(&esp6_offload, IPPROTO_ESP); +} + +static void __exit esp6_offload_exit(void) +{ + inet6_del_offload(&esp6_offload, IPPROTO_ESP); +} + +module_init(esp6_offload_init); +module_exit(esp6_offload_exit); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Steffen Klassert <steffen.klassert@secunet.com>"); diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c index b578956..8ed16c8 100644 --- a/net/ipv6/xfrm6_input.c +++ b/net/ipv6/xfrm6_input.c @@ -44,6 +44,11 @@ int xfrm6_transport_finish(struct sk_buff *skb, int async) ipv6_hdr(skb)->payload_len = htons(skb->len); __skb_push(skb, skb->data - skb_network_header(skb)); + if (skb_xfrm_gro(skb)) { + skb_mac_header_rebuild(skb); + return -1; + } + NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING, dev_net(skb->dev), NULL, skb, skb->dev, NULL, ip6_rcv_finish); diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 6e3f025..893263e 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -193,13 +193,17 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) int decaps = 0; int async = 0; - /* A negative encap_type indicates async resumption. */ if (encap_type < 0) { - async = 1; - x = xfrm_input_state(skb); - seq = XFRM_SKB_CB(skb)->seq.input.low; - family = x->outer_mode->afinfo->family; - goto resume; + /* An encap_type of -1 indicates async resumption. */ + if (encap_type == -1) { + async = 1; + x = xfrm_input_state(skb); + seq = XFRM_SKB_CB(skb)->seq.input.low; + family = x->outer_mode->afinfo->family; + goto resume; + } + /* encap_type < -1 indicates a GRO call. */ + encap_type = 0; } daddr = (xfrm_address_t *)(skb_network_header(skb) + @@ -374,7 +378,16 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) netif_rx(skb); return 0; } else { - return x->inner_mode->afinfo->transport_finish(skb, async); + int xfrm_gro = skb_xfrm_gro(skb); + + err = x->inner_mode->afinfo->transport_finish(skb, async); + if (xfrm_gro) { + skb_dst_drop(skb); + netif_rx(skb); + return 0; + } + + return err; } drop_unlock: -- 1.9.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-02-10 7:39 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-02-07 9:14 [PATCH RFC ipsec-next 0/5] IPsec GRO layer decapsulation Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 1/5] xfrm: Add a secpath_set helper Steffen Klassert 2017-02-08 18:18 ` David Miller 2017-02-07 9:14 ` [PATCH RFC ipsec-next 2/5] net: Add a skb_gro_flush_final helper Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 3/5] net: Prepare gro for packet consuming gro callbacks Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 4/5] xfrm: Export xfrm_parse_spi Steffen Klassert 2017-02-07 9:14 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert 2017-02-07 19:45 ` Eric Dumazet 2017-02-08 11:43 ` Steffen Klassert 2017-02-08 18:19 ` David Miller 2017-02-10 7:39 ` Steffen Klassert -- strict thread matches above, loose matches on Subject: below -- 2017-01-04 8:23 [PATCH RFC ipsec-next] IPsec offload, part one Steffen Klassert 2017-01-04 8:23 ` [PATCH RFC ipsec-next 5/5] esp: Add a software GRO codepath Steffen Klassert
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).