[RFC PATCH net-next 0/7] net: airoha: add EN7581 SOE ESP packet offload

[RFC PATCH net-next 0/7] net: airoha: add EN7581 SOE ESP packet offload

[Jihong Min]

Add Secure Offload Engine (SOE) support for the Airoha EN7581 Ethernet
driver. SOE provides inline ESP packet offload for native ESP and NAT-T
traffic, with the Ethernet/QDMA path used to submit packets to the SOE
block and the PPE path used to bind eligible ESP flows. NETIF_F_GSO_ESP
and NETIF_F_HW_ESP_TX_CSUM are intentionally left out for now and will be
revisited separately for feasibility.

This is posted as RFC because the code was originally developed and tested
against an OpenWrt 6.18 Airoha tree, not against the current upstream
net-next driver. The original OpenWrt commit used as the source for this
RFC is available at:
https://github.com/hurryman2212/OpenW1700k-test/commit/7c1b5e662f7790b3d23ed143beadc1dcbf6d15f7

The SOE part is intentionally linked into the airoha Ethernet module
instead of being exposed as an independent crypto or platform driver. The
user-visible ESP offload control is a netdev capability: xfrmdev_ops and
NETIF_F_HW_ESP live on the target netdev, and the feature can be controlled
through the usual netdev feature path. SOE also shares the FE/QDMA/PPE
datapath, private queues, DSA conduit handling and netdev lifetime owned by
airoha_eth.

Patch 1 adds xdo_dev_packet_xmit() because the existing XFRM packet
offload transmit path does not provide a hook for hardware whose ESP engine
is reached through device-specific packet forwarding. SOE needs to consume
the skb, add a hardware hop descriptor, steer it to a private QDMA path and
return the final transmit status. Drivers that do not implement the
optional callback keep the existing XFRM output behavior.

Jihong Min (7):
  xfrm: allow packet offload drivers to own transmit
  dt-bindings: net: airoha: add EN7581 SOE
  arm64: dts: airoha: add EN7581 SOE node
  net: airoha: add SOE registers and driver state
  net: airoha: add QDMA support for SOE packets
  net: airoha: add PPE support for SOE flows
  net: airoha: add SOE XFRM packet offload support

 .../bindings/net/airoha,en7581-soe.yaml       |   48 +
 MAINTAINERS                                   |    1 +
 arch/arm64/boot/dts/airoha/en7581.dtsi        |    6 +
 drivers/net/ethernet/airoha/Kconfig           |   13 +
 drivers/net/ethernet/airoha/Makefile          |    1 +
 drivers/net/ethernet/airoha/airoha_eth.c      |  668 +++++-
 drivers/net/ethernet/airoha/airoha_eth.h      |   40 +
 drivers/net/ethernet/airoha/airoha_ppe.c      |  606 +++++-
 drivers/net/ethernet/airoha/airoha_regs.h     |   16 +
 drivers/net/ethernet/airoha/airoha_soe.c      | 1896 +++++++++++++++++
 drivers/net/ethernet/airoha/airoha_soe.h      |  126 ++
 include/linux/netdevice.h                     |    8 +
 include/linux/soc/airoha/airoha_offload.h     |    5 +
 net/xfrm/xfrm_output.c                        |   11 +
 14 files changed, 3342 insertions(+), 103 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml
 create mode 100644 drivers/net/ethernet/airoha/airoha_soe.c
 create mode 100644 drivers/net/ethernet/airoha/airoha_soe.h

-- 
2.53.0

[RFC PATCH net-next 1/7] xfrm: allow packet offload drivers to own transmit

[Jihong Min]

Packet offload drivers can currently program state and validate whether an skb can be offloaded, but they cannot take ownership of a packet that needs driver-specific TX preparation before the regular XFRM output path continues.

Add an optional xdo_dev_packet_xmit() callback. Drivers that implement it consume the skb and return the final TX status; all other drivers keep the existing XFRM output path.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 include/linux/netdevice.h |  8 ++++++++
 net/xfrm/xfrm_output.c    | 11 +++++++++++
 2 files changed, 19 insertions(+)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 7f4f0837c09f..1552eb81ddf0 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1048,6 +1048,14 @@ struct xfrmdev_ops {
 	int	(*xdo_dev_policy_add) (struct xfrm_policy *x, struct netlink_ext_ack *extack);
 	void	(*xdo_dev_policy_delete) (struct xfrm_policy *x);
 	void	(*xdo_dev_policy_free) (struct xfrm_policy *x);
+	/* Optional packet-offload TX path for devices that need
+	 * driver-specific transmit preparation instead of continuing through
+	 * the regular XFRM output path, such as adding offload metadata or
+	 * steering the packet to a private transmit queue. The driver consumes
+	 * skb and returns the final transmit status.
+	 */
+	int	(*xdo_dev_packet_xmit)(struct sk_buff *skb,
+				       struct xfrm_state *x);
 };
 #endif
 
diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c
index cc35c2fcbbe0..9f11559b0221 100644
--- a/net/xfrm/xfrm_output.c
+++ b/net/xfrm/xfrm_output.c
@@ -770,6 +770,17 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb)
 	}
 
 	if (x->xso.type == XFRM_DEV_OFFLOAD_PACKET) {
+#ifdef CONFIG_XFRM_OFFLOAD
+		const struct xfrmdev_ops *ops;
+#endif
+
+#ifdef CONFIG_XFRM_OFFLOAD
+		ops = x->xso.dev->xfrmdev_ops;
+		/* Callback validates, consumes skb and returns final TX status. */
+		if (ops && ops->xdo_dev_packet_xmit)
+			return ops->xdo_dev_packet_xmit(skb, x);
+#endif
+
 		if (!xfrm_dev_offload_ok(skb, x)) {
 			XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
 			kfree_skb(skb);
-- 
2.53.0

[RFC PATCH net-next 2/7] dt-bindings: net: airoha: add EN7581 SOE

[Jihong Min]

Document the EN7581 Secure Offload Engine register window used by the Ethernet driver for ESP packet offload, and add the new binding to the Airoha Ethernet MAINTAINERS entry.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 .../bindings/net/airoha,en7581-soe.yaml       | 48 +++++++++++++++++++
 MAINTAINERS                                   |  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml

diff --git a/Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml b/Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml
new file mode 100644
index 000000000000..24aecafecc70
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml
@@ -0,0 +1,48 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/airoha,en7581-soe.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Airoha EN7581 Secure Offload Engine
+
+maintainers:
+  - Lorenzo Bianconi <lorenzo@kernel.org>
+
+description:
+  The Secure Offload Engine provides inline ESP packet offload resources used
+  by the Airoha Ethernet controller.
+
+properties:
+  compatible:
+    const: airoha,en7581-soe
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    soc {
+      #address-cells = <2>;
+      #size-cells = <2>;
+
+      soe@1fbfa000 {
+        compatible = "airoha,en7581-soe";
+        reg = <0 0x1fbfa000 0 0x268>;
+        interrupts = <GIC_SPI 79 IRQ_TYPE_LEVEL_HIGH>;
+      };
+    };
+...
diff --git a/MAINTAINERS b/MAINTAINERS
index cc1dde0c9067..7c338e670572 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -757,6 +757,7 @@ L:	linux-mediatek@lists.infradead.org (moderated for non-subscribers)
 L:	netdev@vger.kernel.org
 S:	Maintained
 F:	Documentation/devicetree/bindings/net/airoha,en7581-eth.yaml
+F:	Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml
 F:	drivers/net/ethernet/airoha/
 
 AIROHA PCIE PHY DRIVER
-- 
2.53.0

[RFC PATCH net-next 3/7] arm64: dts: airoha: add EN7581 SOE node

[Jihong Min]

Describe the EN7581 SOE register window and interrupt so the Ethernet driver can discover and initialize the packet offload engine.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 arch/arm64/boot/dts/airoha/en7581.dtsi | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/arm64/boot/dts/airoha/en7581.dtsi b/arch/arm64/boot/dts/airoha/en7581.dtsi
index ff6908a76e8e..a3c1033d2437 100644
--- a/arch/arm64/boot/dts/airoha/en7581.dtsi
+++ b/arch/arm64/boot/dts/airoha/en7581.dtsi
@@ -347,6 +347,12 @@ i2c1: i2c@1fbf8100 {
 			status = "disabled";
 		};
 
+		soe: soe@1fbfa000 {
+			compatible = "airoha,en7581-soe";
+			reg = <0x0 0x1fbfa000 0x0 0x268>;
+			interrupts = <GIC_SPI 79 IRQ_TYPE_LEVEL_HIGH>;
+		};
+
 		eth: ethernet@1fb50000 {
 			compatible = "airoha,en7581-eth";
 			reg = <0 0x1fb50000 0 0x2600>,
-- 
2.53.0

[RFC PATCH net-next 4/7] net: airoha: add SOE registers and driver state

[Jihong Min]

Add the FE/PPE/QDMA register definitions and shared driver state needed
by the Secure Offload Engine. Add a small SOE-facing header with stubs so
later Ethernet and PPE changes remain buildable until the provider is
enabled.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 drivers/net/ethernet/airoha/airoha_eth.h  |  40 +++++++
 drivers/net/ethernet/airoha/airoha_regs.h |  16 +++
 drivers/net/ethernet/airoha/airoha_soe.h  | 126 ++++++++++++++++++++++
 3 files changed, 182 insertions(+)
 create mode 100644 drivers/net/ethernet/airoha/airoha_soe.h

diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/ethernet/airoha/airoha_eth.h
index 46b1c31939de..c5f09aedd7e7 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.h
+++ b/drivers/net/ethernet/airoha/airoha_eth.h
@@ -16,6 +16,8 @@
 #include <linux/soc/airoha/airoha_offload.h>
 #include <net/dsa.h>
 
+struct airoha_soe;
+
 #define AIROHA_MAX_NUM_GDM_PORTS	4
 #define AIROHA_MAX_NUM_GDM_DEVS		2
 #define AIROHA_MAX_NUM_QDMA		2
@@ -189,18 +191,31 @@ struct airoha_queue {
 	spinlock_t lock;
 	struct airoha_queue_entry *entry;
 	struct airoha_qdma_desc *desc;
+	/* Preserved for RX ring reprogramming; dmam_alloc_coherent() owns it. */
+	dma_addr_t desc_dma;
 	u16 head;
 	u16 tail;
 
 	int queued;
 	int ndesc;
+	/* SOE RX rings may use coherent slots instead of page_pool fragments. */
+	int rx_alloc_ndesc;
 	int free_thr;
 	int buf_size;
 	bool txq_stopped;
+	bool rx_coherent;
+	bool rx_drop_chain;
+	void *rx_coherent_buf;
+	dma_addr_t rx_coherent_dma;
+	size_t rx_coherent_buf_size;
 
 	struct napi_struct napi;
 	struct page_pool *page_pool;
 	struct sk_buff *skb;
+	/* First SOE descriptor metadata is kept while a scattered frame is built. */
+	int rx_frame_descs;
+	u32 soe_rx_msg0;
+	u32 soe_rx_msg2;
 
 	struct list_head tx_list;
 };
@@ -434,6 +449,16 @@ struct airoha_foe_stats64 {
 	u64 packets;
 };
 
+struct airoha_ppe_soe_meta {
+	unsigned long expires;
+	u32 key_hash;
+	u16 foe_hash;
+	u8 valid;
+	u8 sa_index;
+	u8 hop;
+	u8 seen;
+};
+
 struct airoha_flow_data {
 	struct ethhdr eth;
 
@@ -552,6 +577,11 @@ struct airoha_gdm_dev {
 
 	u32 flags;
 	int nbq;
+	/* Prevent toggling NETIF_F_HW_ESP while programmed SAs still exist. */
+	atomic_t soe_xfrm_state_count;
+	/* Private SOE submit path into this GDM's active QDMA instance. */
+	int (*soe_xmit_skb)(struct airoha_gdm_dev *dev, struct sk_buff *skb,
+			    u32 msg0, u32 msg1, u32 msg2);
 
 	struct airoha_hw_stats stats;
 };
@@ -581,6 +611,7 @@ struct airoha_ppe {
 
 	struct hlist_head *foe_flow;
 	u16 *foe_check_time;
+	struct airoha_ppe_soe_meta *soe_meta;
 
 	struct airoha_foe_stats *foe_stats;
 	dma_addr_t foe_stats_dma;
@@ -621,6 +652,7 @@ struct airoha_eth {
 
 	struct airoha_qdma qdma[AIROHA_MAX_NUM_QDMA];
 	struct airoha_gdm_port *ports[AIROHA_MAX_NUM_GDM_PORTS];
+	struct airoha_soe *soe;
 };
 
 u32 airoha_rr(void __iomem *base, u32 offset);
@@ -676,6 +708,14 @@ static inline bool airoha_is_7583(struct airoha_eth *eth)
 int airoha_get_fe_port(struct airoha_gdm_dev *dev);
 bool airoha_is_valid_gdm_dev(struct airoha_eth *eth,
 			     struct airoha_gdm_dev *dev);
+int airoha_qdma_xmit_skb(struct airoha_gdm_dev *dev, struct sk_buff *skb,
+			 u32 msg0, u32 msg1, u32 msg2);
+void airoha_ppe_soe_mark_skb(struct airoha_ppe_dev *dev, struct sk_buff *skb,
+			     u16 hash, u8 sa_index, u8 hop);
+bool airoha_ppe_soe_skb_marked(struct sk_buff *skb);
+void airoha_ppe_soe_xmit_skb(struct airoha_ppe_dev *dev, struct sk_buff *skb,
+			     struct net_device *netdev);
+void airoha_ppe_soe_flush_sa(struct airoha_ppe *ppe, u8 sa_index);
 
 void airoha_ppe_set_cpu_port(struct airoha_gdm_dev *dev, u8 ppe_id, u8 fport);
 bool airoha_ppe_is_enabled(struct airoha_eth *eth, int index);
diff --git a/drivers/net/ethernet/airoha/airoha_regs.h b/drivers/net/ethernet/airoha/airoha_regs.h
index 436f3c8779c1..27e158d0fa4b 100644
--- a/drivers/net/ethernet/airoha/airoha_regs.h
+++ b/drivers/net/ethernet/airoha/airoha_regs.h
@@ -82,6 +82,10 @@
 #define PSE_SHARE_USED_MTHD_MASK	GENMASK(31, 16)
 #define PSE_SHARE_USED_HTHD_MASK	GENMASK(15, 0)
 
+/* TDMA/SOE port 7 needs shared-buffer flow control enabled in the PSE. */
+#define REG_PSE_FC_CFG			0x0098
+#define PSE_TDMA_SHARE_BUF_DIS_MASK	BIT(23)
+
 #define REG_GDM_MISC_CFG		0x0148
 #define GDM2_RDM_ACK_WAIT_PREF_MASK	BIT(9)
 #define GDM2_CHN_VLD_MODE_MASK		BIT(5)
@@ -252,6 +256,8 @@
 #define PPE_GLO_CFG_EN_MASK			BIT(0)
 
 #define REG_PPE_PPE_FLOW_CFG(_n)		(((_n) ? PPE2_BASE : PPE1_BASE) + 0x204)
+#define PPE_FLOW_CFG_IP6_IPSEC_MASK		BIT(28)
+#define PPE_FLOW_CFG_IP4_IPSEC_MASK		BIT(27)
 #define PPE_FLOW_CFG_IP6_HASH_GRE_KEY_MASK	BIT(20)
 #define PPE_FLOW_CFG_IP4_HASH_GRE_KEY_MASK	BIT(19)
 #define PPE_FLOW_CFG_IP4_HASH_FLOW_LABEL_MASK	BIT(18)
@@ -851,6 +857,8 @@
 #define QDMA_DESC_NEXT_ID_MASK		GENMASK(15, 0)
 /* TX MSG0 */
 #define QDMA_ETH_TXMSG_MIC_IDX_MASK	BIT(30)
+/* SOE submit metadata: msg0 carries SA index, msg1 selects port 7 OQ8/OQ9. */
+#define QDMA_ETH_TXMSG_SOE_SA_MASK	GENMASK(29, 24)
 #define QDMA_ETH_TXMSG_SP_TAG_MASK	GENMASK(29, 14)
 #define QDMA_ETH_TXMSG_ICO_MASK		BIT(13)
 #define QDMA_ETH_TXMSG_UCO_MASK		BIT(12)
@@ -873,6 +881,11 @@
 
 /* RX MSG0 */
 #define QDMA_ETH_RXMSG_SPTAG		GENMASK(21, 14)
+/* SOE completion metadata can use the full 16-bit SP tag word. */
+#define QDMA_ETH_RXMSG_SPTAG_FULL	GENMASK(29, 14)
+/* SOE completion metadata returned by the QDMA RX descriptor. */
+#define QDMA_ETH_RXMSG_SOE_MASK		BIT(10)
+#define QDMA_ETH_RXMSG_HOP_FLAGS_MASK	GENMASK(2, 0)
 /* RX MSG1 */
 #define QDMA_ETH_RXMSG_DEI_MASK		BIT(31)
 #define QDMA_ETH_RXMSG_IP6_MASK		BIT(30)
@@ -883,6 +896,9 @@
 #define QDMA_ETH_RXMSG_SPORT_MASK	GENMASK(25, 21)
 #define QDMA_ETH_RXMSG_CRSN_MASK	GENMASK(20, 16)
 #define QDMA_ETH_RXMSG_PPE_ENTRY_MASK	GENMASK(15, 0)
+/* RX MSG2 */
+/* SW_UDF carries the SA index for SOE completion frames. */
+#define QDMA_ETH_RXMSG_SW_UDF_MASK	GENMASK(31, 24)
 
 struct airoha_qdma_desc {
 	__le32 rsv;
diff --git a/drivers/net/ethernet/airoha/airoha_soe.h b/drivers/net/ethernet/airoha/airoha_soe.h
new file mode 100644
index 000000000000..0bde2e9c6b5b
--- /dev/null
+++ b/drivers/net/ethernet/airoha/airoha_soe.h
@@ -0,0 +1,126 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Ethernet-facing declarations for the Airoha Secure Offload Engine (SOE)
+ * packet offload provider.
+ *
+ * airoha_eth owns SOE lifetime and calls these helpers to expose xfrm
+ * ESP/NAT-T offload on its netdevs. When CONFIG_NET_AIROHA_SOE is disabled,
+ * the stubs keep the Ethernet driver buildable without SOE support.
+ */
+
+#ifndef AIROHA_SOE_H
+#define AIROHA_SOE_H
+
+#include <linux/bitops.h>
+#include <linux/errno.h>
+#include <linux/kconfig.h>
+#include <linux/netdev_features.h>
+#include <linux/types.h>
+
+struct airoha_soe;
+struct airoha_soe_sa;
+struct airoha_eth;
+struct airoha_gdm_dev;
+struct device;
+struct dst_entry;
+struct net_device;
+struct netlink_ext_ack;
+struct sk_buff;
+struct xfrm_state;
+
+#define AIROHA_SOE_FEATURE_ESP		BIT(0)
+
+typedef int (*airoha_soe_xmit_skb_t)(struct airoha_gdm_dev *dev,
+				     struct sk_buff *skb, u32 msg0, u32 msg1,
+				     u32 msg2);
+
+#if IS_ENABLED(CONFIG_NET_AIROHA_SOE)
+int airoha_soe_init(struct airoha_eth *eth);
+void airoha_soe_deinit(struct airoha_eth *eth);
+bool airoha_soe_available(struct airoha_soe *soe);
+u32 airoha_soe_features(struct airoha_soe *soe);
+void airoha_soe_build_netdev(struct net_device *dev,
+			     airoha_soe_xmit_skb_t xmit_skb);
+void airoha_soe_teardown_netdev(struct net_device *dev);
+int airoha_soe_set_features(struct net_device *dev,
+			    netdev_features_t features);
+bool airoha_soe_rx_skb(struct airoha_soe *soe, struct sk_buff *skb,
+		       unsigned int sa_index, u32 hop_flags);
+bool airoha_soe_rx_plain_skb(struct airoha_gdm_dev *dev,
+			     struct sk_buff *skb, struct net_device *rx_dev,
+			     u16 foe_hash, u32 foe_reason, bool foe_valid);
+bool airoha_soe_has_pending_rx(struct airoha_soe *soe);
+int airoha_soe_xfrm_ppe_info(const struct dst_entry *dst, u8 *sa_index,
+			     u8 *hop);
+int airoha_soe_xmit(struct airoha_soe_sa *sa, struct airoha_gdm_dev *dev,
+		    struct sk_buff *skb, struct xfrm_state *x);
+#else
+static inline int airoha_soe_init(struct airoha_eth *eth)
+{
+	return 0;
+}
+
+static inline void airoha_soe_deinit(struct airoha_eth *eth)
+{
+}
+
+static inline bool airoha_soe_available(struct airoha_soe *soe)
+{
+	return false;
+}
+
+static inline u32 airoha_soe_features(struct airoha_soe *soe)
+{
+	return 0;
+}
+
+static inline void airoha_soe_build_netdev(struct net_device *dev,
+					   airoha_soe_xmit_skb_t xmit_skb)
+{
+}
+
+static inline void airoha_soe_teardown_netdev(struct net_device *dev)
+{
+}
+
+static inline int airoha_soe_set_features(struct net_device *dev,
+					  netdev_features_t features)
+{
+	return 0;
+}
+
+static inline bool airoha_soe_rx_skb(struct airoha_soe *soe,
+				     struct sk_buff *skb,
+				     unsigned int sa_index, u32 hop_flags)
+{
+	return false;
+}
+
+static inline bool airoha_soe_rx_plain_skb(struct airoha_gdm_dev *dev,
+					   struct sk_buff *skb,
+					   struct net_device *rx_dev,
+					   u16 foe_hash, u32 foe_reason,
+					   bool foe_valid)
+{
+	return false;
+}
+
+static inline bool airoha_soe_has_pending_rx(struct airoha_soe *soe)
+{
+	return false;
+}
+
+static inline int airoha_soe_xfrm_ppe_info(const struct dst_entry *dst,
+					   u8 *sa_index, u8 *hop)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int airoha_soe_xmit(struct airoha_soe_sa *sa,
+				  struct airoha_gdm_dev *dev,
+				  struct sk_buff *skb, struct xfrm_state *x)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
+#endif /* AIROHA_SOE_H */
-- 
2.53.0

[RFC PATCH net-next 5/7] net: airoha: add QDMA support for SOE packets

[Jihong Min]

Add QDMA RX/TX plumbing for SOE packets, including the SOE RX ring,
coherent RX slots, SOE completion decoding, and the private QDMA submit
helper used by the SOE provider. Wire the Ethernet netdev feature and
lifetime hooks through the SOE stubs.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 drivers/net/ethernet/airoha/airoha_eth.c | 668 ++++++++++++++++++++---
 1 file changed, 591 insertions(+), 77 deletions(-)

diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
index 5f1a118875fb..42fd30c12ed7 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.c
+++ b/drivers/net/ethernet/airoha/airoha_eth.c
@@ -6,6 +6,7 @@
 #include <linux/of.h>
 #include <linux/of_net.h>
 #include <linux/of_reserved_mem.h>
+#include <linux/moduleparam.h>
 #include <linux/platform_device.h>
 #include <linux/tcp.h>
 #include <linux/u64_stats_sync.h>
@@ -16,6 +17,67 @@
 
 #include "airoha_regs.h"
 #include "airoha_eth.h"
+#include "airoha_soe.h"
+
+/* QDMA1 uses a different RX IRQ bank layout than QDMA0 on EN7581. */
+#define AIROHA_QDMA_WAN_RX_IRQ0_BANK_PIN_MASK 0x0000839f
+#define AIROHA_QDMA_WAN_RX_IRQ1_BANK_PIN_MASK 0x7f800400
+#define AIROHA_QDMA_WAN_RX_IRQ2_BANK_PIN_MASK 0x00000000
+#define AIROHA_QDMA_WAN_RX_IRQ3_BANK_PIN_MASK 0x00000040
+
+static unsigned int airoha_qdma0_rx_irq_bank_mask[AIROHA_MAX_NUM_IRQ_BANKS] = {
+	RX_IRQ0_BANK_PIN_MASK,
+	RX_IRQ1_BANK_PIN_MASK,
+	RX_IRQ2_BANK_PIN_MASK,
+	RX_IRQ3_BANK_PIN_MASK,
+};
+
+static unsigned int airoha_qdma1_rx_irq_bank_mask[AIROHA_MAX_NUM_IRQ_BANKS] = {
+	AIROHA_QDMA_WAN_RX_IRQ0_BANK_PIN_MASK,
+	AIROHA_QDMA_WAN_RX_IRQ1_BANK_PIN_MASK,
+	AIROHA_QDMA_WAN_RX_IRQ2_BANK_PIN_MASK,
+	AIROHA_QDMA_WAN_RX_IRQ3_BANK_PIN_MASK,
+};
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+#define AIROHA_SOE_RX_RING 10
+#define AIROHA_SOE_RX_ALLOC_NDESC 2048
+#define AIROHA_SOE_RX_NDESC_DEFAULT 512
+#define AIROHA_SOE_RX_BUF_SIZE 4096
+#define AIROHA_SOE_RX_BUF_SIZE_MIN 128
+#define AIROHA_SOE_RX_BUF_SIZE_MAX 16384
+#define AIROHA_SOE_DECRYPT_FOE_REASON_MASK \
+	(BIT(PPE_CPU_REASON_UN_HIT) | BIT(PPE_CPU_REASON_HIT_UNBIND) | \
+	 BIT(PPE_CPU_REASON_HIT_UNBIND_RATE_REACHED))
+
+static int airoha_soe_rx_poll_desc_budget = AIROHA_SOE_RX_NDESC_DEFAULT;
+static int airoha_soe_rx_max_frame_descs = 128;
+static int airoha_soe_probe_rx_ndesc = AIROHA_SOE_RX_NDESC_DEFAULT;
+static int airoha_soe_probe_rx_buf_size = AIROHA_SOE_RX_BUF_SIZE;
+static bool airoha_soe_probe_rx_coherent = true;
+static int airoha_soe_probe_rx_scatter = 1;
+
+module_param_named(soe_rx_poll_desc_budget, airoha_soe_rx_poll_desc_budget, int,
+		   0600);
+MODULE_PARM_DESC(soe_rx_poll_desc_budget,
+		 "Maximum SOE RX descriptors consumed per poll");
+module_param_named(soe_rx_max_frame_descs, airoha_soe_rx_max_frame_descs, int,
+		   0600);
+MODULE_PARM_DESC(soe_rx_max_frame_descs,
+		 "Maximum descriptors per SOE RX frame before dropping the chain");
+module_param_named(soe_probe_rx_ndesc, airoha_soe_probe_rx_ndesc, int, 0600);
+MODULE_PARM_DESC(soe_probe_rx_ndesc, "SOE RX descriptor count");
+module_param_named(soe_probe_rx_buf_size, airoha_soe_probe_rx_buf_size, int,
+		   0600);
+MODULE_PARM_DESC(soe_probe_rx_buf_size, "SOE RX buffer size");
+module_param_named(soe_probe_rx_coherent, airoha_soe_probe_rx_coherent, bool,
+		   0600);
+MODULE_PARM_DESC(soe_probe_rx_coherent, "Use coherent SOE RX buffers");
+module_param_named(soe_probe_rx_scatter, airoha_soe_probe_rx_scatter, int,
+		   0600);
+MODULE_PARM_DESC(soe_probe_rx_scatter,
+		 "SOE RX scatter mode: 0 disabled, 1 enabled");
+#endif
 
 u32 airoha_rr(void __iomem *base, u32 offset)
 {
@@ -71,6 +133,100 @@ static void airoha_qdma_irq_disable(struct airoha_irq_bank *irq_bank,
 	airoha_qdma_set_irqmask(irq_bank, index, mask, 0);
 }
 
+static unsigned int *airoha_qdma_rx_irq_bank_masks(struct airoha_qdma *qdma)
+{
+	struct airoha_eth *eth = qdma->eth;
+	int id = qdma - &eth->qdma[0];
+
+	return id ? airoha_qdma1_rx_irq_bank_mask :
+		    airoha_qdma0_rx_irq_bank_mask;
+}
+
+static u32 airoha_qdma_rx_irq_bank_mask(struct airoha_qdma *qdma, int bank)
+{
+	unsigned int *masks = airoha_qdma_rx_irq_bank_masks(qdma);
+
+	if (bank < 0 || bank >= AIROHA_MAX_NUM_IRQ_BANKS)
+		return 0;
+
+	return READ_ONCE(masks[bank]);
+}
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+static u32 airoha_qdma_rx_irq_all_bank_mask(struct airoha_qdma *qdma)
+{
+	u32 mask = 0;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++)
+		mask |= airoha_qdma_rx_irq_bank_mask(qdma, i);
+
+	return mask;
+}
+#endif
+
+static void airoha_qdma_apply_rx_irq_bank_masks(struct airoha_qdma *qdma)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) {
+		struct airoha_irq_bank *irq_bank = &qdma->irq_banks[i];
+		u32 mask = airoha_qdma_rx_irq_bank_mask(qdma, i);
+
+		airoha_qdma_set_irqmask(irq_bank, QDMA_INT_REG_IDX0,
+					RX_COHERENT_LOW_INT_MASK,
+					INT_RX0_MASK(mask));
+		airoha_qdma_set_irqmask(irq_bank, QDMA_INT_REG_IDX1,
+					RX_NO_CPU_DSCP_LOW_INT_MASK |
+						RX_DONE_LOW_INT_MASK,
+					INT_RX1_MASK(mask));
+		airoha_qdma_set_irqmask(irq_bank, QDMA_INT_REG_IDX2,
+					RX_NO_CPU_DSCP_HIGH_INT_MASK |
+						RX_DONE_HIGH_INT_MASK,
+					INT_RX2_MASK(mask));
+		airoha_qdma_set_irqmask(irq_bank, QDMA_INT_REG_IDX3,
+					RX_COHERENT_HIGH_INT_MASK,
+					INT_RX3_MASK(mask));
+	}
+}
+
+static void airoha_qdma_set_rx_done_irq(struct airoha_qdma *qdma, int qid,
+					bool enable)
+{
+	int i, intr_reg;
+	u32 mask;
+
+	intr_reg = qid < RX_DONE_HIGH_OFFSET ? QDMA_INT_REG_IDX1 :
+					       QDMA_INT_REG_IDX2;
+	mask = BIT(qid % RX_DONE_HIGH_OFFSET);
+
+	for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) {
+		if (!(BIT(qid) & airoha_qdma_rx_irq_bank_mask(qdma, i)))
+			continue;
+
+		if (enable)
+			airoha_qdma_irq_enable(&qdma->irq_banks[i], intr_reg,
+					       mask);
+		else
+			airoha_qdma_irq_disable(&qdma->irq_banks[i], intr_reg,
+						mask);
+	}
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (qid != AIROHA_SOE_RX_RING)
+		return;
+
+	if (BIT(qid) & airoha_qdma_rx_irq_all_bank_mask(qdma))
+		return;
+
+	/* SOE RX10 may not be covered by the shared bank mask; use bank 0. */
+	if (enable)
+		airoha_qdma_irq_enable(&qdma->irq_banks[0], intr_reg, mask);
+	else
+		airoha_qdma_irq_disable(&qdma->irq_banks[0], intr_reg, mask);
+#endif
+}
+
 static int airoha_set_macaddr(struct airoha_gdm_dev *dev, const u8 *addr)
 {
 	u8 ref_addr[ETH_ALEN] __aligned(2);
@@ -532,6 +688,11 @@ static int airoha_fe_init(struct airoha_eth *eth)
 	airoha_fe_wr(eth, REG_FE_CDM1_OQ_MAP2, BIT(4));
 	airoha_fe_wr(eth, REG_FE_CDM1_OQ_MAP3, BIT(28));
 
+	/* SOE/TDMA output depends on PSE shared-buffer flow control instead
+	 * of leaving port-local sharing disabled.
+	 */
+	airoha_fe_clear(eth, REG_PSE_FC_CFG, PSE_TDMA_SHARE_BUF_DIS_MASK);
+
 	airoha_fe_vip_setup(eth);
 	airoha_fe_pse_ports_init(eth);
 
@@ -597,20 +758,29 @@ static int airoha_qdma_fill_rx_queue(struct airoha_queue *q)
 		int offset;
 		u32 val;
 
-		page = page_pool_dev_alloc_frag(q->page_pool, &offset,
-						q->buf_size);
-		if (!page)
-			break;
+		if (q->rx_coherent) {
+			/* Coherent slots avoid page_pool recycling for SOE frames. */
+			offset = q->head * q->buf_size;
+			e->buf = q->rx_coherent_buf + offset;
+			e->dma_addr = q->rx_coherent_dma + offset;
+			e->dma_len = q->buf_size;
+		} else {
+			page = page_pool_dev_alloc_frag(q->page_pool, &offset,
+							q->buf_size);
+			if (!page)
+				break;
+
+			offset += AIROHA_RX_HEADROOM;
+			e->buf = page_address(page) + offset;
+			e->dma_addr = page_pool_get_dma_addr(page) + offset;
+			e->dma_len =
+				SKB_WITH_OVERHEAD(AIROHA_RX_LEN(q->buf_size));
+		}
 
 		q->head = (q->head + 1) % q->ndesc;
 		q->queued++;
 		nframes++;
 
-		offset += AIROHA_RX_HEADROOM;
-		e->buf = page_address(page) + offset;
-		e->dma_addr = page_pool_get_dma_addr(page) + offset;
-		e->dma_len = SKB_WITH_OVERHEAD(AIROHA_RX_LEN(q->buf_size));
-
 		val = FIELD_PREP(QDMA_DESC_LEN_MASK, e->dma_len);
 		WRITE_ONCE(desc->ctrl, cpu_to_le32(val));
 		WRITE_ONCE(desc->addr, cpu_to_le32(e->dma_addr));
@@ -652,92 +822,210 @@ airoha_qdma_get_gdm_dev(struct airoha_eth *eth, struct airoha_qdma_desc *desc)
 	return port->devs[d] ? port->devs[d] : ERR_PTR(-ENODEV);
 }
 
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+static bool airoha_qdma_rx_is_soe(u32 msg0)
+{
+	u32 hop_flags = FIELD_GET(QDMA_ETH_RXMSG_HOP_FLAGS_MASK, msg0);
+
+	return hop_flags >= 3 && hop_flags <= 7;
+}
+#endif
+
 static int airoha_qdma_rx_process(struct airoha_queue *q, int budget)
 {
-	enum dma_data_direction dir = page_pool_get_dma_dir(q->page_pool);
+	enum dma_data_direction dir;
 	struct airoha_qdma *qdma = q->qdma;
 	struct airoha_eth *eth = qdma->eth;
 	int qid = q - &qdma->q_rx[0];
+	int desc_budget = q->ndesc;
+	int desc_done = 0;
 	int done = 0;
 
-	while (done < budget) {
+	dir = q->rx_coherent ? DMA_FROM_DEVICE :
+			       page_pool_get_dma_dir(q->page_pool);
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (airoha_soe_rx_poll_desc_budget > 0)
+		desc_budget = min(airoha_soe_rx_poll_desc_budget, q->ndesc);
+#endif
+
+	while (q->queued > 0 && done < budget && desc_done < desc_budget) {
 		struct airoha_queue_entry *e = &q->entry[q->tail];
 		struct airoha_qdma_desc *desc = &q->desc[q->tail];
-		u32 hash, reason, msg1, desc_ctrl;
-		struct airoha_gdm_dev *dev;
+		u32 hash, reason, msg0, msg1, msg2, desc_ctrl;
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+		/* Scattered SOE completions only tag the first descriptor. */
+		bool partial_soe = q->skb && !q->skb->dev;
+#endif
+		struct airoha_gdm_dev *dev = NULL;
+		struct net_device *rx_dev = NULL;
 		struct net_device *netdev;
+		bool soe_pkt = false;
 		int data_len, len;
-		struct page *page;
+		struct page *page = NULL;
 
 		desc_ctrl = le32_to_cpu(READ_ONCE(desc->ctrl));
 		if (!(desc_ctrl & QDMA_DESC_DONE_MASK))
 			break;
 
 		dma_rmb();
+		desc_done++;
 
 		q->tail = (q->tail + 1) % q->ndesc;
 		q->queued--;
 
-		dma_sync_single_for_cpu(eth->dev, e->dma_addr, e->dma_len,
-					dir);
+		if (!q->rx_coherent)
+			dma_sync_single_for_cpu(eth->dev, e->dma_addr,
+						e->dma_len, dir);
+
+		if (!q->rx_coherent)
+			page = virt_to_head_page(e->buf);
+
+		if (q->rx_drop_chain) {
+			if (!FIELD_GET(QDMA_DESC_MORE_MASK, desc_ctrl)) {
+				q->rx_drop_chain = false;
+				q->rx_frame_descs = 0;
+				done++;
+			}
+			if (!q->rx_coherent)
+				page_pool_put_full_page(q->page_pool, page,
+							true);
+			continue;
+		}
 
-		page = virt_to_head_page(e->buf);
 		len = FIELD_GET(QDMA_DESC_LEN_MASK, desc_ctrl);
-		data_len = q->skb ? AIROHA_RX_LEN(q->buf_size) : e->dma_len;
+		data_len = q->skb && !q->rx_coherent ?
+				   AIROHA_RX_LEN(q->buf_size) :
+				   e->dma_len;
 		if (!len || data_len < len)
 			goto free_frag;
 
-		dev = airoha_qdma_get_gdm_dev(eth, desc);
-		if (IS_ERR(dev))
-			goto free_frag;
+		msg0 = le32_to_cpu(READ_ONCE(desc->msg0));
+		msg1 = le32_to_cpu(READ_ONCE(desc->msg1));
+		msg2 = le32_to_cpu(READ_ONCE(desc->msg2));
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+		soe_pkt = partial_soe || airoha_qdma_rx_is_soe(msg0);
+#endif
+		if (!soe_pkt) {
+			dev = airoha_qdma_get_gdm_dev(eth, desc);
+			if (IS_ERR(dev))
+				goto free_frag;
+			netdev = netdev_from_priv(dev);
+			rx_dev = netdev;
+		}
 
-		netdev = netdev_from_priv(dev);
 		if (!q->skb) { /* first buffer */
-			q->skb = napi_build_skb(e->buf - AIROHA_RX_HEADROOM,
-						q->buf_size);
+			if (q->rx_coherent) {
+				q->skb = napi_alloc_skb(&q->napi, len);
+				if (q->skb)
+					skb_put_data(q->skb, e->buf, len);
+			} else {
+				void *buf = e->buf - AIROHA_RX_HEADROOM;
+
+				q->skb = napi_build_skb(buf, q->buf_size);
+			}
 			if (!q->skb)
 				goto free_frag;
 
-			skb_reserve(q->skb, AIROHA_RX_HEADROOM);
-			__skb_put(q->skb, len);
-			skb_mark_for_recycle(q->skb);
-			q->skb->dev = netdev;
-			q->skb->protocol = eth_type_trans(q->skb, netdev);
-			q->skb->ip_summed = CHECKSUM_UNNECESSARY;
+			q->rx_drop_chain = false;
+			q->rx_frame_descs = 1;
+			if (!q->rx_coherent) {
+				skb_reserve(q->skb, AIROHA_RX_HEADROOM);
+				__skb_put(q->skb, len);
+				skb_mark_for_recycle(q->skb);
+			}
+			q->skb->dev = soe_pkt ? NULL : netdev;
+			q->skb->ip_summed = soe_pkt ? CHECKSUM_NONE :
+						      CHECKSUM_UNNECESSARY;
 			skb_record_rx_queue(q->skb, qid);
+			if (soe_pkt) {
+				q->soe_rx_msg0 = msg0;
+				q->soe_rx_msg2 = msg2;
+			}
+			if (!soe_pkt)
+				q->skb->protocol = eth_type_trans(q->skb,
+								  netdev);
 		} else { /* scattered frame */
-			struct skb_shared_info *shinfo = skb_shinfo(q->skb);
-			int nr_frags = shinfo->nr_frags;
-
-			if (nr_frags >= ARRAY_SIZE(shinfo->frags))
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+			if (soe_pkt && airoha_soe_rx_max_frame_descs > 0 &&
+			    q->rx_frame_descs >=
+				    airoha_soe_rx_max_frame_descs) {
+				q->rx_drop_chain =
+					FIELD_GET(QDMA_DESC_MORE_MASK, desc_ctrl);
 				goto free_frag;
-
-			skb_add_rx_frag(q->skb, nr_frags, page,
-					e->buf - page_address(page), len,
-					q->buf_size);
+			}
+#endif
+			q->rx_frame_descs++;
+			if (q->rx_coherent) {
+				if (skb_tailroom(q->skb) < len) {
+					unsigned int needed;
+
+					needed = len - skb_tailroom(q->skb);
+					if (pskb_expand_head(q->skb, 0, needed,
+							     GFP_ATOMIC))
+						goto free_frag;
+				}
+				skb_put_data(q->skb, e->buf, len);
+			} else {
+				struct skb_shared_info *shinfo =
+					skb_shinfo(q->skb);
+				int nr_frags = shinfo->nr_frags;
+
+				if (nr_frags >= ARRAY_SIZE(shinfo->frags))
+					goto free_frag;
+
+				skb_add_rx_frag(q->skb, nr_frags, page,
+						e->buf - page_address(page),
+						len, q->buf_size);
+			}
 		}
 
 		if (FIELD_GET(QDMA_DESC_MORE_MASK, desc_ctrl))
 			continue;
 
+		q->rx_drop_chain = false;
+		q->rx_frame_descs = 0;
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+		if (soe_pkt) {
+			u32 hop_flags = FIELD_GET(QDMA_ETH_RXMSG_HOP_FLAGS_MASK,
+						  q->soe_rx_msg0);
+			u32 sa_index = FIELD_GET(QDMA_ETH_RXMSG_SW_UDF_MASK,
+						 q->soe_rx_msg2);
+
+			done++;
+			if (!airoha_soe_rx_skb(eth->soe, q->skb, sa_index,
+					       hop_flags))
+				dev_kfree_skb(q->skb);
+			q->skb = NULL;
+			continue;
+		}
+#endif
+
 		if (netdev_uses_dsa(netdev)) {
 			struct airoha_gdm_port *port = dev->port;
+			struct dsa_port *cpu_dp = netdev->dsa_ptr;
+			u32 sptag = FIELD_GET(QDMA_ETH_RXMSG_SPTAG, msg0);
 
 			/* PPE module requires untagged packets to work
 			 * properly and it provides DSA port index via the
 			 * DMA descriptor. Report DSA tag to the DSA stack
 			 * via skb dst info.
 			 */
-			u32 msg0 = le32_to_cpu(READ_ONCE(desc->msg0));
-			u32 sptag = FIELD_GET(QDMA_ETH_RXMSG_SPTAG, msg0);
-
 			if (sptag < ARRAY_SIZE(port->dsa_meta) &&
 			    port->dsa_meta[sptag])
 				skb_dst_set_noref(q->skb,
 						  &port->dsa_meta[sptag]->dst);
+
+			if (cpu_dp && cpu_dp->ds) {
+				struct dsa_port *dp =
+					dsa_to_port(cpu_dp->ds, sptag);
+
+				if (dp && dsa_port_is_user(dp) &&
+				    dp->cpu_dp == cpu_dp && dp->user)
+					rx_dev = dp->user;
+			}
 		}
 
-		msg1 = le32_to_cpu(READ_ONCE(desc->msg1));
 		hash = FIELD_GET(AIROHA_RXD4_FOE_ENTRY, msg1);
 		if (hash != AIROHA_RXD4_FOE_ENTRY)
 			skb_set_hash(q->skb, jhash_1word(hash, 0),
@@ -749,18 +1037,54 @@ static int airoha_qdma_rx_process(struct airoha_queue *q, int budget)
 					     false);
 
 		done++;
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+		/* Native ESP/NAT-T packets enter through the normal RX path
+		 * first. If they match an inbound packet-offload SA, consume
+		 * the encrypted skb here and resubmit it to SOE before GRO
+		 * takes ownership of the packet. SOE decrypts the original
+		 * ESP/NAT-T packet; after GRO or the normal RX stack processes
+		 * the skb, it is no longer a suitable hardware decrypt
+		 * candidate. Keep the RX FOE hash/reason so the decrypt
+		 * completion can later bind the PPE flow after egress is known.
+		 */
+		if (hash != AIROHA_RXD4_FOE_ENTRY) {
+			bool foe_valid = false;
+
+			if (reason < 32)
+				foe_valid = AIROHA_SOE_DECRYPT_FOE_REASON_MASK &
+					    BIT(reason);
+			if (airoha_soe_rx_plain_skb(dev, q->skb, rx_dev, hash,
+						    reason, foe_valid)) {
+				q->skb = NULL;
+				continue;
+			}
+		} else if (airoha_soe_rx_plain_skb(dev, q->skb, rx_dev, hash,
+						   reason, false)) {
+			q->skb = NULL;
+			continue;
+		}
+#endif
 		napi_gro_receive(&q->napi, q->skb);
 		q->skb = NULL;
 		continue;
 free_frag:
+		q->rx_drop_chain = !!FIELD_GET(QDMA_DESC_MORE_MASK, desc_ctrl);
+		q->rx_frame_descs = 0;
+		done++;
 		if (q->skb) {
 			dev_kfree_skb(q->skb);
 			q->skb = NULL;
 		}
-		page_pool_put_full_page(q->page_pool, page, true);
+		if (!q->rx_coherent)
+			page_pool_put_full_page(q->page_pool, page, true);
 	}
 	airoha_qdma_fill_rx_queue(q);
 
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (desc_done && !done)
+		return 1;
+#endif
+
 	return done;
 }
 
@@ -776,17 +1100,9 @@ static int airoha_qdma_rx_napi_poll(struct napi_struct *napi, int budget)
 
 	if (done < budget && napi_complete(napi)) {
 		struct airoha_qdma *qdma = q->qdma;
-		int i, qid = q - &qdma->q_rx[0];
-		int intr_reg = qid < RX_DONE_HIGH_OFFSET ? QDMA_INT_REG_IDX1
-							 : QDMA_INT_REG_IDX2;
-
-		for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) {
-			if (!(BIT(qid) & RX_IRQ_BANK_PIN_MASK(i)))
-				continue;
+		int qid = q - &qdma->q_rx[0];
 
-			airoha_qdma_irq_enable(&qdma->irq_banks[i], intr_reg,
-					       BIT(qid % RX_DONE_HIGH_OFFSET));
-		}
+		airoha_qdma_set_rx_done_irq(qdma, qid, true);
 	}
 
 	return done;
@@ -795,7 +1111,7 @@ static int airoha_qdma_rx_napi_poll(struct napi_struct *napi, int budget)
 static int airoha_qdma_init_rx_queue(struct airoha_queue *q,
 				     struct airoha_qdma *qdma, int ndesc)
 {
-	const struct page_pool_params pp_params = {
+	struct page_pool_params pp_params = {
 		.order = 0,
 		.pool_size = 256,
 		.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV,
@@ -809,8 +1125,21 @@ static int airoha_qdma_init_rx_queue(struct airoha_queue *q,
 	int qid = q - &qdma->q_rx[0], thr;
 	dma_addr_t dma_addr;
 
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (qid == AIROHA_SOE_RX_RING) {
+		ndesc = max(ndesc, AIROHA_SOE_RX_ALLOC_NDESC);
+		if (airoha_soe_probe_rx_ndesc > 0)
+			ndesc = clamp(airoha_soe_probe_rx_ndesc, 1,
+				      AIROHA_SOE_RX_ALLOC_NDESC);
+	}
+#endif
+
 	q->buf_size = PAGE_SIZE / 2;
 	q->qdma = qdma;
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (qid == AIROHA_SOE_RX_RING)
+		q->rx_coherent = airoha_soe_probe_rx_coherent;
+#endif
 
 	q->entry = devm_kzalloc(eth->dev, ndesc * sizeof(*q->entry),
 				GFP_KERNEL);
@@ -821,19 +1150,45 @@ static int airoha_qdma_init_rx_queue(struct airoha_queue *q,
 				      &dma_addr, GFP_KERNEL);
 	if (!q->desc)
 		return -ENOMEM;
+	q->desc_dma = dma_addr;
 
-	q->page_pool = page_pool_create(&pp_params);
-	if (IS_ERR(q->page_pool)) {
-		int err = PTR_ERR(q->page_pool);
+	if (!q->rx_coherent) {
+		q->page_pool = page_pool_create(&pp_params);
+		if (IS_ERR(q->page_pool)) {
+			int err = PTR_ERR(q->page_pool);
 
-		q->page_pool = NULL;
-		return err;
+			q->page_pool = NULL;
+			return err;
+		}
+	}
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (qid == AIROHA_SOE_RX_RING) {
+		size_t buf_size;
+		int max_buf_size;
+
+		q->rx_alloc_ndesc = ndesc;
+		max_buf_size = q->rx_coherent ? AIROHA_SOE_RX_BUF_SIZE_MAX :
+						AIROHA_SOE_RX_BUF_SIZE;
+		q->buf_size = clamp(airoha_soe_probe_rx_buf_size,
+				    AIROHA_SOE_RX_BUF_SIZE_MIN, max_buf_size);
+		if (q->rx_coherent) {
+			buf_size = q->buf_size * ndesc;
+			q->rx_coherent_buf =
+				dmam_alloc_coherent(eth->dev, buf_size,
+						    &q->rx_coherent_dma,
+						    GFP_KERNEL);
+			if (!q->rx_coherent_buf)
+				return -ENOMEM;
+			q->rx_coherent_buf_size = buf_size;
+		}
 	}
+#endif
 
 	q->ndesc = ndesc;
 	netif_napi_add(eth->napi_dev, &q->napi, airoha_qdma_rx_napi_poll);
 
-	airoha_qdma_wr(qdma, REG_RX_RING_BASE(qid), dma_addr);
+	airoha_qdma_wr(qdma, REG_RX_RING_BASE(qid), q->desc_dma);
 	airoha_qdma_rmw(qdma, REG_RX_RING_SIZE(qid),
 			RX_RING_SIZE_MASK,
 			FIELD_PREP(RX_RING_SIZE_MASK, ndesc));
@@ -843,7 +1198,14 @@ static int airoha_qdma_init_rx_queue(struct airoha_queue *q,
 			FIELD_PREP(RX_RING_THR_MASK, thr));
 	airoha_qdma_rmw(qdma, REG_RX_DMA_IDX(qid), RX_RING_DMA_IDX_MASK,
 			FIELD_PREP(RX_RING_DMA_IDX_MASK, q->head));
-	airoha_qdma_set(qdma, REG_RX_SCATTER_CFG(qid), RX_RING_SG_EN_MASK);
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (qid == AIROHA_SOE_RX_RING && !airoha_soe_probe_rx_scatter)
+		airoha_qdma_clear(qdma, REG_RX_SCATTER_CFG(qid),
+				  RX_RING_SG_EN_MASK);
+	else
+#endif
+		airoha_qdma_set(qdma, REG_RX_SCATTER_CFG(qid),
+				RX_RING_SG_EN_MASK);
 
 	airoha_qdma_fill_rx_queue(q);
 
@@ -859,11 +1221,14 @@ static void airoha_qdma_cleanup_rx_queue(struct airoha_queue *q)
 	while (q->queued) {
 		struct airoha_queue_entry *e = &q->entry[q->tail];
 		struct airoha_qdma_desc *desc = &q->desc[q->tail];
-		struct page *page = virt_to_head_page(e->buf);
 
-		dma_sync_single_for_cpu(eth->dev, e->dma_addr, e->dma_len,
-					page_pool_get_dma_dir(q->page_pool));
-		page_pool_put_full_page(q->page_pool, page, false);
+		if (!q->rx_coherent) {
+			struct page *page = virt_to_head_page(e->buf);
+
+			dma_sync_single_for_cpu(eth->dev, e->dma_addr, e->dma_len,
+						page_pool_get_dma_dir(q->page_pool));
+			page_pool_put_full_page(q->page_pool, page, false);
+		}
 		/* Reset DMA descriptor */
 		WRITE_ONCE(desc->ctrl, 0);
 		WRITE_ONCE(desc->addr, 0);
@@ -1045,8 +1410,24 @@ static int airoha_qdma_tx_napi_poll(struct napi_struct *napi, int budget)
 			airoha_qdma_rmw(qdma, REG_IRQ_CLEAR_LEN(id),
 					IRQ_CLEAR_LEN_MASK, 0x80);
 		airoha_qdma_rmw(qdma, REG_IRQ_CLEAR_LEN(id),
-				IRQ_CLEAR_LEN_MASK, (done & 0x7f));
+				IRQ_CLEAR_LEN_MASK,
+				(done & 0x7f));
+	}
+
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (done && airoha_soe_has_pending_rx(eth->soe)) {
+		int i;
+
+		/* SOE decrypt completions can lag behind TX cleanup IRQs. */
+		for (i = 0; i < ARRAY_SIZE(eth->qdma); i++) {
+			struct airoha_queue *rxq =
+				&eth->qdma[i].q_rx[AIROHA_SOE_RX_RING];
+
+			if (rxq->ndesc)
+				napi_schedule(&rxq->napi);
+		}
 	}
+#endif
 
 	if (done < budget && napi_complete(napi))
 		airoha_qdma_irq_enable(&qdma->irq_banks[0], QDMA_INT_REG_IDX0,
@@ -1346,16 +1727,11 @@ static int airoha_qdma_hw_init(struct airoha_qdma *qdma)
 	for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) {
 		/* clear pending irqs */
 		airoha_qdma_wr(qdma, REG_INT_STATUS(i), 0xffffffff);
-		/* setup rx irqs */
-		airoha_qdma_irq_enable(&qdma->irq_banks[i], QDMA_INT_REG_IDX0,
-				       INT_RX0_MASK(RX_IRQ_BANK_PIN_MASK(i)));
-		airoha_qdma_irq_enable(&qdma->irq_banks[i], QDMA_INT_REG_IDX1,
-				       INT_RX1_MASK(RX_IRQ_BANK_PIN_MASK(i)));
-		airoha_qdma_irq_enable(&qdma->irq_banks[i], QDMA_INT_REG_IDX2,
-				       INT_RX2_MASK(RX_IRQ_BANK_PIN_MASK(i)));
-		airoha_qdma_irq_enable(&qdma->irq_banks[i], QDMA_INT_REG_IDX3,
-				       INT_RX3_MASK(RX_IRQ_BANK_PIN_MASK(i)));
 	}
+	airoha_qdma_apply_rx_irq_bank_masks(qdma);
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	airoha_qdma_set_rx_done_irq(qdma, AIROHA_SOE_RX_RING, true);
+#endif
 	/* setup tx irqs */
 	airoha_qdma_irq_enable(&qdma->irq_banks[0], QDMA_INT_REG_IDX0,
 			       TX_COHERENT_LOW_INT_MASK | INT_TX_MASK);
@@ -2176,6 +2552,110 @@ int airoha_get_fe_port(struct airoha_gdm_dev *dev)
 	}
 }
 
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+int airoha_qdma_xmit_skb(struct airoha_gdm_dev *dev, struct sk_buff *skb,
+			 u32 msg0, u32 msg1, u32 msg2)
+{
+	struct net_device *netdev = netdev_from_priv(dev);
+	struct airoha_qdma *qdma = dev->qdma;
+	u32 nr_frags, len;
+	struct airoha_queue_entry *e;
+	struct netdev_queue *txq;
+	struct airoha_queue *q;
+	LIST_HEAD(tx_list);
+	int i = 0, qid;
+	void *data;
+	u16 index;
+
+	qid = airoha_qdma_get_txq(qdma, skb_get_queue_mapping(skb));
+	q = &qdma->q_tx[qid];
+	if (WARN_ON_ONCE(!q->ndesc))
+		return -ENODEV;
+
+	spin_lock_bh(&q->lock);
+
+	txq = skb_get_tx_queue(netdev, skb);
+	nr_frags = 1 + skb_shinfo(skb)->nr_frags;
+	if (q->queued + nr_frags >= q->ndesc) {
+		netif_tx_stop_queue(txq);
+		q->txq_stopped = true;
+		spin_unlock_bh(&q->lock);
+		return -EBUSY;
+	}
+
+	len = skb_headlen(skb);
+	data = skb->data;
+
+	e = list_first_entry(&q->tx_list, struct airoha_queue_entry, list);
+	index = e - q->entry;
+
+	while (true) {
+		struct airoha_qdma_desc *desc = &q->desc[index];
+		skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+		dma_addr_t addr;
+		u32 val;
+
+		addr = dma_map_single(netdev->dev.parent, data, len,
+				      DMA_TO_DEVICE);
+		if (unlikely(dma_mapping_error(netdev->dev.parent, addr)))
+			goto error_unmap;
+
+		list_move_tail(&e->list, &tx_list);
+		e->skb = i == nr_frags - 1 ? skb : NULL;
+		e->dma_addr = addr;
+		e->dma_len = len;
+
+		e = list_first_entry(&q->tx_list, struct airoha_queue_entry,
+				     list);
+		index = e - q->entry;
+
+		val = FIELD_PREP(QDMA_DESC_LEN_MASK, len);
+		if (i < nr_frags - 1)
+			val |= FIELD_PREP(QDMA_DESC_MORE_MASK, 1);
+		WRITE_ONCE(desc->ctrl, cpu_to_le32(val));
+		WRITE_ONCE(desc->addr, cpu_to_le32(addr));
+		val = FIELD_PREP(QDMA_DESC_NEXT_ID_MASK, index);
+		WRITE_ONCE(desc->data, cpu_to_le32(val));
+		WRITE_ONCE(desc->msg0, cpu_to_le32(msg0));
+		WRITE_ONCE(desc->msg1, cpu_to_le32(msg1));
+		WRITE_ONCE(desc->msg2, cpu_to_le32(msg2));
+
+		if (++i == nr_frags)
+			break;
+
+		data = skb_frag_address(frag);
+		len = skb_frag_size(frag);
+	}
+	q->queued += i;
+
+	skb_tx_timestamp(skb);
+	netdev_tx_sent_queue(txq, skb->len);
+	if (q->ndesc - q->queued < q->free_thr) {
+		netif_tx_stop_queue(txq);
+		q->txq_stopped = true;
+	}
+
+	/* SOE submits do not run in the regular ndo_start_xmit batching path. */
+	airoha_qdma_rmw(qdma, REG_TX_CPU_IDX(qid), TX_RING_CPU_IDX_MASK,
+			FIELD_PREP(TX_RING_CPU_IDX_MASK, index));
+
+	spin_unlock_bh(&q->lock);
+
+	return 0;
+
+error_unmap:
+	list_for_each_entry(e, &tx_list, list) {
+		dma_unmap_single(netdev->dev.parent, e->dma_addr, e->dma_len,
+				 DMA_TO_DEVICE);
+		e->dma_addr = 0;
+	}
+	list_splice(&tx_list, &q->tx_list);
+	spin_unlock_bh(&q->lock);
+
+	return -ENOMEM;
+}
+#endif
+
 static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
 				   struct net_device *netdev)
 {
@@ -2185,6 +2665,7 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
 	struct airoha_queue_entry *e;
 	struct netdev_queue *txq;
 	struct airoha_queue *q;
+	bool soe_decrypt_skb = false;
 	LIST_HEAD(tx_list);
 	int i = 0, qid;
 	void *data;
@@ -2223,6 +2704,11 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
 	       FIELD_PREP(QDMA_ETH_TXMSG_FPORT_MASK, fport) |
 	       FIELD_PREP(QDMA_ETH_TXMSG_METER_MASK, 0x7f);
 
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (dev->eth->ppe)
+		soe_decrypt_skb = airoha_ppe_soe_skb_marked(skb);
+#endif
+
 	q = &qdma->q_tx[qid];
 	if (WARN_ON_ONCE(!q->ndesc))
 		goto error;
@@ -2293,13 +2779,24 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
 		q->txq_stopped = true;
 	}
 
-	if (netif_xmit_stopped(txq) || !netdev_xmit_more())
+	if (netif_xmit_stopped(txq) || !netdev_xmit_more() ||
+	    soe_decrypt_skb)
 		airoha_qdma_rmw(qdma, REG_TX_CPU_IDX(qid),
 				TX_RING_CPU_IDX_MASK,
 				FIELD_PREP(TX_RING_CPU_IDX_MASK, index));
 
 	spin_unlock_bh(&q->lock);
 
+	/* SOE decrypt flow binding needs the final egress netdev and QDMA
+	 * descriptor context. The SOE RX path marks only candidate packets; bind
+	 * only after the current plaintext packet has been queued and kicked so
+	 * the newly bound decrypt entry cannot race the CPU-transmitted packet.
+	 */
+#if IS_REACHABLE(CONFIG_NET_AIROHA_SOE)
+	if (soe_decrypt_skb)
+		airoha_ppe_soe_xmit_skb(&dev->eth->ppe->dev, skb, netdev);
+#endif
+
 	return NETDEV_TX_OK;
 
 error_unmap:
@@ -3096,6 +3593,12 @@ static int airoha_dev_tc_setup(struct net_device *dev,
 	}
 }
 
+static int airoha_dev_set_features(struct net_device *netdev,
+				   netdev_features_t features)
+{
+	return airoha_soe_set_features(netdev, features);
+}
+
 static const struct net_device_ops airoha_netdev_ops = {
 	.ndo_init		= airoha_dev_init,
 	.ndo_open		= airoha_dev_open,
@@ -3105,6 +3608,7 @@ static const struct net_device_ops airoha_netdev_ops = {
 	.ndo_start_xmit		= airoha_dev_xmit,
 	.ndo_get_stats64        = airoha_dev_get_stats64,
 	.ndo_set_mac_address	= airoha_dev_set_macaddr,
+	.ndo_set_features	= airoha_dev_set_features,
 	.ndo_setup_tc		= airoha_dev_tc_setup,
 };
 
@@ -3230,6 +3734,7 @@ static int airoha_alloc_gdm_device(struct airoha_eth *eth,
 	dev->eth = eth;
 	dev->nbq = nbq;
 	port->devs[index] = dev;
+	airoha_soe_build_netdev(netdev, airoha_qdma_xmit_skb);
 
 	return 0;
 }
@@ -3409,10 +3914,14 @@ static int airoha_probe(struct platform_device *pdev)
 	strscpy(eth->napi_dev->name, "qdma_eth", sizeof(eth->napi_dev->name));
 	platform_set_drvdata(pdev, eth);
 
-	err = airoha_hw_init(pdev, eth);
+	err = airoha_soe_init(eth);
 	if (err)
 		goto error_netdev_free;
 
+	err = airoha_hw_init(pdev, eth);
+	if (err)
+		goto error_soe_deinit;
+
 	for (i = 0; i < ARRAY_SIZE(eth->qdma); i++)
 		airoha_qdma_start_napi(&eth->qdma[i]);
 
@@ -3457,11 +3966,14 @@ static int airoha_probe(struct platform_device *pdev)
 			netdev = netdev_from_priv(dev);
 			if (netdev->reg_state == NETREG_REGISTERED)
 				unregister_netdev(netdev);
+			airoha_soe_teardown_netdev(netdev);
 			of_node_put(netdev->dev.of_node);
 		}
 		airoha_metadata_dst_free(port);
 	}
 	airoha_hw_cleanup(eth);
+error_soe_deinit:
+	airoha_soe_deinit(eth);
 error_netdev_free:
 	free_netdev(eth->napi_dev);
 	platform_set_drvdata(pdev, NULL);
@@ -3492,12 +4004,14 @@ static void airoha_remove(struct platform_device *pdev)
 				continue;
 
 			netdev = netdev_from_priv(dev);
+			airoha_soe_teardown_netdev(netdev);
 			unregister_netdev(netdev);
 			of_node_put(netdev->dev.of_node);
 		}
 		airoha_metadata_dst_free(port);
 	}
 	airoha_hw_cleanup(eth);
+	airoha_soe_deinit(eth);
 
 	free_netdev(eth->napi_dev);
 	platform_set_drvdata(pdev, NULL);
-- 
2.53.0

[RFC PATCH net-next 6/7] net: airoha: add PPE support for SOE flows

[Jihong Min]

Add PPE metadata handling for SOE flows so decrypted packets can carry
their original FOE/SA context until the normal egress path is known, and
so XFRM flowtable entries can be programmed with SOE SA and hop
information.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 drivers/net/ethernet/airoha/airoha_ppe.c  | 606 +++++++++++++++++++++-
 include/linux/soc/airoha/airoha_offload.h |   5 +
 2 files changed, 585 insertions(+), 26 deletions(-)

diff --git a/drivers/net/ethernet/airoha/airoha_ppe.c b/drivers/net/ethernet/airoha/airoha_ppe.c
index 91bcc55a6ac6..faa5f04d4c7b 100644
--- a/drivers/net/ethernet/airoha/airoha_ppe.c
+++ b/drivers/net/ethernet/airoha/airoha_ppe.c
@@ -6,18 +6,77 @@
 
 #include <linux/ip.h>
 #include <linux/ipv6.h>
+#include <linux/kstrtox.h>
+#include <linux/moduleparam.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>
 #include <linux/rhashtable.h>
+#include <linux/sysfs.h>
+#include <linux/tcp.h>
+#include <linux/udp.h>
 #include <net/ipv6.h>
+#include <net/netfilter/nf_flow_table.h>
 #include <net/pkt_cls.h>
+#include <net/route.h>
 
 #include "airoha_regs.h"
 #include "airoha_eth.h"
+#include "airoha_soe.h"
 
 static DEFINE_MUTEX(flow_offload_mutex);
 static DEFINE_SPINLOCK(ppe_lock);
 
+#define AIROHA_FOE_HOP0			GENMASK(31, 29)
+#define AIROHA_FOE_HOP1			GENMASK(28, 26)
+#define AIROHA_FOE_HOP2			GENMASK(25, 23)
+#define AIROHA_FOE_HOP3			GENMASK(22, 20)
+#define AIROHA_FOE_HOP_MASK						\
+	(AIROHA_FOE_HOP0 | AIROHA_FOE_HOP1 | AIROHA_FOE_HOP2 |		\
+	 AIROHA_FOE_HOP3)
+#define AIROHA_PPE_SOE_DEFAULT_TUNNEL_MTU	1500
+#define AIROHA_PPE_SOE_MAGIC_IPSEC		0x72a1
+#define AIROHA_PPE_SOE_PORT_AG			0x3f
+#define AIROHA_PPE_SOE_CHANNEL			2
+#define AIROHA_PPE_SOE_META_TIMEOUT_MS		1000
+#define AIROHA_PPE_SOE_MAGIC_GDM4		0x729a
+#define AIROHA_PPE_SOE_MARK_MAGIC		0x5e000000
+#define AIROHA_PPE_SOE_MARK_MAGIC_MASK		0xff000000
+#define AIROHA_PPE_SOE_MARK_HASH_MASK		0x00ffff00
+#define AIROHA_PPE_DEFAULT_BIND_RATE		0x1e
+#define AIROHA_PPE_SOE_BIND_DELAY_PACKETS	2
+#define AIROHA_PPE_FORCE_COMMIT_PROBE_WINDOW	0
+
+struct airoha_ppe_soe_tuple_info {
+	unsigned int tunnel_mtu;
+	u8 sa_index;
+	u8 hop;
+};
+
+static unsigned int airoha_ppe_bind_rate = AIROHA_PPE_DEFAULT_BIND_RATE;
+static unsigned int airoha_ppe_soe_inline_bind_delay_packets =
+	AIROHA_PPE_SOE_BIND_DELAY_PACKETS;
+static unsigned int airoha_ppe_soe_inline_force_commit_probe_window =
+	AIROHA_PPE_FORCE_COMMIT_PROBE_WINDOW;
+static struct airoha_ppe *airoha_ppe_active;
+
+static int airoha_ppe_set_bind_rate(const char *val,
+				    const struct kernel_param *kp);
+static int airoha_ppe_get_bind_rate(char *buf, const struct kernel_param *kp);
+
+module_param_named(soe_inline_force_commit_probe_window,
+		   airoha_ppe_soe_inline_force_commit_probe_window, uint, 0600);
+MODULE_PARM_DESC(soe_inline_force_commit_probe_window,
+		 "Adjacent FOE slots searched before force-commit");
+module_param_named(soe_inline_bind_delay_packets,
+		   airoha_ppe_soe_inline_bind_delay_packets, uint, 0600);
+MODULE_PARM_DESC(soe_inline_bind_delay_packets,
+		 "CPU-visible SOE decrypt packets before binding FOE entry");
+module_param_call(ppe_bind_rate, airoha_ppe_set_bind_rate,
+		  airoha_ppe_get_bind_rate, NULL, 0600);
+__MODULE_PARM_TYPE(ppe_bind_rate, "uint");
+MODULE_PARM_DESC(ppe_bind_rate,
+		 "PPE bind-rate threshold for L2B and bind fields");
+
 static const struct rhashtable_params airoha_flow_table_params = {
 	.head_offset = offsetof(struct airoha_flow_table_entry, node),
 	.key_offset = offsetof(struct airoha_flow_table_entry, cookie),
@@ -78,6 +137,17 @@ bool airoha_ppe_is_enabled(struct airoha_eth *eth, int index)
 	return airoha_fe_rr(eth, REG_PPE_GLO_CFG(index)) & PPE_GLO_CFG_EN_MASK;
 }
 
+static void airoha_ppe_apply_bind_rate(struct airoha_eth *eth, int ppe_idx)
+{
+	u32 rate = min_t(u32, READ_ONCE(airoha_ppe_bind_rate),
+			 FIELD_MAX(PPE_BIND_RATE_BIND_MASK));
+
+	airoha_fe_rmw(eth, REG_PPE_BIND_RATE(ppe_idx),
+		      PPE_BIND_RATE_L2B_BIND_MASK | PPE_BIND_RATE_BIND_MASK,
+		      FIELD_PREP(PPE_BIND_RATE_L2B_BIND_MASK, rate) |
+			      FIELD_PREP(PPE_BIND_RATE_BIND_MASK, rate));
+}
+
 static u32 airoha_ppe_get_timestamp(struct airoha_ppe *ppe)
 {
 	return airoha_fe_get(ppe->eth, REG_FE_FOE_TS,
@@ -157,15 +227,14 @@ static void airoha_ppe_hw_init(struct airoha_ppe *ppe)
 			      FIELD_PREP(PPE_DRAM_TB_NUM_ENTRY_MASK,
 					 dram_num_entries));
 
-		airoha_fe_rmw(eth, REG_PPE_BIND_RATE(i),
-			      PPE_BIND_RATE_L2B_BIND_MASK |
-			      PPE_BIND_RATE_BIND_MASK,
-			      FIELD_PREP(PPE_BIND_RATE_L2B_BIND_MASK, 0x1e) |
-			      FIELD_PREP(PPE_BIND_RATE_BIND_MASK, 0x1e));
+		airoha_ppe_apply_bind_rate(eth, i);
 
 		airoha_fe_wr(eth, REG_PPE_HASH_SEED(i), PPE_HASH_SEED);
 		airoha_fe_clear(eth, REG_PPE_PPE_FLOW_CFG(i),
 				PPE_FLOW_CFG_IP6_6RD_MASK);
+		airoha_fe_set(eth, REG_PPE_PPE_FLOW_CFG(i),
+			      PPE_FLOW_CFG_IP4_IPSEC_MASK |
+				      PPE_FLOW_CFG_IP6_IPSEC_MASK);
 
 		for (p = 0; p < ARRAY_SIZE(eth->ports); p++)
 			airoha_fe_rmw(eth, REG_PPE_MTU(i, p),
@@ -509,6 +578,162 @@ static int airoha_ppe_foe_entry_set_ipv6_tuple(struct airoha_foe_entry *hwe,
 	return 0;
 }
 
+static int airoha_ppe_soe_fill_inner_ipv4_data(struct sk_buff *skb,
+					       struct airoha_flow_data *data,
+					       int *type, int *l4proto)
+{
+	unsigned int ip_offset = ETH_HLEN;
+	union {
+		struct tcphdr tcp;
+		struct udphdr udp;
+	} ports;
+	struct iphdr iph_buf, *iph;
+	unsigned int l4_offset;
+	struct udphdr *udp;
+	struct tcphdr *tcp;
+
+	if (skb_headlen(skb) < ETH_HLEN)
+		return -EINVAL;
+
+	memcpy(&data->eth, skb->data, ETH_HLEN);
+	if (data->eth.h_proto != htons(ETH_P_IP))
+		return -EAFNOSUPPORT;
+
+	iph = skb_header_pointer(skb, ip_offset, sizeof(iph_buf), &iph_buf);
+	if (!iph || iph->ihl < 5 || iph->version != 4)
+		return -EINVAL;
+
+	l4_offset = ip_offset + iph->ihl * 4;
+	data->v4.src_addr = iph->saddr;
+	data->v4.dst_addr = iph->daddr;
+	*l4proto = iph->protocol;
+
+	switch (iph->protocol) {
+	case IPPROTO_TCP:
+		tcp = skb_header_pointer(skb, l4_offset, sizeof(ports.tcp),
+					 &ports.tcp);
+		if (!tcp)
+			return -EINVAL;
+
+		data->src_port = tcp->source;
+		data->dst_port = tcp->dest;
+		*type = PPE_PKT_TYPE_IPV4_HNAPT;
+		break;
+	case IPPROTO_UDP:
+		udp = skb_header_pointer(skb, l4_offset, sizeof(ports.udp),
+					 &ports.udp);
+		if (!udp)
+			return -EINVAL;
+
+		data->src_port = udp->source;
+		data->dst_port = udp->dest;
+		*type = PPE_PKT_TYPE_IPV4_HNAPT;
+		break;
+	default:
+		*type = PPE_PKT_TYPE_IPV4_ROUTE;
+		break;
+	}
+
+	return 0;
+}
+
+static int airoha_ppe_foe_entry_set_soe_fields(struct airoha_foe_entry *hwe,
+					       u8 sa_index, u8 hop,
+					       unsigned int tunnel_mtu)
+{
+	int type;
+
+	if (hop > FIELD_MAX(AIROHA_FOE_HOP0))
+		return -ERANGE;
+
+	type = FIELD_GET(AIROHA_FOE_IB1_BIND_PACKET_TYPE, hwe->ib1);
+	switch (type) {
+	case PPE_PKT_TYPE_IPV4_HNAPT:
+	case PPE_PKT_TYPE_IPV4_ROUTE:
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	tunnel_mtu = min_t(unsigned int, tunnel_mtu,
+			   FIELD_MAX(AIROHA_FOE_TUNNEL_MTU));
+
+	/* SOE FOE entries store the hop selector and SA index here. */
+	hwe->ipv4.rsv[0] &= ~AIROHA_FOE_HOP_MASK;
+	hwe->ipv4.rsv[0] |= FIELD_PREP(AIROHA_FOE_HOP0, hop);
+
+	hwe->ipv4.data &= ~(AIROHA_FOE_ACTDP | AIROHA_FOE_TUNNEL_ID);
+	hwe->ipv4.data |= AIROHA_FOE_TUNNEL |
+			  FIELD_PREP(AIROHA_FOE_ACTDP, sa_index);
+
+	hwe->ipv4.l2.meter &= ~AIROHA_FOE_TUNNEL_MTU;
+	hwe->ipv4.l2.meter |= FIELD_PREP(AIROHA_FOE_TUNNEL_MTU, tunnel_mtu);
+
+	hwe->ib1 &= ~AIROHA_FOE_IB1_BIND_TUNNEL_DECAP;
+
+	return 0;
+}
+
+static int
+airoha_ppe_soe_get_tuple_info(const struct flow_offload_tuple *tuple,
+			      struct airoha_ppe_soe_tuple_info *info)
+{
+	int err;
+
+	if (!tuple || !info)
+		return -EINVAL;
+	if (tuple->xmit_type != FLOW_OFFLOAD_XMIT_XFRM)
+		return -EOPNOTSUPP;
+
+	err = airoha_soe_xfrm_ppe_info(tuple->dst_cache, &info->sa_index,
+				       &info->hop);
+	if (err)
+		return err;
+
+	info->tunnel_mtu = tuple->mtu ? tuple->mtu :
+				      AIROHA_PPE_SOE_DEFAULT_TUNNEL_MTU;
+
+	return 0;
+}
+
+static int
+airoha_ppe_foe_entry_set_soe_info(struct airoha_foe_entry *hwe,
+				  const struct flow_offload_tuple *tuple)
+{
+	struct airoha_ppe_soe_tuple_info info;
+	int err;
+
+	if (!tuple)
+		return 0;
+	if (tuple->xmit_type != FLOW_OFFLOAD_XMIT_XFRM)
+		return 0;
+
+	err = airoha_ppe_soe_get_tuple_info(tuple, &info);
+	if (err)
+		return err;
+
+	err = airoha_ppe_foe_entry_set_soe_fields(hwe, info.sa_index,
+						  info.hop, info.tunnel_mtu);
+	if (err)
+		return err;
+
+	/* XFRM packet-offload entries are not plain Ethernet/IP entries:
+	 * the PPE must tag them as SOE/IPsec work and submit them through the
+	 * SOE-facing channel/port aggregation path. Without these fields the
+	 * entry can still become BND, but traffic falls back to the slow path
+	 * instead of the inline encrypt/decrypt datapath.
+	 */
+	hwe->ipv4.l2.common.etype = AIROHA_PPE_SOE_MAGIC_IPSEC;
+	hwe->ipv4.data &= ~AIROHA_FOE_CHANNEL;
+	hwe->ipv4.data |= FIELD_PREP(AIROHA_FOE_CHANNEL,
+				     AIROHA_PPE_SOE_CHANNEL);
+	hwe->ipv4.ib2 &= ~AIROHA_FOE_IB2_PORT_AG;
+	hwe->ipv4.ib2 |= FIELD_PREP(AIROHA_FOE_IB2_PORT_AG,
+				    AIROHA_PPE_SOE_PORT_AG);
+
+	return 0;
+}
+
 static u32 airoha_ppe_foe_get_entry_hash(struct airoha_ppe *ppe,
 					 struct airoha_foe_entry *hwe)
 {
@@ -633,6 +858,9 @@ static void airoha_ppe_foe_flow_stats_update(struct airoha_ppe *ppe,
 		meter = &hwe->ipv4.l2.meter;
 	}
 
+	if (*data & AIROHA_FOE_TUNNEL)
+		return;
+
 	pse_port = FIELD_GET(AIROHA_FOE_IB2_PSE_PORT, *ib2);
 	if (pse_port == FE_PSE_PORT_CDM4)
 		return;
@@ -868,16 +1096,129 @@ airoha_ppe_foe_commit_subflow_entry(struct airoha_ppe *ppe,
 	return 0;
 }
 
-static void airoha_ppe_foe_insert_entry(struct airoha_ppe *ppe,
-					struct sk_buff *skb,
-					u32 hash, bool rx_wlan)
+static void airoha_ppe_soe_meta_store(struct airoha_ppe_soe_meta *meta,
+				      u32 key_hash, u16 foe_hash, u8 sa_index,
+				      u8 hop)
 {
-	struct airoha_flow_table_entry *e;
-	struct airoha_foe_bridge br = {};
-	struct airoha_foe_entry *hwe;
-	bool commit_done = false;
-	struct hlist_node *n;
-	u32 index, state;
+	u8 seen = 1;
+
+	if (READ_ONCE(meta->valid) &&
+	    READ_ONCE(meta->key_hash) == key_hash &&
+	    READ_ONCE(meta->foe_hash) == foe_hash &&
+	    READ_ONCE(meta->sa_index) == sa_index &&
+	    READ_ONCE(meta->hop) == hop &&
+	    time_before_eq(jiffies, READ_ONCE(meta->expires)))
+		seen = min_t(u8, READ_ONCE(meta->seen) + 1, U8_MAX);
+
+	WRITE_ONCE(meta->key_hash, key_hash);
+	WRITE_ONCE(meta->foe_hash, foe_hash);
+	WRITE_ONCE(meta->sa_index, sa_index);
+	WRITE_ONCE(meta->hop, hop);
+	WRITE_ONCE(meta->seen, seen);
+	WRITE_ONCE(meta->expires,
+		   jiffies + msecs_to_jiffies(AIROHA_PPE_SOE_META_TIMEOUT_MS));
+	WRITE_ONCE(meta->valid, 1);
+}
+
+void airoha_ppe_soe_mark_skb(struct airoha_ppe_dev *dev, struct sk_buff *skb,
+			     u16 hash, u8 sa_index, u8 hop)
+{
+	struct airoha_ppe *ppe;
+	u32 ppe_hash_mask;
+
+	if (!dev || !skb)
+		return;
+
+	ppe = dev->priv;
+	if (!ppe || !ppe->soe_meta)
+		return;
+
+	ppe_hash_mask = airoha_ppe_get_total_num_entries(ppe) - 1;
+	if (hash > ppe_hash_mask)
+		return;
+
+	/* SOE decrypt completion is CPU-visible before normal routing has
+	 * selected the plaintext egress netdev. Keep the original encrypted FOE
+	 * hash and SA hop briefly on the skb so airoha_dev_xmit() can finish
+	 * the PPE entry once the final egress descriptor is known.
+	 */
+	airoha_ppe_soe_meta_store(&ppe->soe_meta[hash], hash, hash, sa_index,
+				  hop);
+	ppe->foe_check_time[hash] = 0;
+
+	skb->mark &= ~(AIROHA_PPE_SOE_MARK_MAGIC_MASK |
+		       AIROHA_PPE_SOE_MARK_HASH_MASK);
+	skb->mark |= AIROHA_PPE_SOE_MARK_MAGIC |
+		     FIELD_PREP(AIROHA_PPE_SOE_MARK_HASH_MASK, hash);
+}
+
+bool airoha_ppe_soe_skb_marked(struct sk_buff *skb)
+{
+	return skb && ((skb->mark & AIROHA_PPE_SOE_MARK_MAGIC_MASK) ==
+		       AIROHA_PPE_SOE_MARK_MAGIC);
+}
+
+void airoha_ppe_soe_xmit_skb(struct airoha_ppe_dev *dev, struct sk_buff *skb,
+			     struct net_device *netdev)
+{
+	struct airoha_foe_entry entry, tmpl, *hwe;
+	struct airoha_flow_data data = {};
+	struct airoha_ppe_soe_meta *meta;
+	u32 ppe_hash_mask, key_hash;
+	struct airoha_gdm_dev *gdm;
+	struct airoha_ppe *ppe;
+	unsigned long expires;
+	u16 hash;
+	int err, l4proto, type;
+	u8 sa_index, hop;
+	u8 seen;
+
+	if (!dev || !skb || !netdev)
+		return;
+
+	if ((skb->mark & AIROHA_PPE_SOE_MARK_MAGIC_MASK) !=
+	    AIROHA_PPE_SOE_MARK_MAGIC)
+		return;
+
+	ppe = dev->priv;
+	if (!ppe || !ppe->soe_meta)
+		goto clear_mark;
+
+	ppe_hash_mask = airoha_ppe_get_total_num_entries(ppe) - 1;
+	key_hash = FIELD_GET(AIROHA_PPE_SOE_MARK_HASH_MASK, skb->mark);
+	if (key_hash > ppe_hash_mask)
+		goto clear_mark;
+
+	meta = &ppe->soe_meta[key_hash];
+	if (!READ_ONCE(meta->valid))
+		goto clear_mark;
+
+	if (READ_ONCE(meta->key_hash) != key_hash)
+		goto clear_mark;
+
+	expires = READ_ONCE(meta->expires);
+	if (time_after(jiffies, expires)) {
+		WRITE_ONCE(meta->valid, 0);
+		goto clear_mark;
+	}
+
+	hash = READ_ONCE(meta->foe_hash);
+	if (hash > ppe_hash_mask) {
+		WRITE_ONCE(meta->valid, 0);
+		goto clear_mark;
+	}
+
+	seen = READ_ONCE(meta->seen);
+	if (seen <= READ_ONCE(airoha_ppe_soe_inline_bind_delay_packets))
+		goto clear_mark;
+
+	err = airoha_ppe_soe_fill_inner_ipv4_data(skb, &data, &type, &l4proto);
+	if (err)
+		goto clear_mark;
+
+	sa_index = READ_ONCE(meta->sa_index);
+	hop = READ_ONCE(meta->hop);
+	WRITE_ONCE(meta->valid, 0);
 
 	spin_lock_bh(&ppe_lock);
 
@@ -885,13 +1226,120 @@ static void airoha_ppe_foe_insert_entry(struct airoha_ppe *ppe,
 	if (!hwe)
 		goto unlock;
 
-	state = FIELD_GET(AIROHA_FOE_IB1_BIND_STATE, hwe->ib1);
-	if (state == AIROHA_FOE_STATE_BIND)
+	switch (FIELD_GET(AIROHA_FOE_IB1_BIND_PACKET_TYPE, hwe->ib1)) {
+	case PPE_PKT_TYPE_IPV4_HNAPT:
+	case PPE_PKT_TYPE_IPV4_ROUTE:
+		break;
+	default:
 		goto unlock;
+	}
 
-	index = airoha_ppe_foe_get_entry_hash(ppe, hwe);
-	hlist_for_each_entry_safe(e, n, &ppe->foe_flow[index], list) {
+	err = airoha_ppe_foe_entry_prepare(ppe->eth, &tmpl, netdev, type,
+					   &data, l4proto);
+	if (err)
+		goto unlock;
+
+	memcpy(&entry, hwe, sizeof(entry));
+	entry.ib1 &= ~(AIROHA_FOE_IB1_BIND_STATE |
+		       AIROHA_FOE_IB1_BIND_KEEPALIVE |
+		       AIROHA_FOE_IB1_BIND_TIMESTAMP);
+	entry.ib1 |= FIELD_PREP(AIROHA_FOE_IB1_BIND_STATE,
+				AIROHA_FOE_STATE_BIND) |
+		     AIROHA_FOE_IB1_BIND_TTL;
+	entry.ib1 = (entry.ib1 & (AIROHA_FOE_IB1_BIND_PACKET_TYPE |
+				  AIROHA_FOE_IB1_BIND_UDP)) |
+		    (tmpl.ib1 & ~(AIROHA_FOE_IB1_BIND_PACKET_TYPE |
+				  AIROHA_FOE_IB1_BIND_UDP));
+	entry.ipv4.ib2 = tmpl.ipv4.ib2;
+	entry.ipv4.data = tmpl.ipv4.data;
+	memcpy(&entry.ipv4.l2, &tmpl.ipv4.l2, sizeof(entry.ipv4.l2));
+
+	gdm = netdev_priv(netdev);
+	if (gdm->port && gdm->port->id == AIROHA_GDM4_IDX)
+		entry.ipv4.l2.common.etype = AIROHA_PPE_SOE_MAGIC_GDM4;
+
+	if (FIELD_GET(AIROHA_FOE_IB1_BIND_PACKET_TYPE, entry.ib1) ==
+	    PPE_PKT_TYPE_IPV4_HNAPT)
+		memcpy(&entry.ipv4.new_tuple, &entry.ipv4.orig_tuple,
+		       sizeof(entry.ipv4.new_tuple));
+
+	/* Commit the original decrypt entry only after the normal transmit path
+	 * has provided the final plaintext egress descriptor. Binding it at SOE
+	 * RX completion would miss this device-specific L2/PSE state.
+	 */
+	err = airoha_ppe_foe_entry_set_soe_fields(&entry, sa_index, hop,
+						  AIROHA_PPE_SOE_DEFAULT_TUNNEL_MTU);
+	if (!err)
+		airoha_ppe_foe_commit_entry(ppe, &entry, hash, false);
+
+unlock:
+	spin_unlock_bh(&ppe_lock);
+clear_mark:
+	skb->mark &= ~(AIROHA_PPE_SOE_MARK_MAGIC_MASK |
+		       AIROHA_PPE_SOE_MARK_HASH_MASK);
+}
+
+void airoha_ppe_soe_flush_sa(struct airoha_ppe *ppe, u8 sa_index)
+{
+	u32 num_entries, hash;
+
+	if (!ppe)
+		return;
+
+	num_entries = airoha_ppe_get_total_num_entries(ppe);
+
+	spin_lock_bh(&ppe_lock);
+	for (hash = 0; hash < num_entries; hash++) {
+		struct airoha_foe_entry *hwe;
+		u32 state, type;
+
+		hwe = airoha_ppe_foe_get_entry_locked(ppe, hash);
+		if (!hwe)
+			continue;
+
+		state = FIELD_GET(AIROHA_FOE_IB1_BIND_STATE, hwe->ib1);
+		if (state != AIROHA_FOE_STATE_BIND)
+			continue;
+
+		type = FIELD_GET(AIROHA_FOE_IB1_BIND_PACKET_TYPE, hwe->ib1);
+		if (type != PPE_PKT_TYPE_IPV4_HNAPT &&
+		    type != PPE_PKT_TYPE_IPV4_ROUTE)
+			continue;
+
+		if (!(hwe->ipv4.data & AIROHA_FOE_TUNNEL))
+			continue;
+
+		if (FIELD_GET(AIROHA_FOE_ACTDP, hwe->ipv4.data) != sa_index)
+			continue;
+
+		/* NAT-T data and IKE control both use UDP/4500. A stale SOE
+		 * bound entry can otherwise keep sending later IKE_AUTH packets
+		 * to the SOE path after the SA has been deleted.
+		 */
+		hwe->ib1 &= ~AIROHA_FOE_IB1_BIND_STATE;
+		hwe->ib1 |= FIELD_PREP(AIROHA_FOE_IB1_BIND_STATE,
+				       AIROHA_FOE_STATE_INVALID);
+		airoha_ppe_foe_commit_entry(ppe, hwe, hash, false);
+	}
+	spin_unlock_bh(&ppe_lock);
+}
+
+static bool airoha_ppe_foe_try_flow_commit_bucket(struct airoha_ppe *ppe,
+						  struct airoha_foe_entry *hwe,
+						  u32 hash, u32 probe_index,
+						  bool rx_wlan,
+						  bool allow_l2_subflow)
+{
+	struct airoha_flow_table_entry *e;
+	struct hlist_node *n;
+	bool commit_done = false;
+	u32 state;
+
+	hlist_for_each_entry_safe(e, n, &ppe->foe_flow[probe_index], list) {
 		if (e->type == FLOW_TYPE_L2_SUBFLOW) {
+			if (!allow_l2_subflow)
+				continue;
+
 			state = FIELD_GET(AIROHA_FOE_IB1_BIND_STATE, hwe->ib1);
 			if (state != AIROHA_FOE_STATE_BIND) {
 				e->hash = 0xffff;
@@ -908,6 +1356,51 @@ static void airoha_ppe_foe_insert_entry(struct airoha_ppe *ppe,
 		e->hash = hash;
 	}
 
+	return commit_done;
+}
+
+static void airoha_ppe_foe_insert_entry(struct airoha_ppe *ppe,
+					struct sk_buff *skb,
+					u32 hash, bool rx_wlan)
+{
+	struct airoha_flow_table_entry *e;
+	struct airoha_foe_bridge br = {};
+	struct airoha_foe_entry *hwe;
+	bool commit_done = false;
+	u32 index, mask, state, window;
+	unsigned int i;
+
+	spin_lock_bh(&ppe_lock);
+
+	hwe = airoha_ppe_foe_get_entry_locked(ppe, hash);
+	if (!hwe)
+		goto unlock;
+
+	state = FIELD_GET(AIROHA_FOE_IB1_BIND_STATE, hwe->ib1);
+	if (state == AIROHA_FOE_STATE_BIND)
+		goto unlock;
+
+	index = airoha_ppe_foe_get_entry_hash(ppe, hwe);
+	commit_done =
+		airoha_ppe_foe_try_flow_commit_bucket(ppe, hwe, hash, index,
+						      rx_wlan, true);
+
+	mask = airoha_ppe_get_total_num_entries(ppe) - 1;
+	window = min_t(u32,
+		       READ_ONCE(airoha_ppe_soe_inline_force_commit_probe_window),
+		       mask);
+	for (i = 1; !commit_done && i <= window; i++) {
+		u32 candidates[2] = { (index + i) & mask, (index - i) & mask };
+		unsigned int j;
+
+		for (j = 0; !commit_done && j < ARRAY_SIZE(candidates); j++) {
+			commit_done =
+				airoha_ppe_foe_try_flow_commit_bucket(ppe, hwe,
+								      hash, candidates[j],
+								      rx_wlan, false);
+		}
+	}
+
 	if (commit_done)
 		goto unlock;
 
@@ -940,8 +1433,9 @@ airoha_ppe_foe_l2_flow_commit_entry(struct airoha_ppe *ppe,
 				       airoha_l2_flow_table_params);
 }
 
-static int airoha_ppe_foe_flow_commit_entry(struct airoha_ppe *ppe,
-					    struct airoha_flow_table_entry *e)
+static int
+airoha_ppe_foe_flow_commit_entry(struct airoha_ppe *ppe,
+				 struct airoha_flow_table_entry *e)
 {
 	int type = FIELD_GET(AIROHA_FOE_IB1_BIND_PACKET_TYPE, e->data.ib1);
 	u32 hash;
@@ -1057,6 +1551,7 @@ static int airoha_ppe_entry_idle_time(struct airoha_ppe *ppe,
 static int airoha_ppe_flow_offload_replace(struct airoha_eth *eth,
 					   struct flow_cls_offload *f)
 {
+	const struct flow_offload_tuple *tuple = (const void *)f->cookie;
 	struct flow_rule *rule = flow_cls_offload_flow_rule(f);
 	struct airoha_flow_table_entry *e;
 	struct airoha_flow_data data = {};
@@ -1183,7 +1678,9 @@ static int airoha_ppe_flow_offload_replace(struct airoha_eth *eth,
 		flow_rule_match_ipv4_addrs(rule, &addrs);
 		data.v4.src_addr = addrs.key->src;
 		data.v4.dst_addr = addrs.key->dst;
-		airoha_ppe_foe_entry_set_ipv4_tuple(&hwe, &data, false);
+		err = airoha_ppe_foe_entry_set_ipv4_tuple(&hwe, &data, false);
+		if (err)
+			return err;
 	}
 
 	if (addr_type == FLOW_DISSECTOR_KEY_IPV6_ADDRS) {
@@ -1228,6 +1725,10 @@ static int airoha_ppe_flow_offload_replace(struct airoha_eth *eth,
 			return err;
 	}
 
+	err = airoha_ppe_foe_entry_set_soe_info(&hwe, tuple);
+	if (err)
+		return err;
+
 	e = kzalloc_obj(*e);
 	if (!e)
 		return -ENOMEM;
@@ -1350,16 +1851,26 @@ static int airoha_ppe_flow_offload_cmd(struct airoha_eth *eth,
 	return -EOPNOTSUPP;
 }
 
-static int airoha_ppe_flush_sram_entries(struct airoha_ppe *ppe)
+static int airoha_ppe_flush_entries(struct airoha_ppe *ppe)
 {
+	u32 ppe_num_entries = airoha_ppe_get_total_num_entries(ppe);
 	u32 sram_num_entries = airoha_ppe_get_total_sram_num_entries(ppe);
 	struct airoha_foe_entry *hwe = ppe->foe;
 	int i, err = 0;
 
+	memset(hwe, 0, ppe_num_entries * sizeof(*hwe));
+	if (ppe->foe_stats) {
+		u32 ppe_num_stats_entries =
+			airoha_ppe_get_total_num_stats_entries(ppe);
+
+		memset(ppe->foe_stats, 0,
+		       ppe_num_stats_entries * sizeof(*ppe->foe_stats));
+	}
+	dma_wmb();
+
 	for (i = 0; i < sram_num_entries; i++) {
 		int err;
 
-		memset(&hwe[i], 0, sizeof(*hwe));
 		err = airoha_ppe_foe_commit_sram_entry(ppe, i);
 		if (err)
 			break;
@@ -1368,6 +1879,37 @@ static int airoha_ppe_flush_sram_entries(struct airoha_ppe *ppe)
 	return err;
 }
 
+static int airoha_ppe_set_bind_rate(const char *val,
+				    const struct kernel_param *kp)
+{
+	struct airoha_ppe *ppe;
+	unsigned long rate;
+	int err, i;
+
+	err = kstrtoul(val, 0, &rate);
+	if (err)
+		return err;
+	if (rate > FIELD_MAX(PPE_BIND_RATE_BIND_MASK))
+		return -ERANGE;
+
+	WRITE_ONCE(airoha_ppe_bind_rate, (unsigned int)rate);
+
+	mutex_lock(&flow_offload_mutex);
+	ppe = READ_ONCE(airoha_ppe_active);
+	if (ppe) {
+		for (i = 0; i < ppe->eth->soc->num_ppe; i++)
+			airoha_ppe_apply_bind_rate(ppe->eth, i);
+	}
+	mutex_unlock(&flow_offload_mutex);
+
+	return 0;
+}
+
+static int airoha_ppe_get_bind_rate(char *buf, const struct kernel_param *kp)
+{
+	return sysfs_emit(buf, "%u\n", READ_ONCE(airoha_ppe_bind_rate));
+}
+
 static struct airoha_npu *airoha_ppe_npu_get(struct airoha_eth *eth)
 {
 	struct airoha_npu *npu = airoha_npu_get(eth->dev);
@@ -1601,12 +2143,20 @@ int airoha_ppe_init(struct airoha_eth *eth)
 			return -ENOMEM;
 	}
 
-	ppe->foe_check_time = devm_kzalloc(eth->dev, ppe_num_entries,
-					   GFP_KERNEL);
+	ppe->foe_check_time =
+		devm_kzalloc(eth->dev,
+			     ppe_num_entries * sizeof(*ppe->foe_check_time),
+			     GFP_KERNEL);
 	if (!ppe->foe_check_time)
 		return -ENOMEM;
 
-	err = airoha_ppe_flush_sram_entries(ppe);
+	ppe->soe_meta = devm_kzalloc(eth->dev,
+				     ppe_num_entries * sizeof(*ppe->soe_meta),
+				     GFP_KERNEL);
+	if (!ppe->soe_meta)
+		return -ENOMEM;
+
+	err = airoha_ppe_flush_entries(ppe);
 	if (err)
 		return err;
 
@@ -1622,6 +2172,8 @@ int airoha_ppe_init(struct airoha_eth *eth)
 	if (err)
 		goto error_l2_flow_table_destroy;
 
+	WRITE_ONCE(airoha_ppe_active, ppe);
+
 	return 0;
 
 error_l2_flow_table_destroy:
@@ -1636,6 +2188,8 @@ void airoha_ppe_deinit(struct airoha_eth *eth)
 {
 	struct airoha_npu *npu;
 
+	WRITE_ONCE(airoha_ppe_active, NULL);
+
 	mutex_lock(&flow_offload_mutex);
 
 	npu = rcu_replace_pointer(eth->npu, NULL,
diff --git a/include/linux/soc/airoha/airoha_offload.h b/include/linux/soc/airoha/airoha_offload.h
index 7589fccfeef6..120dbd274c89 100644
--- a/include/linux/soc/airoha/airoha_offload.h
+++ b/include/linux/soc/airoha/airoha_offload.h
@@ -11,7 +11,12 @@
 #include <linux/workqueue.h>
 
 enum {
+	PPE_CPU_REASON_UN_HIT = 0x0d,
+	PPE_CPU_REASON_HIT_UNBIND = 0x0e,
 	PPE_CPU_REASON_HIT_UNBIND_RATE_REACHED = 0x0f,
+	PPE_CPU_REASON_HIT_BIND_FORCE_CPU = 0x16,
+	PPE_CPU_REASON_HIT_BIND_EXCEED_MTU = 0x1c,
+	PPE_CPU_REASON_NOT_THROUGH_PPE = 0x1e,
 };
 
 struct airoha_ppe_dev {
-- 
2.53.0

[RFC PATCH net-next 7/7] net: airoha: add SOE XFRM packet offload support

[Jihong Min]

Add the EN7581 Secure Offload Engine provider. The provider programs ESP
SAs, exposes NETIF_F_HW_ESP through xfrmdev_ops, submits encrypt and
decrypt packets through the QDMA SOE path, and handles SOE completion
delivery.

Mirror the XFRM ops to DSA user devices whose CPU conduit is an Airoha
netdev so packet offload remains available through switch ports.

Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
 drivers/net/ethernet/airoha/Kconfig      |   13 +
 drivers/net/ethernet/airoha/Makefile     |    1 +
 drivers/net/ethernet/airoha/airoha_soe.c | 1896 ++++++++++++++++++++++
 3 files changed, 1910 insertions(+)
 create mode 100644 drivers/net/ethernet/airoha/airoha_soe.c

diff --git a/drivers/net/ethernet/airoha/Kconfig b/drivers/net/ethernet/airoha/Kconfig
index ad3ce501e7a5..a20e9dd0bfde 100644
--- a/drivers/net/ethernet/airoha/Kconfig
+++ b/drivers/net/ethernet/airoha/Kconfig
@@ -31,4 +31,17 @@ config NET_AIROHA_FLOW_STATS
 	help
 	  Enable Aiorha flowtable statistic counters.
 
+config NET_AIROHA_SOE
+	bool "Airoha SOE ESP offload support"
+	depends on NET_AIROHA
+	depends on INET
+	select XFRM
+	select XFRM_OFFLOAD
+	help
+	  Enable support for the Airoha Secure Offload Engine used by
+	  the Ethernet driver for ESP packet offload. This option only
+	  adds the provider and netdev plumbing; ESP offload is still
+	  advertised at runtime only when the SOE block and required
+	  packet offload path are available.
+
 endif #NET_VENDOR_AIROHA
diff --git a/drivers/net/ethernet/airoha/Makefile b/drivers/net/ethernet/airoha/Makefile
index 94468053e34b..b68b8f614b0e 100644
--- a/drivers/net/ethernet/airoha/Makefile
+++ b/drivers/net/ethernet/airoha/Makefile
@@ -6,4 +6,5 @@
 obj-$(CONFIG_NET_AIROHA) += airoha-eth.o
 airoha-eth-y := airoha_eth.o airoha_ppe.o
 airoha-eth-$(CONFIG_DEBUG_FS) += airoha_ppe_debugfs.o
+airoha-eth-$(CONFIG_NET_AIROHA_SOE) += airoha_soe.o
 obj-$(CONFIG_NET_AIROHA_NPU) += airoha_npu.o
diff --git a/drivers/net/ethernet/airoha/airoha_soe.c b/drivers/net/ethernet/airoha/airoha_soe.c
new file mode 100644
index 000000000000..3a240ed44d7f
--- /dev/null
+++ b/drivers/net/ethernet/airoha/airoha_soe.c
@@ -0,0 +1,1896 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Airoha Secure Offload Engine (SOE) provider for the Ethernet driver.
+ *
+ * This file owns the EN7581 SOE packet-offload glue used by airoha_eth:
+ * xfrm state programming, hop-descriptor TX metadata, SOE RX completion
+ * decoding, and DSA proxy netdev binding. The SOE block is reached through
+ * the FE/QDMA packet fabric and is initialized by the Ethernet driver rather
+ * than by a separate platform driver.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitfield.h>
+#include <linux/completion.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/iopoll.h>
+#include <linux/io.h>
+#include <linux/ipv6.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/moduleparam.h>
+#include <linux/netdevice.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/rcupdate.h>
+#include <linux/refcount.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/string.h>
+#include <linux/udp.h>
+#include <linux/unaligned.h>
+
+#include <net/dst.h>
+#include <net/esp.h>
+#include <net/gso.h>
+#include <net/ip.h>
+#include <net/net_namespace.h>
+#include <net/xfrm.h>
+
+#include "airoha_eth.h"
+#include "airoha_regs.h"
+#include "airoha_soe.h"
+
+#define AIROHA_SOE_NUM_SA 32
+#define AIROHA_SOE_QDMA_HOP_DESC_LEN 32
+#define AIROHA_SOE_KEY_WORDS 8
+#define AIROHA_SOE_ADDR_WORDS 4
+#define AIROHA_SOE_SA_TIMEOUT_US 1000
+#define AIROHA_SOE_SA_FREE_TIMEOUT HZ
+#define AIROHA_SOE_HOP_DESC0_ENCRYPT 0xffffff81ULL
+#define AIROHA_SOE_HOP_DESC0_DECRYPT 0xffffff82ULL
+#define AIROHA_SOE_HOP_DESC1 0x1ff00000000ULL
+#define AIROHA_SOE_QDMA_TX_RING 2
+#define AIROHA_SOE_TXMSG2_DEFAULT 0xff00ffff
+
+/* This is the packet/IPsec SOE window at 0x1fbfa000. EN7581 E2 exposes
+ * this register block for packet processing, not standalone crypto offload.
+ */
+#define AIROHA_SOE_GLB_CFG 0x000
+#define AIROHA_SOE_GLB_CFG_ENC_EN BIT(0)
+#define AIROHA_SOE_GLB_CFG_DEC_EN BIT(1)
+#define AIROHA_SOE_CONT_ICV_CTRL 0x004
+#define AIROHA_SOE_INT_EN 0x020
+#define AIROHA_SOE_INT_STS 0x024
+#define AIROHA_SOE_INT_ALL GENMASK(15, 0)
+#define AIROHA_SOE_CNT_CLR 0x04c
+#define AIROHA_SOE_CNT_CLR_ALL BIT(0)
+#define AIROHA_SOE_SA_CTRL 0x100
+#define AIROHA_SOE_SA_DONE 0x104
+#define AIROHA_SOE_SA_CMD 0x110
+#define AIROHA_SOE_BCNT_THSHD_32_SOFT 0x114
+#define AIROHA_SOE_BCNT_THSHD_64_SOFT 0x118
+#define AIROHA_SOE_SA_SPI 0x11c
+#define AIROHA_SOE_SA_UDP_PORT 0x120
+#define AIROHA_SOE_SA_ENC_KEY(n) (0x124 + (n) * 4)
+#define AIROHA_SOE_SA_HMAC_KEY(n) (0x144 + (n) * 4)
+#define AIROHA_SOE_SA_SRC_ADDR(n) (0x164 + (n) * 4)
+#define AIROHA_SOE_SA_DST_ADDR(n) (0x174 + (n) * 4)
+#define AIROHA_SOE_ICV_OK_LO_CNT 0x184
+#define AIROHA_SOE_ICV_OK_HI_CNT 0x188
+#define AIROHA_SOE_ICV_FAIL_LO_CNT 0x18c
+#define AIROHA_SOE_ICV_FAIL_HI_CNT 0x190
+#define AIROHA_SOE_CON_ICV_FAIL_CNT 0x194
+#define AIROHA_SOE_SEQ_NUM_LO 0x198
+#define AIROHA_SOE_SEQ_NUM_HI 0x19c
+#define AIROHA_SOE_BCNT_LO 0x1a0
+#define AIROHA_SOE_BCNT_HI 0x1a4
+#define AIROHA_SOE_FLOW_LAB_DSCP 0x1a8
+#define AIROHA_SOE_BCNT_80 0x1ac
+#define AIROHA_SOE_BCNT_THSHD_80 0x1b0
+#define AIROHA_SOE_BCNT_THSHD_32_HARD 0x1b4
+#define AIROHA_SOE_BCNT_THSHD_64_HARD 0x1b8
+#define AIROHA_SOE_SEQ_THSHD_32_SOFT 0x1bc
+#define AIROHA_SOE_SEQ_THSHD_64_SOFT 0x1c0
+#define AIROHA_SOE_SEQ_THSHD_32_HARD 0x1c4
+#define AIROHA_SOE_SEQ_THSHD_64_HARD 0x1c8
+#define AIROHA_SOE_SA_CTRL_WR BIT(0)
+#define AIROHA_SOE_SA_CTRL_IDX GENMASK(15, 8)
+#define AIROHA_SOE_SA_DONE_W1C BIT(0)
+
+#define AIROHA_SOE_SA_CMD_ENC BIT(0)
+#define AIROHA_SOE_SA_CMD_CIPHER GENMASK(3, 1)
+#define AIROHA_SOE_SA_CMD_HASH GENMASK(6, 4)
+#define AIROHA_SOE_SA_CMD_AES_KEY_LEN GENMASK(8, 7)
+#define AIROHA_SOE_SA_CMD_ESN_EN BIT(9)
+#define AIROHA_SOE_SA_CMD_OUT_IPV6 BIT(10)
+#define AIROHA_SOE_SA_CMD_ESP_MODE BIT(11) /* 0=tunnel, 1=transport */
+#define AIROHA_SOE_SA_CMD_NAT_EN BIT(12)
+#define AIROHA_SOE_SA_CMD_ANTI_RPLY_EN BIT(13)
+#define AIROHA_SOE_SA_CMD_ANTI_RPLY_WDW GENMASK(15, 14)
+#define AIROHA_SOE_SA_CMD_SN_ERR_DROP BIT(16)
+#define AIROHA_SOE_SA_CMD_PAD_ERR_DROP BIT(17)
+#define AIROHA_SOE_SA_CMD_ICV_ERR_DROP BIT(18)
+#define AIROHA_SOE_SA_CMD_GCM_ICV_LEN GENMASK(25, 24)
+#define AIROHA_SOE_SA_CMD_DEC_UDP_PARSER_EN BIT(29)
+#define AIROHA_SOE_SA_CMD_VLD BIT(31)
+
+#define AIROHA_SOE_CIPHER_AES_CBC 1
+#define AIROHA_SOE_CIPHER_AES_GCM 2
+#define AIROHA_SOE_HASH_HMAC_SHA1_96 1
+#define AIROHA_SOE_HASH_HMAC_SHA256_128 2
+#define AIROHA_SOE_AES_KEY_128 0
+#define AIROHA_SOE_AES_KEY_192 1
+#define AIROHA_SOE_AES_KEY_256 2
+#define AIROHA_SOE_QDMA_QUEUE_ENCRYPT 8
+#define AIROHA_SOE_QDMA_QUEUE_DECRYPT 9
+#define AIROHA_SOE_NATT_PORT 4500
+#define AIROHA_SOE_HOP_FLAG_ENCRYPTED 3
+#define AIROHA_SOE_HOP_FLAG_DECRYPTED 4
+#define AIROHA_SOE_HOP_FLAG_ERROR_BASE 5
+#define AIROHA_SOE_HOP_INFO_ENCRYPT 2
+#define AIROHA_SOE_HOP_INFO_DECRYPT 3
+
+static unsigned int airoha_soe_rx_trace_packets;
+module_param_named(soe_rx_trace_packets, airoha_soe_rx_trace_packets, uint,
+		   0600);
+MODULE_PARM_DESC(soe_rx_trace_packets,
+		 "Number of SOE RX completion IPv4 headers to log");
+
+enum airoha_soe_ctx_dir {
+	AIROHA_SOE_CTX_OUT,
+	AIROHA_SOE_CTX_IN,
+};
+
+struct airoha_soe_ctx {
+	struct list_head list;
+	enum airoha_soe_ctx_dir dir;
+	union {
+		struct dst_entry *dst;
+		struct {
+			struct xfrm_state *x;
+			struct airoha_gdm_dev *gdm_dev;
+			struct net_device *dev;
+			__be32 saddr;
+			__be16 sport;
+			u16 foe_hash;
+			u32 foe_reason;
+			u8 sa_index;
+			bool foe_valid;
+			u32 mark;
+		} rx;
+	};
+};
+
+struct airoha_soe_sa {
+	struct airoha_soe *soe;
+	unsigned int index;
+	u32 cmd;
+	u32 spi;
+
+	spinlock_t lock; /* Protects in-flight context queues and dead. */
+	struct list_head tx_queue;
+	struct list_head rx_queue;
+	struct completion idle;
+	unsigned int inflight;
+	bool dead;
+};
+
+struct airoha_soe_xfrm_state {
+	struct airoha_gdm_dev *dev;
+	struct airoha_soe *soe;
+	struct airoha_soe_sa *sa;
+	bool counted;
+};
+
+struct airoha_soe_sa_cfg {
+	u32 cmd;
+	u32 spi;
+	u32 udp_port;
+	u32 enc_key[AIROHA_SOE_KEY_WORDS];
+	u32 hmac_key[AIROHA_SOE_KEY_WORDS];
+	u32 src_addr[AIROHA_SOE_ADDR_WORDS];
+	u32 dst_addr[AIROHA_SOE_ADDR_WORDS];
+	u64 soft_byte_limit;
+	u64 hard_byte_limit;
+	u64 soft_packet_limit;
+	u64 hard_packet_limit;
+};
+
+struct airoha_soe_rx_info {
+	int packet_len;
+	bool encap;
+	__be16 sport;
+	__be16 dport;
+	__be32 spi;
+};
+
+struct airoha_soe {
+	struct device *dev;
+	void __iomem *base;
+
+	/* Serialize SA table programming and software slot ownership. */
+	struct mutex sa_lock;
+	unsigned long sa_map;
+	struct airoha_soe_sa __rcu *sa[AIROHA_SOE_NUM_SA];
+	atomic_t pending_rx;
+
+	spinlock_t state_lock; /* Protects dead against concurrent users. */
+	refcount_t refcnt;
+	struct completion released;
+	bool dead;
+};
+
+static const struct xfrmdev_ops airoha_soe_xfrmdev_ops;
+static const struct xfrmdev_ops airoha_soe_dsa_xfrmdev_ops;
+
+static struct airoha_soe *airoha_soe_get_ref(struct airoha_soe *soe)
+{
+	unsigned long flags;
+	bool alive;
+
+	if (!soe)
+		return NULL;
+
+	spin_lock_irqsave(&soe->state_lock, flags);
+	alive = !soe->dead && refcount_inc_not_zero(&soe->refcnt);
+	spin_unlock_irqrestore(&soe->state_lock, flags);
+
+	return alive ? soe : NULL;
+}
+
+static void airoha_soe_put_ref(struct airoha_soe *soe)
+{
+	if (soe && refcount_dec_and_test(&soe->refcnt))
+		complete(&soe->released);
+}
+
+bool airoha_soe_available(struct airoha_soe *soe)
+{
+	unsigned long flags;
+	bool available;
+
+	if (!soe)
+		return false;
+
+	spin_lock_irqsave(&soe->state_lock, flags);
+	available = !soe->dead;
+	spin_unlock_irqrestore(&soe->state_lock, flags);
+
+	return available;
+}
+
+u32 airoha_soe_features(struct airoha_soe *soe)
+{
+	return airoha_soe_available(soe) ? AIROHA_SOE_FEATURE_ESP : 0;
+}
+
+static u64 airoha_soe_limit(u64 limit)
+{
+	return limit == XFRM_INF ? U64_MAX : limit;
+}
+
+static int airoha_soe_wait_sa_done(struct airoha_soe *soe)
+{
+	u32 done;
+	int err;
+
+	err = readl_poll_timeout(soe->base + AIROHA_SOE_SA_DONE, done,
+				 done & AIROHA_SOE_SA_DONE_W1C, 1,
+				 AIROHA_SOE_SA_TIMEOUT_US);
+	writel(0, soe->base + AIROHA_SOE_SA_CTRL);
+	writel(AIROHA_SOE_SA_DONE_W1C, soe->base + AIROHA_SOE_SA_DONE);
+
+	return err;
+}
+
+static int airoha_soe_commit_sa(struct airoha_soe *soe, unsigned int index)
+{
+	u32 ctrl;
+
+	/* SA registers are a single staging window committed by index. */
+	writel(AIROHA_SOE_SA_DONE_W1C, soe->base + AIROHA_SOE_SA_DONE);
+	ctrl = FIELD_PREP(AIROHA_SOE_SA_CTRL_IDX, index) |
+	       AIROHA_SOE_SA_CTRL_WR;
+	writel(ctrl, soe->base + AIROHA_SOE_SA_CTRL);
+
+	return airoha_soe_wait_sa_done(soe);
+}
+
+static void airoha_soe_write_key(void __iomem *base, u32 reg, const u32 *key)
+{
+	unsigned int i;
+
+	for (i = 0; i < AIROHA_SOE_KEY_WORDS; i++)
+		writel(key[i], base + reg + i * sizeof(u32));
+}
+
+static void airoha_soe_write_addr(void __iomem *base, u32 reg, const u32 *addr)
+{
+	unsigned int i;
+
+	for (i = 0; i < AIROHA_SOE_ADDR_WORDS; i++)
+		writel(addr[i], base + reg + i * sizeof(u32));
+}
+
+static int airoha_soe_program_sa_locked(struct airoha_soe *soe,
+					unsigned int index,
+					const struct airoha_soe_sa_cfg *cfg)
+{
+	void __iomem *base = soe->base;
+
+	writel(cfg->cmd | AIROHA_SOE_SA_CMD_VLD, base + AIROHA_SOE_SA_CMD);
+	writel(lower_32_bits(cfg->soft_byte_limit),
+	       base + AIROHA_SOE_BCNT_THSHD_32_SOFT);
+	writel(upper_32_bits(cfg->soft_byte_limit),
+	       base + AIROHA_SOE_BCNT_THSHD_64_SOFT);
+	writel(cfg->spi, base + AIROHA_SOE_SA_SPI);
+	writel(cfg->udp_port, base + AIROHA_SOE_SA_UDP_PORT);
+	airoha_soe_write_key(base, AIROHA_SOE_SA_ENC_KEY(0), cfg->enc_key);
+	airoha_soe_write_key(base, AIROHA_SOE_SA_HMAC_KEY(0), cfg->hmac_key);
+	airoha_soe_write_addr(base, AIROHA_SOE_SA_SRC_ADDR(0), cfg->src_addr);
+	airoha_soe_write_addr(base, AIROHA_SOE_SA_DST_ADDR(0), cfg->dst_addr);
+
+	writel(0, base + AIROHA_SOE_ICV_OK_LO_CNT);
+	writel(0, base + AIROHA_SOE_ICV_OK_HI_CNT);
+	writel(0, base + AIROHA_SOE_ICV_FAIL_LO_CNT);
+	writel(0, base + AIROHA_SOE_ICV_FAIL_HI_CNT);
+	writel(0, base + AIROHA_SOE_CON_ICV_FAIL_CNT);
+	writel(0, base + AIROHA_SOE_SEQ_NUM_LO);
+	writel(0, base + AIROHA_SOE_SEQ_NUM_HI);
+	writel(0, base + AIROHA_SOE_BCNT_LO);
+	writel(0, base + AIROHA_SOE_BCNT_HI);
+	writel(0, base + AIROHA_SOE_FLOW_LAB_DSCP);
+	writel(0, base + AIROHA_SOE_BCNT_80);
+	writel(0xffffffff, base + AIROHA_SOE_BCNT_THSHD_80);
+	writel(lower_32_bits(cfg->hard_byte_limit),
+	       base + AIROHA_SOE_BCNT_THSHD_32_HARD);
+	writel(upper_32_bits(cfg->hard_byte_limit),
+	       base + AIROHA_SOE_BCNT_THSHD_64_HARD);
+	writel(lower_32_bits(cfg->soft_packet_limit),
+	       base + AIROHA_SOE_SEQ_THSHD_32_SOFT);
+	writel(upper_32_bits(cfg->soft_packet_limit),
+	       base + AIROHA_SOE_SEQ_THSHD_64_SOFT);
+	writel(lower_32_bits(cfg->hard_packet_limit),
+	       base + AIROHA_SOE_SEQ_THSHD_32_HARD);
+	writel(upper_32_bits(cfg->hard_packet_limit),
+	       base + AIROHA_SOE_SEQ_THSHD_64_HARD);
+
+	return airoha_soe_commit_sa(soe, index);
+}
+
+static int airoha_soe_clear_sa_locked(struct airoha_soe *soe,
+				      unsigned int index)
+{
+	struct airoha_soe_sa_cfg cfg = {};
+
+	return airoha_soe_program_sa_locked(soe, index, &cfg);
+}
+
+static void airoha_soe_copy_words(u32 *dst, const u8 *src, unsigned int bits)
+{
+	unsigned int words = bits / (BITS_PER_BYTE * sizeof(u32));
+	unsigned int i;
+
+	for (i = 0; i < words && i < AIROHA_SOE_KEY_WORDS; i++)
+		dst[i] = get_unaligned_be32(src + i * sizeof(u32));
+}
+
+static int airoha_soe_aes_key_len(unsigned int bits,
+				  struct netlink_ext_ack *extack, u32 *val)
+{
+	switch (bits) {
+	case 128:
+		*val = AIROHA_SOE_AES_KEY_128;
+		return 0;
+	case 192:
+		*val = AIROHA_SOE_AES_KEY_192;
+		return 0;
+	case 256:
+		*val = AIROHA_SOE_AES_KEY_256;
+		return 0;
+	default:
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports AES-128/192/256 keys only");
+		return -EOPNOTSUPP;
+	}
+}
+
+static int airoha_soe_build_algo(struct xfrm_state *x,
+				 struct airoha_soe_sa_cfg *cfg,
+				 struct netlink_ext_ack *extack)
+{
+	u32 key_len;
+	u32 field;
+	int err;
+
+	if (x->aead) {
+		if (strcmp(x->aead->alg_name, "rfc4106(gcm(aes))")) {
+			NL_SET_ERR_MSG_MOD(extack,
+					   "SOE supports rfc4106(gcm(aes)) AEAD only");
+			return -EOPNOTSUPP;
+		}
+
+		if (x->aead->alg_key_len < 32) {
+			NL_SET_ERR_MSG_MOD(extack, "invalid AEAD key length");
+			return -EINVAL;
+		}
+
+		key_len = x->aead->alg_key_len - 32;
+		err = airoha_soe_aes_key_len(key_len, extack, &field);
+		if (err)
+			return err;
+
+		cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_CIPHER,
+				       AIROHA_SOE_CIPHER_AES_GCM);
+		cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_AES_KEY_LEN, field);
+		switch (x->aead->alg_icv_len) {
+		case 64:
+			field = 0;
+			break;
+		case 96:
+			field = 1;
+			break;
+		case 128:
+			field = 2;
+			break;
+		default:
+			NL_SET_ERR_MSG_MOD(extack,
+					   "SOE supports 64/96/128-bit GCM ICV only");
+			return -EOPNOTSUPP;
+		}
+		cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_GCM_ICV_LEN, field);
+		airoha_soe_copy_words(cfg->enc_key, x->aead->alg_key, key_len);
+		cfg->hmac_key[0] =
+			get_unaligned_be32(x->aead->alg_key + key_len / 8);
+		return 0;
+	}
+
+	if (!x->ealg || strcmp(x->ealg->alg_name, "cbc(aes)")) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports cbc(aes) encryption only");
+		return -EOPNOTSUPP;
+	}
+
+	err = airoha_soe_aes_key_len(x->ealg->alg_key_len, extack, &field);
+	if (err)
+		return err;
+
+	cfg->cmd |=
+		FIELD_PREP(AIROHA_SOE_SA_CMD_CIPHER, AIROHA_SOE_CIPHER_AES_CBC);
+	cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_AES_KEY_LEN, field);
+	airoha_soe_copy_words(cfg->enc_key, x->ealg->alg_key,
+			      x->ealg->alg_key_len);
+
+	if (!x->aalg) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE CBC mode requires HMAC authentication");
+		return -EOPNOTSUPP;
+	}
+
+	if (!strcmp(x->aalg->alg_name, "hmac(sha1)")) {
+		if (x->aalg->alg_key_len != 160 ||
+		    x->aalg->alg_trunc_len != 96) {
+			NL_SET_ERR_MSG_MOD(extack,
+					   "SOE supports HMAC-SHA1-96 only");
+			return -EOPNOTSUPP;
+		}
+		field = AIROHA_SOE_HASH_HMAC_SHA1_96;
+	} else if (!strcmp(x->aalg->alg_name, "hmac(sha256)")) {
+		if (x->aalg->alg_key_len != 256 ||
+		    x->aalg->alg_trunc_len != 128) {
+			NL_SET_ERR_MSG_MOD(extack,
+					   "SOE supports HMAC-SHA256-128 only");
+			return -EOPNOTSUPP;
+		}
+		field = AIROHA_SOE_HASH_HMAC_SHA256_128;
+	} else {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports HMAC-SHA1/SHA256 only");
+		return -EOPNOTSUPP;
+	}
+
+	cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_HASH, field);
+	airoha_soe_copy_words(cfg->hmac_key, x->aalg->alg_key,
+			      x->aalg->alg_key_len);
+
+	return 0;
+}
+
+static int airoha_soe_build_replay(struct xfrm_state *x,
+				   struct airoha_soe_sa_cfg *cfg,
+				   struct netlink_ext_ack *extack)
+{
+	u32 window;
+
+	if ((x->props.flags & XFRM_STATE_ESN) ||
+	    x->repl_mode == XFRM_REPLAY_MODE_ESN) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE ESN is not supported yet");
+		return -EOPNOTSUPP;
+	}
+
+	window = x->replay_esn ? x->replay_esn->replay_window :
+				 x->props.replay_window;
+	if (!window)
+		return 0;
+
+	cfg->cmd |= AIROHA_SOE_SA_CMD_ANTI_RPLY_EN;
+	cfg->cmd |= FIELD_PREP(AIROHA_SOE_SA_CMD_ANTI_RPLY_WDW,
+			       min_t(u32, (window - 1) / 64, 3));
+
+	return 0;
+}
+
+static int airoha_soe_build_sa(struct xfrm_state *x,
+			       struct airoha_soe_sa_cfg *cfg,
+			       struct netlink_ext_ack *extack)
+{
+	int err;
+
+	if (x->xso.type != XFRM_DEV_OFFLOAD_PACKET) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports XFRM packet offload only");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->xso.dir != XFRM_DEV_OFFLOAD_OUT &&
+	    x->xso.dir != XFRM_DEV_OFFLOAD_IN) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE supports in/out SAs only");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->id.proto != IPPROTO_ESP) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE supports ESP only");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->props.family != AF_INET || x->outer_mode.family != AF_INET) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE bring-up supports IPv4 outer tunnel only");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->props.mode != XFRM_MODE_TUNNEL) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE supports tunnel mode only");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->encap && x->encap->encap_type != UDP_ENCAP_ESPINUDP) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports native ESP or UDP_ENCAP_ESPINUDP");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->tfcpad) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE does not support TFC padding");
+		return -EOPNOTSUPP;
+	}
+
+	cfg->cmd = AIROHA_SOE_SA_CMD_SN_ERR_DROP |
+		   AIROHA_SOE_SA_CMD_PAD_ERR_DROP |
+		   AIROHA_SOE_SA_CMD_ICV_ERR_DROP;
+	if (x->xso.dir == XFRM_DEV_OFFLOAD_OUT) {
+		cfg->cmd |= AIROHA_SOE_SA_CMD_ENC;
+		if (x->encap)
+			cfg->cmd |= AIROHA_SOE_SA_CMD_NAT_EN;
+		cfg->src_addr[0] = be32_to_cpu(x->props.saddr.a4);
+		cfg->dst_addr[0] = be32_to_cpu(x->id.daddr.a4);
+	} else if (x->encap) {
+		/* RX submit passes the full UDP/4500 packet to SOE. Ask the
+		 * decrypt parser to consume the UDP header before ESP decap.
+		 */
+		cfg->cmd |= AIROHA_SOE_SA_CMD_DEC_UDP_PARSER_EN;
+	}
+
+	err = airoha_soe_build_algo(x, cfg, extack);
+	if (err)
+		return err;
+
+	err = airoha_soe_build_replay(x, cfg, extack);
+	if (err)
+		return err;
+
+	cfg->spi = be32_to_cpu(x->id.spi);
+	if (x->encap) {
+		/* The NAT-T port word stores dport above sport. */
+		cfg->udp_port = (u32)ntohs(x->encap->encap_dport) << 16 |
+				ntohs(x->encap->encap_sport);
+	}
+	cfg->soft_byte_limit = airoha_soe_limit(x->lft.soft_byte_limit);
+	cfg->hard_byte_limit = airoha_soe_limit(x->lft.hard_byte_limit);
+	cfg->soft_packet_limit = airoha_soe_limit(x->lft.soft_packet_limit);
+	cfg->hard_packet_limit = airoha_soe_limit(x->lft.hard_packet_limit);
+
+	return 0;
+}
+
+static int airoha_soe_alloc_sa(struct airoha_soe *soe, struct xfrm_state *x,
+			       struct netlink_ext_ack *extack,
+			       struct airoha_soe_sa **sa)
+{
+	struct airoha_soe_sa_cfg cfg = {};
+	struct airoha_soe_sa *new_sa;
+	unsigned int i;
+	int err;
+
+	if (!soe || !sa || !airoha_soe_available(soe)) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE provider is unavailable");
+		return -ENODEV;
+	}
+
+	err = airoha_soe_build_sa(x, &cfg, extack);
+	if (err)
+		return err;
+
+	new_sa = kzalloc_obj(*new_sa, GFP_KERNEL);
+	if (!new_sa)
+		return -ENOMEM;
+
+	mutex_lock(&soe->sa_lock);
+	for (i = 0; i < AIROHA_SOE_NUM_SA; i++) {
+		if (!(soe->sa_map & BIT(i)))
+			break;
+	}
+	if (i == AIROHA_SOE_NUM_SA) {
+		mutex_unlock(&soe->sa_lock);
+		kfree(new_sa);
+		return -ENOSPC;
+	}
+
+	err = airoha_soe_program_sa_locked(soe, i, &cfg);
+	if (err) {
+		mutex_unlock(&soe->sa_lock);
+		kfree(new_sa);
+		return err;
+	}
+
+	new_sa->soe = soe;
+	new_sa->index = i;
+	new_sa->cmd = cfg.cmd;
+	new_sa->spi = cfg.spi;
+	spin_lock_init(&new_sa->lock);
+	INIT_LIST_HEAD(&new_sa->tx_queue);
+	INIT_LIST_HEAD(&new_sa->rx_queue);
+	init_completion(&new_sa->idle);
+	rcu_assign_pointer(soe->sa[i], new_sa);
+	soe->sa_map |= BIT(i);
+	mutex_unlock(&soe->sa_lock);
+
+	*sa = new_sa;
+	return 0;
+}
+
+static void airoha_soe_mark_sa_dead(struct airoha_soe_sa *sa)
+{
+	if (!sa)
+		return;
+
+	spin_lock_bh(&sa->lock);
+	sa->dead = true;
+	if (!sa->inflight)
+		complete(&sa->idle);
+	spin_unlock_bh(&sa->lock);
+}
+
+static void airoha_soe_free_ctx(struct airoha_soe_ctx *ctx)
+{
+	if (!ctx)
+		return;
+
+	if (ctx->dir == AIROHA_SOE_CTX_OUT)
+		dst_release(ctx->dst);
+	else
+		xfrm_state_put(ctx->rx.x);
+	kfree(ctx);
+}
+
+static void airoha_soe_purge_ctx_list(struct list_head *head)
+{
+	struct airoha_soe_ctx *ctx, *tmp;
+
+	list_for_each_entry_safe(ctx, tmp, head, list) {
+		list_del(&ctx->list);
+		airoha_soe_free_ctx(ctx);
+	}
+}
+
+static void airoha_soe_forget_rx_ctx_list(struct airoha_soe_sa *sa)
+{
+	if (!list_empty(&sa->rx_queue))
+		atomic_sub((int)list_count_nodes(&sa->rx_queue),
+			   &sa->soe->pending_rx);
+}
+
+static void airoha_soe_abort_sa(struct airoha_soe_sa *sa)
+{
+	LIST_HEAD(rx_queue);
+	LIST_HEAD(tx_queue);
+
+	if (!sa)
+		return;
+
+	spin_lock_bh(&sa->lock);
+	sa->dead = true;
+	airoha_soe_forget_rx_ctx_list(sa);
+	list_splice_init(&sa->tx_queue, &tx_queue);
+	list_splice_init(&sa->rx_queue, &rx_queue);
+	sa->inflight = 0;
+	complete(&sa->idle);
+	spin_unlock_bh(&sa->lock);
+
+	airoha_soe_purge_ctx_list(&tx_queue);
+	airoha_soe_purge_ctx_list(&rx_queue);
+}
+
+static void airoha_soe_free_sa(struct airoha_soe_sa *sa)
+{
+	LIST_HEAD(rx_queue);
+	LIST_HEAD(tx_queue);
+	struct airoha_soe *soe;
+
+	if (!sa)
+		return;
+
+	soe = sa->soe;
+	airoha_soe_mark_sa_dead(sa);
+	if (!wait_for_completion_timeout(&sa->idle, AIROHA_SOE_SA_FREE_TIMEOUT))
+		dev_warn(soe->dev,
+			 "timed out waiting for SOE SA%u in-flight packets\n",
+			 sa->index);
+
+	mutex_lock(&soe->sa_lock);
+	if (sa->index < AIROHA_SOE_NUM_SA &&
+	    rcu_access_pointer(soe->sa[sa->index]) == sa) {
+		airoha_soe_clear_sa_locked(soe, sa->index);
+		RCU_INIT_POINTER(soe->sa[sa->index], NULL);
+		soe->sa_map &= ~BIT(sa->index);
+	}
+	mutex_unlock(&soe->sa_lock);
+	synchronize_rcu();
+
+	spin_lock_bh(&sa->lock);
+	airoha_soe_forget_rx_ctx_list(sa);
+	list_splice_init(&sa->tx_queue, &tx_queue);
+	list_splice_init(&sa->rx_queue, &rx_queue);
+	spin_unlock_bh(&sa->lock);
+	airoha_soe_purge_ctx_list(&tx_queue);
+	airoha_soe_purge_ctx_list(&rx_queue);
+
+	kfree(sa);
+}
+
+static struct airoha_soe_ctx *airoha_soe_pop_ctx(struct airoha_soe_sa *sa,
+						 enum airoha_soe_ctx_dir dir)
+{
+	struct list_head *head;
+	struct airoha_soe_ctx *ctx = NULL;
+
+	head = dir == AIROHA_SOE_CTX_OUT ? &sa->tx_queue : &sa->rx_queue;
+
+	spin_lock_bh(&sa->lock);
+	if (!list_empty(head)) {
+		ctx = list_first_entry(head, struct airoha_soe_ctx, list);
+		list_del(&ctx->list);
+		if (dir == AIROHA_SOE_CTX_IN)
+			atomic_dec(&sa->soe->pending_rx);
+	}
+
+	if (ctx && !WARN_ON_ONCE(!sa->inflight)) {
+		sa->inflight--;
+		if (sa->dead && !sa->inflight)
+			complete(&sa->idle);
+	}
+	spin_unlock_bh(&sa->lock);
+
+	return ctx;
+}
+
+static int airoha_soe_prepare_ip_headers(struct sk_buff *skb)
+{
+	unsigned int hdr_len;
+
+	if (!pskb_may_pull(skb, 1))
+		return -EINVAL;
+
+	switch (skb->data[0] & 0xf0) {
+	case 0x40:
+		hdr_len = sizeof(struct iphdr);
+		skb->protocol = htons(ETH_P_IP);
+		break;
+	case 0x60:
+		hdr_len = sizeof(struct ipv6hdr);
+		skb->protocol = htons(ETH_P_IPV6);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	if (!pskb_may_pull(skb, hdr_len))
+		return -EINVAL;
+
+	skb_reset_network_header(skb);
+	skb_set_transport_header(skb, hdr_len);
+
+	return 0;
+}
+
+static void airoha_soe_trace_rx_complete(struct sk_buff *skb,
+					 const struct airoha_soe_ctx *ctx,
+					 const struct xfrm_state *x)
+{
+	unsigned int trace = READ_ONCE(airoha_soe_rx_trace_packets);
+	const struct iphdr *iph;
+
+	if (!trace || skb->protocol != htons(ETH_P_IP))
+		return;
+
+	iph = ip_hdr(skb);
+	pr_info("airoha_eth: SOE RX complete dev=%s saddr=%pI4 daddr=%pI4 proto=%u len=%u mark=0x%x spi=0x%08x natt=%u foe=%u hash=0x%04x sa=%u\n",
+		ctx->rx.dev->name, &iph->saddr, &iph->daddr, iph->protocol,
+		ntohs(iph->tot_len), skb->mark, ntohl(x->id.spi),
+		x->encap ? 1 : 0, ctx->rx.foe_valid, ctx->rx.foe_hash,
+		ctx->rx.sa_index);
+	WRITE_ONCE(airoha_soe_rx_trace_packets, trace - 1);
+}
+
+static int airoha_soe_push_l2_header(struct sk_buff *skb)
+{
+	static const u8 ipv4_l2_header[ETH_HLEN] = {
+		0x00, 0x0c, 0xe7, 0x20, 0x21, 0x12, 0x00,
+		0x0c, 0xe7, 0x20, 0x22, 0x62, 0x08, 0x00,
+	};
+	static const u8 ipv6_l2_header[ETH_HLEN] = {
+		0x00, 0x0c, 0xe7, 0x20, 0x21, 0x12, 0x00,
+		0x0c, 0xe7, 0x20, 0x22, 0x62, 0x86, 0xdd,
+	};
+	const u8 *l2_header;
+	int err;
+
+	err = airoha_soe_prepare_ip_headers(skb);
+	if (err)
+		return err;
+
+	if (skb->protocol == htons(ETH_P_IP))
+		l2_header = ipv4_l2_header;
+	else
+		l2_header = ipv6_l2_header;
+
+	/* TDMA/SOE port 7 expects an Ethernet-looking frame before the SOE hop. */
+	memcpy(skb_push(skb, ETH_HLEN), l2_header, ETH_HLEN);
+
+	return 0;
+}
+
+static void airoha_soe_push_hop_desc(struct sk_buff *skb, unsigned int sa_index,
+				     bool encrypt, int foe_idx)
+{
+	u32 hop_direction = encrypt ? AIROHA_SOE_HOP_INFO_ENCRYPT :
+				      AIROHA_SOE_HOP_INFO_DECRYPT;
+	u64 desc3 = ((u64)(u16)((hop_direction << 4) | 0x80) << 48) |
+		    ((u64)(sa_index & 0x3f) << 40) | 0x05dc0000ULL;
+	u64 desc2 = 0;
+	__le64 desc[4] = {};
+
+	if (foe_idx >= 0)
+		desc2 = (u64)(foe_idx & 0xffff) << 32;
+
+	desc[0] = cpu_to_le64(encrypt ? AIROHA_SOE_HOP_DESC0_ENCRYPT :
+					AIROHA_SOE_HOP_DESC0_DECRYPT);
+	desc[1] = cpu_to_le64(AIROHA_SOE_HOP_DESC1);
+	desc[2] = cpu_to_le64(desc2);
+	desc[3] = cpu_to_le64(desc3);
+	((u8 *)desc)[28] = sa_index;
+
+	/* The FE/QDMA hop descriptor is consumed by PSE port 7 before SOE. */
+	memcpy(skb_push(skb, AIROHA_SOE_QDMA_HOP_DESC_LEN), desc, sizeof(desc));
+}
+
+static int airoha_soe_submit_skb(struct airoha_soe_sa *sa,
+				 struct airoha_gdm_dev *dev,
+				 struct sk_buff *skb,
+				 struct airoha_soe_ctx *ctx)
+{
+	struct net_device *netdev = netdev_from_priv(dev);
+	u32 queue = ctx->dir == AIROHA_SOE_CTX_OUT ?
+			    AIROHA_SOE_QDMA_QUEUE_ENCRYPT :
+			    AIROHA_SOE_QDMA_QUEUE_DECRYPT;
+	bool encrypt = ctx->dir == AIROHA_SOE_CTX_OUT;
+	unsigned int headroom = AIROHA_SOE_QDMA_HOP_DESC_LEN + ETH_HLEN;
+	struct list_head *head;
+	u32 msg0, msg1;
+	int foe_idx = -1;
+	int err;
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		err = skb_checksum_help(skb);
+		if (err)
+			return err;
+	}
+
+	err = skb_cow_head(skb, headroom);
+	if (err)
+		return err;
+
+	err = airoha_soe_push_l2_header(skb);
+	if (err)
+		return err;
+
+	msg0 = FIELD_PREP(QDMA_ETH_TXMSG_SOE_SA_MASK, sa->index & 0x3f);
+	msg1 = FIELD_PREP(QDMA_ETH_TXMSG_METER_MASK, 0x7f) |
+	       FIELD_PREP(QDMA_ETH_TXMSG_FPORT_MASK, 7) |
+	       FIELD_PREP(QDMA_ETH_TXMSG_NBOQ_MASK, queue) |
+	       QDMA_ETH_TXMSG_HOP_MASK |
+	       FIELD_PREP(QDMA_ETH_TXMSG_ACNT_G1_MASK, 0x1f) |
+	       FIELD_PREP(QDMA_ETH_TXMSG_ACNT_G0_MASK, 0x3f);
+
+	if (ctx->dir == AIROHA_SOE_CTX_IN && ctx->rx.foe_valid &&
+	    ctx->rx.foe_hash != AIROHA_RXD4_FOE_ENTRY)
+		foe_idx = ctx->rx.foe_hash;
+
+	airoha_soe_push_hop_desc(skb, sa->index, encrypt, foe_idx);
+
+	skb->dev = netdev;
+	skb_set_queue_mapping(skb, AIROHA_SOE_QDMA_TX_RING);
+
+	if (!dev->soe_xmit_skb)
+		return -ENODEV;
+
+	head = ctx->dir == AIROHA_SOE_CTX_OUT ? &sa->tx_queue : &sa->rx_queue;
+	spin_lock_bh(&sa->lock);
+	if (sa->dead) {
+		spin_unlock_bh(&sa->lock);
+		return -ENOENT;
+	}
+
+	/* Completion descriptors carry only SA/hop flags, so keep skb context here. */
+	list_add_tail(&ctx->list, head);
+	sa->inflight++;
+	if (ctx->dir == AIROHA_SOE_CTX_IN)
+		atomic_inc(&sa->soe->pending_rx);
+	reinit_completion(&sa->idle);
+
+	err = dev->soe_xmit_skb(dev, skb, msg0, msg1,
+				AIROHA_SOE_TXMSG2_DEFAULT);
+	if (err) {
+		list_del(&ctx->list);
+		if (ctx->dir == AIROHA_SOE_CTX_IN)
+			atomic_dec(&sa->soe->pending_rx);
+		sa->inflight--;
+		if (sa->dead && !sa->inflight)
+			complete(&sa->idle);
+	}
+	spin_unlock_bh(&sa->lock);
+
+	return err;
+}
+
+int airoha_soe_xmit(struct airoha_soe_sa *sa, struct airoha_gdm_dev *dev,
+		    struct sk_buff *skb, struct xfrm_state *x)
+{
+	struct airoha_soe_ctx *ctx;
+	struct dst_entry *path;
+	struct dst_entry *dst;
+	int err;
+
+	if (!sa || !dev || !skb || !x || x->xso.dir != XFRM_DEV_OFFLOAD_OUT)
+		return -EINVAL;
+
+	if (skb_is_gso(skb))
+		return -EOPNOTSUPP;
+
+	dst = skb_dst(skb);
+	if (!dst)
+		return -EHOSTUNREACH;
+
+	path = xfrm_dst_path(dst);
+	if (!path)
+		return -EHOSTUNREACH;
+
+	ctx = kzalloc_obj(*ctx, GFP_ATOMIC);
+	if (!ctx)
+		return -ENOMEM;
+
+	ctx->dir = AIROHA_SOE_CTX_OUT;
+	dst_hold(path);
+	ctx->dst = path;
+
+	err = airoha_soe_submit_skb(sa, dev, skb, ctx);
+	if (err) {
+		airoha_soe_free_ctx(ctx);
+		return err;
+	}
+
+	return 0;
+}
+
+static bool airoha_soe_rx_parse_ipv4(struct sk_buff *skb,
+				     struct airoha_soe_rx_info *info)
+{
+	struct ip_esp_hdr *esph;
+	struct udphdr *uh;
+	struct iphdr *iph;
+	int iphlen;
+	int udp_len;
+	int packet_len;
+
+	if (skb->protocol != htons(ETH_P_IP)) {
+		if (!pskb_may_pull(skb, 1) || (skb->data[0] >> 4) != 4)
+			return false;
+
+		skb->protocol = htons(ETH_P_IP);
+	}
+
+	if (!pskb_may_pull(skb, sizeof(*iph)))
+		return false;
+
+	iph = ip_hdr(skb);
+	if (iph->version != 4 || ip_is_fragment(iph))
+		return false;
+
+	iphlen = iph->ihl * 4;
+	packet_len = ntohs(iph->tot_len);
+	if (iphlen < sizeof(*iph) || packet_len > skb->len)
+		return false;
+
+	if (iph->protocol == IPPROTO_ESP) {
+		if (packet_len <= iphlen + sizeof(*esph) ||
+		    !pskb_may_pull(skb, iphlen + sizeof(*esph)))
+			return false;
+
+		esph = (struct ip_esp_hdr *)(skb->data + iphlen);
+		if (!esph->spi)
+			return false;
+
+		info->packet_len = packet_len;
+		info->encap = false;
+		info->sport = 0;
+		info->dport = 0;
+		info->spi = esph->spi;
+
+		return true;
+	}
+
+	if (iph->protocol != IPPROTO_UDP ||
+	    !pskb_may_pull(skb, iphlen + sizeof(*uh) + sizeof(*esph)))
+		return false;
+
+	uh = (struct udphdr *)(skb->data + iphlen);
+	udp_len = ntohs(uh->len);
+	if (uh->dest != htons(AIROHA_SOE_NATT_PORT) ||
+	    udp_len <= sizeof(*uh) + sizeof(*esph) ||
+	    iphlen + udp_len != packet_len || packet_len > skb->len)
+		return false;
+
+	esph = (struct ip_esp_hdr *)(skb->data + iphlen + sizeof(*uh));
+	if (!esph->spi)
+		return false;
+
+	info->packet_len = packet_len;
+	info->encap = true;
+	info->sport = uh->source;
+	info->dport = uh->dest;
+	info->spi = esph->spi;
+
+	return true;
+}
+
+/* Plain ESP/NAT-T first arrives as normal RX, then is bounced to SOE decrypt. */
+bool airoha_soe_rx_plain_skb(struct airoha_gdm_dev *dev, struct sk_buff *skb,
+			     struct net_device *rx_dev, u16 foe_hash,
+			     u32 foe_reason, bool foe_valid)
+{
+	struct airoha_soe_xfrm_state *state;
+	struct airoha_soe_rx_info info = {};
+	struct airoha_soe_ctx *ctx;
+	xfrm_address_t daddr = {};
+	struct xfrm_state *x;
+	int err;
+
+	if (!dev || !skb || !rx_dev)
+		return false;
+
+	if (!dev->eth->soe || !(rx_dev->features & NETIF_F_HW_ESP))
+		return false;
+
+	if (!atomic_read(&dev->soe_xfrm_state_count))
+		return false;
+
+	/* The packet is still in the driver RX path after eth_type_trans(). */
+	skb_reset_network_header(skb);
+	if (!airoha_soe_rx_parse_ipv4(skb, &info))
+		return false;
+
+	if (skb->len != info.packet_len && pskb_trim(skb, info.packet_len))
+		return false;
+
+	daddr.a4 = ip_hdr(skb)->daddr;
+	x = xfrm_input_state_lookup(dev_net(rx_dev), skb->mark, &daddr,
+				    info.spi, IPPROTO_ESP, AF_INET);
+	if (!x)
+		return false;
+
+	if (x->xso.dir != XFRM_DEV_OFFLOAD_IN)
+		goto put_state;
+	if (x->xso.type != XFRM_DEV_OFFLOAD_PACKET)
+		goto put_state;
+	if (x->xso.dev != rx_dev)
+		goto put_state;
+	if ((info.encap &&
+	     (!x->encap || x->encap->encap_type != UDP_ENCAP_ESPINUDP)) ||
+	    (!info.encap && x->encap))
+		goto put_state;
+
+	if (info.encap && info.dport != x->encap->encap_dport)
+		goto put_state;
+
+	state = (struct airoha_soe_xfrm_state *)x->xso.offload_handle;
+	if (!state || state->dev != dev || !state->sa)
+		goto put_state;
+
+	ctx = kzalloc_obj(*ctx, GFP_ATOMIC);
+	if (!ctx)
+		goto put_state;
+
+	ctx->dir = AIROHA_SOE_CTX_IN;
+	ctx->rx.x = x;
+	ctx->rx.gdm_dev = dev;
+	ctx->rx.dev = rx_dev;
+	ctx->rx.saddr = ip_hdr(skb)->saddr;
+	ctx->rx.sport = info.sport;
+	ctx->rx.foe_hash = foe_hash;
+	ctx->rx.foe_reason = foe_reason;
+	ctx->rx.sa_index = state->sa->index;
+	ctx->rx.foe_valid = foe_valid;
+	ctx->rx.mark = skb->mark;
+
+		err = airoha_soe_submit_skb(state->sa, dev, skb, ctx);
+	if (err) {
+		airoha_soe_free_ctx(ctx);
+		goto drop_state;
+	}
+
+	return true;
+
+drop_state:
+	kfree_skb(skb);
+	return true;
+put_state:
+	xfrm_state_put(x);
+	return false;
+}
+
+static bool airoha_soe_complete_out(struct sk_buff *skb,
+				    struct airoha_soe_ctx *ctx)
+{
+	struct dst_entry *dst = ctx->dst;
+	struct net *net;
+	int err;
+
+	ctx->dst = NULL;
+	if (!pskb_may_pull(skb, ETH_HLEN + 1))
+		goto drop;
+	skb_pull(skb, ETH_HLEN);
+
+	err = airoha_soe_prepare_ip_headers(skb);
+	if (err)
+		goto drop;
+
+	/* Re-enter dst_output() with the original dst after hardware ESP encode. */
+	skb->protocol = htons(ETH_P_IP);
+	skb_dst_drop(skb);
+	skb_dst_set(skb, dst);
+	skb->ignore_df = 1;
+	net = dev_net(dst->dev);
+	kfree(ctx);
+	dst_output(net, NULL, skb);
+
+	return true;
+
+drop:
+	dst_release(dst);
+	kfree(ctx);
+	kfree_skb(skb);
+	return true;
+}
+
+static bool airoha_soe_complete_in(struct sk_buff *skb,
+				   struct airoha_soe_ctx *ctx)
+{
+	struct xfrm_state *x = ctx->rx.x;
+	struct net_device *rx_dev = ctx->rx.dev;
+	struct xfrm_offload *xo;
+	struct sec_path *sp;
+	int err;
+
+	if (!pskb_may_pull(skb, ETH_HLEN + 1))
+		goto drop;
+	skb_pull(skb, ETH_HLEN);
+
+	err = airoha_soe_prepare_ip_headers(skb);
+	if (err)
+		goto drop;
+
+	skb->dev = rx_dev;
+	skb->mark = ctx->rx.mark;
+	skb->ip_summed = CHECKSUM_NONE;
+	skb_reset_mac_header(skb);
+	skb_reset_mac_len(skb);
+	skb->pkt_type = PACKET_HOST;
+	skb->encapsulation = 0;
+	skb_dst_drop(skb);
+
+	if (x->encap && x->encap->encap_type == UDP_ENCAP_ESPINUDP &&
+	    (ctx->rx.saddr != x->props.saddr.a4 ||
+	     ctx->rx.sport != x->encap->encap_sport)) {
+		xfrm_address_t ipaddr = {
+			.a4 = ctx->rx.saddr,
+		};
+
+		km_new_mapping(x, &ipaddr, ctx->rx.sport);
+	}
+
+	/* Tell xfrm_input() equivalent consumers that hardware already decrypted. */
+	sp = secpath_set(skb);
+	if (!sp)
+		goto drop;
+
+	if (sp->len == XFRM_MAX_DEPTH) {
+		secpath_reset(skb);
+		goto drop;
+	}
+
+	sp->xvec[sp->len++] = x;
+	sp->olen++;
+	ctx->rx.x = NULL;
+	xo = xfrm_offload(skb);
+	if (!xo) {
+		secpath_reset(skb);
+		goto drop;
+	}
+
+	xo->flags = CRYPTO_DONE;
+	xo->status = CRYPTO_SUCCESS;
+
+	airoha_soe_trace_rx_complete(skb, ctx, x);
+
+	/* SOE decrypt completion reaches the CPU before the routed plaintext
+	 * packet has selected its final egress port. Preserve the original FOE
+	 * hash and SA hop until the Ethernet xmit path can bind that decrypt
+	 * entry with the completed L2/PSE descriptor.
+	 */
+	if (ctx->rx.foe_valid)
+		airoha_ppe_soe_mark_skb(&ctx->rx.gdm_dev->eth->ppe->dev, skb,
+					ctx->rx.foe_hash, ctx->rx.sa_index,
+					AIROHA_SOE_HOP_INFO_DECRYPT);
+
+	kfree(ctx);
+	netif_rx(skb);
+
+	return true;
+
+drop:
+	airoha_soe_free_ctx(ctx);
+	kfree_skb(skb);
+	return true;
+}
+
+bool airoha_soe_rx_skb(struct airoha_soe *soe, struct sk_buff *skb,
+		       unsigned int sa_index, u32 hop_flags)
+{
+	struct airoha_soe_ctx *ctx;
+	struct airoha_soe_sa *sa;
+
+	if (!soe || !skb || sa_index >= AIROHA_SOE_NUM_SA)
+		return false;
+
+	rcu_read_lock();
+	sa = rcu_dereference(soe->sa[sa_index]);
+	if (!sa) {
+		rcu_read_unlock();
+		return false;
+	}
+
+	if (hop_flags >= AIROHA_SOE_HOP_FLAG_ERROR_BASE) {
+		ctx = airoha_soe_pop_ctx(sa, AIROHA_SOE_CTX_OUT);
+		if (!ctx)
+			ctx = airoha_soe_pop_ctx(sa, AIROHA_SOE_CTX_IN);
+		rcu_read_unlock();
+		airoha_soe_free_ctx(ctx);
+		kfree_skb(skb);
+		return true;
+	}
+
+	if (hop_flags == AIROHA_SOE_HOP_FLAG_ENCRYPTED) {
+		ctx = airoha_soe_pop_ctx(sa, AIROHA_SOE_CTX_OUT);
+		rcu_read_unlock();
+		if (!ctx) {
+			kfree_skb(skb);
+			return true;
+		}
+		return airoha_soe_complete_out(skb, ctx);
+	}
+
+	if (hop_flags == AIROHA_SOE_HOP_FLAG_DECRYPTED) {
+		ctx = airoha_soe_pop_ctx(sa, AIROHA_SOE_CTX_IN);
+		rcu_read_unlock();
+		if (!ctx) {
+			kfree_skb(skb);
+			return true;
+		}
+		return airoha_soe_complete_in(skb, ctx);
+	}
+
+	rcu_read_unlock();
+	return false;
+}
+
+bool airoha_soe_has_pending_rx(struct airoha_soe *soe)
+{
+	if (!soe)
+		return false;
+
+	return !!atomic_read(&soe->pending_rx);
+}
+
+int airoha_soe_xfrm_ppe_info(const struct dst_entry *dst, u8 *sa_index, u8 *hop)
+{
+	struct airoha_soe_xfrm_state *state;
+	struct net_device *netdev;
+	struct xfrm_state *x;
+
+	if (!dst || !sa_index || !hop)
+		return -EINVAL;
+
+	x = dst_xfrm(dst);
+	if (!x || x->xso.type != XFRM_DEV_OFFLOAD_PACKET)
+		return -EOPNOTSUPP;
+
+	state = (struct airoha_soe_xfrm_state *)x->xso.offload_handle;
+	if (!state || !state->sa)
+		return -ENODEV;
+
+	if (!state->dev)
+		return -ENODEV;
+
+	netdev = netdev_from_priv(state->dev);
+	if (netdev != x->xso.dev || !(netdev->features & NETIF_F_HW_ESP))
+		return -ENODEV;
+
+	switch (x->xso.dir) {
+	case XFRM_DEV_OFFLOAD_OUT:
+		*hop = AIROHA_SOE_HOP_INFO_ENCRYPT;
+		break;
+	case XFRM_DEV_OFFLOAD_IN:
+		*hop = AIROHA_SOE_HOP_INFO_DECRYPT;
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	*sa_index = state->sa->index;
+
+	return 0;
+}
+
+static int airoha_soe_xfrm_state_add(struct net_device *dev,
+				     struct xfrm_state *x,
+				     struct netlink_ext_ack *extack)
+{
+	struct airoha_soe_xfrm_state *state;
+	struct airoha_gdm_dev *gdm_dev;
+	struct airoha_soe *soe;
+	gfp_t gfp;
+	int err;
+
+	if (dev->xfrmdev_ops != &airoha_soe_xfrmdev_ops ||
+	    !(dev->features & NETIF_F_HW_ESP))
+		return -EOPNOTSUPP;
+
+	gdm_dev = netdev_priv(dev);
+	soe = airoha_soe_get_ref(gdm_dev->eth->soe);
+	if (!soe)
+		return -ENODEV;
+
+	gfp = (x->xso.flags & XFRM_DEV_OFFLOAD_FLAG_ACQ) ? GFP_ATOMIC :
+							   GFP_KERNEL;
+	state = kzalloc_obj(*state, gfp);
+	if (!state) {
+		airoha_soe_put_ref(soe);
+		return -ENOMEM;
+	}
+
+	state->dev = gdm_dev;
+	state->soe = soe;
+
+	if (x->xso.flags & XFRM_DEV_OFFLOAD_FLAG_ACQ)
+		goto out;
+
+	err = airoha_soe_alloc_sa(soe, x, extack, &state->sa);
+	if (err)
+		goto err_free;
+
+	atomic_inc(&gdm_dev->soe_xfrm_state_count);
+	state->counted = true;
+out:
+	x->xso.offload_handle = (unsigned long)state;
+	return 0;
+
+err_free:
+	kfree(state);
+	airoha_soe_put_ref(soe);
+	return err;
+}
+
+static void airoha_soe_xfrm_state_delete(struct net_device *dev,
+					 struct xfrm_state *x)
+{
+	struct airoha_soe_xfrm_state *state;
+
+	state = (struct airoha_soe_xfrm_state *)x->xso.offload_handle;
+	if (state && state->sa) {
+		airoha_ppe_soe_flush_sa(state->dev->eth->ppe, state->sa->index);
+		airoha_soe_abort_sa(state->sa);
+	}
+}
+
+static void airoha_soe_xfrm_state_free(struct net_device *dev,
+				       struct xfrm_state *x)
+{
+	struct airoha_soe_xfrm_state *state;
+
+	state = (struct airoha_soe_xfrm_state *)xchg(&x->xso.offload_handle, 0);
+	if (!state)
+		return;
+
+	if (state->sa) {
+		airoha_ppe_soe_flush_sa(state->dev->eth->ppe,
+					state->sa->index);
+		airoha_soe_free_sa(state->sa);
+	}
+	if (state->counted)
+		atomic_dec(&state->dev->soe_xfrm_state_count);
+	airoha_soe_put_ref(state->soe);
+	kfree(state);
+}
+
+static bool airoha_soe_xfrm_offload_ok(struct sk_buff *skb,
+				       struct xfrm_state *x)
+{
+	struct airoha_soe_xfrm_state *state;
+	struct net_device *dev = x->xso.dev;
+
+	if (!dev || !(dev->features & NETIF_F_HW_ESP))
+		return false;
+
+	if (x->xso.type != XFRM_DEV_OFFLOAD_PACKET ||
+	    x->xso.dir != XFRM_DEV_OFFLOAD_OUT)
+		return false;
+
+	state = (struct airoha_soe_xfrm_state *)x->xso.offload_handle;
+
+	return state && state->sa;
+}
+
+static int airoha_soe_xfrm_packet_xmit_gso(struct sk_buff *skb,
+					   struct xfrm_state *x,
+					   struct airoha_soe_xfrm_state *state)
+{
+	struct sk_buff *segs, *nskb;
+	int err;
+
+	segs = skb_gso_segment(skb, 0);
+	if (IS_ERR(segs)) {
+		XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTERROR);
+		kfree_skb(skb);
+		return PTR_ERR(segs);
+	}
+
+	consume_skb(skb);
+
+	skb_list_walk_safe(segs, skb, nskb) {
+		skb_mark_not_on_list(skb);
+		err = airoha_soe_xmit(state->sa, state->dev, skb, x);
+		if (err) {
+			XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTERROR);
+			kfree_skb(skb);
+			kfree_skb_list(nskb);
+			return err;
+		}
+	}
+
+	return 0;
+}
+
+static int airoha_soe_xfrm_packet_xmit(struct sk_buff *skb,
+				       struct xfrm_state *x)
+{
+	struct airoha_soe_xfrm_state *state;
+	struct net_device *netdev;
+	int err = -EHOSTUNREACH;
+
+	state = (struct airoha_soe_xfrm_state *)x->xso.offload_handle;
+	if (!state || !state->sa || !state->dev)
+		goto drop;
+
+	netdev = netdev_from_priv(state->dev);
+	if (netdev->xfrmdev_ops != &airoha_soe_xfrmdev_ops ||
+	    !(netdev->features & NETIF_F_HW_ESP))
+		goto drop;
+
+	if (skb_is_gso(skb))
+		return airoha_soe_xfrm_packet_xmit_gso(skb, x, state);
+
+	err = airoha_soe_xmit(state->sa, state->dev, skb, x);
+	if (err)
+		goto drop;
+
+	return 0;
+
+drop:
+	XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTERROR);
+	kfree_skb(skb);
+	return err;
+}
+
+static int airoha_soe_xfrm_policy_add(struct xfrm_policy *x,
+				      struct netlink_ext_ack *extack)
+{
+	if (x->xdo.type != XFRM_DEV_OFFLOAD_PACKET) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE supports XFRM packet policies only");
+		return -EOPNOTSUPP;
+	}
+
+	if (xfrm_policy_id2dir(x->index) >= XFRM_POLICY_MAX) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE does not offload socket policies");
+		return -EOPNOTSUPP;
+	}
+
+	if (x->xfrm_nr != 1 ||
+	    x->xfrm_vec[0].id.proto != IPPROTO_ESP) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "SOE offloads ESP policies only");
+		return -EOPNOTSUPP;
+	}
+
+	if (!x->xdo.dev || !(x->xdo.dev->features & NETIF_F_HW_ESP)) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE ESP offload is disabled");
+		return -EOPNOTSUPP;
+	}
+
+	switch (x->xdo.dir) {
+	case XFRM_DEV_OFFLOAD_IN:
+	case XFRM_DEV_OFFLOAD_OUT:
+		return 0;
+	default:
+		NL_SET_ERR_MSG_MOD(extack, "SOE supports in/out policies only");
+		return -EOPNOTSUPP;
+	}
+}
+
+static struct net_device *airoha_soe_dsa_conduit_get(struct net_device *dev)
+{
+	struct net_device *conduit;
+	struct dsa_port *dp;
+
+	if (!dsa_user_dev_check(dev))
+		return NULL;
+
+	/* DSA users expose XFRM, but SOE is attached to their CPU conduit. */
+	dp = dsa_port_from_netdev(dev);
+	if (IS_ERR(dp) || !dp->cpu_dp)
+		return NULL;
+
+	conduit = dsa_port_to_conduit(dp);
+	if (!conduit || conduit->xfrmdev_ops != &airoha_soe_xfrmdev_ops)
+		return NULL;
+
+	dev_hold(conduit);
+
+	return conduit;
+}
+
+static int airoha_soe_dsa_xfrm_state_add(struct net_device *dev,
+					 struct xfrm_state *x,
+					 struct netlink_ext_ack *extack)
+{
+	struct net_device *conduit;
+	int err;
+
+	conduit = airoha_soe_dsa_conduit_get(dev);
+	if (!conduit) {
+		NL_SET_ERR_MSG_MOD(extack, "SOE DSA conduit is unavailable");
+		return -EOPNOTSUPP;
+	}
+
+	err = airoha_soe_xfrm_state_add(conduit, x, extack);
+	dev_put(conduit);
+
+	return err;
+}
+
+static void airoha_soe_dsa_xfrm_state_delete(struct net_device *dev,
+					     struct xfrm_state *x)
+{
+	airoha_soe_xfrm_state_delete(dev, x);
+}
+
+static void airoha_soe_dsa_xfrm_state_free(struct net_device *dev,
+					   struct xfrm_state *x)
+{
+	airoha_soe_xfrm_state_free(dev, x);
+}
+
+static bool airoha_soe_dsa_xfrm_offload_ok(struct sk_buff *skb,
+					   struct xfrm_state *x)
+{
+	return airoha_soe_xfrm_offload_ok(skb, x);
+}
+
+static int airoha_soe_dsa_xfrm_policy_add(struct xfrm_policy *x,
+					  struct netlink_ext_ack *extack)
+{
+	return airoha_soe_xfrm_policy_add(x, extack);
+}
+
+static int airoha_soe_dsa_xfrm_packet_xmit(struct sk_buff *skb,
+					   struct xfrm_state *x)
+{
+	return airoha_soe_xfrm_packet_xmit(skb, x);
+}
+
+static const struct xfrmdev_ops airoha_soe_xfrmdev_ops = {
+	.xdo_dev_state_add = airoha_soe_xfrm_state_add,
+	.xdo_dev_state_delete = airoha_soe_xfrm_state_delete,
+	.xdo_dev_state_free = airoha_soe_xfrm_state_free,
+	.xdo_dev_offload_ok = airoha_soe_xfrm_offload_ok,
+	.xdo_dev_policy_add = airoha_soe_xfrm_policy_add,
+	.xdo_dev_packet_xmit = airoha_soe_xfrm_packet_xmit,
+};
+
+static const struct xfrmdev_ops airoha_soe_dsa_xfrmdev_ops = {
+	.xdo_dev_state_add = airoha_soe_dsa_xfrm_state_add,
+	.xdo_dev_state_delete = airoha_soe_dsa_xfrm_state_delete,
+	.xdo_dev_state_free = airoha_soe_dsa_xfrm_state_free,
+	.xdo_dev_offload_ok = airoha_soe_dsa_xfrm_offload_ok,
+	.xdo_dev_policy_add = airoha_soe_dsa_xfrm_policy_add,
+	.xdo_dev_packet_xmit = airoha_soe_dsa_xfrm_packet_xmit,
+};
+
+static void airoha_soe_dsa_proxy_enable(struct net_device *dev)
+{
+	struct net_device *conduit;
+
+	conduit = airoha_soe_dsa_conduit_get(dev);
+	if (!conduit)
+		return;
+
+	if (dev->xfrmdev_ops && dev->xfrmdev_ops != &airoha_soe_dsa_xfrmdev_ops)
+		goto out;
+
+	/* Mirror ESP capability onto DSA users while programming SAs on the conduit. */
+	dev->xfrmdev_ops = &airoha_soe_dsa_xfrmdev_ops;
+	dev->hw_features |= NETIF_F_HW_ESP;
+	dev->hw_enc_features |= NETIF_F_HW_ESP;
+	dev->wanted_features |= NETIF_F_HW_ESP;
+
+	conduit->wanted_features |= NETIF_F_HW_ESP;
+	netdev_update_features(conduit);
+	netdev_update_features(dev);
+out:
+	dev_put(conduit);
+}
+
+static void airoha_soe_dsa_proxy_clear(struct net_device *dev)
+{
+	if (dev->xfrmdev_ops != &airoha_soe_dsa_xfrmdev_ops)
+		return;
+
+	dev->wanted_features &= ~NETIF_F_HW_ESP;
+	dev->hw_features &= ~NETIF_F_HW_ESP;
+	dev->hw_enc_features &= ~NETIF_F_HW_ESP;
+	netdev_update_features(dev);
+	dev->xfrmdev_ops = NULL;
+}
+
+static void airoha_soe_dsa_proxy_scan(bool enable)
+{
+	struct net_device *dev;
+
+	for_each_netdev(&init_net, dev) {
+		if (enable)
+			airoha_soe_dsa_proxy_enable(dev);
+		else
+			airoha_soe_dsa_proxy_clear(dev);
+	}
+}
+
+static int airoha_soe_netdev_event(struct notifier_block *nb,
+				   unsigned long event, void *ptr)
+{
+	switch (event) {
+	case NETDEV_REGISTER:
+	case NETDEV_CHANGENAME:
+		airoha_soe_dsa_proxy_scan(true);
+		break;
+	}
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block airoha_soe_netdev_notifier = {
+	.notifier_call = airoha_soe_netdev_event,
+};
+
+void airoha_soe_build_netdev(struct net_device *netdev,
+			     airoha_soe_xmit_skb_t xmit_skb)
+{
+	struct airoha_gdm_dev *dev = netdev_priv(netdev);
+
+	atomic_set(&dev->soe_xfrm_state_count, 0);
+	dev->soe_xmit_skb = xmit_skb;
+
+	if (!xmit_skb ||
+	    !(airoha_soe_features(dev->eth->soe) & AIROHA_SOE_FEATURE_ESP))
+		return;
+
+	netdev->xfrmdev_ops = &airoha_soe_xfrmdev_ops;
+	netdev->hw_features |= NETIF_F_HW_ESP;
+	netdev->hw_enc_features |= NETIF_F_HW_ESP;
+}
+
+void airoha_soe_teardown_netdev(struct net_device *netdev)
+{
+	struct airoha_gdm_dev *dev = netdev_priv(netdev);
+
+	if (netdev->xfrmdev_ops == &airoha_soe_xfrmdev_ops)
+		netdev->xfrmdev_ops = NULL;
+	dev->soe_xmit_skb = NULL;
+}
+
+int airoha_soe_set_features(struct net_device *netdev,
+			    netdev_features_t features)
+{
+	netdev_features_t changed = (netdev->features ^ features) &
+				    NETIF_F_HW_ESP;
+	struct airoha_gdm_dev *dev = netdev_priv(netdev);
+
+	if (!changed)
+		return 0;
+
+	if ((features & NETIF_F_HW_ESP) &&
+	    !(airoha_soe_features(dev->eth->soe) & AIROHA_SOE_FEATURE_ESP))
+		return -EOPNOTSUPP;
+
+	if (atomic_read(&dev->soe_xfrm_state_count)) {
+		netdev_err(netdev,
+			   "cannot change ESP features with active SAs\n");
+		return -EBUSY;
+	}
+
+	return 0;
+}
+
+static struct device_node *airoha_soe_find_node(struct airoha_eth *eth)
+{
+	struct device_node *parent, *np;
+
+	if (!eth->dev->of_node)
+		return NULL;
+
+	parent = of_get_parent(eth->dev->of_node);
+	if (!parent)
+		return NULL;
+
+	/* SOE is a sibling DT node; Ethernet owns the provider lifetime. */
+	for_each_child_of_node(parent, np) {
+		if (!of_device_is_available(np) ||
+		    !of_device_is_compatible(np, "airoha,en7581-soe"))
+			continue;
+
+		of_node_put(parent);
+		return np;
+	}
+
+	of_node_put(parent);
+
+	return NULL;
+}
+
+int airoha_soe_init(struct airoha_eth *eth)
+{
+	struct device *dev = eth->dev;
+	struct device_node *np;
+	struct resource res;
+	struct airoha_soe *soe;
+	void __iomem *base;
+	int err;
+
+	np = airoha_soe_find_node(eth);
+	if (!np)
+		return 0;
+
+	err = of_address_to_resource(np, 0, &res);
+	if (err)
+		goto put_node;
+
+	base = devm_ioremap_resource(dev, &res);
+	if (IS_ERR(base)) {
+		err = PTR_ERR(base);
+		goto put_node;
+	}
+
+	soe = devm_kzalloc(dev, sizeof(*soe), GFP_KERNEL);
+	if (!soe) {
+		err = -ENOMEM;
+		goto put_node;
+	}
+
+	soe->dev = dev;
+	soe->base = base;
+	mutex_init(&soe->sa_lock);
+	spin_lock_init(&soe->state_lock);
+	refcount_set(&soe->refcnt, 1);
+	init_completion(&soe->released);
+
+	/* Enable the packet engines; reset leaves SOE present but idle. */
+	writel(AIROHA_SOE_INT_ALL, base + AIROHA_SOE_INT_STS);
+	writel(AIROHA_SOE_CNT_CLR_ALL, base + AIROHA_SOE_CNT_CLR);
+	writel(AIROHA_SOE_INT_ALL, base + AIROHA_SOE_INT_EN);
+	writel(AIROHA_SOE_GLB_CFG_ENC_EN | AIROHA_SOE_GLB_CFG_DEC_EN,
+	       base + AIROHA_SOE_GLB_CFG);
+
+	err = register_netdevice_notifier(&airoha_soe_netdev_notifier);
+	if (err)
+		goto disable_soe;
+
+	eth->soe = soe;
+
+	rtnl_lock();
+	airoha_soe_dsa_proxy_scan(true);
+	rtnl_unlock();
+
+	of_node_put(np);
+
+	return 0;
+
+disable_soe:
+	writel(0, base + AIROHA_SOE_GLB_CFG);
+	writel(0, base + AIROHA_SOE_INT_EN);
+	writel(0xffffffff, base + AIROHA_SOE_INT_STS);
+put_node:
+	of_node_put(np);
+
+	return err;
+}
+
+void airoha_soe_deinit(struct airoha_eth *eth)
+{
+	struct airoha_soe *soe = eth->soe;
+	unsigned long flags;
+
+	if (!soe)
+		return;
+
+	eth->soe = NULL;
+
+	spin_lock_irqsave(&soe->state_lock, flags);
+	soe->dead = true;
+	spin_unlock_irqrestore(&soe->state_lock, flags);
+
+	rtnl_lock();
+	airoha_soe_dsa_proxy_scan(false);
+	rtnl_unlock();
+	unregister_netdevice_notifier(&airoha_soe_netdev_notifier);
+
+	airoha_soe_put_ref(soe);
+	wait_for_completion(&soe->released);
+
+	writel(0, soe->base + AIROHA_SOE_GLB_CFG);
+	writel(0, soe->base + AIROHA_SOE_INT_EN);
+	writel(0xffffffff, soe->base + AIROHA_SOE_INT_STS);
+}
-- 
2.53.0

Re: [RFC PATCH net-next 0/7] net: airoha: add EN7581 SOE ESP packet offload

[Jihong Min]

On 6/14/26 13:00, Jihong Min wrote:
> Add Secure Offload Engine (SOE) support for the Airoha EN7581 Ethernet
> driver. SOE provides inline ESP packet offload for native ESP and NAT-T
> traffic, with the Ethernet/QDMA path used to submit packets to the SOE
> block and the PPE path used to bind eligible ESP flows. NETIF_F_GSO_ESP
> and NETIF_F_HW_ESP_TX_CSUM are intentionally left out for now and will be
> revisited separately for feasibility.
> 
> This is posted as RFC because the code was originally developed and tested
> against an OpenWrt 6.18 Airoha tree, not against the current upstream
> net-next driver. The original OpenWrt commit used as the source for this
> RFC is available at:
> https://github.com/hurryman2212/OpenW1700k-test/commit/7c1b5e662f7790b3d23ed143beadc1dcbf6d15f7
> 
> The SOE part is intentionally linked into the airoha Ethernet module
> instead of being exposed as an independent crypto or platform driver. The
> user-visible ESP offload control is a netdev capability: xfrmdev_ops and
> NETIF_F_HW_ESP live on the target netdev, and the feature can be controlled
> through the usual netdev feature path. SOE also shares the FE/QDMA/PPE
> datapath, private queues, DSA conduit handling and netdev lifetime owned by
> airoha_eth.
> 
> Patch 1 adds xdo_dev_packet_xmit() because the existing XFRM packet
> offload transmit path does not provide a hook for hardware whose ESP engine
> is reached through device-specific packet forwarding. SOE needs to consume
> the skb, add a hardware hop descriptor, steer it to a private QDMA path and
> return the final transmit status. Drivers that do not implement the
> optional callback keep the existing XFRM output behavior.
> 
> Jihong Min (7):
>   xfrm: allow packet offload drivers to own transmit
>   dt-bindings: net: airoha: add EN7581 SOE
>   arm64: dts: airoha: add EN7581 SOE node
>   net: airoha: add SOE registers and driver state
>   net: airoha: add QDMA support for SOE packets
>   net: airoha: add PPE support for SOE flows
>   net: airoha: add SOE XFRM packet offload support
> 
>  .../bindings/net/airoha,en7581-soe.yaml       |   48 +
>  MAINTAINERS                                   |    1 +
>  arch/arm64/boot/dts/airoha/en7581.dtsi        |    6 +
>  drivers/net/ethernet/airoha/Kconfig           |   13 +
>  drivers/net/ethernet/airoha/Makefile          |    1 +
>  drivers/net/ethernet/airoha/airoha_eth.c      |  668 +++++-
>  drivers/net/ethernet/airoha/airoha_eth.h      |   40 +
>  drivers/net/ethernet/airoha/airoha_ppe.c      |  606 +++++-
>  drivers/net/ethernet/airoha/airoha_regs.h     |   16 +
>  drivers/net/ethernet/airoha/airoha_soe.c      | 1896 +++++++++++++++++
>  drivers/net/ethernet/airoha/airoha_soe.h      |  126 ++
>  include/linux/netdevice.h                     |    8 +
>  include/linux/soc/airoha/airoha_offload.h     |    5 +
>  net/xfrm/xfrm_output.c                        |   11 +
>  14 files changed, 3342 insertions(+), 103 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/net/airoha,en7581-soe.yaml
>  create mode 100644 drivers/net/ethernet/airoha/airoha_soe.c
>  create mode 100644 drivers/net/ethernet/airoha/airoha_soe.h
> 

I noticed, after posting this RFC, that I forgot to include the
following trailer while preparing the latest patch series:

Assisted-by: Codex:gpt-5.5

These patches were written and tested with AI assistance, although I've
reviewed the resulting code and test results. I'll include the trailer
properly in future revisions or submissions. Sorry.


Sincerely,
Jihong Min