From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57B1C37C112 for ; Tue, 6 Oct 2026 10:03:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.180.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791280982; cv=none; b=K6OFOrEwIQKD2Fxv7dqgROvpr+LzKIyuMcK61XTsvzlxdJqjxixaPYZdCDxFejE61hTP9nhLqhkPOKAekTFT5ymKveaukv8PzfWpRBd51yeUlVtBs9+Qu+3ZXtKlTBKBU98K/rS82dxmggIKUOzI2t9Fv7bdRNXwKHXi8/gjSFA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791280982; c=relaxed/simple; bh=rlG4gBq9hSd8jZaRbTERxvVHIpEVqUFZY/mUc14FHx0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GM+KCaIC00KLMstYOkOtzvEVbXe5eDACTvc8MMUaT/S5riThuIv2EfKcyOGf5ZM3O/PYUNAxsxjyh7PZYMuuhipdHHuDur501yJv4OvPSOeJjVdfUVSb4uuCzfLWbY5AbqgJQzritvHUvDcKW/83vf78OR7dpbqN2Fa8QJso24I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com; spf=pass smtp.mailfrom=oss.qualcomm.com; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b=IN5oh3/s; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b=KUo2cs0x; arc=none smtp.client-ip=205.220.180.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b="IN5oh3/s"; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="KUo2cs0x" Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6966eISk548492 for ; Tue, 6 Oct 2026 10:02:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= 0qhBE5M+IMw0fom1auSpZ+/gMrGIPvg4PYKmlKUoZF0=; b=IN5oh3/sW6rmTSoi kz0Sb+vAeAgHvgMHvqToEpIaoIIpmUrlV+e9S4xXG1b/b8OmDJ9re6gaL8TzthCj skKxYU/Fmr+XXrycr1aw+L8pohhbprliwvkWEoHEODRp7+X1hXXFhPSpDSkVNaUm 81To3v12my9csMo6Cq1X07rAu6NFsLX0689KotbxExX9wRQZ/nA0t3kzUjPX3VUu H/ShL76Jh1dzq4aRnZ+g2h2fVc/PXLZiqaMxwUJSY0Cfdz9RwyXM0rnuJOjwXShi 1uv1S9Cg9NtYLu5jQ6zimGZQeVYYcGBhOntWxnlMidIxg3eHQTMwPGpOgvKI5MRd +rwjcg== Received: from mail-ua1-f69.google.com (mail-ua1-f69.google.com [209.85.222.69]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4h49tvvqqn-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Tue, 06 Oct 2026 10:02:59 +0000 (GMT) Received: by mail-ua1-f69.google.com with SMTP id a1e0cc1a2514c-98c88cefeb3so821887241.3 for ; Tue, 06 Oct 2026 03:02:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1791280978; x=1791885778; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0qhBE5M+IMw0fom1auSpZ+/gMrGIPvg4PYKmlKUoZF0=; b=KUo2cs0xyv0lcf2CWZ+Jio8cXEj5t8re9I23o+FPzur2Wj0PPEqgMq8M/j73r5nHNL /NqVoDDXNgGkMKaJhx5LVjW1xCdAz2IbaP6bOgwptUgkInG2O8yRG18dIJMvMDctI/Vz DjnAmADBuVwLMpHzjyuoVaORYoEnP8HcsO+ZnTNh+KLoCxxj7lQ1DeDox+n48qYN3t2k FTLo0PJCqzrsc8RBoTGFCYJtO7lOj8V2VsuY8iZjseNmQLCmxqnG9sFjt7mSnBkstRId JeWMebApNYKbLIOFUBKhHJIUQpa+FpsbNrwQHiR1DaCJpfSBE2bkAkGxy77RHMKTTvhk ehLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791280978; x=1791885778; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0qhBE5M+IMw0fom1auSpZ+/gMrGIPvg4PYKmlKUoZF0=; b=KBVZDyfRbtB9950a6EPVyhdkKURNDWV942hbPzM6kLCP8I6RIfVXGiy8fcHzYRerQ/ vuHhW0XKs1++vFHMYXggdAF0CCuO3r5Lo+j7t+SPbkhMPdr0cKWOVAHqdWEY//15EfOR ceZKhrEXhXG+5WPtJZQhMiiHYUKcHHv6QL9qi1p5hcWZ09HiEO7ebbsnFV7Ul4tQwBFg WJ4Ave3oNtkdTcKYSbZeyWxL6NS2tevJ5BKEBxcRTxFEB5OgkRIIKA5aaorYPG9LaV3Q wJmxjXBeaNJaSILqC+1Vrg+O37bW4d7ALY9iXVmchz0VxAO/CJJRWZzUyuLHmZZjaROo BvEA== X-Gm-Message-State: AFq9FYI1GvUBPcIfkUN6Etyrnnqe6YsyvC4b7x8Nv0gO2en2XS7/DxeT e/Plr5xNpobRylpE99SY+QU9AN8GEyE93XqijUbXTNCfHW6V3v4lG0HyBNQtlJAZzmkvWvx0LB2 DWI/eHmFHt6S+F/n/NOhfi780+8Rix5eBQ/m0o9+nYzUB8Wtce/uZxKyXuXI= X-Gm-Gg: AYBFou180lKDC6VN21oQ8SztaPgzslATlhsAI8DZoZNcn2sv7aeJT8Q4ZbMt1RbSCHl tGVKodqPJcH/Y8j3smUO9SLFmUu71QPc5wlndfIpcbfshYdEfHoQXIqKH5QkYq1XJakdlfZSxfQ PVePHf9Kiq9XjbkyGrxkSmRhj3FWQzeVZNO/hkrPBiTEVpQbkVuIETw13G/P1sFyifgoPbyThlx s96iX/FZDfz4DEMrJ1F2wxnLsgwQzB1jYgO8UdJefGeno/0fWBg5dRXw44B2ZwpuUOmOG6sImcm IvxN9alJy5tOJrgVUCR7i89waVFHxyHAzW8F412HCo0R1Um7PisjiL5/E9PxS7XJ3lnKc3tojdM eMcuVaP+p3MHaxA== X-Received: by 2002:a05:6102:f9b:b0:7b2:f3ff:6b34 with SMTP id ada2fe7eead31-7c877cee27cmr193003137.1.1791280978186; Tue, 06 Oct 2026 03:02:58 -0700 (PDT) X-Received: by 2002:a05:6102:f9b:b0:7b2:f3ff:6b34 with SMTP id ada2fe7eead31-7c877cee27cmr192994137.1.1791280977522; Tue, 06 Oct 2026 03:02:57 -0700 (PDT) Received: from localhost ([188.216.77.92]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-48c6fa8d2cfsm518654f8f.13.2026.10.06.03.02.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2026 03:02:56 -0700 (PDT) From: Lorenzo Bianconi Date: Tue, 06 Oct 2026 12:02:21 +0200 Subject: [PATCH net-next v4 2/2] net: stmmac: rework stmmac_rx to support XDP rx multi-buff Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261006-stmmac-rx-mb-v4-2-efa4ca974e3d@oss.qualcomm.com> References: <20261006-stmmac-rx-mb-v4-0-efa4ca974e3d@oss.qualcomm.com> In-Reply-To: <20261006-stmmac-rx-mb-v4-0-efa4ca974e3d@oss.qualcomm.com> To: Maxime Chevallier , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Maxime Coquelin , Alexandre Torgue , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Stanislav Fomichev , Jose Abreu , Ong Boon Leong Cc: netdev@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, linux-arm-kernel@lists.infradead.org, bpf@vger.kernel.org, Lorenzo Bianconi X-Mailer: b4 0.14.3 X-Authority-Analysis: v=2.4 cv=aN9lOr9m c=1 sm=1 tr=0 ts=6ac4c753 cx=c_pps a=UbhLPJ621ZpgOD2l3yZY1w==:117 a=WpTaRW6qxYHRGzLzQsVYzg==:17 a=IkcTkHD0fZMA:10 a=660iZSQnnn4A:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=3WHJM1ZQz_JShphwDgj5:22 a=EUspDBNiAAAA:8 a=v9h4JUQAhp93ei3YNAAA:9 a=QEXdDO2ut3YA:10 a=TOPH6uDL9cOC6tEoww4z:22 X-Proofpoint-GUID: JfHjZ3XzBbmFl2cHhYFvvE9MVlg76z-d X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYxMDA2MDAzOSBTYWx0ZWRfX3WK5GOLldj2R lvz5MpXH3V3R+TzdMxBM1arxtP5bY5yb08rTG+ajJ5/UstlA2x0juWm5IUINU7FR5WK02MVwiMW xMY5MwjkUPP93b/2pgfyZqHeSG18D/E6Z1bT2SD9VhxBWgOUFcYK/X8TQW+eINbV40pAG4dMmQY 90nIzGzJ4B/8UY+74peeRM6Vm8VjK9pBGl4CdaraofEtdZTd74DxGqQNRk+ZdRUjpTpd+m5yPjM wKsjMOVgdW8Vto3PntArSa5DRbH3f1iuhuRUHDOfRK3gbMzm7h9LB8Hj7w6gNsNmuUYvOBkuntJ 5XV050CmSaWG2FV8VV4NQ/NhAgNjkrPV8/E28mxAPtbgbs2eeGL14ioHoL0G4TgUVyikgkAccQE Cv8VbU/9l5t2HieoOLQuBJbDhBwMAF/+qY6HjRtblhnbM4e5qIpjqzjGSgocEGE172+XDam42uB 4x3DdMg6Wl9NKkNA66w== X-Proofpoint-Spam-Info: AW1haW4tMjYxMDA2MDAzOSBTYWx0ZWRfX9gL0HLwo2qZz O7cr88HTeMzu0eK8HEjmgqEKyg39SvsWygvBZZAi0mZLX06a9lC97V5SwWIB7nSc7++qUEgo+2a MLvMgYO6m2oCvX6U9B6F+dDdQdOEdTM= X-Proofpoint-ORIG-GUID: JfHjZ3XzBbmFl2cHhYFvvE9MVlg76z-d X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-10-06_02,2026-10-05_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 clxscore=1015 spamscore=0 phishscore=0 malwarescore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 bulkscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2609040000 definitions=main-2610060039 Build the xdp_buff by accumulating all the descriptors that make up a frame, so the XDP program runs on the full (possibly fragmented) packet instead of just the first buffer. When the frame is not consumed by the program, assemble the skb from the head buffer and the collected fragments via napi_build_skb()/xdp_update_skb_frags_info(). Note that XDP still rejects an MTU larger than ETH_DATA_LEN (stmmac_xdp_set_prog(), stmmac_change_mtu()) and disables SPH, so an ordinary frame fits in a single buffer. This path is therefore exercised by frames that do span several descriptors (for example jumbo frames the MAC admits through its jumbo-enable bit) and is groundwork for extending XDP to jumbo/SPH; it does not enable XDP jumbo support by itself. To do so, store the in-progress xdp_buff in rx_q->state instead of the partially built skb, so the accumulated head and fragments survive a NAPI poll boundary (mid-frame dma_own or dirty_rx break). The state is saved only while a frame is in progress and cleared once it completes, leaving it untouched when the poll does not process anything (e.g. netpoll invoked with a zero budget). In addition: - advertise NETDEV_XDP_ACT_RX_SG, since the RX path can now deliver non-linear buffers to XDP; - build the skb head with napi_build_skb() passing xdp->frame_sz, so skb_shinfo() lands on the same shared_info the fragments were accumulated into; - attach fragments with xdp_buff_add_frag(), which initializes all the shared_info fields and takes care of the pfmemalloc bit; - release the buffers collected in the xdp_buff when a frame is dropped on RX errors; the previous code relied on dev_kfree_skb() recycling the skb frags, which no longer applies now that the head and fragments live in an xdp_buff rather than a partially built skb; - drop the whole frame when it would exceed MAX_SKB_FRAGS, where the previous skb_add_rx_frag() calls were unbounded; - strip the Ethernet FCS from the accumulated xdp_buff with a driver-local tail shrink (stmmac_xdp_shrink_tail()) for both linear and fragmented frames, instead of trimming the skb or the first descriptor length. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/stmicro/stmmac/stmmac.h | 3 +- drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 331 ++++++++++++++-------- 2 files changed, 219 insertions(+), 115 deletions(-) diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h index 4fc96b317d79..0148891dbcb4 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h @@ -132,7 +132,8 @@ struct stmmac_rx_queue { dma_addr_t dma_rx_phy; unsigned int state_saved; struct { - struct sk_buff *skb; + struct xdp_buff xdp; + unsigned int frames; unsigned int len; unsigned int error; } state; diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 28e9f8438f93..742866466526 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -158,6 +158,9 @@ static void stmmac_flush_tx_descriptors(struct stmmac_priv *priv, int queue); static void stmmac_set_dma_operation_mode(struct stmmac_priv *priv, u32 txmode, u32 rxmode, u32 chan); static void stmmac_vlan_restore(struct stmmac_priv *priv); +static void stmmac_xdp_put_buff(struct stmmac_rx_queue *rx_q, + struct xdp_buff *xdp, int sync_len, + bool allow_direct); #ifdef CONFIG_DEBUG_FS static const struct net_device_ops stmmac_netdev_ops; @@ -2180,8 +2183,10 @@ static void __free_dma_rx_desc_resources(struct stmmac_priv *priv, else dma_free_rx_skbufs(priv, dma_conf, queue); - dev_kfree_skb_any(rx_q->state.skb); - rx_q->state.skb = NULL; + if (rx_q->state.frames && !rx_q->xsk_pool) { + stmmac_xdp_put_buff(rx_q, &rx_q->state.xdp, -1, false); + rx_q->state.frames = 0; + } rx_q->state_saved = false; rx_q->buf_alloc_num = 0; rx_q->xsk_pool = NULL; @@ -5758,6 +5763,118 @@ static int stmmac_rx_zc(struct stmmac_priv *priv, int limit, u32 queue) return failure ? limit : (int)count; } +static void +stmmac_xdp_put_buff(struct stmmac_rx_queue *rx_q, struct xdp_buff *xdp, + int sync_len, bool allow_direct) +{ + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); + int i; + + if (likely(!xdp_buff_has_frags(xdp))) + goto out; + + for (i = 0; i < sinfo->nr_frags; i++) + page_pool_put_full_page(rx_q->page_pool, + skb_frag_page(&sinfo->frags[i]), + allow_direct); +out: + page_pool_put_page(rx_q->page_pool, virt_to_head_page(xdp->data), + sync_len, allow_direct); +} + +static struct sk_buff *stmmac_build_skb(struct xdp_buff *xdp) +{ + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); + u32 metasize = xdp->data - xdp->data_meta; + struct sk_buff *skb; + u8 num_frags = 0; + + if (unlikely(xdp_buff_has_frags(xdp))) + num_frags = sinfo->nr_frags; + + skb = napi_build_skb(xdp->data_hard_start, xdp->frame_sz); + if (!skb) + return NULL; + + skb_mark_for_recycle(skb); + skb_reserve(skb, xdp->data - xdp->data_hard_start); + skb_put(skb, xdp->data_end - xdp->data); + if (metasize) + skb_metadata_set(skb, metasize); + + if (unlikely(xdp_buff_has_frags(xdp))) + xdp_update_skb_frags_info(skb, num_frags, sinfo->xdp_frags_size, + num_frags * xdp->frame_sz, + xdp_buff_get_skb_flags(xdp)); + return skb; +} + +static bool stmmac_build_xdp_frags(struct stmmac_priv *priv, + struct stmmac_rx_queue *rx_q, + unsigned int len, struct page *page, + unsigned int offset, + enum dma_data_direction dma_dir, + struct xdp_buff *xdp) +{ + dma_addr_t dma_addr = page_pool_get_dma_addr(page) + offset; + + dma_sync_single_for_cpu(priv->device, dma_addr, len, dma_dir); + if (!xdp_buff_add_frag(xdp, page_to_netmem(page), offset, len, + xdp->frame_sz)) { + page_pool_put_full_page(rx_q->page_pool, page, true); + return false; + } + + return true; +} + +static int stmmac_xdp_shrink_tail(struct stmmac_rx_queue *rx_q, + struct xdp_buff *xdp, int offset) +{ + struct skb_shared_info *sinfo; + int i; + + if (unlikely(offset < 0 || + offset > (int)xdp_get_buff_len(xdp) - ETH_HLEN)) + return -EINVAL; + + if (likely(!xdp_buff_has_frags(xdp))) { + xdp->data_end -= offset; + return 0; + } + + sinfo = xdp_get_shared_info_from_buff(xdp); + for (i = sinfo->nr_frags - 1; i >= 0 && offset > 0; i--) { + skb_frag_t *frag = &sinfo->frags[i]; + int delta = min_t(int, offset, skb_frag_size(frag)); + + if (delta == skb_frag_size(frag)) { + /* The whole frag is consumed by the strip: return it + * to the page pool right away. Its ring slot still + * carries the page DMA address until stmmac_rx_refill() + * re-arms it at the end of the NAPI poll. This is + * safe because HW cannot touch the slot until then. + */ + page_pool_put_full_page(rx_q->page_pool, + skb_frag_page(frag), true); + sinfo->nr_frags--; + } else { + skb_frag_size_sub(frag, delta); + } + + sinfo->xdp_frags_size -= delta; + offset -= delta; + } + + if (unlikely(!sinfo->nr_frags)) { + xdp_buff_clear_frags_flag(xdp); + xdp_buff_clear_frag_pfmemalloc(xdp); + xdp->data_end -= offset; + } + + return 0; +} + /** * stmmac_rx - manage the receive process * @priv: driver private structure @@ -5771,21 +5888,20 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue) u32 rx_errors = 0, rx_dropped = 0, rx_bytes = 0, rx_packets = 0; struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[queue]; struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[queue]; + unsigned int frames = 0, next_entry = rx_q->cur_rx; struct stmmac_channel *ch = &priv->channel[queue]; unsigned int count = 0, error = 0, len = 0; - unsigned int next_entry = rx_q->cur_rx; bool in_progress = rx_q->state_saved; enum dma_data_direction dma_dir; int coe = priv->hw->rx_csum; - unsigned int desc_size; - struct sk_buff *skb = NULL; struct stmmac_xdp_buff ctx; - bool fcs_stripped = false; + unsigned int desc_size; int xdp_status = 0; int bufsz; dma_dir = page_pool_get_dma_dir(rx_q->page_pool); - bufsz = DIV_ROUND_UP(priv->dma_conf.dma_buf_sz, PAGE_SIZE) * PAGE_SIZE; + bufsz = rx_q->napi_skb_frag_size; + ctx.priv = priv; if (netif_msg_rx_status(priv)) { void *rx_head = stmmac_get_rx_desc(priv, rx_q, 0); @@ -5798,23 +5914,25 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue) } if (rx_q->state_saved) { - skb = rx_q->state.skb; + ctx.xdp = rx_q->state.xdp; error = rx_q->state.error; + frames = rx_q->state.frames; len = rx_q->state.len; - rx_q->state.skb = NULL; rx_q->state_saved = false; + rx_q->state.frames = 0; } while (count < limit) { unsigned int buf1_len = 0, buf2_len = 0; + unsigned int pre_len, sync_len; enum pkt_hash_types hash_type; struct stmmac_rx_buffer *buf; struct dma_desc *np, *p; + struct sk_buff *skb; int entry, status; u32 hash; if (!in_progress) { - skb = NULL; error = 0; len = 0; } @@ -5851,19 +5969,24 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue) if (priv->extend_desc) stmmac_rx_extended_status(priv, &priv->xstats, rx_q->dma_erx + entry); if (unlikely(status == discard_frame)) { - page_pool_put_page(rx_q->page_pool, buf->page, 0, true); - buf->page = NULL; error = 1; if (!priv->hwts_rx_en) rx_errors++; } - if (unlikely(error && (status & rx_not_ls))) - goto read_again; - if (unlikely(error)) { - dev_kfree_skb(skb); - goto next; + page_pool_put_page(rx_q->page_pool, buf->page, 0, true); + buf->page = NULL; + if (buf->sec_page) { + page_pool_put_page(rx_q->page_pool, + buf->sec_page, 0, true); + buf->sec_page = NULL; + } + + if (status & rx_not_ls) + goto read_again; + + goto error_free_frag; } /* Buffer is good. Go on. */ @@ -5873,121 +5996,89 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue) buf2_len = stmmac_rx_buf2_len(priv, p, status, len); len += buf2_len; - /* ACS is disabled; strip manually. */ - if (likely(!(status & rx_not_ls))) - len -= ETH_FCS_LEN; - - if (!skb) { - unsigned int pre_len, sync_len; - - /* Each frame starts here: reset the FCS handling */ - fcs_stripped = false; - + if (!frames) { dma_sync_single_for_cpu(priv->device, buf->addr, buf1_len, dma_dir); net_prefetch(page_address(buf->page) + buf->page_offset); - if (stmmac_xdp_is_enabled(priv) && !buf2_len) { - buf1_len -= ETH_FCS_LEN; - fcs_stripped = true; - } - xdp_init_buff(&ctx.xdp, bufsz, &rx_q->xdp_rxq); xdp_prepare_buff(&ctx.xdp, page_address(buf->page), buf->page_offset, buf1_len, true); - - pre_len = ctx.xdp.data_end - ctx.xdp.data_hard_start - - buf->page_offset; - - ctx.priv = priv; - ctx.desc = p; - ctx.ndesc = np; - - skb = stmmac_xdp_run_prog(priv, &ctx.xdp); - /* Due xdp_adjust_tail: DMA sync for_device - * cover max len CPU touch - */ - sync_len = ctx.xdp.data_end - ctx.xdp.data_hard_start - - buf->page_offset; - sync_len = max(sync_len, pre_len); - - /* For Not XDP_PASS verdict */ - if (IS_ERR(skb)) { - unsigned int xdp_res = -PTR_ERR(skb); - - if (xdp_res & STMMAC_XDP_CONSUMED) { - page_pool_put_page(rx_q->page_pool, - virt_to_head_page(ctx.xdp.data), - sync_len, true); - buf->page = NULL; - rx_dropped++; - goto next; - } else if (xdp_res & (STMMAC_XDP_TX | - STMMAC_XDP_REDIRECT)) { - xdp_status |= xdp_res; - buf->page = NULL; - goto next; - } - } - } - - if (!skb) { - unsigned int head_pad_len; - - /* XDP program may expand or reduce tail */ - buf1_len = ctx.xdp.data_end - ctx.xdp.data; - - skb = napi_build_skb(page_address(buf->page), - rx_q->napi_skb_frag_size); - if (!skb) { - page_pool_recycle_direct(rx_q->page_pool, - buf->page); - buf->page = NULL; - rx_dropped++; - count++; - goto drain_data; - } - - /* XDP program may adjust header */ - head_pad_len = ctx.xdp.data - ctx.xdp.data_hard_start; - skb_reserve(skb, head_pad_len); - skb_put(skb, buf1_len); - skb_mark_for_recycle(skb); buf->page = NULL; } else if (buf1_len) { - dma_sync_single_for_cpu(priv->device, buf->addr, - buf1_len, dma_dir); - skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, - buf->page, buf->page_offset, buf1_len, - priv->dma_conf.dma_buf_sz); + if (!stmmac_build_xdp_frags(priv, rx_q, buf1_len, + buf->page, + buf->page_offset, + dma_dir, &ctx.xdp)) { + if (!error) + rx_dropped++; + error = 1; + } buf->page = NULL; } if (buf2_len) { - dma_sync_single_for_cpu(priv->device, buf->sec_addr, - buf2_len, dma_dir); - skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, - buf->sec_page, 0, buf2_len, - priv->dma_conf.dma_buf_sz); + if (!stmmac_build_xdp_frags(priv, rx_q, buf2_len, + buf->sec_page, 0, + dma_dir, &ctx.xdp)) { + if (!error) + rx_dropped++; + error = 1; + } buf->sec_page = NULL; } + frames++; -drain_data: if (likely(status & rx_not_ls)) goto read_again; - if (!skb) - continue; - /* Got entire packet into SKB. Finish it. */ + /* ACS is disabled; strip manually. */ + len -= ETH_FCS_LEN; + if (stmmac_xdp_shrink_tail(rx_q, &ctx.xdp, ETH_FCS_LEN)) { + if (!error) + rx_dropped++; + error = 1; + } + + if (unlikely(error)) + goto error_free_frag; + + pre_len = ctx.xdp.data_end - ctx.xdp.data_hard_start; + + ctx.desc = p; + ctx.ndesc = np; + + skb = stmmac_xdp_run_prog(priv, &ctx.xdp); + /* Due xdp_adjust_tail: DMA sync for_device + * cover max len CPU touch + */ + sync_len = ctx.xdp.data_end - ctx.xdp.data_hard_start; + sync_len = max(sync_len, pre_len); + + /* For Not XDP_PASS verdict */ + if (IS_ERR(skb)) { + unsigned int xdp_res = -PTR_ERR(skb); + + if (xdp_res & STMMAC_XDP_CONSUMED) { + stmmac_xdp_put_buff(rx_q, &ctx.xdp, sync_len, + true); + rx_dropped++; + } else if (xdp_res & (STMMAC_XDP_TX | + STMMAC_XDP_REDIRECT)) { + xdp_status |= xdp_res; + } - /* Remove FCS if needed */ - if (!fcs_stripped && pskb_trim(skb, len)) { - dev_kfree_skb_any(skb); - rx_dropped++; goto next; } + skb = stmmac_build_skb(&ctx.xdp); + if (!skb) { + rx_dropped++; + goto error_free_frag; + } + + /* Got entire packet into SKB. Finish it. */ stmmac_get_rx_hwtstamp(priv, p, np, skb); if (priv->hw->hw_vlan_en) @@ -6014,15 +6105,23 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue) rx_bytes += len; next: in_progress = false; - skb = NULL; + frames = 0; + count++; + continue; +error_free_frag: + if (frames) + stmmac_xdp_put_buff(rx_q, &ctx.xdp, -1, true); + in_progress = false; + frames = 0; count++; } - if (in_progress || skb) { - rx_q->state_saved = true; - rx_q->state.skb = skb; + if (in_progress || frames) { + rx_q->state.xdp = ctx.xdp; + rx_q->state.frames = frames; rx_q->state.error = error; rx_q->state.len = len; + rx_q->state_saved = true; } stmmac_finalize_xdp_rx(priv, xdp_status); @@ -8022,6 +8121,7 @@ static int __stmmac_dvr_probe(struct device *device, ndev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_RXCSUM; ndev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | + NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_XSK_ZEROCOPY; ret = stmmac_tc_init(priv, priv); @@ -8338,9 +8438,12 @@ static void stmmac_reset_rx_queue(struct stmmac_priv *priv, u32 queue) { struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[queue]; - dev_kfree_skb_any(rx_q->state.skb); - rx_q->state.skb = NULL; + if (rx_q->state.frames && !rx_q->xsk_pool) { + stmmac_xdp_put_buff(rx_q, &rx_q->state.xdp, -1, false); + rx_q->state.frames = 0; + } rx_q->state_saved = false; + rx_q->cur_rx = 0; rx_q->dirty_rx = 0; } -- 2.55.0