From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B795A233941 for ; Sat, 11 Jul 2026 10:49:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783766968; cv=none; b=N/O3DGkNcobZZGoiRa1vMJrdvgp8DV6dGdx8KgdNOwnO49c7ow50NhDCf5VaVORTQE1H9WVg/1v/Hp48VsrAB3xnb0DFbmTnxQyau02+HB7Bta0FO1ZCzaKM1QUlHVHKUkk/HYdHy+WLGCJ5t8bvqkb1olZjeIpVyJNzZTxwntQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783766968; c=relaxed/simple; bh=OEvIyS+mKiIWCrKAbZR0LHE1ZtBrvPvlwVaplICVy9Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xuc4a8MXnDfycKKfe3K9wtims7LkRJHSuvuQFaJi5bvl46pjJt9EqXsMhsG0E4Y+eRbIqDou4in5rFHE7NrT/L6O/niJ44fH9Um7VvEKtUgAOfnuqEf+mjDdK5zFAa7zUuV1LMW4IXbtnzG0h2EShDHSZ9WhKQA7XP6dpn4SIWE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=k+CqBuhq; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k+CqBuhq" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-c15ba3a2b4bso213807066b.1 for ; Sat, 11 Jul 2026 03:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783766960; x=1784371760; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qms3rEKyX2JB93BBm5Zfxr9oAEK8d+GwUbmaIMWOy7c=; b=k+CqBuhqSEJcdv+cz8DkxGumbr/AeR3jAllYDJEBl3FLs1JxuUVQ0kqqjwkfekFQed E/QRKBg6Azdu1Ouc0G/Zc1J+n/nm5VxyZKkIfuHaI/h9gpV082ZB9K1oeqs20XV1df/k ZMKdkRoNmUUN0c9Z9UxqV93qDWheSxemNf6RQkujlOkqU4f9cEWYZOXXKOUOw4WobXWH rqXVyrXZkhJZmJrNI0QcZxVT3fupdjnLlzmebZTxdeI46itJW1qlfI8O6l/vQxG7v1Br D6+RKMU6M2060XDplQ5xSQPhP/AjbDT2yqGqr7YtmHBebMz3GYfoZDGMpGOuTZ1Kb2Q9 3x6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783766960; x=1784371760; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=qms3rEKyX2JB93BBm5Zfxr9oAEK8d+GwUbmaIMWOy7c=; b=DZ32P8uxhtbCM3OV0AgD2bJKoOzAMunUo5laTF95tAm2ntORdey9i6i+No6p8vn0bP SV9MurvWwv09DKi7pkk/59oXKr2pFFjGlHla1aETzNooHZbuLKOIfHurhX+hsuVOtC4z skGQr3wzDgvtwguZdWhr1gTqZITjDiRXun+SRbeTj74pWiHdBZv3zaOP1+GiF3RnScUM DcXU5wqb6TbCi3eCZP5A0spWyFHZmS1MAon9Mil/bn0AFj95X4GDCwmeOPxXqpO64Lay 7nT7w2sH3XRDSFtBwzxOjXHI3jK0h14fUk6QDTungAe1bZyqSGLHiagbND9ZewQWth4P WcpA== X-Gm-Message-State: AOJu0Yy0drFfKpwH4EnnxQs4vXhgJVQPoMbvHqHsUXfZ3ypa/rrl3mfl jk34bKROMc14P14iUcVOXd8w4DVG/cbQrgxmmgpP2XPMtnv0YesFVuwYPC8qbQ== X-Gm-Gg: AfdE7cmQQ/kY58k9/+RNVmcI2iKKy7KNvMJDMfOxjxOdqOv+fg942IZCSxi/WHPZ5B8 6YSqGUSbRcIdvC6H8aoCH4aISsEs0zoJHcXgU3mngMQR30gnvZdGnceg2B9+cPO6ZZEnVRChwWi OvddxM8J7/4idVjRpQuXe9Q2oFLk2raoAf16MhQ+Y2qpN8ZFjzWy8VJvV1399fToN7ZjM4uCvQ0 YXzo3f/1GiC7UngvypqhkTbOsMIZ+0UE+Ii44PCfMlCQsRdHSmZS+249n/hLm6shg0ms0fFl8NK aQqF/ps2ROZXGsbIS0756ahHxiVCHOiiI+R7J/6QPtz9Kyk7xguKNAczppZ2taOJD+heblXIsax ke1mV/rpy79PiqIFJfLGK5XLLluKRsfKS+MQKvmURzhrt3+WtgYFYq2SyfNSnfpJN4HcfVj67dB ta9vFhoccBzV5g5nj5JKtNM3wBB0Gw6HXK8jYtiTTkmp9IhVrSR8hSFw1GVIaE0cTPs1qjGDd6k FISlkmsUcJWZf5rxoz3AdNJO4pszRDgaEYvO37jKzKu4d6mu2/eeS6s+AHa X-Received: by 2002:a17:907:97d4:b0:c15:efb1:9ff7 with SMTP id a640c23a62f3a-c161e91a30fmr84454466b.28.1783766960456; Sat, 11 Jul 2026 03:49:20 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-222-132.dab.02.net. [82.132.222.132]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c15beb53b86sm609123166b.25.2026.07.11.03.49.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 11 Jul 2026 03:49:19 -0700 (PDT) From: Pavel Begunkov To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Jamal Hadi Salim , io-uring@vger.kernel.org, asml.silence@gmail.com Subject: [RFC 04/10] io_uring/zcrx: prepare areas to be exported for tx Date: Sat, 11 Jul 2026 11:48:33 +0100 Message-ID: <60d1b1b7b56986791e4975c79632c42d15c2e57f.1783614400.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit We're going to piggy back tx on top of zcrx, for that we need a separate niov list without the pp field set and make sure net core can reference them via get_netmem, which pins the zcrx instance. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/net.h | 10 ++++++ io_uring/zcrx.c | 65 +++++++++++++++++++++++++++++++++--- io_uring/zcrx.h | 10 ++++++ net/core/skbuff.c | 3 ++ 4 files changed, 83 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring/net.h b/include/linux/io_uring/net.h index b58f39fed4d5..19f31f0d38d5 100644 --- a/include/linux/io_uring/net.h +++ b/include/linux/io_uring/net.h @@ -2,6 +2,8 @@ #ifndef _LINUX_IO_URING_NET_H #define _LINUX_IO_URING_NET_H +#include + struct io_uring_cmd; #if defined(CONFIG_IO_URING) @@ -15,4 +17,12 @@ static inline int io_uring_cmd_sock(struct io_uring_cmd *cmd, } #endif +#if defined(CONFIG_IO_URING_ZCRX) +void zcrx_ref_niov(struct net_iov *niov); +#else +static inline void zcrx_ref_niov(struct net_iov *niov) +{ +} +#endif + #endif diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index f501fc75d7b6..28398f0d0014 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -100,6 +101,8 @@ static int io_populate_area_dma(struct io_zcrx_ifq *ifq, if (net_mp_niov_set_dma_addr(niov, dma)) return -EFAULT; + if (net_mp_niov_set_dma_addr(&area->tx_niovs[niov_idx], dma)) + return -EFAULT; sg_len -= niov_size; dma += niov_size; niov_idx++; @@ -118,7 +121,7 @@ static void io_release_dmabuf(struct io_zcrx_mem *mem) if (mem->sgt) dma_buf_unmap_attachment_unlocked(mem->attach, mem->sgt, - DMA_FROM_DEVICE); + DMA_BIDIRECTIONAL); if (mem->attach) dma_buf_detach(mem->dmabuf, mem->attach); if (mem->dmabuf) @@ -162,7 +165,7 @@ static int io_import_dmabuf(struct io_zcrx_ifq *ifq, goto err; } - mem->sgt = dma_buf_map_attachment_unlocked(mem->attach, DMA_FROM_DEVICE); + mem->sgt = dma_buf_map_attachment_unlocked(mem->attach, DMA_BIDIRECTIONAL); if (IS_ERR(mem->sgt)) { ret = PTR_ERR(mem->sgt); mem->sgt = NULL; @@ -226,7 +229,7 @@ static int io_import_umem(struct io_zcrx_ifq *ifq, if (ifq->dev) { ret = dma_map_sgtable(ifq->dev, &mem->page_sg_table, - DMA_FROM_DEVICE, IO_DMA_ATTR); + DMA_BIDIRECTIONAL, IO_DMA_ATTR); if (ret < 0) goto out_err; mapped = true; @@ -247,7 +250,7 @@ static int io_import_umem(struct io_zcrx_ifq *ifq, out_err: if (mapped) dma_unmap_sgtable(ifq->dev, &mem->page_sg_table, - DMA_FROM_DEVICE, IO_DMA_ATTR); + DMA_BIDIRECTIONAL, IO_DMA_ATTR); sg_free_table(&mem->page_sg_table); unpin_user_pages(pages, nr_pages); kvfree(pages); @@ -310,7 +313,7 @@ static void io_zcrx_unmap_area(struct io_zcrx_ifq *ifq, io_release_dmabuf(&area->mem); } else { dma_unmap_sgtable(ifq->dev, &area->mem.page_sg_table, - DMA_FROM_DEVICE, IO_DMA_ATTR); + DMA_BIDIRECTIONAL, IO_DMA_ATTR); } } @@ -494,6 +497,11 @@ static int io_zcrx_create_area(struct io_zcrx_ifq *ifq, if (!area->nia.niovs) goto err; + area->tx_niovs = kvmalloc_objs(area->tx_niovs[0], nr_iovs, + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!area->tx_niovs) + goto err; + area->freelist = kvmalloc_array(nr_iovs, sizeof(area->freelist[0]), GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!area->freelist) @@ -510,6 +518,7 @@ static int io_zcrx_create_area(struct io_zcrx_ifq *ifq, net_iov_init(niov, &area->nia, NET_IOV_IOURING); area->freelist[i] = i; atomic_set(&area->user_refs[i], 0); + net_iov_init(&area->tx_niovs[i], &area->nia, NET_IOV_IOURING); } if (ifq->dev) { @@ -741,6 +750,13 @@ static const struct file_operations zcrx_box_fops = { .release = zcrx_box_release, }; +void zcrx_ref_niov(struct net_iov *niov) +{ + struct io_zcrx_ifq *ifq = io_zcrx_iov_to_area(niov)->ifq; + + percpu_ref_get(&ifq->refs); +} + static int zcrx_export(struct io_ring_ctx *ctx, struct io_zcrx_ifq *ifq, struct zcrx_ctrl *ctrl, void __user *arg) { @@ -1817,3 +1833,42 @@ int io_zcrx_recv(struct io_kiocb *req, struct io_zcrx_ifq *ifq, sock_rps_record_flow(sk); return io_zcrx_tcp_recvmsg(req, ifq, sk, flags, issue_flags, len); } + +int io_zcrx_fill_tx_skb(struct sk_buff *skb, struct io_zcrx_ifq *zcrx, + struct iov_iter *from, size_t length) +{ + int i = skb_shinfo(skb)->nr_frags; + unsigned niovs_emitted = 0; + struct io_zcrx_area *area = zcrx->area; + unsigned niov_size = 1U << zcrx->niov_shift; + + if (i && skb_frags_readable(skb)) + return -EINVAL; + length = min(length, iov_iter_count(from)); + + while (length) { + struct net_iov *niov; + size_t offset, size, niov_off; + + if (i == MAX_SKB_FRAGS) { + percpu_ref_get_many(&zcrx->refs, niovs_emitted); + return -EMSGSIZE; + } + + offset = (size_t)iter_iov_addr(from); + niov = &area->tx_niovs[offset >> zcrx->niov_shift]; + niov_off = offset & (niov_size - 1); + size = min(length, niov_size - niov_off); + size = min(size, iter_iov_len(from)); + + skb_add_rx_frag_netmem(skb, i, net_iov_to_netmem(niov), niov_off, + size, size); + iov_iter_advance(from, size); + length -= size; + i++; + niovs_emitted++; + } + + percpu_ref_get_many(&zcrx->refs, niovs_emitted); + return 0; +} diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 205b9b40c74d..406e0399bd7b 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -32,6 +32,7 @@ struct io_zcrx_area { struct net_iov_area nia; struct io_zcrx_ifq *ifq; atomic_t *user_refs; + struct net_iov *tx_niovs; bool is_mapped; u16 area_id; @@ -96,6 +97,10 @@ int io_zcrx_recv(struct io_kiocb *req, struct io_zcrx_ifq *ifq, unsigned issue_flags, unsigned int *len); struct io_mapped_region *io_zcrx_get_region(struct io_ring_ctx *ctx, unsigned int id); + +int io_zcrx_fill_tx_skb(struct sk_buff *skb, struct io_zcrx_ifq *zcrx, + struct iov_iter *from, size_t length); + #else static inline int io_register_zcrx(struct io_ring_ctx *ctx, struct io_uring_zcrx_ifq_reg __user *arg) @@ -124,6 +129,11 @@ static inline int io_zcrx_ctrl(struct io_ring_ctx *ctx, { return -EOPNOTSUPP; } +static inline int io_zcrx_fill_tx_skb(struct sk_buff *skb, struct io_zcrx_ifq *zcrx, + struct iov_iter *from, size_t length) +{ + return -EOPNOTSUPP; +} #endif int io_recvzc(struct io_kiocb *req, unsigned int issue_flags); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 18dabb4e9cfa..f871e3f2299b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -92,6 +92,7 @@ #include #include #include +#include #include "dev.h" #include "devmem.h" @@ -7476,6 +7477,8 @@ void __get_netmem(netmem_ref netmem) if (net_is_devmem_iov(niov)) net_devmem_get_net_iov(netmem_to_net_iov(netmem)); + else if (niov->type == NET_IOV_IOURING) + zcrx_ref_niov(niov); } EXPORT_SYMBOL(__get_netmem); -- 2.54.0