From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 743D5256C6C for ; Fri, 19 Sep 2025 18:09:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758305372; cv=none; b=kUtdF2KOUEwBFyzi6kdghisiUFKryQwwVhvbwTIuQ/FUW246CX+WXdzC2NbTlY9iRVbYRTbvSZ5jNjMWzyd6NOafxai6Pr9+KnOvzgA2HHJK9+QCzfbfxU+HKvhJLwCqZqR4nxW1eNgjKGgZI16V5LZf76TI3u+xxNoFJr2LoL8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758305372; c=relaxed/simple; bh=mOdqaIQDWGokgR6nAetO40xL2IYhPDADdHeev4bl8mc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aeXXypIM9mRgGULohgttW3BRVLhimuUSeHVq6YU7UgZrG7wGxcdbhkMQ7x7wtlh7t4ttkP/FVgdoi6noFYqypVgp2teEdtTpukhZBG3wYQ1zgMatyqiIhZIbeyJhEzfUF9hCxNaUMeq7n2KM4mFDS1Gcv9MU+GGkkvPfqv0kf6E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LOgVsHAj; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LOgVsHAj" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-b54dd647edcso2247003a12.1 for ; Fri, 19 Sep 2025 11:09:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1758305370; x=1758910170; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x9wB4tgeF+7WSUk+APXIQdcXv6nxisAwKEv7VWjbrUU=; b=LOgVsHAjpfge0/NN2v1iy7Q/xWqG7lpceL17Vh3VrVqGkx10fGWyDVt3K77+1ebIdl UR/KqE1zmDjhWhpEOuyXqfqcTKKw6+2N1XMn6wvS/VvRIywmZgmvATfPlEn5YlExfytR nhvWJd8qcb8IUFSE0bFq7SfjwxnhvLKtfshndlGyQHtatQyIcIfTialEWuZqE2ss3h9k 4rx+DxiWUE4HGY/8R6FGvvPHW5Oh4wWh71PdgnjTOzhu6LdTf6T6rHsk9pzbm3Ouu+uS MJksYFUjk0y9uF856k8ac1mmEaMo/dlvh+0Mb607F7msKPTLRbDjm+toUIh4zvzeX8nl 9Wpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758305370; x=1758910170; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x9wB4tgeF+7WSUk+APXIQdcXv6nxisAwKEv7VWjbrUU=; b=J85NyWb+ZUG6ZrGkIYBv6zaJKlg1aZxSpPMP0svslSLP0sHjUeWUxiBPBzV6v+CWRw 20Lr+8QiUIB9hff7Ex6rJgHZ9zXbSmpc7pYL3BTZmlfgv+sMHOTIEiT5J6Bii7I4G8Wn lQaGku4prczcZKEeafzVYLAWF2gN+5XcxO4ZPgx+Uc/DOzyINwKQXDS+cEjEHyHwRt08 3VLrChHtnQlbGAo01TeWfrPtwwmM8G3+qFrwTxbAbTgmBKGGoemhGmDm2JHUrf0jTl0S ednaUTnesT5J7W29YjxIFnIwKUQEs8Fh6Vu5x7aA9uXjFL9j9X1ynOZQRx1nODLQRmrH OT0Q== X-Gm-Message-State: AOJu0YyS2Si6VpS05C28BzcbAHbyfDY1W7xIf1EqGKjedTk9QCAIhLHa /+gN1DzQ/12/uCkPsPPQK6eg3cTgI6yHuas2rqInHlAhh1Hn05AzwAS9 X-Gm-Gg: ASbGncv3wHIQn4Z+BvQa9rwxA3vmC/eirJis+lLDpax8xIbc0rOr8bW9WUp0vL/slX3 YPZeB1s7zInfKoY3mTCVd+KAn+1gBYiIEv+ZPUwtCYqe4ILM9PhdC24so5uOdTmUZnlDHDgdLCn nDFdFbgO9bYaz5FTkVdzdCwAJi31wJnMEGhXbM3abnksFd2ybS6SZF6O0h81JfycxxGmXRt2Upv jsoOpstxbb15seSD4rE/qE25EC0o48PvErhFmsbwjZgBBBUIV3BQxxDlEG5QtGUOGCJeav8ieX2 lPG4ml/mxOqyXrgooyLHHItd225blkCRiCou0XvmC0A2RrirBOKMfd8K8W4NmWE9wWq0msmYwrz nlz1hEn0LmA5tdg== X-Google-Smtp-Source: AGHT+IGD1VuAGHTS06Q6aq9wXJAQe9bhRR8zbRcZswN3JbbdMYMnQNrARfzVoY1s4zUgjeYqRg5gQg== X-Received: by 2002:a17:902:ec8e:b0:248:79d4:93be with SMTP id d9443c01a7336-269ba5161famr59087925ad.30.1758305369739; Fri, 19 Sep 2025 11:09:29 -0700 (PDT) Received: from localhost ([2a03:2880:ff:50::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-269802de979sm61438045ad.69.2025.09.19.11.09.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Sep 2025 11:09:29 -0700 (PDT) From: Amery Hung To: bpf@vger.kernel.org Cc: netdev@vger.kernel.org, alexei.starovoitov@gmail.com, andrii@kernel.org, daniel@iogearbox.net, paul.chaignon@gmail.com, kuba@kernel.org, stfomichev@gmail.com, martin.lau@kernel.org, mohsin.bashr@gmail.com, noren@nvidia.com, dtatulea@nvidia.com, saeedm@nvidia.com, tariqt@nvidia.com, mbloch@nvidia.com, maciej.fijalkowski@intel.com, kernel-team@meta.com Subject: [PATCH bpf-next v4 2/6] bpf: Support pulling non-linear xdp data Date: Fri, 19 Sep 2025 11:09:22 -0700 Message-ID: <20250919180926.1760403-3-ameryhung@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20250919180926.1760403-1-ameryhung@gmail.com> References: <20250919180926.1760403-1-ameryhung@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Add kfunc, bpf_xdp_pull_data(), to support pulling data from xdp fragments. Similar to bpf_skb_pull_data(), bpf_xdp_pull_data() makes the first len bytes of data directly readable and writable in bpf programs. If the "len" argument is larger than the linear data size, data in fragments will be copied to the linear data area when there is enough room. Specifically, the kfunc will try to use the tailroom first. When the tailroom is not enough, metadata and data will be shifted down to make room for pulling data. A use case of the kfunc is to decapsulate headers residing in xdp fragments. It is possible for a NIC driver to place headers in xdp fragments. To keep using direct packet access for parsing and decapsulating headers, users can pull headers into the linear data area by calling bpf_xdp_pull_data() and then pop the header with bpf_xdp_adjust_head(). Reviewed-by: Jakub Kicinski Signed-off-by: Amery Hung --- net/core/filter.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/net/core/filter.c b/net/core/filter.c index 0b82cb348ce0..0e8d63bf1d30 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -12212,6 +12212,96 @@ __bpf_kfunc int bpf_sock_ops_enable_tx_tstamp(struct bpf_sock_ops_kern *skops, return 0; } +/** + * bpf_xdp_pull_data() - Pull in non-linear xdp data. + * @x: &xdp_md associated with the XDP buffer + * @len: length of data to be made directly accessible in the linear part + * + * Pull in data in case the XDP buffer associated with @x is non-linear and + * not all @len are in the linear data area. + * + * Direct packet access allows reading and writing linear XDP data through + * packet pointers (i.e., &xdp_md->data + offsets). The amount of data which + * ends up in the linear part of the xdp_buff depends on the NIC and its + * configuration. When a frag-capable XDP program wants to directly access + * headers that may be in the non-linear area, call this kfunc to make sure + * the data is available in the linear area. Alternatively, use dynptr or + * bpf_xdp_{load,store}_bytes() to access data without pulling. + * + * This kfunc can also be used with bpf_xdp_adjust_head() to decapsulate + * headers in the non-linear data area. + * + * A call to this kfunc may reduce headroom. If there is not enough tailroom + * in the linear data area, metadata and data will be shifted down. + * + * A call to this kfunc is susceptible to change the buffer geometry. + * Therefore, at load time, all checks on pointers previously done by the + * verifier are invalidated and must be performed again, if the kfunc is used + * in combination with direct packet access. + * + * Return: + * * %0 - success + * * %-EINVAL - invalid len + */ +__bpf_kfunc int bpf_xdp_pull_data(struct xdp_md *x, u32 len) +{ + struct xdp_buff *xdp = (struct xdp_buff *)x; + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); + int i, delta, shift, headroom, tailroom, n_frags_free = 0; + void *data_hard_end = xdp_data_hard_end(xdp); + int data_len = xdp->data_end - xdp->data; + void *start; + + if (len <= data_len) + return 0; + + if (unlikely(len > xdp_get_buff_len(xdp))) + return -EINVAL; + + start = xdp_data_meta_unsupported(xdp) ? xdp->data : xdp->data_meta; + + headroom = start - xdp->data_hard_start - sizeof(struct xdp_frame); + tailroom = data_hard_end - xdp->data_end; + + delta = len - data_len; + if (unlikely(delta > tailroom + headroom)) + return -EINVAL; + + shift = delta - tailroom; + if (shift > 0) { + memmove(start - shift, start, xdp->data_end - start); + + xdp->data_meta -= shift; + xdp->data -= shift; + xdp->data_end -= shift; + } + + for (i = 0; i < sinfo->nr_frags && delta; i++) { + skb_frag_t *frag = &sinfo->frags[i]; + u32 shrink = min_t(u32, delta, skb_frag_size(frag)); + + memcpy(xdp->data_end, skb_frag_address(frag), shrink); + + xdp->data_end += shrink; + sinfo->xdp_frags_size -= shrink; + delta -= shrink; + if (bpf_xdp_shrink_data(xdp, frag, shrink, false)) + n_frags_free++; + } + + if (unlikely(n_frags_free)) { + memmove(sinfo->frags, sinfo->frags + n_frags_free, + (sinfo->nr_frags - n_frags_free) * sizeof(skb_frag_t)); + + sinfo->nr_frags -= n_frags_free; + + if (!sinfo->nr_frags) + xdp_buff_clear_frags_flag(xdp); + } + + return 0; +} + __bpf_kfunc_end_defs(); int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags, @@ -12239,6 +12329,7 @@ BTF_KFUNCS_END(bpf_kfunc_check_set_skb_meta) BTF_KFUNCS_START(bpf_kfunc_check_set_xdp) BTF_ID_FLAGS(func, bpf_dynptr_from_xdp) +BTF_ID_FLAGS(func, bpf_xdp_pull_data) BTF_KFUNCS_END(bpf_kfunc_check_set_xdp) BTF_KFUNCS_START(bpf_kfunc_check_set_sock_addr) -- 2.47.3