[PATCH net-next] net: datagram: Bypass usercopy checks for kernel iterators

[PATCH net-next] net: datagram: Bypass usercopy checks for kernel iterators

[Chuck Lever]

From: Chuck Lever <chuck.lever@oracle.com>

Profiling NFSD under an iozone workload showed that hardened
usercopy checks consume roughly 1.3% of CPU in the TCP receive path.
These checks validate memory regions during copies, but provide no
security benefit when both source (skb data) and destination (kernel
pages in BVEC/KVEC iterators) reside in kernel address space.

Modify simple_copy_to_iter() and crc32c_and_copy_to_iter() to call
_copy_to_iter() directly when the destination is a kernel-only
iterator, bypassing the usercopy hardening validation. User-backed
iterators (ITER_UBUF, ITER_IOVEC) continue to use copy_to_iter()
with full validation.

This benefits kernel consumers of TCP receive such as the NFS client
and server and NVMe-TCP, which use ITER_BVEC for their receive
buffers.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/core/datagram.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/net/core/datagram.c b/net/core/datagram.c
index c285c6465923..e83cf0125008 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -490,7 +490,10 @@ static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes,
 	u32 *crcp = _crcp;
 	size_t copied;
 
-	copied = copy_to_iter(addr, bytes, i);
+	if (user_backed_iter(i))
+		copied = copy_to_iter(addr, bytes, i);
+	else
+		copied = _copy_to_iter(addr, bytes, i);
 	*crcp = crc32c(*crcp, addr, copied);
 	return copied;
 }
@@ -515,10 +518,17 @@ int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offset,
 EXPORT_SYMBOL(skb_copy_and_crc32c_datagram_iter);
 #endif /* CONFIG_NET_CRC32C */
 
+/*
+ * Bypass usercopy hardening for kernel-only iterators: no data
+ * crosses the user/kernel boundary, so the slab whitelist check
+ * on the source buffer is unnecessary overhead.
+ */
 static size_t simple_copy_to_iter(const void *addr, size_t bytes,
 		void *data __always_unused, struct iov_iter *i)
 {
-	return copy_to_iter(addr, bytes, i);
+	if (user_backed_iter(i))
+		return copy_to_iter(addr, bytes, i);
+	return _copy_to_iter(addr, bytes, i);
 }
 
 /**
-- 
2.52.0

Re: [PATCH net-next] net: datagram: Bypass usercopy checks for kernel iterators

[Jakub Kicinski]

On Wed, 25 Feb 2026 11:25:32 -0500 Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> Profiling NFSD under an iozone workload showed that hardened
> usercopy checks consume roughly 1.3% of CPU in the TCP receive path.
> These checks validate memory regions during copies, but provide no
> security benefit when both source (skb data) and destination (kernel
> pages in BVEC/KVEC iterators) reside in kernel address space.
> 
> Modify simple_copy_to_iter() and crc32c_and_copy_to_iter() to call
> _copy_to_iter() directly when the destination is a kernel-only
> iterator, bypassing the usercopy hardening validation. User-backed
> iterators (ITER_UBUF, ITER_IOVEC) continue to use copy_to_iter()
> with full validation.
> 
> This benefits kernel consumers of TCP receive such as the NFS client
> and server and NVMe-TCP, which use ITER_BVEC for their receive
> buffers.

If it makes such a difference why not make copy_to_iter()
check the iter type? Why force callers to check it?

> diff --git a/net/core/datagram.c b/net/core/datagram.c
> index c285c6465923..e83cf0125008 100644
> --- a/net/core/datagram.c
> +++ b/net/core/datagram.c
> @@ -490,7 +490,10 @@ static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes,
>  	u32 *crcp = _crcp;
>  	size_t copied;
>  
> -	copied = copy_to_iter(addr, bytes, i);
> +	if (user_backed_iter(i))
> +		copied = copy_to_iter(addr, bytes, i);
> +	else
> +		copied = _copy_to_iter(addr, bytes, i);
>  	*crcp = crc32c(*crcp, addr, copied);
>  	return copied;
>  }
> @@ -515,10 +518,17 @@ int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offset,
>  EXPORT_SYMBOL(skb_copy_and_crc32c_datagram_iter);
>  #endif /* CONFIG_NET_CRC32C */
>  
> +/*
> + * Bypass usercopy hardening for kernel-only iterators: no data
> + * crosses the user/kernel boundary, so the slab whitelist check
> + * on the source buffer is unnecessary overhead.
> + */
>  static size_t simple_copy_to_iter(const void *addr, size_t bytes,
>  		void *data __always_unused, struct iov_iter *i)
>  {
> -	return copy_to_iter(addr, bytes, i);
> +	if (user_backed_iter(i))
> +		return copy_to_iter(addr, bytes, i);
> +	return _copy_to_iter(addr, bytes, i);
>  }
>  
>  /**

Re: [PATCH net-next] net: datagram: Bypass usercopy checks for kernel iterators

[Paolo Abeni]

On 2/25/26 5:25 PM, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Profiling NFSD under an iozone workload showed that hardened
> usercopy checks consume roughly 1.3% of CPU in the TCP receive path.
> These checks validate memory regions during copies, but provide no
> security benefit when both source (skb data) and destination (kernel
> pages in BVEC/KVEC iterators) reside in kernel address space.

Are you sure? AFAICS:

size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)
{
        if (check_copy_size(addr, bytes, false))
                return _copy_from_iter(addr, bytes, i);

calls check_copy_size() on the source address, and the latter:

static __always_inline __must_check bool
check_copy_size(const void *addr, size_t bytes, bool is_source)
{
        int sz = __builtin_object_size(addr, 0);
        if (unlikely(sz >= 0 && sz < bytes)) {
        if (!__builtin_constant_p(bytes))
                        copy_overflow(sz, bytes);
                else if (is_source)$
                        __bad_copy_from();
                else
                        __bad_copy_to();
                return false;
        }
        if (WARN_ON_ONCE(bytes > INT_MAX))
                return false;

Validates vs overflow regardless of the source address being in kernel
space or user-space.

FTR, I also observe a relevant overhead in check_copy_size(), especially
for oldish CPUs.

/P

Re: [PATCH net-next] net: datagram: Bypass usercopy checks for kernel iterators

[Chuck Lever]

On Tue, Mar 3, 2026, at 4:42 AM, Paolo Abeni wrote:
> On 2/25/26 5:25 PM, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>>
>> Profiling NFSD under an iozone workload showed that hardened
>> usercopy checks consume roughly 1.3% of CPU in the TCP receive path.
>> These checks validate memory regions during copies, but provide no
>> security benefit when both source (skb data) and destination (kernel
>> pages in BVEC/KVEC iterators) reside in kernel address space.
>
> Are you sure? AFAICS:
>
> size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)
> {
>         if (check_copy_size(addr, bytes, false))
>                 return _copy_from_iter(addr, bytes, i);
>
> calls check_copy_size() on the source address, and the latter:
>
> static __always_inline __must_check bool
> check_copy_size(const void *addr, size_t bytes, bool is_source)
> {
>         int sz = __builtin_object_size(addr, 0);
>         if (unlikely(sz >= 0 && sz < bytes)) {
>         if (!__builtin_constant_p(bytes))
>                         copy_overflow(sz, bytes);
>                 else if (is_source)$
>                         __bad_copy_from();
>                 else
>                         __bad_copy_to();
>                 return false;
>         }
>         if (WARN_ON_ONCE(bytes > INT_MAX))
>                 return false;
>
> Validates vs overflow regardless of the source address being in kernel
> space or user-space.
>
> FTR, I also observe a relevant overhead in check_copy_size(), especially
> for oldish CPUs.
>
> /P

Paolo, thanks for the review. I'll post a refreshed patch shortly that
attempts to address your comments, and that modifies the common helper,
as Jakub suggested earlier.


-- 
Chuck Lever