[PATCH v1 2/2] epoll: Use __user_write_access_begin() and unsafe_put_user() in epoll_put_uevent().

Kuniyuki Iwashima
2025-10-23T00:05:41Z

epoll_put_uevent() calls __put_user() twice, which are inlined
to two calls of out-of-line functions, __put_user_nocheck_4()
and __put_user_nocheck_8().

Both functions wrap mov with a stac/clac pair, which is expensive
on an AMD EPYC 7B12 64-Core Processor platform.

__put_user_nocheck_4 /proc/kcore [Percent: local period]
Percent │
89.91 │ stac
0.19 │ mov %eax,(%rcx)
0.15 │ xor %ecx,%ecx
9.69 │ clac
0.06 │ ← retq

This was remarkable while testing neper/tcp_rr with 1000 flows per
thread.

Overhead Shared O Symbol
10.08% [kernel] [k] _copy_to_iter
7.12% [kernel] [k] ip6_output
6.40% [kernel] [k] sock_poll
5.71% [kernel] [k] move_addr_to_user
4.39% [kernel] [k] __put_user_nocheck_4
...
1.06% [kernel] [k] ep_try_send_events
... ^- epoll_put_uevent() was inlined
0.78% [kernel] [k] __put_user_nocheck_8

Use __user_write_access_begin() and unsafe_put_user() in
epoll_put_uevent().

We see 1% improvement on tcp_rr throughput by just saving a single
stac/clac pair.

Another option would be to use can_do_masked_user_access()
and masked_user_access_begin(), but we saw ~5% regression with
unnecessary 3 operations for address masking, which is already
checked by ep_check_params().

Suggested-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
---
include/linux/eventpoll.h | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h
index ccb478eb174b..efc0aa2d496f 100644
--- a/include/linux/eventpoll.h
+++ b/include/linux/eventpoll.h
@@ -82,11 +82,14 @@ static inline struct epoll_event __user *
epoll_put_uevent(__poll_t revents, __u64 data,
struct epoll_event __user *uevent)
{
- if (__put_user(revents, &uevent->events) ||
- __put_user(data, &uevent->data))
- return NULL;
-
- return uevent+1;
+ __user_write_access_begin(uevent, sizeof(*uevent));
+ unsafe_put_user(revents, &uevent->events, efault);
+ unsafe_put_user(data, &uevent->data, efault);
+ user_access_end();
+ return uevent + 1;
+efault:
+ user_access_end();
+ return NULL;
}
#endif

--
2.51.1.814.gb8fa24458f-goog