[PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[Andy Pan via B4 Relay]

From: Andy Pan <i@andypan.me>

For the moment, the edge-triggered epoll generates an event for each
receipt of a chunk of data, that is to say, epoll_wait() will return
and tell us a monitored file descriptor is ready whenever there is a
new activity on that FD since we were last informed about that FD.
This is not a real _edge_ implementation for epoll, but it's been
working this way for years and plenty of projects are relying on it
to eliminate the overhead of one system call of read(2) per wakeup event.

There are several renowned open-source projects relying on this feature
for notification function (with eventfd): register eventfd with EPOLLET
and avoid calling read(2) on the eventfd when there is wakeup event (eventfd being written).
Examples: nginx [1], netty [2], tokio [3], libevent [4], ect. [5]
These projects are widely used in today's Internet infrastructures.
Thus, changing this behavior of epoll ET will fundamentally break them
and cause a significant negative impact.
Linux has changed it for pipe before [6], breaking some Android libraries,
which had got "reverted" somehow. [7] [8]

Nevertheless, the paragraph in the manual pages describing this
characteristic of epoll ET seems ambiguous, I think a more explict
sentence should be used to clarify it. We're improving the notification
mechanism for libuv recently by exploiting this feature with eventfd,
which brings us a significant performance boost. [9]

Therefore, we (as well as the maintainers of nginx, netty, tokio, etc.)
would have a sense of security to build an enhanced notification function
based on this feature if there is a guarantee of retaining this implementation
of epoll ET for the backward compatibility in the man pages.

[1]: https://github.com/nginx/nginx/blob/efc6a217b92985a1ee211b6bb7337cd2f62deb90/src/event/modules/ngx_epoll_module.c#L386-L457
[2]: https://github.com/netty/netty/pull/9192
[3]: https://github.com/tokio-rs/mio/blob/309daae21ecb1d46203a7dbc0cf4c80310240cba/src/sys/unix/waker.rs#L111-L143
[4]: https://github.com/libevent/libevent/blob/525f5d0a14c9c103be750f2ca175328c25505ea4/event.c#L2597-L2614
[5]: https://github.com/libuv/libuv/pull/4400#issuecomment-2123798748
[6]: https://lkml.iu.edu/hypermail/linux/kernel/2010.1/04363.html
[7]: https://github.com/torvalds/linux/commit/3a34b13a88caeb2800ab44a4918f230041b37dd9
[8]: https://github.com/torvalds/linux/commit/3b844826b6c6affa80755254da322b017358a2f4
[9]: https://github.com/libuv/libuv/pull/4400#issuecomment-2103232402

Signed-off-by: Andy Pan <i@andypan.me>
---
Changes in v3:
- Updated the git commit description
- Link to v2: https://lore.kernel.org/r/20240727-epoll-et-desc-v2-1-c99b2ac66775@andypan.me

Changes in v2:
- Added the git commit description based on feedback
- Link to v1: https://lore.kernel.org/r/20240727-epoll-et-desc-v1-1-390bafc678b9@andypan.me
---
 man/man7/epoll.7 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/man/man7/epoll.7 b/man/man7/epoll.7
index 951500131..361d9db99 100644
--- a/man/man7/epoll.7
+++ b/man/man7/epoll.7
@@ -172,6 +172,7 @@ .SS Level-triggered and edge-triggered
 Since even with edge-triggered
 .BR epoll ,
 multiple events can be generated upon receipt of multiple chunks of data,
+that is, an event will be generated upon each receipt of a chunk of data,
 the caller has the option to specify the
 .B EPOLLONESHOT
 flag, to tell

---
base-commit: cbc0a111e4dceea2037c51098de33e6bc8c16a5c
change-id: 20240727-epoll-et-desc-04ea9a590f3b

Best regards,
-- 
Andy Pan <i@andypan.me>

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 4169 bytes --]

Hi Andy,

On Tue, Jul 30, 2024 at 10:48:28AM GMT, Andy Pan via B4 Relay wrote:
> From: Andy Pan <i@andypan.me>
> 
> For the moment, the edge-triggered epoll generates an event for each
> receipt of a chunk of data, that is to say, epoll_wait() will return
> and tell us a monitored file descriptor is ready whenever there is a
> new activity on that FD since we were last informed about that FD.
> This is not a real _edge_ implementation for epoll, but it's been
> working this way for years and plenty of projects are relying on it
> to eliminate the overhead of one system call of read(2) per wakeup event.
> 
> There are several renowned open-source projects relying on this feature
> for notification function (with eventfd): register eventfd with EPOLLET
> and avoid calling read(2) on the eventfd when there is wakeup event (eventfd being written).
> Examples: nginx [1], netty [2], tokio [3], libevent [4], ect. [5]
> These projects are widely used in today's Internet infrastructures.
> Thus, changing this behavior of epoll ET will fundamentally break them
> and cause a significant negative impact.
> Linux has changed it for pipe before [6], breaking some Android libraries,
> which had got "reverted" somehow. [7] [8]
> 
> Nevertheless, the paragraph in the manual pages describing this
> characteristic of epoll ET seems ambiguous, I think a more explict
> sentence should be used to clarify it. We're improving the notification
> mechanism for libuv recently by exploiting this feature with eventfd,
> which brings us a significant performance boost. [9]
> 
> Therefore, we (as well as the maintainers of nginx, netty, tokio, etc.)
> would have a sense of security to build an enhanced notification function
> based on this feature if there is a guarantee of retaining this implementation
> of epoll ET for the backward compatibility in the man pages.
> 
> [1]: https://github.com/nginx/nginx/blob/efc6a217b92985a1ee211b6bb7337cd2f62deb90/src/event/modules/ngx_epoll_module.c#L386-L457
> [2]: https://github.com/netty/netty/pull/9192
> [3]: https://github.com/tokio-rs/mio/blob/309daae21ecb1d46203a7dbc0cf4c80310240cba/src/sys/unix/waker.rs#L111-L143
> [4]: https://github.com/libevent/libevent/blob/525f5d0a14c9c103be750f2ca175328c25505ea4/event.c#L2597-L2614
> [5]: https://github.com/libuv/libuv/pull/4400#issuecomment-2123798748
> [6]: https://lkml.iu.edu/hypermail/linux/kernel/2010.1/04363.html
> [7]: https://github.com/torvalds/linux/commit/3a34b13a88caeb2800ab44a4918f230041b37dd9
> [8]: https://github.com/torvalds/linux/commit/3b844826b6c6affa80755254da322b017358a2f4
> [9]: https://github.com/libuv/libuv/pull/4400#issuecomment-2103232402
> 
> Signed-off-by: Andy Pan <i@andypan.me>
> ---
> Changes in v3:
> - Updated the git commit description
> - Link to v2: https://lore.kernel.org/r/20240727-epoll-et-desc-v2-1-c99b2ac66775@andypan.me
> 
> Changes in v2:
> - Added the git commit description based on feedback
> - Link to v1: https://lore.kernel.org/r/20240727-epoll-et-desc-v1-1-390bafc678b9@andypan.me
> ---
>  man/man7/epoll.7 | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/man/man7/epoll.7 b/man/man7/epoll.7
> index 951500131..361d9db99 100644
> --- a/man/man7/epoll.7
> +++ b/man/man7/epoll.7
> @@ -172,6 +172,7 @@ .SS Level-triggered and edge-triggered
>  Since even with edge-triggered
>  .BR epoll ,

This paragraph is not only about EPOLLET.

>  multiple events can be generated upon receipt of multiple chunks of data,
> +that is, an event will be generated upon each receipt of a chunk of data,

Is this strictly true in all modes?  The commit message seems to say
that this is the behavior of EPOLLET, but this paragraph is not only
about EPOLLET.  Is this correct here, or should it go somewhere else?

Have a lovely day!
Alex

>  the caller has the option to specify the
>  .B EPOLLONESHOT
>  flag, to tell
> 
> ---
> base-commit: cbc0a111e4dceea2037c51098de33e6bc8c16a5c
> change-id: 20240727-epoll-et-desc-04ea9a590f3b
> 
> Best regards,
> -- 
> Andy Pan <i@andypan.me>
> 
> 

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[潘少]

Hi Alejandro,
> Since even with edge-triggered epoll, multiple events can be
> generated upon receipt of multiple chunks of data, the caller has
> the option to specify the EPOLLONESHOT flag, to tell epoll to
> disable the associated file descriptor after the receipt of an
> event with epoll_wait(2).  When the EPOLLONESHOT flag is
> specified, it is the caller's responsibility to rearm the file
> descriptor using epoll_ctl(2) with EPOLL_CTL_MOD.

This paragraph in man pages of epoll starts with "Since even with edge-triggered epoll", thus I assume the following sentences of "multiple events can be generated upon receipt of multiple chunks of data" and "that is, an event will be generated upon each receipt of a chunk of data" ought to be talking about EPOLLET naturally. Have I misunderstood something?

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]

[TO += mtk]

Archive: <https://lore.kernel.org/linux-man/tencent_311E460546C884C26FB32E3C@qq.com/T/>

Hi,

On Wed, Jul 31, 2024 at 06:54:26PM GMT, 潘少 wrote:
> Hi Alejandro,
> > Since even with edge-triggered epoll, multiple events can be
> > generated upon receipt of multiple chunks of data, the caller has
> > the option to specify the EPOLLONESHOT flag, to tell epoll to
> > disable the associated file descriptor after the receipt of an
> > event with epoll_wait(2).  When the EPOLLONESHOT flag is
> > specified, it is the caller's responsibility to rearm the file
> > descriptor using epoll_ctl(2) with EPOLL_CTL_MOD.
> 
> This paragraph in man pages of epoll starts with "Since even with
> edge-triggered epoll", thus I assume the following sentences of
> "multiple events can be generated upon receipt of multiple chunks of
> data" and "that is, an event will be generated upon each receipt of a
> chunk of data" ought to be talking about EPOLLET naturally. Have I
> misunderstood something?

I understand it as:

	Since _even_ with edge-triggered epoll [and thus also with other
	epolls], multiple events can be generated ...

the stuff after the comma applied to at least more than one epoll typer.;
That's maybe why it's so generic.

However, I think a clarification like to one you suggest could be
appropriate in the page.  I've added Michael to this mail, who may be
better placed to help with these APIs.

Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

Hi Andy,

On Wed, Jul 31, 2024 at 07:16:38PM GMT, i@andypan.me wrote:
> OK, I get your point now. In that case, should I start a new paragraph
> where I put the clarification? If so, where would you suggest I begin
> writing that new paragraph? Thanks!

Thank you!  I don't have an opinion at the moment.  Feel free to chose
where you want; we'll see.

Cheers,
Alex

> 
> Best regards,
> Andy Pan

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[潘少]

OK, I get your point now. In that case, should I start a new paragraph where I put the clarification? If so, where would you suggest I begin writing that new paragraph? Thanks! (Sorry for the duplicate email if you've already got my last one, it got rejected by the email gourps of linux-man and linux-api due to HTML content in that email, so I'm sending it again with only plain text.)

Best regards,
Andy Pan

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[潘少]

Hi Alejandro,
I believe I've found a more suitable place to put this added clarification sentence. Please check out the [PATCH v4], thanks!

-----------

Best regards,
Andy Pan

Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

[潘少]

Updated to [PATCH v5].
-----------
Best regards,
Andy Pan


------------------ Original ------------------
From:  "潘少"<i@andypan.me>;
Date:  Wed, Jul 31, 2024 11:22 PMTo:  "Alejandro Colomar"<alx@kernel.org>; 
Cc:  "Michael Kerrisk"<mtk.manpages@gmail.com>; "linux-man"<linux-man@vger.kernel.org>; "linux-api"<linux-api@vger.kernel.org>; 
Subject:  Re: [PATCH v3] epoll.7: clarify the event distribution under edge-triggered mode

Hi Alejandro,
I believe I've found a more suitable place to put this added clarification sentence. Please check out the [PATCH v4], thanks!
-----------
Best regards,Andy Pan