PROBLEM: epoll_wait() does not return events when running in multiple threads

[Sergey Nikitin <nikitins@oktetlabs.ru>]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Hi!

epoll does not report an event to all the threads running epoll_wait() 
on the same epoll descriptor.
The behavior appeared in recent kernel versions starting with 5.6 probably.

How to reproduce:
- create a pair of sockets
- create epoll instance
- register the socket on the epoll instance, listen for EPOLLIN events
- start 2 threads running epoll_wait()
- send some data to the socket
- see that epoll_wait() within one of the threads reported an event, 
unlike another.

I attached a python script reproducing the issue.
Here's the output on my environment:
1. Fail case
   $ cat /proc/version
   Linux version 5.7.9-200.fc32.x86_64 
(mockbuild@bkernel01.iad2.fedoraproject.org) (gcc version 10.1.1 
20200507 (Red Hat 10.1.1-1) (GCC), GNU ld version 2.34-3.fc32) #1 SMP 
Fri Jul 17 16:23:37 UTC 2020
   $ ./multiple_same_epfd.py
   MainThread: created epfd5
   Thread-1 epfd5: start polling
   Thread-2 epfd5: start polling
   MainThread: Send some data
   Thread-2 epfd5: got events: 1
   Thread-1 epfd5: got events: 0
2. Pass case
   $ cat /proc/version
   Linux version 5.4.17-200.fc31.x86_64 
(mockbuild@bkernel04.phx2.fedoraproject.org) (gcc version 9.2.1 20190827 
(Red Hat 9.2.1-1) (GCC)) #1 SMP Sat Feb 1 19:00:13 UTC 2020
   $ ./multiple_same_epfd.py
   MainThread: created epfd5
   Thread-1 epfd5: start polling
   Thread-2 epfd5: start polling
   MainThread: Send some data
   Thread-2 epfd5: got events: 1
   Thread-1 epfd5: got events: 1

I created a Bugzilla bug also:
https://bugzilla.kernel.org/show_bug.cgi?id=208943

-- 
Best regards,
Sergey Nikitin


[-- Attachment #2: multiple_same_epfd.py --]
[-- Type: text/x-python, Size: 1197 bytes --]

#!/usr/bin/python3

import select
import socket
import threading

# Mutex to print messages from multiple threads
lock = threading.Lock()


def epoll_wait_thread(epfd):
    lock.acquire()
    print(threading.currentThread().getName(), " epfd", epfd.fileno(), ": start polling", sep='')
    lock.release()
    events = epfd.poll(3)
    lock.acquire()
    print(threading.currentThread().getName(), " epfd", epfd.fileno(), ": got events: ", len(events), sep='')
    lock.release()


# Create a connection
s1, s2 = socket.socketpair(socket.AF_UNIX)

# Create epoll descriptor and register a socket
epfd = select.epoll()
epfd.register(s1.fileno(), select.EPOLLIN)
print(threading.currentThread().getName(), ": created epfd", epfd.fileno(), sep='')

# Start 2 threads with epoll_wait() routine
threads = []
for i in range(2):
    thread = threading.Thread(target=epoll_wait_thread, args=(epfd,))
    thread.start()
    threads.append(thread)

# Send some data to unblock epoll_wait() threads
lock.acquire()
print(threading.currentThread().getName(), ": Send some data", sep='')
lock.release()
s2.sendall(b'qwerty')

# Cleanup
for thread in threads:
    thread.join()
epfd.close()
s1.close()
s2.close()