stack trace of threads that wake up a specific target userspace thread blocking in epoll_wait

[Jun Wang <junwang123@gmail.com>]

Hi Everyone,

I have a profiling tool question that could go beyond perf.

In an application, there are threads blocking on epoll_wait(). The
file descriptors are sockets connecting to other servers in a data
center with less than 1ms network latency. On normal systems, the
delay to wake up epoll_wait is much less than 1ms. It is noticed on a
system that sometimes it takes an unbelievable 30ms delay to wake up
the thread blocking in epoll_wait(). To be exact, the delay of
reference here is between tcp_rcv_established() and epoll_wait.return.

I wonder whether there is a way to dump out the stack trace of
threads, or some other kernel entities, that wake up that thread. The
idea is to figure out why there is a huge delay by looking at how the
delayed is ended, and by whom/what.

epoll wakeup internels in the kernel space can be referred to at
http://www.slideshare.net/llj098/epoll-from-the-kernel-side.

As a reference, there is a dtrace script that does this on Solaris.
https://github.com/brendangregg/dtrace-cloud-tools/blob/master/system/cv_wakeup_slow.d

Would perf/systemTap or other Linux tools be helpful?

Thanks,
Jun