epoll improvements

epoll improvements

[Nev Ikte]

Hi,

I've a server using epoll and I've noticed that when
the traffic is low, the epoll_wait() latency goes up.

I've tried to reproduce it with a single client
and basically, if ep_poll() is able to find an event or the timeout is 0,
the latency is down to 5usec, otherwise if it enters the waitqueue
the latency goes up to 10-25usec, which impact the application performance.

Looking at the code there is something like a todo ("Is it worth to try to dig for events ?")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/eventpoll.c?id=refs/tags/v3.17#n1650

Anyone knows how the situation can be improved?
in theory having something like a 50usec spin loop 
checking for events before entering in the waitqueue should improve the situation.

suggestion? patches?
thanks!

epoll improvements

[Greg KH]

On Thu, Oct 16, 2014 at 02:09:05PM +0200, Nev Ikte wrote:
> Hi,
> 
> I've a server using epoll and I've noticed that when
> the traffic is low, the epoll_wait() latency goes up.
> 
> I've tried to reproduce it with a single client
> and basically, if ep_poll() is able to find an event or the timeout is 0,
> the latency is down to 5usec, otherwise if it enters the waitqueue
> the latency goes up to 10-25usec, which impact the application performance.
> 
> Looking at the code there is something like a todo ("Is it worth to try to dig for events ?")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/eventpoll.c?id=refs/tags/v3.17#n1650
> 
> Anyone knows how the situation can be improved?
> in theory having something like a 50usec spin loop 
> checking for events before entering in the waitqueue should improve the situation.
> 
> suggestion? patches?

The epoll code is subtle, tricky, and very very difficult to modify.  I
suggest trying your suggestion and seeing if it helps any.  But be
prepared to go down a long rabbit-hole in this area of the kernel.

Enjoy the trip!  :)

greg k-h

epoll improvements

[Jason Ball]

The usual solution in low latency environments where the higher delays
is an issue is to simply spin on a poll.  If your polling multiple
sockets then you need to cater for this in your code and avoid the
kernel paths wherever possible.    Depending on the hardware
configuration consistent low latency performance is very possible,
I've done it on a number of occasions, you need to avoid the kernel at
all costs.

Of course I know nothing of your specific requirements...

Cheers





On Thu, Oct 16, 2014 at 11:17 PM, Greg KH <greg@kroah.com> wrote:
> On Thu, Oct 16, 2014 at 02:09:05PM +0200, Nev Ikte wrote:
>> Hi,
>>
>> I've a server using epoll and I've noticed that when
>> the traffic is low, the epoll_wait() latency goes up.
>>
>> I've tried to reproduce it with a single client
>> and basically, if ep_poll() is able to find an event or the timeout is 0,
>> the latency is down to 5usec, otherwise if it enters the waitqueue
>> the latency goes up to 10-25usec, which impact the application performance.
>>
>> Looking at the code there is something like a todo ("Is it worth to try to dig for events ?")
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/eventpoll.c?id=refs/tags/v3.17#n1650
>>
>> Anyone knows how the situation can be improved?
>> in theory having something like a 50usec spin loop
>> checking for events before entering in the waitqueue should improve the situation.
>>
>> suggestion? patches?
>
> The epoll code is subtle, tricky, and very very difficult to modify.  I
> suggest trying your suggestion and seeing if it helps any.  But be
> prepared to go down a long rabbit-hole in this area of the kernel.
>
> Enjoy the trip!  :)
>
> greg k-h
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



-- 
--

Teach your kids Science, or somebody else will :/

jason at ball.net
vk2vjb at google.com
callsign: vk2vjb

epoll improvements

[Valdis.Kletnieks at vt.edu]

On Thu, 16 Oct 2014 14:09:05 +0200, "Nev Ikte" said:

> I've tried to reproduce it with a single client
> and basically, if ep_poll() is able to find an event or the timeout is 0,
> the latency is down to 5usec, otherwise if it enters the waitqueue
> the latency goes up to 10-25usec, which impact the application performance.

Is it possible that what you're actually measuring here is your CPU's
latency coming out of a sleep state if every task is in a wait queue?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141016/0fecc742/attachment.bin 

epoll improvements

[michi1 at michaelblizek.twilightparadox.com]

Hi!

On 13:02 Thu 16 Oct     , Valdis.Kletnieks at vt.edu wrote:
> On Thu, 16 Oct 2014 14:09:05 +0200, "Nev Ikte" said:
> 
> > I've tried to reproduce it with a single client
> > and basically, if ep_poll() is able to find an event or the timeout is 0,
> > the latency is down to 5usec, otherwise if it enters the waitqueue
> > the latency goes up to 10-25usec, which impact the application performance.
> 
> Is it possible that what you're actually measuring here is your CPU's
> latency coming out of a sleep state if every task is in a wait queue?

I have seen wakeup latencies of ~100usec on some systems. You may want to
disable sleep states and see of performance gets better. Take a look at:

Documentation/power/pm_qos_interface.txt

Also, you probably want disable frequency scaling:

/sys/devices/system/cpu/cpu.../cpufreq/scaling_governor should return
"performance" for every cpu. You can change it with
eche "performance" > /sys/...

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com