No SO_BUSY_POLL for UDP?

No SO_BUSY_POLL for UDP?

[Lukas Erlacher]

Hello netdev,

I am trying to benchmark SO_BUSY_POLL and it appears not to be functioning with UDP.

I have been trying with netperf as well as with a small udp echo server and a load generator; I am trying to reproduce the Intel benchmarks[1] so I am using X520 NICs with rx-usecs set to 100.

Performing TCP_RR netperf test with /proc/sys/net/core/busy_{read,poll} set to 0 produces this:

> root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t TCP_RR -l 5
> MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> Local /Remote
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 16384  87380  1        1       5.00     9762.51

With busy_* set to 50 produces this:

> root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t TCP_RR -l 5
> MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> Local /Remote
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 16384  87380  1        1       5.00     42127.63
> 16384  87380

 From that I conclude that the system manages 9700 round-trips per second without busy poll set and 42127 with busy poll enabled. So, busy polling works with TCP.

However, no matter what busy_* is set to, UDP_RR produces this:

> root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t UDP_RR -l 5
> MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> Local /Remote
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 212992 212992 1        1       5.00     9763.20
> 212992 212992

Around 9700 round-trips per second no matter the busy_poll setting.

The Intel whitepaper on busy polling[1] shows a UDP benchmark, so I am assuming that busy polling should be available for UDP sockets. Is this a regression or was the feature intentionally removed? As best I can tell the behaviour is identical on kernels 3.11, 3.16 and 3.19, but on 3.11 and 3.16 I have only tested it with my custom udp echo server.

Best regards,
Lukas Erlacher

[1] http://www.intel.in/content/dam/www/public/us/en/documents/white-papers/open-source-kernel-enhancements-paper.pdf

Re: No SO_BUSY_POLL for UDP?

[Eric Dumazet]

On Tue, 2015-03-17 at 23:26 +0100, Lukas Erlacher wrote:
> Hello netdev,
> 
> I am trying to benchmark SO_BUSY_POLL and it appears not to be functioning with UDP.
> 
> I have been trying with netperf as well as with a small udp echo server and a load generator; I am trying to reproduce the Intel benchmarks[1] so I am using X520 NICs with rx-usecs set to 100.
> 
> Performing TCP_RR netperf test with /proc/sys/net/core/busy_{read,poll} set to 0 produces this:
> 
> > root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t TCP_RR -l 5
> > MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> > Local /Remote
> > Socket Size   Request  Resp.   Elapsed  Trans.
> > Send   Recv   Size     Size    Time     Rate
> > bytes  Bytes  bytes    bytes   secs.    per sec
> >
> > 16384  87380  1        1       5.00     9762.51
> 
> With busy_* set to 50 produces this:
> 
> > root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t TCP_RR -l 5
> > MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> > Local /Remote
> > Socket Size   Request  Resp.   Elapsed  Trans.
> > Send   Recv   Size     Size    Time     Rate
> > bytes  Bytes  bytes    bytes   secs.    per sec
> >
> > 16384  87380  1        1       5.00     42127.63
> > 16384  87380
> 
>  From that I conclude that the system manages 9700 round-trips per second without busy poll set and 42127 with busy poll enabled. So, busy polling works with TCP.
> 
> However, no matter what busy_* is set to, UDP_RR produces this:
> 
> > root@klaipeda ~/MoonGen (git)-[master] # netperf -H 192.168.1.2,udpv4 -t UDP_RR -l 5
> > MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.2 () port 0 AF_INET : demo : first burst 0
> > Local /Remote
> > Socket Size   Request  Resp.   Elapsed  Trans.
> > Send   Recv   Size     Size    Time     Rate
> > bytes  Bytes  bytes    bytes   secs.    per sec
> >
> > 212992 212992 1        1       5.00     9763.20
> > 212992 212992
> 
> Around 9700 round-trips per second no matter the busy_poll setting.
> 
> The Intel whitepaper on busy polling[1] shows a UDP benchmark, so I am assuming that busy polling should be available for UDP sockets. Is this a regression or was the feature intentionally removed? As best I can tell the behaviour is identical on kernels 3.11, 3.16 and 3.19, but on 3.11 and 3.16 I have only tested it with my custom udp echo server.
> 
> Best regards,
> Lukas Erlacher
> 
> [1] http://www.intel.in/content/dam/www/public/us/en/documents/white-papers/open-source-kernel-enhancements-paper.pdf


busypoll should be available on _connected_ UDP sockets. Otherwise, we
would have to look all RX queues of the NIC, not a single one.

Try netperf  -t UDP_RR -- -N -n

    -n                Use the connected socket for UDP locally
    -N                Use the connected socket for UDP remotely

(Although I believe netserver->netperf is ignoring the -N option, that
is a netperf bug)

check with ss -a|grep udp

Re: No SO_BUSY_POLL for UDP?

[Lukas Erlacher]

On 03/18/2015 12:20 AM, Eric Dumazet wrote:
> 
> busypoll should be available on _connected_ UDP sockets. Otherwise, we
> would have to look all RX queues of the NIC, not a single one.
> 
> Try netperf  -t UDP_RR -- -N -n
> 
>     -n                Use the connected socket for UDP locally
>     -N                Use the connected socket for UDP remotely
> 
> (Although I believe netserver->netperf is ignoring the -N option, that
> is a netperf bug)
> 
> check with ss -a|grep udp
> 

Thank you very much, I was missing that bit of info. With connected
sockets, busy polling works perfectly.

Best,
Lukas Erlacher