* vxlan/veth performance issues on net.git + latest kernels
@ 2013-12-03 15:05 Or Gerlitz
2013-12-03 15:30 ` Eric Dumazet
2013-12-03 17:12 ` Eric Dumazet
0 siblings, 2 replies; 63+ messages in thread
From: Or Gerlitz @ 2013-12-03 15:05 UTC (permalink / raw)
To: Eric Dumazet, Alexei Starovoitov, Pravin B Shelar; +Cc: David Miller, netdev
I've been chasing lately a performance issues which come into play when
combining veth and vxlan over fast Ethernet NIC.
I came across it while working to enable TCP stateless offloads for
vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the
issue without any HWoffloads involved, so it would be easier to discuss
like that (no offloads involved).
The setup involves a stacked {veth --> bridge --> vlxan --> IP stack -->
NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain.
Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs,
vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for
multiple sessions, as long as veth isn't involved. Once veth is used I
can't get to > 7-8Gbs, no matter how many sessions are used. For the
time being, I manually took into account the tunneling overhead and
reduced the veth pair MTU by 50 bytes.
Looking on the kernel TCP counters in a {veth --> bridge --> vxlan -->
NIC} configuration, on the client side I see lots of hits for the
following TCP counters (the numbers are just single sample, I look on
the output of iterative sampling every seconds, e.g using "watch -d -n 1
netstat -st"):
67092 segments retransmited
31461 times recovered from packet loss due to SACK data
Detected reordering 1045142 times using SACK
436215 fast retransmits
59966 forward retransmits
Also on the passive side I see hits for the "Quick ack mode was
activated N times" counter, see below full snapshot of the counters from
both sides.
Without using veth, e.g when running in a {vxlan -> NIC} or {bridge -->
vxlan --> NIC},I see hits only for the "recovered from packet loss due
to SACK data" counter and fastretransmits counter, but not for the
forward retransmits or "Detected reordering N timesusing SACK". Also,
the quick ack mode counter isn't active on the passive side.
I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems
on all. At this point I don't really see a past point to go and apply
bisection. So I hope this counter report can help to shed some light on
the nature of the problem and possible solution, so ideas welcome!!
without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2,
the results
for the net.git are pretty much the same.
18/32/38 NIC
17/30/35 bridge --> NIC
14/23/35 veth --> bridge --> NIC
with vxlan, these are the Gbs results for 1/2/4 streams
6/12/14 vxlan --> IP --> NIC
5/10/14 bridge --> vxlan --> IP --> NIC
6/7/7 veth --> bridge --> vxlan --> IP --> NIC
Also, the 3.12.2 number do get any better also when adding a ported
version of 82d8189826d5 "veth: extend features to support tunneling" on
top of 3.12.2
See @ the end the sequence of commands I use for the environment
Or.
--> TCP counters from active side
# netstat -ts
IcmpMsg:
InType0: 2
InType8: 1
OutType0: 1
OutType3: 4
OutType8: 2
Tcp:
189 active connections openings
4 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
22403193 segments received
541234150 segments send out
14248 segments retransmited
0 bad segments received.
5 resets sent
UdpLite:
TcpExt:
2 invalid SYN cookies received
178 TCP sockets finished time wait in fast timer
10 delayed acks sent
Quick ack mode was activated 1 times
4 packets directly queued to recvmsg prequeue.
3728 packets directly received from backlog
2 packets directly received from prequeue
2524 packets header predicted
4 packets header predicted and directly queued to user
19793310 acknowledgments not containing data received
1216966 predicted acknowledgments
2130 times recovered from packet loss due to SACK data
Detected reordering 73 times using FACK
Detected reordering 11424 times using SACK
55 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 457
2 congestion windows recovered after partial ack
11498 fast retransmits
2748 forward retransmits
2 other TCP timeouts
TCPLossProbes: 4
3 DSACKs sent for old packets
TCPSackShifted: 1037782
TCPSackMerged: 332827
TCPSackShiftFallback: 598055
TCPRcvCoalesce: 380
TCPOFOQueue: 463
TCPSpuriousRtxHostQueues: 192
IpExt:
InNoRoutes: 1
InMcastPkts: 191
OutMcastPkts: 28
InBcastPkts: 25
InOctets: 1789360097
OutOctets: 893757758988
InMcastOctets: 8152
OutMcastOctets: 3044
InBcastOctets: 4259
InNoECTPkts: 30117553
--> TCP counters from passiveside
netstat -ts
IcmpMsg:
InType0: 1
InType8: 2
OutType0: 2
OutType3: 5
OutType8: 1
Tcp:
75 active connections openings
140 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
146888643 segments received
27430160 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
UdpLite:
TcpExt:
3 invalid SYN cookies received
72 TCP sockets finished time wait in fast timer
10 delayed acks sent
3 delayed acks further delayed because of locked socket
Quick ack mode was activated 13548 times
4 packets directly queued to recvmsg prequeue.
2 packets directly received from prequeue
139384763 packets header predicted
2 packets header predicted and directly queued to user
671 acknowledgments not containing data received
938 predicted acknowledgments
TCPLossProbes: 2
TCPLossProbeRecovery: 1
14 DSACKs sent for old packets
TCPBacklogDrop: 848
TCPRcvCoalesce: 118368414
TCPOFOQueue: 3167879
IpExt:
InNoRoutes: 1
InMcastPkts: 184
OutMcastPkts: 26
InBcastPkts: 26
InOctets: 1007364296775
OutOctets: 2433872888
InMcastOctets: 6202
OutMcastOctets: 2888
InBcastOctets: 4597
InNoECTPkts: 702313233
client side (node 144)
----------------------
ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN
ifconfig vxlan42 192.168.42.144/24 up
brctl addbr br-vx
ip link set br-vx up
ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42
ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.144/24 up
ip link set veth1 up
ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450
server side (node 147)
----------------------
ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN
ifconfig vxlan42 192.168.42.147/24 up
brctl addbr br-vx
ip link set br-vx up
ifconfig br-vx 192.168.52.147/24 up
brctl addif br-vx vxlan42
ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.147/24 up
ip link set veth1 up
ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 15:05 vxlan/veth performance issues on net.git + latest kernels Or Gerlitz @ 2013-12-03 15:30 ` Eric Dumazet 2013-12-03 19:55 ` Or Gerlitz 2013-12-03 17:12 ` Eric Dumazet 1 sibling, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 15:30 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 2013-12-03 at 17:05 +0200, Or Gerlitz wrote: > I've been chasing lately a performance issues which come into play when > combining veth and vxlan over fast Ethernet NIC. > > I came across it while working to enable TCP stateless offloads for > vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the > issue without any HWoffloads involved, so it would be easier to discuss > like that (no offloads involved). > > The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> > NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain. > > Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, > vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for > multiple sessions, as long as veth isn't involved. Once veth is used I > can't get to > 7-8Gbs, no matter how many sessions are used. For the > time being, I manually took into account the tunneling overhead and > reduced the veth pair MTU by 50 bytes. > > Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> > NIC} configuration, on the client side I see lots of hits for the > following TCP counters (the numbers are just single sample, I look on > the output of iterative sampling every seconds, e.g using "watch -d -n 1 > netstat -st"): > > 67092 segments retransmited > > 31461 times recovered from packet loss due to SACK data > Detected reordering 1045142 times using SACK > 436215 fast retransmits > 59966 forward retransmits > > Also on the passive side I see hits for the "Quick ack mode was > activated N times" counter, see below full snapshot of the counters from > both sides. > > Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> > vxlan --> NIC},I see hits only for the "recovered from packet loss due > to SACK data" counter and fastretransmits counter, but not for the > forward retransmits or "Detected reordering N timesusing SACK". Also, > the quick ack mode counter isn't active on the passive side. > > I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems > on all. At this point I don't really see a past point to go and apply > bisection. So I hope this counter report can help to shed some light on > the nature of the problem and possible solution, so ideas welcome!! > > without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, > the results > for the net.git are pretty much the same. > > 18/32/38 NIC > 17/30/35 bridge --> NIC > 14/23/35 veth --> bridge --> NIC > > with vxlan, these are the Gbs results for 1/2/4 streams > > 6/12/14 vxlan --> IP --> NIC > 5/10/14 bridge --> vxlan --> IP --> NIC > 6/7/7 veth --> bridge --> vxlan --> IP --> NIC > > Also, the 3.12.2 number do get any better also when adding a ported > version of 82d8189826d5 "veth: extend features to support tunneling" on > top of 3.12.2 > > See @ the end the sequence of commands I use for the environment > > Or. > > > --> TCP counters from active side > > # netstat -ts > IcmpMsg: > InType0: 2 > InType8: 1 > OutType0: 1 > OutType3: 4 > OutType8: 2 > Tcp: > 189 active connections openings > 4 passive connection openings > 0 failed connection attempts > 0 connection resets received > 4 connections established > 22403193 segments received > 541234150 segments send out > 14248 segments retransmited > 0 bad segments received. > 5 resets sent > UdpLite: > TcpExt: > 2 invalid SYN cookies received > 178 TCP sockets finished time wait in fast timer > 10 delayed acks sent > Quick ack mode was activated 1 times > 4 packets directly queued to recvmsg prequeue. > 3728 packets directly received from backlog > 2 packets directly received from prequeue > 2524 packets header predicted > 4 packets header predicted and directly queued to user > 19793310 acknowledgments not containing data received > 1216966 predicted acknowledgments > 2130 times recovered from packet loss due to SACK data > Detected reordering 73 times using FACK > Detected reordering 11424 times using SACK > 55 congestion windows partially recovered using Hoe heuristic > TCPDSACKUndo: 457 > 2 congestion windows recovered after partial ack > 11498 fast retransmits > 2748 forward retransmits > 2 other TCP timeouts > TCPLossProbes: 4 > 3 DSACKs sent for old packets > TCPSackShifted: 1037782 > TCPSackMerged: 332827 > TCPSackShiftFallback: 598055 > TCPRcvCoalesce: 380 > TCPOFOQueue: 463 > TCPSpuriousRtxHostQueues: 192 > IpExt: > InNoRoutes: 1 > InMcastPkts: 191 > OutMcastPkts: 28 > InBcastPkts: 25 > InOctets: 1789360097 > OutOctets: 893757758988 > InMcastOctets: 8152 > OutMcastOctets: 3044 > InBcastOctets: 4259 > InNoECTPkts: 30117553 > > > > --> TCP counters from passiveside > > netstat -ts > IcmpMsg: > InType0: 1 > InType8: 2 > OutType0: 2 > OutType3: 5 > OutType8: 1 > Tcp: > 75 active connections openings > 140 passive connection openings > 0 failed connection attempts > 0 connection resets received > 4 connections established > 146888643 segments received > 27430160 segments send out > 0 segments retransmited > 0 bad segments received. > 6 resets sent > UdpLite: > TcpExt: > 3 invalid SYN cookies received > 72 TCP sockets finished time wait in fast timer > 10 delayed acks sent > 3 delayed acks further delayed because of locked socket > Quick ack mode was activated 13548 times > 4 packets directly queued to recvmsg prequeue. > 2 packets directly received from prequeue > 139384763 packets header predicted > 2 packets header predicted and directly queued to user > 671 acknowledgments not containing data received > 938 predicted acknowledgments > TCPLossProbes: 2 > TCPLossProbeRecovery: 1 > 14 DSACKs sent for old packets > TCPBacklogDrop: 848 Thats bad : Dropping packets on receiver. Check also "ifconfig -a" to see if rxdrop is increasing as well. > TCPRcvCoalesce: 118368414 lack of GRO : receiver seems to not be able to receive as fast as you want. > TCPOFOQueue: 3167879 So many packets are received out of order (because of losses) > IpExt: > InNoRoutes: 1 > InMcastPkts: 184 > OutMcastPkts: 26 > InBcastPkts: 26 > InOctets: 1007364296775 > OutOctets: 2433872888 > InMcastOctets: 6202 > OutMcastOctets: 2888 > InBcastOctets: 4597 > InNoECTPkts: 702313233 > > > client side (node 144) > ---------------------- > > ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN > ifconfig vxlan42 192.168.42.144/24 up > > brctl addbr br-vx > ip link set br-vx up > > ifconfig br-vx 192.168.52.144/24 up > brctl addif br-vx vxlan42 > > ip link add type veth > brctl addif br-vx veth1 > ifconfig veth0 192.168.62.144/24 up > ip link set veth1 up > > ifconfig veth0 mtu 1450 > ifconfig veth1 mtu 1450 > > > server side (node 147) > ---------------------- > > ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN > ifconfig vxlan42 192.168.42.147/24 up > > brctl addbr br-vx > ip link set br-vx up > > ifconfig br-vx 192.168.52.147/24 up > brctl addif br-vx vxlan42 > > > ip link add type veth > brctl addif br-vx veth1 > ifconfig veth0 192.168.62.147/24 up > ip link set veth1 up > > ifconfig veth0 mtu 1450 > ifconfig veth1 mtu 1450 > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 15:30 ` Eric Dumazet @ 2013-12-03 19:55 ` Or Gerlitz 2013-12-03 21:11 ` Joseph Gasparakis 0 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 19:55 UTC (permalink / raw) To: Eric Dumazet, Jerry Chu Cc: Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, Dec 3, 2013 at 5:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 17:05 +0200, Or Gerlitz wrote: >> I've been chasing lately a performance issues which come into play when >> combining veth and vxlan over fast Ethernet NIC. >> >> I came across it while working to enable TCP stateless offloads for >> vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the >> issue without any HWoffloads involved, so it would be easier to discuss >> like that (no offloads involved). >> >> The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> >> NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain. >> >> Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, >> vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for >> multiple sessions, as long as veth isn't involved. Once veth is used I >> can't get to > 7-8Gbs, no matter how many sessions are used. For the >> time being, I manually took into account the tunneling overhead and >> reduced the veth pair MTU by 50 bytes. >> >> Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> >> NIC} configuration, on the client side I see lots of hits for the >> following TCP counters (the numbers are just single sample, I look on >> the output of iterative sampling every seconds, e.g using "watch -d -n 1 >> netstat -st"): >> >> 67092 segments retransmited >> >> 31461 times recovered from packet loss due to SACK data >> Detected reordering 1045142 times using SACK >> 436215 fast retransmits >> 59966 forward retransmits >> >> Also on the passive side I see hits for the "Quick ack mode was >> activated N times" counter, see below full snapshot of the counters from >> both sides. >> >> Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> >> vxlan --> NIC},I see hits only for the "recovered from packet loss due >> to SACK data" counter and fastretransmits counter, but not for the >> forward retransmits or "Detected reordering N timesusing SACK". Also, >> the quick ack mode counter isn't active on the passive side. >> >> I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems >> on all. At this point I don't really see a past point to go and apply >> bisection. So I hope this counter report can help to shed some light on >> the nature of the problem and possible solution, so ideas welcome!! >> >> without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, >> the results >> for the net.git are pretty much the same. >> >> 18/32/38 NIC >> 17/30/35 bridge --> NIC >> 14/23/35 veth --> bridge --> NIC >> >> with vxlan, these are the Gbs results for 1/2/4 streams >> >> 6/12/14 vxlan --> IP --> NIC >> 5/10/14 bridge --> vxlan --> IP --> NIC >> 6/7/7 veth --> bridge --> vxlan --> IP --> NIC >> >> Also, the 3.12.2 number do get any better also when adding a ported >> version of 82d8189826d5 "veth: extend features to support tunneling" on >> top of 3.12.2 >> >> See @ the end the sequence of commands I use for the environment >> >> Or. >> >> >> --> TCP counters from active side >> >> # netstat -ts >> IcmpMsg: >> InType0: 2 >> InType8: 1 >> OutType0: 1 >> OutType3: 4 >> OutType8: 2 >> Tcp: >> 189 active connections openings >> 4 passive connection openings >> 0 failed connection attempts >> 0 connection resets received >> 4 connections established >> 22403193 segments received >> 541234150 segments send out >> 14248 segments retransmited >> 0 bad segments received. >> 5 resets sent >> UdpLite: >> TcpExt: >> 2 invalid SYN cookies received >> 178 TCP sockets finished time wait in fast timer >> 10 delayed acks sent >> Quick ack mode was activated 1 times >> 4 packets directly queued to recvmsg prequeue. >> 3728 packets directly received from backlog >> 2 packets directly received from prequeue >> 2524 packets header predicted >> 4 packets header predicted and directly queued to user >> 19793310 acknowledgments not containing data received >> 1216966 predicted acknowledgments >> 2130 times recovered from packet loss due to SACK data >> Detected reordering 73 times using FACK >> Detected reordering 11424 times using SACK >> 55 congestion windows partially recovered using Hoe heuristic >> TCPDSACKUndo: 457 >> 2 congestion windows recovered after partial ack >> 11498 fast retransmits >> 2748 forward retransmits >> 2 other TCP timeouts >> TCPLossProbes: 4 >> 3 DSACKs sent for old packets >> TCPSackShifted: 1037782 >> TCPSackMerged: 332827 >> TCPSackShiftFallback: 598055 >> TCPRcvCoalesce: 380 >> TCPOFOQueue: 463 >> TCPSpuriousRtxHostQueues: 192 >> IpExt: >> InNoRoutes: 1 >> InMcastPkts: 191 >> OutMcastPkts: 28 >> InBcastPkts: 25 >> InOctets: 1789360097 >> OutOctets: 893757758988 >> InMcastOctets: 8152 >> OutMcastOctets: 3044 >> InBcastOctets: 4259 >> InNoECTPkts: 30117553 >> >> >> >> --> TCP counters from passiveside >> >> netstat -ts >> IcmpMsg: >> InType0: 1 >> InType8: 2 >> OutType0: 2 >> OutType3: 5 >> OutType8: 1 >> Tcp: >> 75 active connections openings >> 140 passive connection openings >> 0 failed connection attempts >> 0 connection resets received >> 4 connections established >> 146888643 segments received >> 27430160 segments send out >> 0 segments retransmited >> 0 bad segments received. >> 6 resets sent >> UdpLite: >> TcpExt: >> 3 invalid SYN cookies received >> 72 TCP sockets finished time wait in fast timer >> 10 delayed acks sent >> 3 delayed acks further delayed because of locked socket >> Quick ack mode was activated 13548 times >> 4 packets directly queued to recvmsg prequeue. >> 2 packets directly received from prequeue >> 139384763 packets header predicted >> 2 packets header predicted and directly queued to user >> 671 acknowledgments not containing data received >> 938 predicted acknowledgments >> TCPLossProbes: 2 >> TCPLossProbeRecovery: 1 >> 14 DSACKs sent for old packets >> TCPBacklogDrop: 848 > > Thats bad : Dropping packets on receiver. > > Check also "ifconfig -a" to see if rxdrop is increasing as well. > >> TCPRcvCoalesce: 118368414 > > lack of GRO : receiver seems to not be able to receive as fast as you want. > >> TCPOFOQueue: 3167879 > > So many packets are received out of order (because of losses) I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out? > >> IpExt: >> InNoRoutes: 1 >> InMcastPkts: 184 >> OutMcastPkts: 26 >> InBcastPkts: 26 >> InOctets: 1007364296775 >> OutOctets: 2433872888 >> InMcastOctets: 6202 >> OutMcastOctets: 2888 >> InBcastOctets: 4597 >> InNoECTPkts: 702313233 >> >> >> client side (node 144) >> ---------------------- >> >> ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN >> ifconfig vxlan42 192.168.42.144/24 up >> >> brctl addbr br-vx >> ip link set br-vx up >> >> ifconfig br-vx 192.168.52.144/24 up >> brctl addif br-vx vxlan42 >> >> ip link add type veth >> brctl addif br-vx veth1 >> ifconfig veth0 192.168.62.144/24 up >> ip link set veth1 up >> >> ifconfig veth0 mtu 1450 >> ifconfig veth1 mtu 1450 >> >> >> server side (node 147) >> ---------------------- >> >> ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN >> ifconfig vxlan42 192.168.42.147/24 up >> >> brctl addbr br-vx >> ip link set br-vx up >> >> ifconfig br-vx 192.168.52.147/24 up >> brctl addif br-vx vxlan42 >> >> >> ip link add type veth >> brctl addif br-vx veth1 >> ifconfig veth0 192.168.62.147/24 up >> ip link set veth1 up >> >> ifconfig veth0 mtu 1450 >> ifconfig veth1 mtu 1450 >> >> > > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 19:55 ` Or Gerlitz @ 2013-12-03 21:11 ` Joseph Gasparakis 2013-12-03 21:09 ` Or Gerlitz 0 siblings, 1 reply; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-03 21:11 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 3 Dec 2013, Or Gerlitz wrote: > On Tue, Dec 3, 2013 at 5:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Tue, 2013-12-03 at 17:05 +0200, Or Gerlitz wrote: > >> I've been chasing lately a performance issues which come into play when > >> combining veth and vxlan over fast Ethernet NIC. > >> > >> I came across it while working to enable TCP stateless offloads for > >> vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the > >> issue without any HWoffloads involved, so it would be easier to discuss > >> like that (no offloads involved). > >> > >> The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> > >> NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain. > >> > >> Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, > >> vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for > >> multiple sessions, as long as veth isn't involved. Once veth is used I > >> can't get to > 7-8Gbs, no matter how many sessions are used. For the > >> time being, I manually took into account the tunneling overhead and > >> reduced the veth pair MTU by 50 bytes. > >> > >> Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> > >> NIC} configuration, on the client side I see lots of hits for the > >> following TCP counters (the numbers are just single sample, I look on > >> the output of iterative sampling every seconds, e.g using "watch -d -n 1 > >> netstat -st"): > >> > >> 67092 segments retransmited > >> > >> 31461 times recovered from packet loss due to SACK data > >> Detected reordering 1045142 times using SACK > >> 436215 fast retransmits > >> 59966 forward retransmits > >> > >> Also on the passive side I see hits for the "Quick ack mode was > >> activated N times" counter, see below full snapshot of the counters from > >> both sides. > >> > >> Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> > >> vxlan --> NIC},I see hits only for the "recovered from packet loss due > >> to SACK data" counter and fastretransmits counter, but not for the > >> forward retransmits or "Detected reordering N timesusing SACK". Also, > >> the quick ack mode counter isn't active on the passive side. > >> > >> I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems > >> on all. At this point I don't really see a past point to go and apply > >> bisection. So I hope this counter report can help to shed some light on > >> the nature of the problem and possible solution, so ideas welcome!! > >> > >> without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, > >> the results > >> for the net.git are pretty much the same. > >> > >> 18/32/38 NIC > >> 17/30/35 bridge --> NIC > >> 14/23/35 veth --> bridge --> NIC > >> > >> with vxlan, these are the Gbs results for 1/2/4 streams > >> > >> 6/12/14 vxlan --> IP --> NIC > >> 5/10/14 bridge --> vxlan --> IP --> NIC > >> 6/7/7 veth --> bridge --> vxlan --> IP --> NIC > >> > >> Also, the 3.12.2 number do get any better also when adding a ported > >> version of 82d8189826d5 "veth: extend features to support tunneling" on > >> top of 3.12.2 > >> > >> See @ the end the sequence of commands I use for the environment > >> > >> Or. > >> > >> > >> --> TCP counters from active side > >> > >> # netstat -ts > >> IcmpMsg: > >> InType0: 2 > >> InType8: 1 > >> OutType0: 1 > >> OutType3: 4 > >> OutType8: 2 > >> Tcp: > >> 189 active connections openings > >> 4 passive connection openings > >> 0 failed connection attempts > >> 0 connection resets received > >> 4 connections established > >> 22403193 segments received > >> 541234150 segments send out > >> 14248 segments retransmited > >> 0 bad segments received. > >> 5 resets sent > >> UdpLite: > >> TcpExt: > >> 2 invalid SYN cookies received > >> 178 TCP sockets finished time wait in fast timer > >> 10 delayed acks sent > >> Quick ack mode was activated 1 times > >> 4 packets directly queued to recvmsg prequeue. > >> 3728 packets directly received from backlog > >> 2 packets directly received from prequeue > >> 2524 packets header predicted > >> 4 packets header predicted and directly queued to user > >> 19793310 acknowledgments not containing data received > >> 1216966 predicted acknowledgments > >> 2130 times recovered from packet loss due to SACK data > >> Detected reordering 73 times using FACK > >> Detected reordering 11424 times using SACK > >> 55 congestion windows partially recovered using Hoe heuristic > >> TCPDSACKUndo: 457 > >> 2 congestion windows recovered after partial ack > >> 11498 fast retransmits > >> 2748 forward retransmits > >> 2 other TCP timeouts > >> TCPLossProbes: 4 > >> 3 DSACKs sent for old packets > >> TCPSackShifted: 1037782 > >> TCPSackMerged: 332827 > >> TCPSackShiftFallback: 598055 > >> TCPRcvCoalesce: 380 > >> TCPOFOQueue: 463 > >> TCPSpuriousRtxHostQueues: 192 > >> IpExt: > >> InNoRoutes: 1 > >> InMcastPkts: 191 > >> OutMcastPkts: 28 > >> InBcastPkts: 25 > >> InOctets: 1789360097 > >> OutOctets: 893757758988 > >> InMcastOctets: 8152 > >> OutMcastOctets: 3044 > >> InBcastOctets: 4259 > >> InNoECTPkts: 30117553 > >> > >> > >> > >> --> TCP counters from passiveside > >> > >> netstat -ts > >> IcmpMsg: > >> InType0: 1 > >> InType8: 2 > >> OutType0: 2 > >> OutType3: 5 > >> OutType8: 1 > >> Tcp: > >> 75 active connections openings > >> 140 passive connection openings > >> 0 failed connection attempts > >> 0 connection resets received > >> 4 connections established > >> 146888643 segments received > >> 27430160 segments send out > >> 0 segments retransmited > >> 0 bad segments received. > >> 6 resets sent > >> UdpLite: > >> TcpExt: > >> 3 invalid SYN cookies received > >> 72 TCP sockets finished time wait in fast timer > >> 10 delayed acks sent > >> 3 delayed acks further delayed because of locked socket > >> Quick ack mode was activated 13548 times > >> 4 packets directly queued to recvmsg prequeue. > >> 2 packets directly received from prequeue > >> 139384763 packets header predicted > >> 2 packets header predicted and directly queued to user > >> 671 acknowledgments not containing data received > >> 938 predicted acknowledgments > >> TCPLossProbes: 2 > >> TCPLossProbeRecovery: 1 > >> 14 DSACKs sent for old packets > >> TCPBacklogDrop: 848 > > > > Thats bad : Dropping packets on receiver. > > > > Check also "ifconfig -a" to see if rxdrop is increasing as well. > > > >> TCPRcvCoalesce: 118368414 > > > > lack of GRO : receiver seems to not be able to receive as fast as you want. > > > >> TCPOFOQueue: 3167879 > > > > So many packets are received out of order (because of losses) > > I see that there's no GRO also for the non-veth tests which involve > vxlan, and over there the receiving side is capable to consume the > packets, do you have rough explaination why adding veth to the chain > is such game changer which makes things to start falling out? > I have seen this before. Here are my findings: The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation. I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix. > > > > > >> IpExt: > >> InNoRoutes: 1 > >> InMcastPkts: 184 > >> OutMcastPkts: 26 > >> InBcastPkts: 26 > >> InOctets: 1007364296775 > >> OutOctets: 2433872888 > >> InMcastOctets: 6202 > >> OutMcastOctets: 2888 > >> InBcastOctets: 4597 > >> InNoECTPkts: 702313233 > >> > >> > >> client side (node 144) > >> ---------------------- > >> > >> ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN > >> ifconfig vxlan42 192.168.42.144/24 up > >> > >> brctl addbr br-vx > >> ip link set br-vx up > >> > >> ifconfig br-vx 192.168.52.144/24 up > >> brctl addif br-vx vxlan42 > >> > >> ip link add type veth > >> brctl addif br-vx veth1 > >> ifconfig veth0 192.168.62.144/24 up > >> ip link set veth1 up > >> > >> ifconfig veth0 mtu 1450 > >> ifconfig veth1 mtu 1450 > >> > >> > >> server side (node 147) > >> ---------------------- > >> > >> ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN > >> ifconfig vxlan42 192.168.42.147/24 up > >> > >> brctl addbr br-vx > >> ip link set br-vx up > >> > >> ifconfig br-vx 192.168.52.147/24 up > >> brctl addif br-vx vxlan42 > >> > >> > >> ip link add type veth > >> brctl addif br-vx veth1 > >> ifconfig veth0 192.168.62.147/24 up > >> ip link set veth1 up > >> > >> ifconfig veth0 mtu 1450 > >> ifconfig veth1 mtu 1450 > >> > >> > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe netdev" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:11 ` Joseph Gasparakis @ 2013-12-03 21:09 ` Or Gerlitz 2013-12-03 21:24 ` Eric Dumazet 2013-12-03 23:13 ` vxlan/veth performance issues on net.git + latest kernels Joseph Gasparakis 0 siblings, 2 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 21:09 UTC (permalink / raw) To: Joseph Gasparakis Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: >>> lack of GRO : receiver seems to not be able to receive as fast as you want. >>>> TCPOFOQueue: 3167879 >>> So many packets are received out of order (because of losses) >> I see that there's no GRO also for the non-veth tests which involve >> vxlan, and over there the receiving side is capable to consume the >> packets, do you have rough explaination why adding veth to the chain >> is such game changer which makes things to start falling out? > I have seen this before. Here are my findings: > > The gso_type is different if the skb comes from veth or not. From veth, > you will see the SKB_GSO_DODGY set. This breaks things as when the > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, > the stack drops it silently. I never got the time to find the root cause > for this, but I know it causes re-transmissions and big performance > degregation. > > I went as far as just quickly hacking a one liner unsetting the DODGY bit > in vxlan.c and that bypassed the issue and recovered the performance > problem, but obviously this is not a real fix. thanks for the heads up, few quick questions/clafications -- -- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver. -- without the hack you did, still packets are sent/received, so what makes the stack to drop only some of them? -- why packets coming from veth would have the SKB_GSO_DODGY bit set? -- so where is now (say net.git or 3.12.x) this one line you commented out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c explicit setting of SKB_GSO_DODGY Also, I am pretty sure the problem exists also when sending/receiving guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I just sticked to the veth flavour b/c its one (== the hypervisor) network stack to debug and not two (+ the guest one). ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:09 ` Or Gerlitz @ 2013-12-03 21:24 ` Eric Dumazet 2013-12-03 21:36 ` Or Gerlitz 2013-12-03 23:13 ` vxlan/veth performance issues on net.git + latest kernels Joseph Gasparakis 1 sibling, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 21:24 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 2013-12-03 at 23:09 +0200, Or Gerlitz wrote: > On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > > >>> lack of GRO : receiver seems to not be able to receive as fast as you want. > >>>> TCPOFOQueue: 3167879 > >>> So many packets are received out of order (because of losses) > > >> I see that there's no GRO also for the non-veth tests which involve > >> vxlan, and over there the receiving side is capable to consume the > >> packets, do you have rough explaination why adding veth to the chain > >> is such game changer which makes things to start falling out? > > > I have seen this before. Here are my findings: > > > > The gso_type is different if the skb comes from veth or not. From veth, > > you will see the SKB_GSO_DODGY set. This breaks things as when the > > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, > > the stack drops it silently. I never got the time to find the root cause > > for this, but I know it causes re-transmissions and big performance > > degregation. > > > > I went as far as just quickly hacking a one liner unsetting the DODGY bit > > in vxlan.c and that bypassed the issue and recovered the performance > > problem, but obviously this is not a real fix. > > thanks for the heads up, few quick questions/clafications -- > > -- you are talking on drops done @ the sender side, correct? Eric was > saying we have evidences that the drops happen on the receiver. I suggested you take a look at the receiver, like "ifconfig -a" I suspect one cpu is 100% in sofirq mode draining packets from the NIC and feeding IP / TCP stack. Because of vxlan encap, all the packets are delivered to a single RX queue (I dont think mlx4 is able to look at inner header to get L4 info) mpstat -P ALL 10 10 ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:24 ` Eric Dumazet @ 2013-12-03 21:36 ` Or Gerlitz 2013-12-03 21:50 ` David Miller 0 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 21:36 UTC (permalink / raw) To: Eric Dumazet Cc: Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, Dec 3, 2013 at 11:24 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 23:09 +0200, Or Gerlitz wrote: >> On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis >> <joseph.gasparakis@intel.com> wrote: >> >> >>> lack of GRO : receiver seems to not be able to receive as fast as you want. >> >>>> TCPOFOQueue: 3167879 >> >>> So many packets are received out of order (because of losses) >> >> >> I see that there's no GRO also for the non-veth tests which involve >> >> vxlan, and over there the receiving side is capable to consume the >> >> packets, do you have rough explaination why adding veth to the chain >> >> is such game changer which makes things to start falling out? >> >> > I have seen this before. Here are my findings: >> > >> > The gso_type is different if the skb comes from veth or not. From veth, >> > you will see the SKB_GSO_DODGY set. This breaks things as when the >> > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, >> > the stack drops it silently. I never got the time to find the root cause >> > for this, but I know it causes re-transmissions and big performance >> > degregation. >> > >> > I went as far as just quickly hacking a one liner unsetting the DODGY bit >> > in vxlan.c and that bypassed the issue and recovered the performance >> > problem, but obviously this is not a real fix. >> >> thanks for the heads up, few quick questions/clafications -- >> >> -- you are talking on drops done @ the sender side, correct? Eric was >> saying we have evidences that the drops happen on the receiver. > > I suggested you take a look at the receiver, like "ifconfig -a" Eric, sorry I am away from the system now, will try to get some access and report back now and if not, tomorow, but > I suspect one cpu is 100% in sofirq mode draining packets from the NIC > and feeding IP / TCP stack. > Because of vxlan encap, all the packets are delivered to a single RX > queue (I dont think mlx4 is able to look at inner header to get L4 info) With the new card, ConnectX3-pro we are able to look on inner headers and do RX/TX checksum and LSO for the encapsulated traffic, this is how I initially got into this problem. But as I wrote earlier, I was able to see the problem w.o activating the offloads for the inner packets. Sorry if I didn't mention that, but from the mlx4_en NIC driver point of view, different stream do map to different RX queues, b/c the HW does RSS on the outer (UDP) header and the sender vxlan code uses few sockets to send multiple streams which each having difference source UDP port. For the "outer RSS" you don't need the -pro card, just make sure the udp_rss module param of mlx4 is set. I also thought that under veth there's contention point which could explain why packets are dropped, but haven't found it. > mpstat -P ALL 10 10 ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:36 ` Or Gerlitz @ 2013-12-03 21:50 ` David Miller 2013-12-03 21:55 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: David Miller @ 2013-12-03 21:50 UTC (permalink / raw) To: or.gerlitz Cc: eric.dumazet, joseph.gasparakis, hkchu, ogerlitz, edumazet, ast, pshelar, netdev From: Or Gerlitz <or.gerlitz@gmail.com> Date: Tue, 3 Dec 2013 23:36:50 +0200 > I also thought that under veth there's contention point which could > explain why packets are dropped, but haven't found it. At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it. I suspect what you will find is that either the cpu is at %100, or sporadic events where GRO is not performed is killing the stream. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:50 ` David Miller @ 2013-12-03 21:55 ` Eric Dumazet 2013-12-03 22:15 ` Or Gerlitz ` (2 more replies) 0 siblings, 3 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 21:55 UTC (permalink / raw) To: David Miller Cc: or.gerlitz, joseph.gasparakis, hkchu, ogerlitz, edumazet, ast, pshelar, netdev On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > At this point I would use drop monitor to figure out in what context > packets are being dropped on the floor. There are scripts provided > with the perf tool to utilize it. Most easy way is to do : perf record -e skb:kfree_skb -a -g sleep 10 perf report ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:55 ` Eric Dumazet @ 2013-12-03 22:15 ` Or Gerlitz 2013-12-03 22:22 ` Or Gerlitz 2013-12-03 23:10 ` Or Gerlitz 2 siblings, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 22:15 UTC (permalink / raw) To: Eric Dumazet Cc: David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > >> At this point I would use drop monitor to figure out in what context >> packets are being dropped on the floor. There are scripts provided >> with the perf tool to utilize it. > > Most easy way is to do : > > perf record -e skb:kfree_skb -a -g sleep 10 > > perf report The version of perf I have on these nodes fail to run the 1st command, anyway, here's some data which was asked by Eric passive side top + plain perf for two streams top - 00:08:09 up 7:53, 3 users, load average: 0.59, 0.43, 0.32 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 5.7%sy, 0.0%ni, 92.2%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st Cpu1 : 0.6%us, 17.2%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 51.7%si, 0.0%st Cpu2 : 0.7%us, 4.3%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st Cpu3 : 0.0%us, 1.7%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 14.4%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.0%hi, 35.6%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8183220k total, 1383012k used, 6800208k free, 10724k buffers Swap: 2097148k total, 0k used, 2097148k free, 127520k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3857 root 20 0 381m 640 468 S 69.2 0.0 6:45.22 iperf 15 root 20 0 0 0 0 S 7.3 0.0 0:09.92 ksoftirqd/1 40 root 20 0 0 0 0 S 7.3 0.0 0:12.10 ksoftirqd/6 20 root 20 0 0 0 0 S 0.3 0.0 0:20.95 ksoftirqd/2 9229 root 20 0 220m 16m 6168 S 0.3 0.2 0:00.78 perf 9578 root 20 0 15084 1156 856 R 0.3 0.0 0:00.01 top 1 root 20 0 23648 1644 1320 S 0.0 0.0 0:00.34 init Samples: 1K of event 'cpu-clock', Event count (approx.): 283500000 5.17% [kernel] [k] fib_table_lookup 5.02% [kernel] [k] __do_softirq 3.63% [kernel] [k] copy_user_generic_unrolled 3.55% perf [.] 0x000000000004c808 3.16% [kernel] [k] __netif_receive_skb_core 3.09% [kernel] [k] _raw_spin_unlock_irqrestore 2.93% [kernel] [k] enqueue_to_backlog 2.85% [kernel] [k] _raw_spin_lock 2.08% [kernel] [k] __pskb_pull_tail 1.93% [kernel] [k] __udp4_lib_lookup 1.85% [kernel] [k] ip_rcv 1.77% [kernel] [k] __slab_free 1.62% [kernel] [k] _raw_spin_unlock_irq 1.54% [kernel] [k] check_leaf 1.47% [kernel] [k] pvclock_clocksource_read 1.47% [kernel] [k] skb_copy_bits 1.31% [kernel] [k] __rcu_read_unlock 1.31% [kernel] [k] tcp_v4_rcv 1.23% [mlx4_en] [k] mlx4_en_process_rx_cq 1.00% [kernel] [k] skb_try_coalesce 1.00% [kernel] [k] napi_gro_frags 1.00% [kernel] [k] __inet_lookup_established 0.93% [mlx4_en] [k] mlx4_en_xmit ifconfig -a listing (eth2 is the NIC over which we run, br1 is the bridge) - no recorded drops [root@r-dcs47-005 ~]# ifconfig -a br1 Link encap:Ethernet HWaddr 1A:10:63:AD:55:4C inet addr:192.168.52.147 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::f88c:60ff:fe19:6d33/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:32120731 errors:0 dropped:0 overruns:0 frame:0 TX packets:987550 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:46552789260 (43.3 GiB) TX bytes:53462960 (50.9 MiB) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4A:05 inet addr:10.212.74.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4a05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:265478 errors:0 dropped:16 overruns:0 frame:0 TX packets:14391 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:28861836 (27.5 MiB) TX bytes:2561303 (2.4 MiB) eth2 Link encap:Ethernet HWaddr 00:02:C9:E9:C0:82 inet addr:192.168.30.147 Bcast:192.168.30.255 Mask:255.255.255.0 inet6 addr: fe80::2:c900:1e9:c082/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:853813090 errors:0 dropped:0 overruns:0 frame:0 TX packets:76377493 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1270293325678 (1.1 TiB) TX bytes:7858334980 (7.3 GiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:48 errors:0 dropped:0 overruns:0 frame:0 TX packets:48 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3236 (3.1 KiB) TX bytes:3236 (3.1 KiB) veth0 Link encap:Ethernet HWaddr EA:4F:C9:1C:5D:EE inet addr:192.168.62.147 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::e84f:c9ff:fe1c:5dee/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:372217768 errors:0 dropped:0 overruns:0 frame:0 TX packets:60732630 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:544449586684 (507.0 GiB) TX bytes:3840742148 (3.5 GiB) veth1 Link encap:Ethernet HWaddr 1A:10:63:AD:55:4C inet6 addr: fe80::1810:63ff:fead:554c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:60732693 errors:0 dropped:0 overruns:0 frame:0 TX packets:372217836 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3840746210 (3.5 GiB) TX bytes:544449686236 (507.0 GiB) vxlan42 Link encap:Ethernet HWaddr FE:EF:4E:C7:0F:06 inet addr:192.168.42.147 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::fcef:4eff:fec7:f06/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:404338687 errors:0 dropped:0 overruns:0 frame:0 TX packets:61720259 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:585791638636 (545.5 GiB) TX bytes:4881734478 (4.5 GiB) active side perf top Samples: 74K of event 'cpu-clock', Event count (approx.): 12653268255 14.74% [kernel] [k] __copy_user_nocache 7.91% [kernel] [k] csum_partial 7.51% [kernel] [k] _raw_spin_lock 6.38% [kernel] [k] _raw_spin_unlock_irqrestore 5.57% [kernel] [k] __do_softirq 4.24% [mlx4_en] [k] mlx4_en_xmit 2.40% [kernel] [k] __copy_skb_header 2.10% [kernel] [k] _raw_spin_unlock_irq 2.04% [kernel] [k] memcpy 1.92% [kernel] [k] fib_table_lookup 1.73% [kernel] [k] tcp_sendmsg 1.64% [kernel] [k] skb_segment 1.52% [kernel] [k] __slab_free 1.09% [kernel] [k] __alloc_skb 0.89% [kernel] [k] __slab_alloc 0.85% [kernel] [k] tcp_ack 0.83% [kernel] [k] __netif_receive_skb_core 0.81% [kernel] [k] __kmalloc_node_track_caller 0.75% [kernel] [k] ip_send_check 0.70% [kernel] [k] put_compound_page 0.70% [kernel] [k] ksize 0.67% [kernel] [k] pvclock_clocksource_read 0.65% [kernel] [k] dev_hard_start_xmit 0.61% [kernel] [k] kmem_cache_alloc_node 0.58% [kernel] [k] dev_queue_xmit 0.55% [kernel] [k] enqueue_to_backlog 0.52% [kernel] [k] __pskb_pull_tail 0.49% [kernel] [k] __iowrite64_copy 0.45% [kernel] [k] dev_queue_xmit_nit 0.45% [kernel] [k] skb_copy_bits 0.44% [kernel] [k] check_leaf 0.44% [kernel] [k] skb_release_data 0.44% [kernel] [k] get_page_from_freelist 0.43% [kernel] [k] inet_gso_segment 0.42% [mlx4_en] [k] mlx4_en_process_rx_cq 0.39% [kernel] [k] process_backlog 0.38% [kernel] [k] pskb_expand_head and top [root@r-dcs44-005 ~]# top top - 00:13:27 up 7:59, 3 users, load average: 1.11, 0.76, 0.44 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 10.3%sy, 0.0%ni, 87.0%id, 0.0%wa, 0.0%hi, 2.7%si, 0.0%st Cpu1 : 0.3%us, 11.7%sy, 0.0%ni, 84.3%id, 0.0%wa, 0.0%hi, 3.7%si, 0.0%st Cpu2 : 0.5%us, 18.4%sy, 0.0%ni, 43.4%id, 0.0%wa, 0.0%hi, 37.8%si, 0.0%st Cpu3 : 0.0%us, 5.1%sy, 0.0%ni, 94.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.3%us, 6.0%sy, 0.0%ni, 93.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Cpu6 : 0.3%us, 7.4%sy, 0.0%ni, 92.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 24.4%sy, 0.0%ni, 42.1%id, 0.0%wa, 0.0%hi, 33.5%si, 0.0%st Mem: 8183236k total, 1378928k used, 6804308k free, 10436k buffers Swap: 2097148k total, 0k used, 2097148k free, 128068k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17220 root 20 0 165m 564 456 S 122.4 0.0 6:14.34 iperf 20 root 20 0 0 0 0 S 1.3 0.0 0:07.85 ksoftirqd/2 45 root 20 0 0 0 0 S 1.3 0.0 0:05.65 ksoftirqd/7 18404 root 20 0 220m 16m 6384 S 0.7 0.2 0:00.62 perf 35 root 20 0 0 0 0 S 0.3 0.0 0:07.73 ksoftirqd/5 1 root 20 0 23520 1624 1316 S 0.0 0.0 0:00.33 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd and ifconfig -a (eth6 is the NIC over which we run) root@r-dcs44-005 ~]# ifconfig -a br1 Link encap:Ethernet HWaddr 12:C0:46:32:46:6A inet addr:192.168.52.144 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::2815:30ff:fe06:b5f9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:987575 errors:0 dropped:0 overruns:0 frame:0 TX packets:842756 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:39638896 (37.8 MiB) TX bytes:45313678898 (42.2 GiB) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4B:05 inet addr:10.212.75.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4b05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:193220 errors:0 dropped:55 overruns:0 frame:0 TX packets:17721 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:17006297 (16.2 MiB) TX bytes:2871741 (2.7 MiB) eth6 Link encap:Ethernet HWaddr 00:02:C9:E9:BB:B2 inet addr:192.168.30.144 Bcast:192.168.30.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:101512678 errors:0 dropped:0 overruns:0 frame:0 TX packets:995876432 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:10707213944 (9.9 GiB) TX bytes:1485292875970 (1.3 TiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:36 errors:0 dropped:0 overruns:0 frame:0 TX packets:36 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2456 (2.3 KiB) TX bytes:2456 (2.3 KiB) veth0 Link encap:Ethernet HWaddr 32:7D:34:FA:3A:A1 inet addr:192.168.62.144 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::307d:34ff:fefa:3aa1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:85847361 errors:0 dropped:0 overruns:0 frame:0 TX packets:72371900 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5431514098 (5.0 GiB) TX bytes:728410479216 (678.3 GiB) veth1 Link encap:Ethernet HWaddr 12:C0:46:32:46:6A inet6 addr: fe80::10c0:46ff:fe32:466a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:72371915 errors:0 dropped:0 overruns:0 frame:0 TX packets:85847368 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:728410821246 (678.3 GiB) TX bytes:5431514476 (5.0 GiB) vxlan42 Link encap:Ethernet HWaddr B2:F9:D2:68:A3:11 inet addr:192.168.42.144 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::b0f9:d2ff:fe68:a311/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:86834927 errors:0 dropped:0 overruns:0 frame:0 TX packets:73214688 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4269288656 (3.9 GiB) TX bytes:774896089976 (721.6 GiB) no drops ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:55 ` Eric Dumazet 2013-12-03 22:15 ` Or Gerlitz @ 2013-12-03 22:22 ` Or Gerlitz 2013-12-03 22:30 ` Hannes Frederic Sowa 2013-12-03 23:10 ` Or Gerlitz 2 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 22:22 UTC (permalink / raw) To: Eric Dumazet Cc: David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > >> At this point I would use drop monitor to figure out in what context >> packets are being dropped on the floor. There are scripts provided >> with the perf tool to utilize it. > > Most easy way is to do : > > perf record -e skb:kfree_skb -a -g sleep 10 some typo here? I tried the perf tool that comes with the net git ./perf record -e skb:kfree_skb -a -g sleep 10 invalid or unsupported event: 'skb:kfree_skb' Run 'perf list' for a list of valid events usage: perf record [<options>] [<command>] or: perf record [<options>] -- <command> [<options>] -e, --event <event> event selector. use 'perf list' to list available events > > perf report > > > anything critical missing here? perf]# make BUILD: Doing 'make -j16' parallel build Auto-detecting system features: ... backtrace: [ on ] ... dwarf: [ on ] ... fortify-source: [ on ] ... glibc: [ on ] ... gtk2: [ OFF ] ... gtk2-infobar: [ OFF ] ... libaudit: [ OFF ] ... libbfd: [ on ] ... libelf: [ on ] ... libelf-getphdrnum: [ on ] ... libelf-mmap: [ on ] ... libnuma: [ OFF ] ... libperl: [ on ] ... libpython: [ on ] ... libpython-version: [ on ] ... libslang: [ on ] ... libunwind: [ OFF ] ... on-exit: [ on ] ... stackprotector: [ on ] ... stackprotector-all: [ on ] ... timerfd: [ on ] config/Makefile:329: No libunwind found, disabling post unwind support. Please install libunwind-dev[el] >= 1.1 config/Makefile:354: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev config/Makefile:381: GTK2 not found, disables GTK2 support. Please install gtk2-devel or libgtk2.0-dev config/Makefile:536: No numa.h found, disables 'perf bench numa mem' benchmark, please install numa-libs-devel or libnuma-dev ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 22:22 ` Or Gerlitz @ 2013-12-03 22:30 ` Hannes Frederic Sowa 2013-12-03 22:35 ` Or Gerlitz 0 siblings, 1 reply; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-03 22:30 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 04, 2013 at 12:22:08AM +0200, Or Gerlitz wrote: > On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > > > >> At this point I would use drop monitor to figure out in what context > >> packets are being dropped on the floor. There are scripts provided > >> with the perf tool to utilize it. > > > > Most easy way is to do : > > > > perf record -e skb:kfree_skb -a -g sleep 10 > > some typo here? I tried the perf tool that comes with the net git > > ./perf record -e skb:kfree_skb -a -g sleep 10 > invalid or unsupported event: 'skb:kfree_skb' > Run 'perf list' for a list of valid events > > usage: perf record [<options>] [<command>] > or: perf record [<options>] -- <command> [<options>] > > -e, --event <event> event selector. use 'perf list' to list > available events -g takes an optional argument. Reorder the arguments: perf record -e skb:kfree_skb -g -a sleep 10 ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 22:30 ` Hannes Frederic Sowa @ 2013-12-03 22:35 ` Or Gerlitz 2013-12-03 22:39 ` Hannes Frederic Sowa 0 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 22:35 UTC (permalink / raw) To: Or Gerlitz, Eric Dumazet, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 4, 2013 at 12:30 AM, Hannes Frederic Sowa <hannes@stressinduktion.org> wrote: > On Wed, Dec 04, 2013 at 12:22:08AM +0200, Or Gerlitz wrote: >> On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: >> > >> >> At this point I would use drop monitor to figure out in what context >> >> packets are being dropped on the floor. There are scripts provided >> >> with the perf tool to utilize it. >> > >> > Most easy way is to do : >> > >> > perf record -e skb:kfree_skb -a -g sleep 10 >> >> some typo here? I tried the perf tool that comes with the net git >> >> ./perf record -e skb:kfree_skb -a -g sleep 10 >> invalid or unsupported event: 'skb:kfree_skb' >> Run 'perf list' for a list of valid events >> >> usage: perf record [<options>] [<command>] >> or: perf record [<options>] -- <command> [<options>] >> >> -e, --event <event> event selector. use 'perf list' to list >> available events > > -g takes an optional argument. Reorder the arguments: > > perf record -e skb:kfree_skb -g -a sleep 10 Sorry, it doesn't help, I get the same error even with the oerf kernel.org tree which has these events (no sign for skb's) List of pre-defined events (to be used in -e): cpu-cycles OR cycles [Hardware event] instructions [Hardware event] cache-references [Hardware event] cache-misses [Hardware event] branch-instructions OR branches [Hardware event] branch-misses [Hardware event] bus-cycles [Hardware event] stalled-cycles-frontend OR idle-cycles-frontend [Hardware event] stalled-cycles-backend OR idle-cycles-backend [Hardware event] ref-cycles [Hardware event] cpu-clock [Software event] task-clock [Software event] page-faults OR faults [Software event] context-switches OR cs [Software event] cpu-migrations OR migrations [Software event] minor-faults [Software event] major-faults [Software event] alignment-faults [Software event] emulation-faults [Software event] L1-dcache-loads [Hardware cache event] L1-dcache-load-misses [Hardware cache event] L1-dcache-stores [Hardware cache event] L1-dcache-store-misses [Hardware cache event] L1-dcache-prefetches [Hardware cache event] L1-dcache-prefetch-misses [Hardware cache event] L1-icache-loads [Hardware cache event] L1-icache-load-misses [Hardware cache event] L1-icache-prefetches [Hardware cache event] L1-icache-prefetch-misses [Hardware cache event] LLC-loads [Hardware cache event] LLC-load-misses [Hardware cache event] LLC-stores [Hardware cache event] LLC-store-misses [Hardware cache event] LLC-prefetches [Hardware cache event] LLC-prefetch-misses [Hardware cache event] dTLB-loads [Hardware cache event] dTLB-load-misses [Hardware cache event] dTLB-stores [Hardware cache event] dTLB-store-misses [Hardware cache event] dTLB-prefetches [Hardware cache event] ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 22:35 ` Or Gerlitz @ 2013-12-03 22:39 ` Hannes Frederic Sowa 0 siblings, 0 replies; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-03 22:39 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 04, 2013 at 12:35:37AM +0200, Or Gerlitz wrote: > Sorry, it doesn't help, I get the same error even with the oerf > kernel.org tree which has these events (no sign for skb's) > > > List of pre-defined events (to be used in -e): > cpu-cycles OR cycles [Hardware event] > instructions [Hardware event] > cache-references [Hardware event] > cache-misses [Hardware event] > branch-instructions OR branches [Hardware event] > branch-misses [Hardware event] > bus-cycles [Hardware event] > stalled-cycles-frontend OR idle-cycles-frontend [Hardware event] > stalled-cycles-backend OR idle-cycles-backend [Hardware event] > ref-cycles [Hardware event] > > cpu-clock [Software event] > task-clock [Software event] > page-faults OR faults [Software event] > context-switches OR cs [Software event] > cpu-migrations OR migrations [Software event] > minor-faults [Software event] > major-faults [Software event] > alignment-faults [Software event] > emulation-faults [Software event] > > L1-dcache-loads [Hardware cache event] > L1-dcache-load-misses [Hardware cache event] > L1-dcache-stores [Hardware cache event] > L1-dcache-store-misses [Hardware cache event] > L1-dcache-prefetches [Hardware cache event] > L1-dcache-prefetch-misses [Hardware cache event] > L1-icache-loads [Hardware cache event] > L1-icache-load-misses [Hardware cache event] > L1-icache-prefetches [Hardware cache event] > L1-icache-prefetch-misses [Hardware cache event] > LLC-loads [Hardware cache event] > LLC-load-misses [Hardware cache event] > LLC-stores [Hardware cache event] > LLC-store-misses [Hardware cache event] > LLC-prefetches [Hardware cache event] > LLC-prefetch-misses [Hardware cache event] > dTLB-loads [Hardware cache event] > dTLB-load-misses [Hardware cache event] > dTLB-stores [Hardware cache event] > dTLB-store-misses [Hardware cache event] > dTLB-prefetches [Hardware cache event] Is this the whole output of perf list? Then you seem to be missing some tracepoint options, CONFIG_TRACEPOINTS e.g.? I can confirm it works for me on a current net build. Greetings, Hannes ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:55 ` Eric Dumazet 2013-12-03 22:15 ` Or Gerlitz 2013-12-03 22:22 ` Or Gerlitz @ 2013-12-03 23:10 ` Or Gerlitz 2013-12-03 23:30 ` Or Gerlitz 2013-12-03 23:59 ` Eric Dumazet 2 siblings, 2 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 23:10 UTC (permalink / raw) To: Eric Dumazet Cc: David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > >> At this point I would use drop monitor to figure out in what context >> packets are being dropped on the floor. There are scripts provided >> with the perf tool to utilize it. > > Most easy way is to do : > > perf record -e skb:kfree_skb -a -g sleep 10 > perf report $ ./perf record -e skb:kfree_skb -g -a sleep 10 $ ./perf report -i perf.data Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406 + 97.13% swapper [kernel.kallsyms] [k] net_tx_action + 1.53% iperf [kernel.kallsyms] [k] net_tx_action + 1.03% perf [kernel.kallsyms] [k] net_tx_action + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action + 0.00% swapper [kernel.kallsyms] [k] kfree_skb + 0.00% sleep [kernel.kallsyms] [k] net_tx_action + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb + 0.00% iperf [kernel.kallsyms] [k] kfree_skb + 0.00% perf [kernel.kallsyms] [k] kfree_skb > > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:10 ` Or Gerlitz @ 2013-12-03 23:30 ` Or Gerlitz 2013-12-03 23:49 ` Hannes Frederic Sowa 2013-12-03 23:59 ` Eric Dumazet 1 sibling, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 23:30 UTC (permalink / raw) To: Eric Dumazet Cc: David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 4, 2013 at 1:10 AM, Or Gerlitz <or.gerlitz@gmail.com> wrote: > On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: >> >>> At this point I would use drop monitor to figure out in what context >>> packets are being dropped on the floor. There are scripts provided >>> with the perf tool to utilize it. >> >> Most easy way is to do : >> >> perf record -e skb:kfree_skb -a -g sleep 10 >> perf report > > $ ./perf record -e skb:kfree_skb -g -a sleep 10 > $ ./perf report -i perf.data > > > Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406 > + 97.13% swapper [kernel.kallsyms] [k] net_tx_action > + 1.53% iperf [kernel.kallsyms] [k] net_tx_action > + 1.03% perf [kernel.kallsyms] [k] net_tx_action > + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action > + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action > + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action > + 0.00% swapper [kernel.kallsyms] [k] kfree_skb > + 0.00% sleep [kernel.kallsyms] [k] net_tx_action > + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb > + 0.00% iperf [kernel.kallsyms] [k] kfree_skb > + 0.00% perf [kernel.kallsyms] [k] kfree_skb I added proper sorting (thanks Rick), here's the passive side report - 99.99% [kernel.kallsyms] [k] net_tx_action ? - net_tx_action ? - 59.20% __libc_recv ? 100.00% 0 ? - 35.61% __write_nocancel ? - 100.00% run_builtin ? main ? __libc_start_main ? - 2.60% __poll ? - 92.62% run_builtin ? main ? __libc_start_main ? - 7.38% 0x7f2d4c4adfd4 ? 0x626370523a32333a ? - 1.86% cmd_record ? run_builtin ? main ? __libc_start_main ? - 0.01% [kernel.kallsyms] [k] kfree_skb ? - kfree_skb ? - 50.00% __connect_nocancel ? 0x64697063612f6e75 ? - 33.33% __libc_recv ? 0 ? - 16.67% __write_nocancel ? run_builtin ? main ? __libc_start_main and the active side report 100.00% [kernel.kallsyms] [k] net_tx_action ? - net_tx_action ? - 76.91% __write_nocancel ? - 100.00% run_builtin ? main ? __libc_start_main ? 15.92% 0x37cc60e4ed ? - 2.69% __poll ? run_builtin ? main ? __libc_start_main ? - 1.92% cmd_record ? run_builtin ? main ? __libc_start_main ? - 1.66% pthread_cond_signal@@GLIBC_2. 3.2 ? 0x10000 ? - 0.52% perf_header__has_feat ? - 73.28% run_builtin ? main ? __libc_start_main ? - 26.72% cmd_record ? run_builtin ? main ? __libc_start_main ? - 0.00% [kernel.kallsyms] [k] kfree_skb ? - kfree_skb ? - 100.00% __GI___connect_internal ? - 50.00% get_mapping ? __nscd_get_map_ref ? 50.00% __nscd_open_socket > > > >> >> >> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:30 ` Or Gerlitz @ 2013-12-03 23:49 ` Hannes Frederic Sowa 0 siblings, 0 replies; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-03 23:49 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 04, 2013 at 01:30:09AM +0200, Or Gerlitz wrote: > On Wed, Dec 4, 2013 at 1:10 AM, Or Gerlitz <or.gerlitz@gmail.com> wrote: > > On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote: > >> > >>> At this point I would use drop monitor to figure out in what context > >>> packets are being dropped on the floor. There are scripts provided > >>> with the perf tool to utilize it. > >> > >> Most easy way is to do : > >> > >> perf record -e skb:kfree_skb -a -g sleep 10 > >> perf report > > > > $ ./perf record -e skb:kfree_skb -g -a sleep 10 > > $ ./perf report -i perf.data > > > > > > Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406 > > + 97.13% swapper [kernel.kallsyms] [k] net_tx_action > > + 1.53% iperf [kernel.kallsyms] [k] net_tx_action > > + 1.03% perf [kernel.kallsyms] [k] net_tx_action > > + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action > > + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action > > + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action > > + 0.00% swapper [kernel.kallsyms] [k] kfree_skb > > + 0.00% sleep [kernel.kallsyms] [k] net_tx_action > > + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb > > + 0.00% iperf [kernel.kallsyms] [k] kfree_skb > > + 0.00% perf [kernel.kallsyms] [k] kfree_skb > > I added proper sorting (thanks Rick), here's the passive side report Btw. the nice helper for the dropwatch is "perf script net_dropmonitor". Greetings, Hannes ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:10 ` Or Gerlitz 2013-12-03 23:30 ` Or Gerlitz @ 2013-12-03 23:59 ` Eric Dumazet 2013-12-04 0:26 ` Alexei Starovoitov ` (2 more replies) 1 sibling, 3 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 23:59 UTC (permalink / raw) To: Or Gerlitz Cc: David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, netdev@vger.kernel.org On Wed, 2013-12-04 at 01:10 +0200, Or Gerlitz wrote: > Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406 > + 97.13% swapper [kernel.kallsyms] [k] net_tx_action > + 1.53% iperf [kernel.kallsyms] [k] net_tx_action > + 1.03% perf [kernel.kallsyms] [k] net_tx_action > + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action > + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action > + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action > + 0.00% swapper [kernel.kallsyms] [k] kfree_skb > + 0.00% sleep [kernel.kallsyms] [k] net_tx_action > + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb > + 0.00% iperf [kernel.kallsyms] [k] kfree_skb > + 0.00% perf [kernel.kallsyms] [k] kfree_skb > Right, I actually have a patch for that, but was waiting for net-next being re-opened : commit 9a731d750dd8bf0b8c20fb1ca53c42317fb4dd37 Author: Eric Dumazet <edumazet@google.com> Date: Mon Nov 25 13:09:20 2013 -0800 net-fixes: introduce dev_consume_skb_any() Some NIC drivers use dev_kfree_skb_any() generic helper to free skbs, both for dropped packets and TX completed ones. To have "perf record -e skb:kfree_skb" give good hints on where packets are dropped (if any), we need to separate the two causes. dev_consume_skb_any() is a helper to free skbs that were properly sent to the wire. Signed-off-by: Eric Dumazet <edumazet@google.com> Google-Bug-Id: 11634401 --- diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c index ec96130533cc..8b8c2171b187 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c @@ -209,9 +209,9 @@ static u16 bnx2x_free_tx_pkt(struct bnx2x *bp, struct bnx2x_fp_txdata *txdata, if (likely(skb)) { (*pkts_compl)++; (*bytes_compl) += skb->len; + dev_consume_skb_any(skb); } - dev_kfree_skb_any(skb); tx_buf->first_bd = 0; tx_buf->skb = NULL; diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c index 8d3945ab7334..03081932e519 100644 --- a/drivers/net/ethernet/intel/e1000e/netdev.c +++ b/drivers/net/ethernet/intel/e1000e/netdev.c @@ -1058,7 +1058,7 @@ static void e1000_put_txbuf(struct e1000_ring *tx_ring, buffer_info->dma = 0; } if (buffer_info->skb) { - dev_kfree_skb_any(buffer_info->skb); + dev_consume_skb_any(buffer_info->skb); buffer_info->skb = NULL; } buffer_info->time_stamp = 0; diff --git a/drivers/net/ethernet/marvell/sky2.c b/drivers/net/ethernet/marvell/sky2.c index 43aa7acd84a6..294825efb248 100644 --- a/drivers/net/ethernet/marvell/sky2.c +++ b/drivers/net/ethernet/marvell/sky2.c @@ -2037,7 +2037,7 @@ static void sky2_tx_complete(struct sky2_port *sky2, u16 done) bytes_compl += skb->len; re->skb = NULL; - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); sky2->tx_next = RING_NEXT(idx, sky2->tx_ring_size); } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index f54ebd5a1702..653484bfae98 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -317,7 +317,7 @@ static u32 mlx4_en_free_tx_desc(struct mlx4_en_priv *priv, } } } - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); return tx_info->nr_txbb; } diff --git a/drivers/net/ethernet/nvidia/forcedeth.c b/drivers/net/ethernet/nvidia/forcedeth.c index 2d045be4b5cf..d44f7b69a6a0 100644 --- a/drivers/net/ethernet/nvidia/forcedeth.c +++ b/drivers/net/ethernet/nvidia/forcedeth.c @@ -2557,7 +2557,7 @@ static int nv_tx_done(struct net_device *dev, int limit) u64_stats_update_end(&np->swstats_tx_syncp); } bytes_compl += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; } @@ -2574,7 +2574,7 @@ static int nv_tx_done(struct net_device *dev, int limit) u64_stats_update_end(&np->swstats_tx_syncp); } bytes_compl += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; } @@ -2625,7 +2625,7 @@ static int nv_tx_done_optimized(struct net_device *dev, int limit) } bytes_cleaned += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 916241d16c67..ab8970693ff3 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -755,7 +755,7 @@ static void free_old_xmit_skbs(struct send_queue *sq) stats->tx_packets++; u64_stats_update_end(&stats->tx_syncp); - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); } } diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 7f0ed423a360..8a7482fa2656 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2380,6 +2380,13 @@ void dev_kfree_skb_irq(struct sk_buff *skb); */ void dev_kfree_skb_any(struct sk_buff *skb); +#define SKB_CONSUMED_MAGIC ((void *)0xDEAD0001) +static inline void dev_consume_skb_any(struct sk_buff *skb) +{ + skb->dev = SKB_CONSUMED_MAGIC; + dev_kfree_skb_any(skb); +} + int netif_rx(struct sk_buff *skb); int netif_rx_ni(struct sk_buff *skb); int netif_receive_skb(struct sk_buff *skb); diff --git a/net/core/dev.c b/net/core/dev.c index ba3b7ea5ebb3..b2b0e5776ce9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3306,7 +3306,10 @@ static void net_tx_action(struct softirq_action *h) clist = clist->next; WARN_ON(atomic_read(&skb->users)); - trace_kfree_skb(skb, net_tx_action); + if (likely(skb->dev == SKB_CONSUMED_MAGIC)) + trace_consume_skb(skb); + else + trace_kfree_skb(skb, net_tx_action); __kfree_skb(skb); } } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:59 ` Eric Dumazet @ 2013-12-04 0:26 ` Alexei Starovoitov 2013-12-04 0:36 ` Eric Dumazet 2013-12-04 6:39 ` David Miller 2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet 2 siblings, 1 reply; 63+ messages in thread From: Alexei Starovoitov @ 2013-12-04 0:26 UTC (permalink / raw) To: Eric Dumazet Cc: Or Gerlitz, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 3:59 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > +#define SKB_CONSUMED_MAGIC ((void *)0xDEAD0001) > +static inline void dev_consume_skb_any(struct sk_buff *skb) > +{ > + skb->dev = SKB_CONSUMED_MAGIC; > + dev_kfree_skb_any(skb); > +} > + > int netif_rx(struct sk_buff *skb); > int netif_rx_ni(struct sk_buff *skb); > int netif_receive_skb(struct sk_buff *skb); > diff --git a/net/core/dev.c b/net/core/dev.c > index ba3b7ea5ebb3..b2b0e5776ce9 100644 > --- a/net/core/dev.c > +++ b/net/core/dev.c > @@ -3306,7 +3306,10 @@ static void net_tx_action(struct softirq_action *h) > clist = clist->next; > > WARN_ON(atomic_read(&skb->users)); > - trace_kfree_skb(skb, net_tx_action); > + if (likely(skb->dev == SKB_CONSUMED_MAGIC)) > + trace_consume_skb(skb); > + else > + trace_kfree_skb(skb, net_tx_action); Could you use some other way to mark skb ? In tracing we might want to examine skb more carefully and not being able to see the device will limit the usability of this tracepoint. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:26 ` Alexei Starovoitov @ 2013-12-04 0:36 ` Eric Dumazet 2013-12-04 0:55 ` Alexei Starovoitov 0 siblings, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-04 0:36 UTC (permalink / raw) To: Alexei Starovoitov Cc: Or Gerlitz, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Tue, 2013-12-03 at 16:26 -0800, Alexei Starovoitov wrote: > > Could you use some other way to mark skb ? I could ;) > In tracing we might want to examine skb more carefully and not being > able to see the device > will limit the usability of this tracepoint. Unfortunately, using skb->dev as a pointer to device would be buggy or expensive (you would need to take a reference on device in order not letting it disappear, as we escape RCU protection) Current TRACE_EVENT for trace_consume_skb() does not use skb->dev. Anyway, this magic is pretty easy to change, I am open to suggestions. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:36 ` Eric Dumazet @ 2013-12-04 0:55 ` Alexei Starovoitov 2013-12-04 1:23 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Alexei Starovoitov @ 2013-12-04 0:55 UTC (permalink / raw) To: Eric Dumazet Cc: Or Gerlitz, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 4:36 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 16:26 -0800, Alexei Starovoitov wrote: > >> >> Could you use some other way to mark skb ? > > I could ;) > >> In tracing we might want to examine skb more carefully and not being >> able to see the device >> will limit the usability of this tracepoint. > > Unfortunately, using skb->dev as a pointer to device would be buggy or > expensive (you would need to take a reference on device in order not > letting it disappear, as we escape RCU protection) well, yes, you might have an skb around when device is already freed when skb_dst_noref. but I'm not suggesting anything expensive. Tracing definitely should not add overhead by doing rcu_lock() or dev_hold(). Instead it can go through skb, skb->dev, skb->dev->xxx via probe_kernel_read(). If dev is gone, it's still safe. > Anyway, this magic is pretty easy to change, I am open to suggestions. you're the expert :) use skb->mark field, since it's unused during freeing path... ? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:55 ` Alexei Starovoitov @ 2013-12-04 1:23 ` Eric Dumazet 2013-12-04 1:59 ` Alexei Starovoitov 2013-12-06 9:06 ` Or Gerlitz 0 siblings, 2 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-04 1:23 UTC (permalink / raw) To: Alexei Starovoitov Cc: Or Gerlitz, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Tue, 2013-12-03 at 16:55 -0800, Alexei Starovoitov wrote: > On Tue, Dec 3, 2013 at 4:36 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Tue, 2013-12-03 at 16:26 -0800, Alexei Starovoitov wrote: > > > >> > >> Could you use some other way to mark skb ? > > > > I could ;) > > > >> In tracing we might want to examine skb more carefully and not being > >> able to see the device > >> will limit the usability of this tracepoint. > > > > Unfortunately, using skb->dev as a pointer to device would be buggy or > > expensive (you would need to take a reference on device in order not > > letting it disappear, as we escape RCU protection) > > well, yes, you might have an skb around when device is already freed > when skb_dst_noref. > but I'm not suggesting anything expensive. Tracing definitely should > not add overhead by doing rcu_lock() or dev_hold(). Instead it can go > through skb, skb->dev, skb->dev->xxx via probe_kernel_read(). If dev > is gone, it's still safe. Its certainly not safe to 'probe'. Its not about faulting inexistent memory, that is the least of the problem. Any kind of information fetched from skb->dev might have been overwritten. You could for example fetch security sensitive data and expose it. > > > Anyway, this magic is pretty easy to change, I am open to suggestions. > > you're the expert :) use skb->mark field, since it's unused during > freeing path... ? cache line miss ;) skb->dev is in the first cache line, where we access skb->next anyway. I could use skb->cb[] like the following patch : diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c index ec96130533cc..8b8c2171b187 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c @@ -209,9 +209,9 @@ static u16 bnx2x_free_tx_pkt(struct bnx2x *bp, struct bnx2x_fp_txdata *txdata, if (likely(skb)) { (*pkts_compl)++; (*bytes_compl) += skb->len; + dev_consume_skb_any(skb); } - dev_kfree_skb_any(skb); tx_buf->first_bd = 0; tx_buf->skb = NULL; diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c index 8d3945ab7334..03081932e519 100644 --- a/drivers/net/ethernet/intel/e1000e/netdev.c +++ b/drivers/net/ethernet/intel/e1000e/netdev.c @@ -1058,7 +1058,7 @@ static void e1000_put_txbuf(struct e1000_ring *tx_ring, buffer_info->dma = 0; } if (buffer_info->skb) { - dev_kfree_skb_any(buffer_info->skb); + dev_consume_skb_any(buffer_info->skb); buffer_info->skb = NULL; } buffer_info->time_stamp = 0; diff --git a/drivers/net/ethernet/marvell/sky2.c b/drivers/net/ethernet/marvell/sky2.c index 43aa7acd84a6..294825efb248 100644 --- a/drivers/net/ethernet/marvell/sky2.c +++ b/drivers/net/ethernet/marvell/sky2.c @@ -2037,7 +2037,7 @@ static void sky2_tx_complete(struct sky2_port *sky2, u16 done) bytes_compl += skb->len; re->skb = NULL; - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); sky2->tx_next = RING_NEXT(idx, sky2->tx_ring_size); } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index f54ebd5a1702..653484bfae98 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -317,7 +317,7 @@ static u32 mlx4_en_free_tx_desc(struct mlx4_en_priv *priv, } } } - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); return tx_info->nr_txbb; } diff --git a/drivers/net/ethernet/nvidia/forcedeth.c b/drivers/net/ethernet/nvidia/forcedeth.c index 2d045be4b5cf..d44f7b69a6a0 100644 --- a/drivers/net/ethernet/nvidia/forcedeth.c +++ b/drivers/net/ethernet/nvidia/forcedeth.c @@ -2557,7 +2557,7 @@ static int nv_tx_done(struct net_device *dev, int limit) u64_stats_update_end(&np->swstats_tx_syncp); } bytes_compl += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; } @@ -2574,7 +2574,7 @@ static int nv_tx_done(struct net_device *dev, int limit) u64_stats_update_end(&np->swstats_tx_syncp); } bytes_compl += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; } @@ -2625,7 +2625,7 @@ static int nv_tx_done_optimized(struct net_device *dev, int limit) } bytes_cleaned += np->get_tx_ctx->skb->len; - dev_kfree_skb_any(np->get_tx_ctx->skb); + dev_consume_skb_any(np->get_tx_ctx->skb); np->get_tx_ctx->skb = NULL; tx_work++; diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 916241d16c67..ab8970693ff3 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -755,7 +755,7 @@ static void free_old_xmit_skbs(struct send_queue *sq) stats->tx_packets++; u64_stats_update_end(&stats->tx_syncp); - dev_kfree_skb_any(skb); + dev_consume_skb_any(skb); } } diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 7f0ed423a360..8b80a58ec1ac 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2374,11 +2374,38 @@ int netif_get_num_default_rss_queues(void); */ void dev_kfree_skb_irq(struct sk_buff *skb); +void __dev_kfree_skb_any(struct sk_buff *skb); + +struct __dev_kfree_skb_cb { + unsigned int reason; +}; + +static inline struct __dev_kfree_skb_cb *get_kfree_skb_cb(const struct sk_buff *skb) +{ + return (struct __dev_kfree_skb_cb *)skb->cb; +} + +enum { + SKB_REASON_CONSUMED, + SKB_REASON_DROPPED, +}; + /* Use this variant in places where it could be invoked * from either hardware interrupt or other context, with hardware interrupts * either disabled or enabled. + * Note that TX completion should use dev_consume_skb_any() */ -void dev_kfree_skb_any(struct sk_buff *skb); +static inline void dev_kfree_skb_any(struct sk_buff *skb) +{ + get_kfree_skb_cb(skb)->reason = SKB_REASON_DROPPED; + __dev_kfree_skb_any(skb); +} + +static inline void dev_consume_skb_any(struct sk_buff *skb) +{ + get_kfree_skb_cb(skb)->reason = SKB_REASON_CONSUMED; + __dev_kfree_skb_any(skb); +} int netif_rx(struct sk_buff *skb); int netif_rx_ni(struct sk_buff *skb); diff --git a/net/core/dev.c b/net/core/dev.c index ba3b7ea5ebb3..3170776e53da 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -2161,14 +2161,14 @@ void dev_kfree_skb_irq(struct sk_buff *skb) } EXPORT_SYMBOL(dev_kfree_skb_irq); -void dev_kfree_skb_any(struct sk_buff *skb) +void __dev_kfree_skb_any(struct sk_buff *skb) { if (in_irq() || irqs_disabled()) dev_kfree_skb_irq(skb); else dev_kfree_skb(skb); } -EXPORT_SYMBOL(dev_kfree_skb_any); +EXPORT_SYMBOL(__dev_kfree_skb_any); /** @@ -3306,7 +3306,10 @@ static void net_tx_action(struct softirq_action *h) clist = clist->next; WARN_ON(atomic_read(&skb->users)); - trace_kfree_skb(skb, net_tx_action); + if (likely(get_kfree_skb_cb(skb)->reason == SKB_REASON_CONSUMED)) + trace_consume_skb(skb); + else + trace_kfree_skb(skb, net_tx_action); __kfree_skb(skb); } } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 1:23 ` Eric Dumazet @ 2013-12-04 1:59 ` Alexei Starovoitov 2013-12-06 9:06 ` Or Gerlitz 1 sibling, 0 replies; 63+ messages in thread From: Alexei Starovoitov @ 2013-12-04 1:59 UTC (permalink / raw) To: Eric Dumazet Cc: Or Gerlitz, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Tue, Dec 3, 2013 at 5:23 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-12-03 at 16:55 -0800, Alexei Starovoitov wrote: >> On Tue, Dec 3, 2013 at 4:36 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > On Tue, 2013-12-03 at 16:26 -0800, Alexei Starovoitov wrote: >> > >> >> >> >> Could you use some other way to mark skb ? >> > >> > I could ;) >> > >> >> In tracing we might want to examine skb more carefully and not being >> >> able to see the device >> >> will limit the usability of this tracepoint. >> > >> > Unfortunately, using skb->dev as a pointer to device would be buggy or >> > expensive (you would need to take a reference on device in order not >> > letting it disappear, as we escape RCU protection) >> >> well, yes, you might have an skb around when device is already freed >> when skb_dst_noref. >> but I'm not suggesting anything expensive. Tracing definitely should >> not add overhead by doing rcu_lock() or dev_hold(). Instead it can go >> through skb, skb->dev, skb->dev->xxx via probe_kernel_read(). If dev >> is gone, it's still safe. > > Its certainly not safe to 'probe'. > > Its not about faulting inexistent memory, that is the least of the > problem. > > Any kind of information fetched from skb->dev might have been > overwritten. > > You could for example fetch security sensitive data and expose it. Of course. Even without walking pointer chains with probe() you could infer all sorts of info from tracepoints. That's why tracing filters are for root only. >> > Anyway, this magic is pretty easy to change, I am open to suggestions. >> >> you're the expert :) use skb->mark field, since it's unused during >> freeing path... ? > > cache line miss ;) > > skb->dev is in the first cache line, where we access skb->next anyway. > > I could use skb->cb[] like the following patch : > > +struct __dev_kfree_skb_cb { > + unsigned int reason; > +}; > + > +static inline struct __dev_kfree_skb_cb *get_kfree_skb_cb(const struct sk_buff *skb) > +{ > + return (struct __dev_kfree_skb_cb *)skb->cb; > +} > + > +enum { > + SKB_REASON_CONSUMED, > + SKB_REASON_DROPPED, > +}; > + > /* Use this variant in places where it could be invoked > * from either hardware interrupt or other context, with hardware interrupts > * either disabled or enabled. > + * Note that TX completion should use dev_consume_skb_any() > */ > -void dev_kfree_skb_any(struct sk_buff *skb); > +static inline void dev_kfree_skb_any(struct sk_buff *skb) > +{ > + get_kfree_skb_cb(skb)->reason = SKB_REASON_DROPPED; > + __dev_kfree_skb_any(skb); > +} > + > +static inline void dev_consume_skb_any(struct sk_buff *skb) > +{ > + get_kfree_skb_cb(skb)->reason = SKB_REASON_CONSUMED; > + __dev_kfree_skb_any(skb); > +} thanks. I think that is much cleaner. Ack. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 1:23 ` Eric Dumazet 2013-12-04 1:59 ` Alexei Starovoitov @ 2013-12-06 9:06 ` Or Gerlitz 2013-12-06 13:36 ` Eric Dumazet 1 sibling, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-06 9:06 UTC (permalink / raw) To: Eric Dumazet Cc: Alexei Starovoitov, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > skb->dev is in the first cache line, where we access skb->next anyway. > I could use skb->cb[] like the following patch : Hi Eric, I applied on the net tree the patch you posted yesterday "net: introduce dev_consume_skb_any()" along with the network drivers part of this patch, unless I got it wrong, I assume both pieces are needed? So I re-run the vxlan/veth test that we suspect goes through packet drops on TX. With the patches applied I have almost no samples of that event $ ./perf report -i perf.data Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action ? + 28.09% swapper [kernel.kallsyms] [k] net_tx_action ? + 28.09% sshd [kernel.kallsyms] [k] net_tx_action ? + 2.25% swapper [kernel.kallsyms] [k] kfree_skb ? + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action ? + 1.12% iperf [kernel.kallsyms] [k] net_tx_action ./perf report -i perf.data --sort dso,symbol Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 + 97.75% [kernel.kallsyms] [k] net_tx_action + 2.25% [kernel.kallsyms] [k] kfree_skb ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 9:06 ` Or Gerlitz @ 2013-12-06 13:36 ` Eric Dumazet 2013-12-07 21:20 ` Or Gerlitz 2013-12-08 12:09 ` Or Gerlitz 0 siblings, 2 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-06 13:36 UTC (permalink / raw) To: Or Gerlitz Cc: Alexei Starovoitov, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote: > On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > skb->dev is in the first cache line, where we access skb->next anyway. > > I could use skb->cb[] like the following patch : > > Hi Eric, I applied on the net tree the patch you posted yesterday > "net: introduce dev_consume_skb_any()" along with the network drivers > part of this patch, unless I got it wrong, I assume both pieces are > needed? > > So I re-run the vxlan/veth test that we suspect goes through packet drops on TX. > > With the patches applied I have almost no samples of that event > > $ ./perf report -i perf.data How did you get this perf.data file ? There are a few drops. > > Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 > + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action > > ? > + 28.09% swapper [kernel.kallsyms] [k] net_tx_action > > ? > + 28.09% sshd [kernel.kallsyms] [k] net_tx_action > > ? > + 2.25% swapper [kernel.kallsyms] [k] kfree_skb > > ? > + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action > > ? > + 1.12% iperf [kernel.kallsyms] [k] net_tx_action > > ./perf report -i perf.data --sort dso,symbol > Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 > + 97.75% [kernel.kallsyms] [k] net_tx_action > + 2.25% [kernel.kallsyms] [k] kfree_skb > -- OK, this means your driver drops few packets in its ndo_start_xmit() handler. Could you give us "ifconfig -a" reports as I already asked ? You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g -e skb:kfree_skb sleep 20 ; perf report) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index f54ebd5a1702..53130f27dec0 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -873,7 +873,7 @@ tx_drop_unmap: } tx_drop: - dev_kfree_skb_any(skb); + kfree_skb(skb); priv->stats.tx_dropped++; return NETDEV_TX_OK; } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 13:36 ` Eric Dumazet @ 2013-12-07 21:20 ` Or Gerlitz 2013-12-08 12:09 ` Or Gerlitz 1 sibling, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-07 21:20 UTC (permalink / raw) To: Eric Dumazet Cc: Alexei Starovoitov, David Miller, Joseph Gasparakis, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On Fri, Dec 6, 2013 at 3:36 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote: >> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > skb->dev is in the first cache line, where we access skb->next anyway. >> > I could use skb->cb[] like the following patch : >> >> Hi Eric, I applied on the net tree the patch you posted yesterday >> "net: introduce dev_consume_skb_any()" along with the network drivers >> part of this patch, unless I got it wrong, I assume both pieces are >> needed? >> >> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX. >> >> With the patches applied I have almost no samples of that event >> >> $ ./perf report -i perf.data > > How did you get this perf.data file ? There are a few drops. Using the command you suggested on the active side while running traffic (iperf tcp) $ ./perf record -e skb:kfree_skb -g -a sleep 10 >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action >> + 28.09% swapper [kernel.kallsyms] [k] net_tx_action >> + 28.09% sshd [kernel.kallsyms] [k] net_tx_action >> + 2.25% swapper [kernel.kallsyms] [k] kfree_skb >> + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action >> + 1.12% iperf [kernel.kallsyms] [k] net_tx_action >> ./perf report -i perf.data --sort dso,symbol >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 97.75% [kernel.kallsyms] [k] net_tx_action >> + 2.25% [kernel.kallsyms] [k] kfree_skb >> -- > OK, this means your driver drops few packets in its ndo_start_xmit() > handler. > > Could you give us "ifconfig -a" reports as I already asked ? will do tomorrow while infront of the setup. When I provided you the info last time http://marc.info/?l=linux-netdev&m=138610891121531&w=2 it included the ifconfig -a output and no drops were seen there > > You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() > to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g > -e skb:kfree_skb sleep 20 ; perf report) yes, tomorrow > > diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > index f54ebd5a1702..53130f27dec0 100644 > --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c > +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > @@ -873,7 +873,7 @@ tx_drop_unmap: > } > > tx_drop: > - dev_kfree_skb_any(skb); > + kfree_skb(skb); > priv->stats.tx_dropped++; > return NETDEV_TX_OK; > } > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 13:36 ` Eric Dumazet 2013-12-07 21:20 ` Or Gerlitz @ 2013-12-08 12:09 ` Or Gerlitz 1 sibling, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-08 12:09 UTC (permalink / raw) To: Eric Dumazet, Or Gerlitz Cc: Alexei Starovoitov, David Miller, Joseph Gasparakis, Jerry Chu, Eric Dumazet, Pravin B Shelar, netdev@vger.kernel.org On 06/12/2013 15:36, Eric Dumazet wrote: > On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote: >> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >>> skb->dev is in the first cache line, where we access skb->next anyway. >>> I could use skb->cb[] like the following patch : >> Hi Eric, I applied on the net tree the patch you posted yesterday >> "net: introduce dev_consume_skb_any()" along with the network drivers >> part of this patch, unless I got it wrong, I assume both pieces are >> needed? >> >> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX. >> >> With the patches applied I have almost no samples of that event >> >> $ ./perf report -i perf.data > How did you get this perf.data file ? There are a few drops. > >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action >> + 28.09% swapper [kernel.kallsyms] [k] net_tx_action >> + 28.09% sshd [kernel.kallsyms] [k] net_tx_action >> + 2.25% swapper [kernel.kallsyms] [k] kfree_skb >> + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action >> + 1.12% iperf [kernel.kallsyms] [k] net_tx_action >> >> ./perf report -i perf.data --sort dso,symbol >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 97.75% [kernel.kallsyms] [k] net_tx_action >> + 2.25% [kernel.kallsyms] [k] kfree_skb >> -- > OK, this means your driver drops few packets in its ndo_start_xmit() handler. I wasn't sure to follow how the above lead you to think the drops occur at the driver -- but, anyway, I applied your other patches and the one below which made the mlx4 driver to make a kfree_skb(skb) call - and don't see mlx4 hits in either the client or server side client side: Samples: 133 of event 'skb:kfree_skb', Event count (approx.): 133 + 40.60% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action + 25.56% iperf [kernel.kallsyms] [k] net_tx_action + 24.06% swapper [kernel.kallsyms] [k] net_tx_action + 3.01% iperf [kernel.kallsyms] [k] kfree_skb + 2.26% swapper [kernel.kallsyms] [k] kfree_skb + 2.26% kworker/2:1 [kernel.kallsyms] [k] net_tx_action + 0.75% rcuc/2 [kernel.kallsyms] [k] net_tx_action + 0.75% kworker/2:2 [kernel.kallsyms] [k] net_tx_action + 0.75% ypbind [kernel.kallsyms] [k] net_tx_action server side: Samples: 57 of event 'skb:kfree_skb', Event count (approx.): 57 + 47.37% swapper [kernel.kallsyms] [k] kfree_skb + 22.81% iperf [kernel.kallsyms] [k] kfree_skb + 8.77% ksoftirqd/2 [kernel.kallsyms] [k] kfree_skb + 7.02% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb + 7.02% ls [kernel.kallsyms] [k] kfree_skb + 3.51% umount.nfs [kernel.kallsyms] [k] kfree_skb + 1.75% udevd [kernel.kallsyms] [k] kfree_skb + 1.75% rpc.idmapd [kernel.kallsyms] [k] kfree_skb I will provide you OOB the full perf report files, but I have scrolled into the hits and didn't see one in mlx4... any idea where/how to take this from here? > > Could you give us "ifconfig -a" reports as I already asked ? sure, I see on both sides there are some drops on 1Gbs NIC which is not part of the test client side (mlx4 NIC is eth6) r-dcs44-005 perf]# ifconfig -a br1 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44 inet addr:192.168.52.144 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::c0c0:45ff:feff:bfed/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:11 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:732 (732.0 b) TX bytes:648 (648.0 b) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4B:05 inet addr:10.212.75.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4b05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:70485 errors:0 dropped:467 overruns:0 frame:0 TX packets:35821 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:58838076 (56.1 MiB) TX bytes:20632115 (19.6 MiB) eth6 Link encap:Ethernet HWaddr 00:02:C9:E9:BB:B2 inet addr:192.168.30.144 Bcast:192.168.30.255 Mask:255.255.255.0 inet6 addr: fe80::2:c900:1e9:bbb2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:58503788 errors:0 dropped:0 overruns:0 frame:0 TX packets:389819285 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6604815254 (6.1 GiB) TX bytes:589822872168 (549.3 GiB) eth7 Link encap:Ethernet HWaddr 52:54:00:86:B6:48 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:137 errors:0 dropped:0 overruns:0 frame:0 TX packets:137 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:13860 (13.5 KiB) TX bytes:13860 (13.5 KiB) veth0 Link encap:Ethernet HWaddr E6:95:68:49:A6:3D inet addr:192.168.62.144 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::e495:68ff:fe49:a63d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:58510472 errors:0 dropped:0 overruns:0 frame:0 TX packets:55440581 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3680050172 (3.4 GiB) TX bytes:552349060836 (514.4 GiB) veth1 Link encap:Ethernet HWaddr 5A:4D:A3:4B:B1:97 inet6 addr: fe80::584d:a3ff:fe4b:b197/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:55440581 errors:0 dropped:0 overruns:0 frame:0 TX packets:58510475 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:552349060836 (514.4 GiB) TX bytes:3680050334 (3.4 GiB) vxlan42 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44 inet addr:192.168.42.144 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::c08:ccff:fe54:7844/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:58510461 errors:0 dropped:0 overruns:0 frame:0 TX packets:55440599 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2860902728 (2.6 GiB) TX bytes:553236159764 (515.2 GiB) server side (mlx4 NIC is eth2) r-dcs47-005 perf]# ifconfig -a br1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB inet addr:192.168.52.147 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::cca:f9ff:fead:4210/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:33 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2172 (2.1 KiB) TX bytes:648 (648.0 b) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4A:05 inet addr:10.212.74.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4a05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15274 errors:0 dropped:18 overruns:0 frame:0 TX packets:9190 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:36265311 (34.5 MiB) TX bytes:5736966 (5.4 MiB) eth2 Link encap:Ethernet HWaddr 00:02:C9:E9:C0:82 inet addr:192.168.30.147 Bcast:192.168.30.255 Mask:255.255.255.0 inet6 addr: fe80::2:c900:1e9:c082/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:442423939 errors:0 dropped:0 overruns:0 frame:0 TX packets:66524628 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:669425541728 (623.4 GiB) TX bytes:7511309282 (6.9 GiB) eth3 Link encap:Ethernet HWaddr 52:54:00:5D:70:D9 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:32 errors:0 dropped:0 overruns:0 frame:0 TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2196 (2.1 KiB) TX bytes:2196 (2.1 KiB) veth0 Link encap:Ethernet HWaddr 56:3D:34:30:86:68 inet addr:192.168.62.147 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::543d:34ff:fe30:8668/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:442491685 errors:0 dropped:0 overruns:0 frame:0 TX packets:66538057 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:647403574534 (602.9 GiB) TX bytes:4185942998 (3.8 GiB) veth1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB inet6 addr: fe80::289b:c5ff:fe5f:faab/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:66538066 errors:0 dropped:0 overruns:0 frame:0 TX packets:442491738 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4185943484 (3.8 GiB) TX bytes:647403652126 (602.9 GiB) vxlan42 Link encap:Ethernet HWaddr F6:E2:99:BD:D6:58 inet addr:192.168.42.147 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::f4e2:99ff:febd:d658/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:442491767 errors:0 dropped:0 overruns:0 frame:0 TX packets:66538082 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:641208829864 (597.1 GiB) TX bytes:5250554092 (4.8 GiB) > > You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() > to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g > -e skb:kfree_skb sleep 20 ; perf report) > > diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > index f54ebd5a1702..53130f27dec0 100644 > --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c > +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > @@ -873,7 +873,7 @@ tx_drop_unmap: > } > > tx_drop: > - dev_kfree_skb_any(skb); > + kfree_skb(skb); > priv->stats.tx_dropped++; > return NETDEV_TX_OK; > } > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:59 ` Eric Dumazet 2013-12-04 0:26 ` Alexei Starovoitov @ 2013-12-04 6:39 ` David Miller 2013-12-04 17:40 ` Eric Dumazet 2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet 2 siblings, 1 reply; 63+ messages in thread From: David Miller @ 2013-12-04 6:39 UTC (permalink / raw) To: eric.dumazet Cc: or.gerlitz, joseph.gasparakis, hkchu, ogerlitz, edumazet, ast, pshelar, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 03 Dec 2013 15:59:59 -0800 > On Wed, 2013-12-04 at 01:10 +0200, Or Gerlitz wrote: > >> Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406 >> + 97.13% swapper [kernel.kallsyms] [k] net_tx_action >> + 1.53% iperf [kernel.kallsyms] [k] net_tx_action >> + 1.03% perf [kernel.kallsyms] [k] net_tx_action >> + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action >> + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action >> + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action >> + 0.00% swapper [kernel.kallsyms] [k] kfree_skb >> + 0.00% sleep [kernel.kallsyms] [k] net_tx_action >> + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb >> + 0.00% iperf [kernel.kallsyms] [k] kfree_skb >> + 0.00% perf [kernel.kallsyms] [k] kfree_skb >> > > Right, I actually have a patch for that, but was waiting for net-next > being re-opened : > > commit 9a731d750dd8bf0b8c20fb1ca53c42317fb4dd37 > Author: Eric Dumazet <edumazet@google.com> > Date: Mon Nov 25 13:09:20 2013 -0800 > > net-fixes: introduce dev_consume_skb_any() I definitely prefer the control block approach to this. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 6:39 ` David Miller @ 2013-12-04 17:40 ` Eric Dumazet 0 siblings, 0 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-04 17:40 UTC (permalink / raw) To: David Miller Cc: or.gerlitz, joseph.gasparakis, hkchu, ogerlitz, edumazet, ast, pshelar, netdev On Wed, 2013-12-04 at 01:39 -0500, David Miller wrote: > I definitely prefer the control block approach to this. I polished the patch to keep this knowledge in net/core/dev.c ^ permalink raw reply [flat|nested] 63+ messages in thread
* [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-03 23:59 ` Eric Dumazet 2013-12-04 0:26 ` Alexei Starovoitov 2013-12-04 6:39 ` David Miller @ 2013-12-05 12:45 ` Eric Dumazet 2013-12-05 14:13 ` Hannes Frederic Sowa 2013-12-06 20:24 ` David Miller 2 siblings, 2 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-05 12:45 UTC (permalink / raw) To: David Miller; +Cc: netdev From: Eric Dumazet <edumazet@google.com> Some network drivers use dev_kfree_skb_any() and dev_kfree_skb_irq() helpers to free skbs, both for dropped packets and TX completed ones. We need to separate the two causes to get better diagnostics given by dropwatch or "perf record -e skb:kfree_skb" This patch provides two new helpers, dev_consume_skb_any() and dev_consume_skb_irq() to be used for consumed skbs. __dev_kfree_skb_irq() is slightly optimized to remove one atomic_dec_and_test() in fast path, and use this_cpu_{r|w} accessors. Signed-off-by: Eric Dumazet <edumazet@google.com> --- include/linux/netdevice.h | 53 +++++++++++++++++++++++++++++------- net/core/dev.c | 45 ++++++++++++++++++++---------- 2 files changed, 74 insertions(+), 24 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 7f0ed423a360..c6d64d20050c 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2368,17 +2368,52 @@ static inline int netif_copy_real_num_queues(struct net_device *to_dev, #define DEFAULT_MAX_NUM_RSS_QUEUES (8) int netif_get_num_default_rss_queues(void); -/* Use this variant when it is known for sure that it - * is executing from hardware interrupt context or with hardware interrupts - * disabled. - */ -void dev_kfree_skb_irq(struct sk_buff *skb); +enum skb_free_reason { + SKB_REASON_CONSUMED, + SKB_REASON_DROPPED, +}; + +void __dev_kfree_skb_irq(struct sk_buff *skb, enum skb_free_reason reason); +void __dev_kfree_skb_any(struct sk_buff *skb, enum skb_free_reason reason); -/* Use this variant in places where it could be invoked - * from either hardware interrupt or other context, with hardware interrupts - * either disabled or enabled. +/* + * It is not allowed to call kfree_skb() or consume_skb() from hardware + * interrupt context or with hardware interrupts being disabled. + * (in_irq() || irqs_disabled()) + * + * We provide four helpers that can be used in following contexts : + * + * dev_kfree_skb_irq(skb) when caller drops a packet from irq context, + * replacing kfree_skb(skb) + * + * dev_consume_skb_irq(skb) when caller consumes a packet from irq context. + * Typically used in place of consume_skb(skb) in TX completion path + * + * dev_kfree_skb_any(skb) when caller doesn't know its current irq context, + * replacing kfree_skb(skb) + * + * dev_consume_skb_any(skb) when caller doesn't know its current irq context, + * and consumed a packet. Used in place of consume_skb(skb) */ -void dev_kfree_skb_any(struct sk_buff *skb); +static inline void dev_kfree_skb_irq(struct sk_buff *skb) +{ + __dev_kfree_skb_irq(skb, SKB_REASON_DROPPED); +} + +static inline void dev_consume_skb_irq(struct sk_buff *skb) +{ + __dev_kfree_skb_irq(skb, SKB_REASON_CONSUMED); +} + +static inline void dev_kfree_skb_any(struct sk_buff *skb) +{ + __dev_kfree_skb_any(skb, SKB_REASON_DROPPED); +} + +static inline void dev_consume_skb_any(struct sk_buff *skb) +{ + __dev_kfree_skb_any(skb, SKB_REASON_CONSUMED); +} int netif_rx(struct sk_buff *skb); int netif_rx_ni(struct sk_buff *skb); diff --git a/net/core/dev.c b/net/core/dev.c index ba3b7ea5ebb3..aa54a742f392 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -2145,30 +2145,42 @@ void __netif_schedule(struct Qdisc *q) } EXPORT_SYMBOL(__netif_schedule); -void dev_kfree_skb_irq(struct sk_buff *skb) +struct dev_kfree_skb_cb { + enum skb_free_reason reason; +}; + +static struct dev_kfree_skb_cb *get_kfree_skb_cb(const struct sk_buff *skb) +{ + return (struct dev_kfree_skb_cb *)skb->cb; +} + +void __dev_kfree_skb_irq(struct sk_buff *skb, enum skb_free_reason reason) { - if (atomic_dec_and_test(&skb->users)) { - struct softnet_data *sd; - unsigned long flags; + unsigned long flags; - local_irq_save(flags); - sd = &__get_cpu_var(softnet_data); - skb->next = sd->completion_queue; - sd->completion_queue = skb; - raise_softirq_irqoff(NET_TX_SOFTIRQ); - local_irq_restore(flags); + if (likely(atomic_read(&skb->users) == 1)) { + smp_rmb(); + atomic_set(&skb->users, 0); + } else if (likely(!atomic_dec_and_test(&skb->users))) { + return; } + get_kfree_skb_cb(skb)->reason = reason; + local_irq_save(flags); + skb->next = __this_cpu_read(softnet_data.completion_queue); + __this_cpu_write(softnet_data.completion_queue, skb); + raise_softirq_irqoff(NET_TX_SOFTIRQ); + local_irq_restore(flags); } -EXPORT_SYMBOL(dev_kfree_skb_irq); +EXPORT_SYMBOL(__dev_kfree_skb_irq); -void dev_kfree_skb_any(struct sk_buff *skb) +void __dev_kfree_skb_any(struct sk_buff *skb, enum skb_free_reason reason) { if (in_irq() || irqs_disabled()) - dev_kfree_skb_irq(skb); + __dev_kfree_skb_irq(skb, reason); else dev_kfree_skb(skb); } -EXPORT_SYMBOL(dev_kfree_skb_any); +EXPORT_SYMBOL(__dev_kfree_skb_any); /** @@ -3306,7 +3318,10 @@ static void net_tx_action(struct softirq_action *h) clist = clist->next; WARN_ON(atomic_read(&skb->users)); - trace_kfree_skb(skb, net_tx_action); + if (likely(get_kfree_skb_cb(skb)->reason == SKB_REASON_CONSUMED)) + trace_consume_skb(skb); + else + trace_kfree_skb(skb, net_tx_action); __kfree_skb(skb); } } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet @ 2013-12-05 14:13 ` Hannes Frederic Sowa 2013-12-05 14:45 ` Eric Dumazet 2013-12-06 20:24 ` David Miller 1 sibling, 1 reply; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-05 14:13 UTC (permalink / raw) To: Eric Dumazet; +Cc: David Miller, netdev On Thu, Dec 05, 2013 at 04:45:08AM -0800, Eric Dumazet wrote: > - local_irq_save(flags); > - sd = &__get_cpu_var(softnet_data); > - skb->next = sd->completion_queue; > - sd->completion_queue = skb; > - raise_softirq_irqoff(NET_TX_SOFTIRQ); > - local_irq_restore(flags); > + if (likely(atomic_read(&skb->users) == 1)) { > + smp_rmb(); Could you give me a hint why this barrier is needed? IMHO the volatile access in atomic_read should get rid of the control dependency so I don't see a need for this barrier. Without the volatile access a compiler-barrier would still suffice, I guess? > + atomic_set(&skb->users, 0); > + } else if (likely(!atomic_dec_and_test(&skb->users))) { > + return; Or does this memory barrier deal with the part below this return? Thanks, Hannes ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 14:13 ` Hannes Frederic Sowa @ 2013-12-05 14:45 ` Eric Dumazet 2013-12-05 15:05 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-05 14:45 UTC (permalink / raw) To: Hannes Frederic Sowa; +Cc: David Miller, netdev On Thu, 2013-12-05 at 15:13 +0100, Hannes Frederic Sowa wrote: > On Thu, Dec 05, 2013 at 04:45:08AM -0800, Eric Dumazet wrote: > > - local_irq_save(flags); > > - sd = &__get_cpu_var(softnet_data); > > - skb->next = sd->completion_queue; > > - sd->completion_queue = skb; > > - raise_softirq_irqoff(NET_TX_SOFTIRQ); > > - local_irq_restore(flags); > > + if (likely(atomic_read(&skb->users) == 1)) { > > + smp_rmb(); > > Could you give me a hint why this barrier is needed? IMHO the volatile > access in atomic_read should get rid of the control dependency so I > don't see a need for this barrier. Without the volatile access a > compiler-barrier would still suffice, I guess? Please take a look at kfree_skb() implementation. If you think a comment is needed there, please feel free to add it. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 14:45 ` Eric Dumazet @ 2013-12-05 15:05 ` Eric Dumazet 2013-12-05 15:44 ` Hannes Frederic Sowa 0 siblings, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-05 15:05 UTC (permalink / raw) To: Hannes Frederic Sowa; +Cc: David Miller, netdev On Thu, 2013-12-05 at 06:45 -0800, Eric Dumazet wrote: > On Thu, 2013-12-05 at 15:13 +0100, Hannes Frederic Sowa wrote: > > On Thu, Dec 05, 2013 at 04:45:08AM -0800, Eric Dumazet wrote: > > > - local_irq_save(flags); > > > - sd = &__get_cpu_var(softnet_data); > > > - skb->next = sd->completion_queue; > > > - sd->completion_queue = skb; > > > - raise_softirq_irqoff(NET_TX_SOFTIRQ); > > > - local_irq_restore(flags); > > > + if (likely(atomic_read(&skb->users) == 1)) { > > > + smp_rmb(); > > > > Could you give me a hint why this barrier is needed? IMHO the volatile > > access in atomic_read should get rid of the control dependency so I > > don't see a need for this barrier. Without the volatile access a > > compiler-barrier would still suffice, I guess? > > Please take a look at kfree_skb() implementation. > > If you think a comment is needed there, please feel free to add it. > My understanding of this (old) barrier here is an implicit wmb in skb_get() This probably needs something like : static inline struct sk_buff *skb_get(struct sk_buff *skb) { smp_mb__before_atomic_inc(); /* check {consume|kfree}_skb() */ atomic_inc(&skb->users); } ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 15:05 ` Eric Dumazet @ 2013-12-05 15:44 ` Hannes Frederic Sowa 2013-12-05 16:38 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-05 15:44 UTC (permalink / raw) To: Eric Dumazet; +Cc: David Miller, netdev On Thu, Dec 05, 2013 at 07:05:52AM -0800, Eric Dumazet wrote: > On Thu, 2013-12-05 at 06:45 -0800, Eric Dumazet wrote: > > On Thu, 2013-12-05 at 15:13 +0100, Hannes Frederic Sowa wrote: > > > On Thu, Dec 05, 2013 at 04:45:08AM -0800, Eric Dumazet wrote: > > > > - local_irq_save(flags); > > > > - sd = &__get_cpu_var(softnet_data); > > > > - skb->next = sd->completion_queue; > > > > - sd->completion_queue = skb; > > > > - raise_softirq_irqoff(NET_TX_SOFTIRQ); > > > > - local_irq_restore(flags); > > > > + if (likely(atomic_read(&skb->users) == 1)) { > > > > + smp_rmb(); > > > > > > Could you give me a hint why this barrier is needed? IMHO the volatile > > > access in atomic_read should get rid of the control dependency so I > > > don't see a need for this barrier. Without the volatile access a > > > compiler-barrier would still suffice, I guess? > > > > Please take a look at kfree_skb() implementation. > > > > If you think a comment is needed there, please feel free to add it. > > > > My understanding of this (old) barrier here is an implicit wmb in > skb_get() > > This probably needs something like : > > static inline struct sk_buff *skb_get(struct sk_buff *skb) > { > smp_mb__before_atomic_inc(); /* check {consume|kfree}_skb() */ > atomic_inc(&skb->users); > } Thanks for the pointer to kfree_skb. I found this commit which added the barrier in kfree_skb (from history.git): commit 09d3e84de438f217510b604a980befd07b0c8262 Author: Herbert Xu <herbert@gondor.apana.org.au> Date: Sat Feb 5 03:23:27 2005 -0800 [NET]: Add missing memory barrier to kfree_skb(). Also kill kfree_skb_fast(), that is a relic from fast switching which was killed off years ago. The bug is that in the case where we do the atomic_read() optimization, we need to make sure that reads of skb state later in __kfree_skb() processing (particularly the skb->list BUG check) are not reordered to occur before the counter read by the cpu. Thanks to Olaf Kirch and Anton Blanchard for discovering and helping fix this bug. Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: David S. Miller <davem@davemloft.net> It makes some sense but I did not grasp the whole ->users dependency picture, yet. I guess the barrier is only needed when refcount drops down to 0 and we don't necessarily need one when incrementing ->users. Thank you, Hannes ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 15:44 ` Hannes Frederic Sowa @ 2013-12-05 16:38 ` Eric Dumazet 2013-12-05 16:54 ` Hannes Frederic Sowa 0 siblings, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-05 16:38 UTC (permalink / raw) To: Hannes Frederic Sowa; +Cc: David Miller, netdev On Thu, 2013-12-05 at 16:44 +0100, Hannes Frederic Sowa wrote: > It makes some sense but I did not grasp the whole ->users dependency > picture, yet. I guess the barrier is only needed when refcount drops > down to 0 and we don't necessarily need one when incrementing ->users. If you are the only user of this skb, really no smp barrier is needed at all. The problem comes when another cpu is working on the skb, and finally releases its reference on it. Before releasing its reference, it must commit all changes it might have done onto skb. Otherwise another cpu might read stale data. The smp_wmb() is done by the atomic_dec_and_test(), as it contains a full barrier. So the smp_rmb() pairs with the barrier done in atomic_dec_and_test() ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 16:38 ` Eric Dumazet @ 2013-12-05 16:54 ` Hannes Frederic Sowa 0 siblings, 0 replies; 63+ messages in thread From: Hannes Frederic Sowa @ 2013-12-05 16:54 UTC (permalink / raw) To: Eric Dumazet; +Cc: David Miller, netdev On Thu, Dec 05, 2013 at 08:38:05AM -0800, Eric Dumazet wrote: > On Thu, 2013-12-05 at 16:44 +0100, Hannes Frederic Sowa wrote: > > > It makes some sense but I did not grasp the whole ->users dependency > > picture, yet. I guess the barrier is only needed when refcount drops > > down to 0 and we don't necessarily need one when incrementing ->users. > > If you are the only user of this skb, really no smp barrier is needed at > all. > > The problem comes when another cpu is working on the skb, and finally > releases its reference on it. > > Before releasing its reference, it must commit all changes it might have > done onto skb. Otherwise another cpu might read stale data. > > The smp_wmb() is done by the atomic_dec_and_test(), as it contains a > full barrier. > > So the smp_rmb() pairs with the barrier done in atomic_dec_and_test() Ha, it all makes sense now! Thanks, Eric! (Sorry for the noise but I find this kind of problems very interesting) ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH net-next] net: introduce dev_consume_skb_any() 2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet 2013-12-05 14:13 ` Hannes Frederic Sowa @ 2013-12-06 20:24 ` David Miller 1 sibling, 0 replies; 63+ messages in thread From: David Miller @ 2013-12-06 20:24 UTC (permalink / raw) To: eric.dumazet; +Cc: netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Thu, 05 Dec 2013 04:45:08 -0800 > From: Eric Dumazet <edumazet@google.com> > > Some network drivers use dev_kfree_skb_any() and dev_kfree_skb_irq() > helpers to free skbs, both for dropped packets and TX completed ones. > > We need to separate the two causes to get better diagnostics > given by dropwatch or "perf record -e skb:kfree_skb" > > This patch provides two new helpers, dev_consume_skb_any() and > dev_consume_skb_irq() to be used for consumed skbs. > > __dev_kfree_skb_irq() is slightly optimized to remove one > atomic_dec_and_test() in fast path, and use this_cpu_{r|w} accessors. > > Signed-off-by: Eric Dumazet <edumazet@google.com> Applied, thanks Eric. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 21:09 ` Or Gerlitz 2013-12-03 21:24 ` Eric Dumazet @ 2013-12-03 23:13 ` Joseph Gasparakis 2013-12-03 23:09 ` Or Gerlitz 1 sibling, 1 reply; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-03 23:13 UTC (permalink / raw) To: Or Gerlitz Cc: Gasparakis, Joseph, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 3 Dec 2013, Or Gerlitz wrote: > On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > > >>> lack of GRO : receiver seems to not be able to receive as fast as you want. > >>>> TCPOFOQueue: 3167879 > >>> So many packets are received out of order (because of losses) > > >> I see that there's no GRO also for the non-veth tests which involve > >> vxlan, and over there the receiving side is capable to consume the > >> packets, do you have rough explaination why adding veth to the chain > >> is such game changer which makes things to start falling out? > > > I have seen this before. Here are my findings: > > > > The gso_type is different if the skb comes from veth or not. From veth, > > you will see the SKB_GSO_DODGY set. This breaks things as when the > > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, > > the stack drops it silently. I never got the time to find the root cause > > for this, but I know it causes re-transmissions and big performance > > degregation. > > > > I went as far as just quickly hacking a one liner unsetting the DODGY bit > > in vxlan.c and that bypassed the issue and recovered the performance > > problem, but obviously this is not a real fix. > > thanks for the heads up, few quick questions/clafications -- > > -- you are talking on drops done @ the sender side, correct? Eric was > saying we have evidences that the drops happen on the receiver. I am *guessing* drops on the Rx are due to the drops at the Tx. See my answer to your next question for more info. > > -- without the hack you did, still packets are sent/received, so what > makes the stack to drop only some of them? > What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs never made it to the driver, they were broken into non GSO smaller skbs by the stack. I think the stack is not handling well the GSO with the DODGY bit set, and that causes it to maybe partially the packet to be emitted, causing the re-transmits (and maybe the drops on your Rx end)? Of course all this is speculation, the fact that I know is that as soon as I was forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds. > -- why packets coming from veth would have the SKB_GSO_DODGY bit set? That is something I would love to know too. I am guessing this is a way for the VM to say it is a non-trusted packet? And maybe all this can be fixed by maybe setting something on the VM through a userspace tool that will stop the veth to set the DODGY bit? > > -- so where is now (say net.git or 3.12.x) this one line you commented > out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c > explicit setting of SKB_GSO_DODGY I did not commit it, as this was just a workaround to prove to myself that the problem I was seing was due to the gso_type, and it would actually just hide the problem and not give a proper solution to it. > > Also, I am pretty sure the problem exists also when sending/receiving > guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I > just sticked to the veth flavour b/c its one (== the hypervisor) > network stack to debug and not two (+ the guest one). > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:13 ` vxlan/veth performance issues on net.git + latest kernels Joseph Gasparakis @ 2013-12-03 23:09 ` Or Gerlitz 2013-12-04 0:35 ` Joseph Gasparakis 0 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 23:09 UTC (permalink / raw) To: Joseph Gasparakis Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Wed, Dec 4, 2013 at 1:13 AM, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > > > On Tue, 3 Dec 2013, Or Gerlitz wrote: > >> On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis >> <joseph.gasparakis@intel.com> wrote: >> >> >>> lack of GRO : receiver seems to not be able to receive as fast as you want. >> >>>> TCPOFOQueue: 3167879 >> >>> So many packets are received out of order (because of losses) >> >> >> I see that there's no GRO also for the non-veth tests which involve >> >> vxlan, and over there the receiving side is capable to consume the >> >> packets, do you have rough explaination why adding veth to the chain >> >> is such game changer which makes things to start falling out? >> >> > I have seen this before. Here are my findings: >> > >> > The gso_type is different if the skb comes from veth or not. From veth, >> > you will see the SKB_GSO_DODGY set. This breaks things as when the >> > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, >> > the stack drops it silently. I never got the time to find the root cause >> > for this, but I know it causes re-transmissions and big performance >> > degregation. >> > >> > I went as far as just quickly hacking a one liner unsetting the DODGY bit >> > in vxlan.c and that bypassed the issue and recovered the performance >> > problem, but obviously this is not a real fix. >> >> thanks for the heads up, few quick questions/clafications -- >> >> -- you are talking on drops done @ the sender side, correct? Eric was >> saying we have evidences that the drops happen on the receiver. > > I am *guessing* drops on the Rx are due to the drops at the Tx. See my > answer to your next question for more info. > >> >> -- without the hack you did, still packets are sent/received, so what >> makes the stack to drop only some of them? >> > > What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs > never made it to the driver, they were broken into non GSO smaller skbs by > the stack. I think the stack is not handling well the GSO with the DODGY > bit set, and that causes it to maybe partially the packet to be emitted, > causing the re-transmits (and maybe the drops on your Rx end)? Of course > all this is speculation, the fact that I know is that as soon as I was > forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds. > >> -- why packets coming from veth would have the SKB_GSO_DODGY bit set? > > That is something I would love to know too. I am guessing this is a way > for the VM to say it is a non-trusted packet? And maybe all this can be > fixed by maybe setting something on the VM through a userspace tool that > will stop the veth to set the DODGY bit? > >> >> -- so where is now (say net.git or 3.12.x) this one line you commented >> out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c >> explicit setting of SKB_GSO_DODGY > > I did not commit it, as this was just a workaround to prove to myself that > the problem I was seing was due to the gso_type, and it would actually > just hide the problem and not give a proper solution to it. > >> >> Also, I am pretty sure the problem exists also when sending/receiving >> guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I >> just sticked to the veth flavour b/c its one (== the hypervisor) >> network stack to debug and not two (+ the guest one). understood, can you point the line/area you hacked, I'd like to try it too and see the impact >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 23:09 ` Or Gerlitz @ 2013-12-04 0:35 ` Joseph Gasparakis 2013-12-04 0:34 ` Alexei Starovoitov ` (2 more replies) 0 siblings, 3 replies; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-04 0:35 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 3 Dec 2013, Or Gerlitz wrote: > On Wed, Dec 4, 2013 at 1:13 AM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > > > > > > On Tue, 3 Dec 2013, Or Gerlitz wrote: > > > >> On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis > >> <joseph.gasparakis@intel.com> wrote: > >> > >> >>> lack of GRO : receiver seems to not be able to receive as fast as you want. > >> >>>> TCPOFOQueue: 3167879 > >> >>> So many packets are received out of order (because of losses) > >> > >> >> I see that there's no GRO also for the non-veth tests which involve > >> >> vxlan, and over there the receiving side is capable to consume the > >> >> packets, do you have rough explaination why adding veth to the chain > >> >> is such game changer which makes things to start falling out? > >> > >> > I have seen this before. Here are my findings: > >> > > >> > The gso_type is different if the skb comes from veth or not. From veth, > >> > you will see the SKB_GSO_DODGY set. This breaks things as when the > >> > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, > >> > the stack drops it silently. I never got the time to find the root cause > >> > for this, but I know it causes re-transmissions and big performance > >> > degregation. > >> > > >> > I went as far as just quickly hacking a one liner unsetting the DODGY bit > >> > in vxlan.c and that bypassed the issue and recovered the performance > >> > problem, but obviously this is not a real fix. > >> > >> thanks for the heads up, few quick questions/clafications -- > >> > >> -- you are talking on drops done @ the sender side, correct? Eric was > >> saying we have evidences that the drops happen on the receiver. > > > > I am *guessing* drops on the Rx are due to the drops at the Tx. See my > > answer to your next question for more info. > > > >> > >> -- without the hack you did, still packets are sent/received, so what > >> makes the stack to drop only some of them? > >> > > > > What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs > > never made it to the driver, they were broken into non GSO smaller skbs by > > the stack. I think the stack is not handling well the GSO with the DODGY > > bit set, and that causes it to maybe partially the packet to be emitted, > > causing the re-transmits (and maybe the drops on your Rx end)? Of course > > all this is speculation, the fact that I know is that as soon as I was > > forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds. > > > >> -- why packets coming from veth would have the SKB_GSO_DODGY bit set? > > > > That is something I would love to know too. I am guessing this is a way > > for the VM to say it is a non-trusted packet? And maybe all this can be > > fixed by maybe setting something on the VM through a userspace tool that > > will stop the veth to set the DODGY bit? > > > >> > >> -- so where is now (say net.git or 3.12.x) this one line you commented > >> out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c > >> explicit setting of SKB_GSO_DODGY > > > > I did not commit it, as this was just a workaround to prove to myself that > > the problem I was seing was due to the gso_type, and it would actually > > just hide the problem and not give a proper solution to it. > > > >> > >> Also, I am pretty sure the problem exists also when sending/receiving > >> guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I > >> just sticked to the veth flavour b/c its one (== the hypervisor) > >> network stack to debug and not two (+ the guest one). > > understood, can you point the line/area you hacked, I'd like to try it > too and see the impact I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked. Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me. I would be curious to see if this is the same issue as mine. It seems like it is. > > >> -- > >> To unsubscribe from this list: send the line "unsubscribe netdev" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:35 ` Joseph Gasparakis @ 2013-12-04 0:34 ` Alexei Starovoitov 2013-12-04 1:29 ` Joseph Gasparakis 2013-12-04 0:44 ` Joseph Gasparakis 2013-12-04 8:35 ` Or Gerlitz 2 siblings, 1 reply; 63+ messages in thread From: Alexei Starovoitov @ 2013-12-04 0:34 UTC (permalink / raw) To: Joseph Gasparakis Cc: Or Gerlitz, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, David Miller, netdev On Tue, Dec 3, 2013 at 4:35 PM, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > > I was printing the gso_type in vxlan_xmit_skb(), right before > iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the > gso_type was different when a VM was involved and when it was not > (although I was transmitting exactly the same packet), and then I replaced > my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had > for non-VM skb> and it all worked. > > Then I looked into what was different between the two gso_types and the > only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. > I am sure I could have been more delicate with the aproach, but hey, it > worked for me. hmm. dodgy should be a normal path from vm. kvm suppose to negotiate vnet_hdr for tap/macvtap and corresponding driver will be remapping virtio_net_gso* flags into skb_gso_* flags plus gso_dodgy. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:34 ` Alexei Starovoitov @ 2013-12-04 1:29 ` Joseph Gasparakis 2013-12-04 1:18 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-04 1:29 UTC (permalink / raw) To: Alexei Starovoitov Cc: Joseph Gasparakis, Or Gerlitz, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, David Miller, netdev On Tue, 3 Dec 2013, Alexei Starovoitov wrote: > On Tue, Dec 3, 2013 at 4:35 PM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > > > > I was printing the gso_type in vxlan_xmit_skb(), right before > > iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the > > gso_type was different when a VM was involved and when it was not > > (although I was transmitting exactly the same packet), and then I replaced > > my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had > > for non-VM skb> and it all worked. > > > > Then I looked into what was different between the two gso_types and the > > only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. > > I am sure I could have been more delicate with the aproach, but hey, it > > worked for me. > > hmm. dodgy should be a normal path from vm. > kvm suppose to negotiate vnet_hdr for tap/macvtap and corresponding > driver will be remapping virtio_net_gso* flags into skb_gso_* flags > plus gso_dodgy. > Then would the fix be as simple as just unsetting the bit in vxlan? Because I am guessing that there is a bug somewhere where the combination of GSO_UDP_TUNNEL and GSO_DODGY. That would solve my problem, hopefully Or's too if it is the same. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 1:29 ` Joseph Gasparakis @ 2013-12-04 1:18 ` Eric Dumazet 0 siblings, 0 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-04 1:18 UTC (permalink / raw) To: Joseph Gasparakis Cc: Alexei Starovoitov, Or Gerlitz, Jerry Chu, Or Gerlitz, Eric Dumazet, Pravin B Shelar, David Miller, netdev On Tue, 2013-12-03 at 17:29 -0800, Joseph Gasparakis wrote: > > Then would the fix be as simple as just unsetting the bit in vxlan? > Because I am guessing that there is a bug somewhere where the combination of > GSO_UDP_TUNNEL and GSO_DODGY. That would solve my problem, hopefully Or's > too if it is the same. Not really ;) We need to handle this bit properly, not pretending it is of no use ;) ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:35 ` Joseph Gasparakis 2013-12-04 0:34 ` Alexei Starovoitov @ 2013-12-04 0:44 ` Joseph Gasparakis 2013-12-04 8:35 ` Or Gerlitz 2 siblings, 0 replies; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-04 0:44 UTC (permalink / raw) To: Joseph Gasparakis Cc: Or Gerlitz, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 3 Dec 2013, Joseph Gasparakis wrote: > > > On Tue, 3 Dec 2013, Or Gerlitz wrote: > > > On Wed, Dec 4, 2013 at 1:13 AM, Joseph Gasparakis > > <joseph.gasparakis@intel.com> wrote: > > > > > > > > > On Tue, 3 Dec 2013, Or Gerlitz wrote: > > > > > >> On Tue, Dec 3, 2013 at 11:11 PM, Joseph Gasparakis > > >> <joseph.gasparakis@intel.com> wrote: > > >> > > >> >>> lack of GRO : receiver seems to not be able to receive as fast as you want. > > >> >>>> TCPOFOQueue: 3167879 > > >> >>> So many packets are received out of order (because of losses) > > >> > > >> >> I see that there's no GRO also for the non-veth tests which involve > > >> >> vxlan, and over there the receiving side is capable to consume the > > >> >> packets, do you have rough explaination why adding veth to the chain > > >> >> is such game changer which makes things to start falling out? > > >> > > >> > I have seen this before. Here are my findings: > > >> > > > >> > The gso_type is different if the skb comes from veth or not. From veth, > > >> > you will see the SKB_GSO_DODGY set. This breaks things as when the > > >> > skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, > > >> > the stack drops it silently. I never got the time to find the root cause > > >> > for this, but I know it causes re-transmissions and big performance > > >> > degregation. > > >> > > > >> > I went as far as just quickly hacking a one liner unsetting the DODGY bit > > >> > in vxlan.c and that bypassed the issue and recovered the performance > > >> > problem, but obviously this is not a real fix. > > >> > > >> thanks for the heads up, few quick questions/clafications -- > > >> > > >> -- you are talking on drops done @ the sender side, correct? Eric was > > >> saying we have evidences that the drops happen on the receiver. > > > > > > I am *guessing* drops on the Rx are due to the drops at the Tx. See my > > > answer to your next question for more info. > > > > > >> > > >> -- without the hack you did, still packets are sent/received, so what > > >> makes the stack to drop only some of them? > > >> > > > > > > What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs > > > never made it to the driver, they were broken into non GSO smaller skbs by > > > the stack. I think the stack is not handling well the GSO with the DODGY > > > bit set, and that causes it to maybe partially the packet to be emitted, > > > causing the re-transmits (and maybe the drops on your Rx end)? Of course > > > all this is speculation, the fact that I know is that as soon as I was > > > forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds. > > > > > >> -- why packets coming from veth would have the SKB_GSO_DODGY bit set? > > > > > > That is something I would love to know too. I am guessing this is a way > > > for the VM to say it is a non-trusted packet? And maybe all this can be > > > fixed by maybe setting something on the VM through a userspace tool that > > > will stop the veth to set the DODGY bit? > > > > > >> > > >> -- so where is now (say net.git or 3.12.x) this one line you commented > > >> out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c > > >> explicit setting of SKB_GSO_DODGY > > > > > > I did not commit it, as this was just a workaround to prove to myself that > > > the problem I was seing was due to the gso_type, and it would actually > > > just hide the problem and not give a proper solution to it. > > > > > >> > > >> Also, I am pretty sure the problem exists also when sending/receiving > > >> guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I > > >> just sticked to the veth flavour b/c its one (== the hypervisor) > > >> network stack to debug and not two (+ the guest one). > > > > understood, can you point the line/area you hacked, I'd like to try it > > too and see the impact > > I was printing the gso_type in vxlan_xmit_skb(), right before > iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the > gso_type was different when a VM was involved and when it was not > (although I was transmitting exactly the same packet), and then I replaced > my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had > for non-VM skb> and it all worked. > > Then I looked into what was different between the two gso_types and the > only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. > I am sure I could have been more delicate with the aproach, but hey, it > worked for me. > > I would be curious to see if this is the same issue as mine. It seems like > it is. > Oh, and if I remember correctly, gso_type without VMs involved was 129 (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and with VM it was 133 (SKB_GSO_UDP_TUNNEL | SKB_GSO_DODGY | SKB_GSO_TCPV4). > > > > >> -- > > >> To unsubscribe from this list: send the line "unsubscribe netdev" in > > >> the body of a message to majordomo@vger.kernel.org > > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > >> > > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 0:35 ` Joseph Gasparakis 2013-12-04 0:34 ` Alexei Starovoitov 2013-12-04 0:44 ` Joseph Gasparakis @ 2013-12-04 8:35 ` Or Gerlitz 2013-12-04 9:24 ` Joseph Gasparakis 2 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-04 8:35 UTC (permalink / raw) To: Joseph Gasparakis Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Wed, Dec 4, 2013 at 2:35 AM, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > I was printing the gso_type in vxlan_xmit_skb(), right before > iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the > gso_type was different when a VM was involved and when it was not > (although I was transmitting exactly the same packet), and then I replaced > my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had > for non-VM skb> and it all worked. > > Then I looked into what was different between the two gso_types and the > only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. > I am sure I could have been more delicate with the aproach, but hey, it > worked for me. > > I would be curious to see if this is the same issue as mine. It seems like it is. nope! with the latest net tree, after handle_offloads is called in vxlan_xmit_skb and before iptunnel_xmit is invoked, skb_shinfo(skb)->gso_type is either 0 or 0x201 which is (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether the session runs over veth device or directly over the bridge, where over veth and > 1 stream we see drops, bad perf, etc. I am very interested in the VM case too, so will check it out and let you know. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 8:35 ` Or Gerlitz @ 2013-12-04 9:24 ` Joseph Gasparakis 2013-12-04 9:41 ` Or Gerlitz 0 siblings, 1 reply; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-04 9:24 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Wed, 4 Dec 2013, Or Gerlitz wrote: > On Wed, Dec 4, 2013 at 2:35 AM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > > > I was printing the gso_type in vxlan_xmit_skb(), right before > > iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the > > gso_type was different when a VM was involved and when it was not > > (although I was transmitting exactly the same packet), and then I replaced > > my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had > > for non-VM skb> and it all worked. > > > > Then I looked into what was different between the two gso_types and the > > only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. > > I am sure I could have been more delicate with the aproach, but hey, it > > worked for me. > > > > I would be curious to see if this is the same issue as mine. It seems like it is. > > nope! with the latest net tree, after handle_offloads is called in > vxlan_xmit_skb and before iptunnel_xmit is invoked, > skb_shinfo(skb)->gso_type is either 0 or 0x201 which is > (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether > the session runs over veth device or directly over the bridge, where > over veth and > 1 stream we see drops, bad perf, etc. > > I am very interested in the VM case too, so will check it out and let you know. > Ok, I was really hoping that would be the same... And just for the record, you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests before it. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 9:24 ` Joseph Gasparakis @ 2013-12-04 9:41 ` Or Gerlitz 2013-12-04 15:20 ` Or Gerlitz [not found] ` <52A197DF.5010806@mellanox.com> 0 siblings, 2 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-04 9:41 UTC (permalink / raw) To: Joseph Gasparakis Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Wed, Dec 4, 2013 at 11:24 AM, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > On Wed, 4 Dec 2013, Or Gerlitz wrote: >> nope! with the latest net tree, after handle_offloads is called in >> vxlan_xmit_skb and before iptunnel_xmit is invoked, >> skb_shinfo(skb)->gso_type is either 0 or 0x201 which is >> (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether >> the session runs over veth device or directly over the bridge, where >> over veth and > 1 stream we see drops, bad perf, etc. >> I am very interested in the VM case too, so will check it out and let you know. > Ok, I was really hoping that would be the same... So when running traffic from VM I do see SKB_GSO_DODGY bit being set! my environment for running VMs with sane peformance was screwed a bit, I will bring it up later today and see if unsetting the bit helps. > And just for the record, > you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was > seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO > support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests > before it. indeed, also, on what kernel did you conducted your tests which you managed to WA the problem with unsetting that bit? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-04 9:41 ` Or Gerlitz @ 2013-12-04 15:20 ` Or Gerlitz [not found] ` <52A197DF.5010806@mellanox.com> 1 sibling, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-04 15:20 UTC (permalink / raw) To: Joseph Gasparakis Cc: Eric Dumazet, Jerry Chu, Or Gerlitz, Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Wed, Dec 4, 2013 at 11:41 AM, Or Gerlitz <or.gerlitz@gmail.com> wrote: > On Wed, Dec 4, 2013 at 11:24 AM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: >> On Wed, 4 Dec 2013, Or Gerlitz wrote: > >>> nope! with the latest net tree, after handle_offloads is called in >>> vxlan_xmit_skb and before iptunnel_xmit is invoked, >>> skb_shinfo(skb)->gso_type is either 0 or 0x201 which is >>> (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether >>> the session runs over veth device or directly over the bridge, where >>> over veth and > 1 stream we see drops, bad perf, etc. >>> I am very interested in the VM case too, so will check it out and let you know. > >> Ok, I was really hoping that would be the same... > > So when running traffic from VM I do see SKB_GSO_DODGY bit being set! > my environment for running VMs with sane peformance was screwed a bit, I will bring it > up later today and see if unsetting the bit helps. so if the DODGY bit is left set, it simply doesn't work! it seems that guest GSO packets are just dropped As for the performance when the bit is unset in the vxlan driver as you suggested, I need to tune the VMs a bit more, later or tomorrow ^ permalink raw reply [flat|nested] 63+ messages in thread
[parent not found: <52A197DF.5010806@mellanox.com>]
* Re: vxlan/veth performance issues on net.git + latest kernels [not found] ` <52A197DF.5010806@mellanox.com> @ 2013-12-06 9:30 ` Or Gerlitz 2013-12-08 12:43 ` Mike Rapoport 2013-12-06 10:30 ` Joseph Gasparakis 1 sibling, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-06 9:30 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend > On 04/12/2013 11:41, Or Gerlitz wrote: > On Wed, Dec 4, 2013 at 11:24 AM, Joseph Gasparakis > <joseph.gasparakis@intel.com> wrote: > >> And just for the record, >> you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was >> seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO >> support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests >> before it. > > indeed, also, on what kernel did you conducted your tests which you managed > to WA the problem with unsetting that bit? Hi Joseph, Really need your response here -- 1. on which kernel did you manage to get along fine vxlan performance wise with this hack? 2. did the hack helped for both veth host traffic or only on PV VM traffic? Currently it doesn't converge with 3.12.x or net.git, with veth/vxlan the DODGE bit isn't set when looking on the skb in the vxlan xmit time, so there's nothing for me to hack there. For VMs without unsetting the bit things don't really work, but unsetting it for itself so far didn't get me far performance wise. BTW guys, I saw the issues with both bridge/openvswitch configuration - seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year. Or. net.git]# grep -rn SKB_GSO_DODGY drivers/net/ net/ipv4 net/core drivers/net/macvtap.c:585: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; drivers/net/tun.c:1135: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; drivers/net/virtio_net.c:497: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; drivers/net/xen-netback/netback.c:1146: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; drivers/net/xen-netfront.c:823: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; net/ipv4/af_inet.c:1264: SKB_GSO_DODGY | net/ipv4/tcp_offload.c:56: SKB_GSO_DODGY | net/ipv4/gre_offload.c:40: SKB_GSO_DODGY | net/ipv4/udp_offload.c:53: if (unlikely(type & ~(SKB_GSO_UDP | SKB_GSO_DODGY | net/core/dev.c:2694: if (shinfo->gso_type & SKB_GSO_DODGY) ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 9:30 ` Or Gerlitz @ 2013-12-08 12:43 ` Mike Rapoport 2013-12-08 13:07 ` Or Gerlitz 0 siblings, 1 reply; 63+ messages in thread From: Mike Rapoport @ 2013-12-08 12:43 UTC (permalink / raw) To: Or Gerlitz Cc: Or Gerlitz, Joseph Gasparakis, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Fri, Dec 06, 2013 at 11:30:37AM +0200, Or Gerlitz wrote: > > On 04/12/2013 11:41, Or Gerlitz wrote: > > BTW guys, I saw the issues with both bridge/openvswitch configuration > - seems that we might have here somehow large breakage of the system > w.r.t vxlan traffic for rates that go over few Gbs -- so would love to > get feedback of any kind from the people that were involved with vxlan > over the last months/year. I've seen similar problems with vxlan traffic. In our scenario I had two VMs running on the same host and both VMs having the { veth --> bridge --> vlxan --> IP stack --> NIC } chain. Running iperf on veth showed rate ~6 times slower than direct NIC <-> NIC. With a hack that forces large gso_size in vxlan's handle_offloads, I've got veth performing only slightly slower than NICs ... The explanation I thought of is that performing the split of the packet as late as possible reduces processing overhead and allows more data to be processed. My $0.02 > > Or. > -- Sincerely yours, Mike. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-08 12:43 ` Mike Rapoport @ 2013-12-08 13:07 ` Or Gerlitz 2013-12-08 14:30 ` Mike Rapoport 0 siblings, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-08 13:07 UTC (permalink / raw) To: Mike Rapoport, Or Gerlitz Cc: Joseph Gasparakis, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On 08/12/2013 14:43, Mike Rapoport wrote: > On Fri, Dec 06, 2013 at 11:30:37AM +0200, Or Gerlitz wrote: >>> On 04/12/2013 11:41, Or Gerlitz wrote: >> BTW guys, I saw the issues with both bridge/openvswitch configuration >> - seems that we might have here somehow large breakage of the system >> w.r.t vxlan traffic for rates that go over few Gbs -- so would love to >> get feedback of any kind from the people that were involved with vxlan >> over the last months/year. > I've seen similar problems with vxlan traffic. In our scenario I had two VMs running on the same host and both VMs having the { veth --> bridge --> vlxan --> IP stack --> NIC } chain. How the VMs were connected to the veth NICs? what kernel were you using? > Running iperf on veth showed rate ~6 times slower than direct NIC <-> NIC. With a hack that forces large gso_size in vxlan's handle_offloads, I've got veth performing only slightly slower than NICs ... The explanation I thought of is that performing the split of the packet as late as possible reduces processing overhead and allows more data to be processed. thanks for the tip! few quick clarifications -- so you artificially enlarged the gso_size of the skb? can you provide the line you added here static int handle_offloads(struct sk_buff *skb) { if (skb_is_gso(skb)) { int err = skb_unclone(skb, GFP_ATOMIC); if (unlikely(err)) return err; skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL; } else if (skb->ip_summed != CHECKSUM_PARTIAL) skb->ip_summed = CHECKSUM_NONE; return 0; } also, why enlarging the gso size for skb's cause the actual segmentation to come into play lower in the stack? Or. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-08 13:07 ` Or Gerlitz @ 2013-12-08 14:30 ` Mike Rapoport 2013-12-08 20:50 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Mike Rapoport @ 2013-12-08 14:30 UTC (permalink / raw) To: Or Gerlitz Cc: Or Gerlitz, Joseph Gasparakis, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Sun, Dec 08, 2013 at 03:07:54PM +0200, Or Gerlitz wrote: > On 08/12/2013 14:43, Mike Rapoport wrote: > > On Fri, Dec 06, 2013 at 11:30:37AM +0200, Or Gerlitz wrote: > >>> On 04/12/2013 11:41, Or Gerlitz wrote: > >> BTW guys, I saw the issues with both bridge/openvswitch configuration > >> - seems that we might have here somehow large breakage of the system > >> w.r.t vxlan traffic for rates that go over few Gbs -- so would love to > >> get feedback of any kind from the people that were involved with vxlan > >> over the last months/year. > > I've seen similar problems with vxlan traffic. In our scenario I had two VMs running on the same host and both VMs having the { veth --> bridge --> vlxan --> IP stack --> NIC } chain. > > How the VMs were connected to the veth NICs? what kernel were you using? > > > > Running iperf on veth showed rate ~6 times slower than direct NIC <-> NIC. With a hack that forces large gso_size in vxlan's handle_offloads, I've got veth performing only slightly slower than NICs ... The explanation I thought of is that performing the split of the packet as late as possible reduces processing overhead and allows more data to be processed. > > thanks for the tip! few quick clarifications -- so you artificially > enlarged the gso_size of the skb? can you provide the line you added here It was something *very* hacky: static int handle_offloads(struct sk_buff *skb) { if (skb_is_gso(skb)) { int err = skb_unclone(skb, GFP_ATOMIC); if (unlikely(err)) return err; skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL; if (skb->len < 64000) skb_shinfo(skb)->gso_size = skb->len; else skb_shinfo(skb)->gso_size = 64000; } else if (skb->ip_summed != CHECKSUM_PARTIAL) skb->ip_summed = CHECKSUM_NONE; return 0; } > also, why enlarging the gso size for skb's cause the actual segmentation > to come into play lower in the stack? > > Or. > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-08 14:30 ` Mike Rapoport @ 2013-12-08 20:50 ` Eric Dumazet 2013-12-08 21:36 ` Eric Dumazet 0 siblings, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-08 20:50 UTC (permalink / raw) To: Mike Rapoport Cc: Or Gerlitz, Or Gerlitz, Joseph Gasparakis, Pravin B Shelar, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Sun, 2013-12-08 at 16:30 +0200, Mike Rapoport wrote: > > It was something *very* hacky: > if (skb->len < 64000) > skb_shinfo(skb)->gso_size = skb->len; > else > skb_shinfo(skb)->gso_size = 64000; This sounds like an 16bit overflow somewhere. This reminds the issue we fix in commit 50bceae9bd356 ("tcp: Reallocate headroom if it would overflow csum_start") You might try to reduce the 0xFFFF to something smaller. diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 993da005e087..8364bcfe1e08 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -2373,7 +2373,7 @@ int __tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb) * beyond what csum_start can cover. */ if (unlikely((NET_IP_ALIGN && ((unsigned long)skb->data & 3)) || - skb_headroom(skb) >= 0xFFFF)) { + skb_headroom(skb) >= 0xFF00)) { struct sk_buff *nskb = __pskb_copy(skb, MAX_TCP_HEADER, GFP_ATOMIC); return nskb ? tcp_transmit_skb(sk, nskb, 0, GFP_ATOMIC) : ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-08 20:50 ` Eric Dumazet @ 2013-12-08 21:36 ` Eric Dumazet 0 siblings, 0 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-08 21:36 UTC (permalink / raw) To: Mike Rapoport Cc: Or Gerlitz, Or Gerlitz, Joseph Gasparakis, Pravin B Shelar, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Sun, 2013-12-08 at 12:50 -0800, Eric Dumazet wrote: > On Sun, 2013-12-08 at 16:30 +0200, Mike Rapoport wrote: > > > > It was something *very* hacky: > > > if (skb->len < 64000) > > skb_shinfo(skb)->gso_size = skb->len; > > else > > skb_shinfo(skb)->gso_size = 64000; > > This sounds like an 16bit overflow somewhere. > > This reminds the issue we fix in commit 50bceae9bd356 > ("tcp: Reallocate headroom if it would overflow csum_start") > > You might try to reduce the 0xFFFF to something smaller. Also try following debugging patch : diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 2718fed53d8c..d6fcb6272d37 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -913,8 +913,12 @@ EXPORT_SYMBOL(skb_clone); static void skb_headers_offset_update(struct sk_buff *skb, int off) { /* Only adjust this if it actually is csum_start rather than csum */ - if (skb->ip_summed == CHECKSUM_PARTIAL) - skb->csum_start += off; + if (skb->ip_summed == CHECKSUM_PARTIAL) { + u32 val = (u32)skb->csum_start + off; + + WARN_ON_ONCE(val > 0xFFFF); + skb->csum_start = val; + } /* {transport,network,mac}_header and tail are relative to skb->head */ skb->transport_header += off; skb->network_header += off; ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels [not found] ` <52A197DF.5010806@mellanox.com> 2013-12-06 9:30 ` Or Gerlitz @ 2013-12-06 10:30 ` Joseph Gasparakis 2013-12-07 21:27 ` Or Gerlitz 2013-12-08 15:21 ` Or Gerlitz 1 sibling, 2 replies; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-06 10:30 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Pravin B Shelar, Or Gerlitz, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, jeffrey.t.kirsher, John Fastabend On Fri, 6 Dec 2013, Or Gerlitz wrote: > On 04/12/2013 11:41, Or Gerlitz wrote: > > On Wed, Dec 4, 2013 at 11:24 AM, Joseph > > Gasparakis<joseph.gasparakis@intel.com> wrote: > > > >And just for the record, > > > >you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was > > > >seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO > > > >support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests > > > before it. > > indeed, also, on what kernel did you conducted your tests which you managed > > to WA the problem with unsetting that bit? > > > Hi Joseph, > > Really need your response here -- I'm sorry Or, I managed to miss your original request... > > 1. on which kernel did you manage to get along fine vxlan performance wise > with this hack? > I was running 3.10.6. > 2. did the hack helped for both veth host traffic or only on PV VM traffic? > No, just VM. I haven't tried veth. If you leave the DODGY bit, does your traffic get droped on Tx, after it leaves vxlan and before it hits your driver, which is what I had seen. Is that right? If you unset it, do you recover? What is the output of your ethtool -k on the interface you are transmitting from? > Currently it doesn't converge with 3.12.x or net.git, with veth/vxlan the > DODGE bit isn't set when looking on the skb in the vxlan xmit time, so there's > nothing for me to hack there. For VMs without unsetting the bit things don't > really work, but unsetting it for itself so far didn't get me far performance > wise. > > BTW guys, I saw the issues with both bridge/openvswitch configuration - seems > that we might have here somehow large breakage of the system w.r.t vxlan > traffic for rates that go over few Gbs -- so would love to get feedback of any > kind from the people that were involved with vxlan over the last months/year. > > Or. > > net.git]# grep -rn SKB_GSO_DODGY drivers/net/ net/ipv4 net/core > drivers/net/macvtap.c:585: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; > drivers/net/tun.c:1135: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; > drivers/net/virtio_net.c:497: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; > drivers/net/xen-netback/netback.c:1146: skb_shinfo(skb)->gso_type |= > SKB_GSO_DODGY; > drivers/net/xen-netfront.c:823: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; > net/ipv4/af_inet.c:1264: SKB_GSO_DODGY | > net/ipv4/tcp_offload.c:56: SKB_GSO_DODGY | > net/ipv4/gre_offload.c:40: SKB_GSO_DODGY | > net/ipv4/udp_offload.c:53: if (unlikely(type & ~(SKB_GSO_UDP | > SKB_GSO_DODGY | > net/core/dev.c:2694: if (shinfo->gso_type & SKB_GSO_DODGY) > > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 10:30 ` Joseph Gasparakis @ 2013-12-07 21:27 ` Or Gerlitz 2013-12-08 18:08 ` Joseph Gasparakis 2013-12-08 15:21 ` Or Gerlitz 1 sibling, 1 reply; 63+ messages in thread From: Or Gerlitz @ 2013-12-07 21:27 UTC (permalink / raw) To: Joseph Gasparakis Cc: Or Gerlitz, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Fri, Dec 6, 2013, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > On Fri, 6 Dec 2013, Or Gerlitz wrote: > >> On 04/12/2013 11:41, Or Gerlitz wrote: >> > On Wed, Dec 4, 2013 at 11:24 AM, Joseph >> > Gasparakis<joseph.gasparakis@intel.com> wrote: >> > > >And just for the record, >> > > >you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was >> > > >seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO >> > > >support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests >> > > before it. >> > indeed, also, on what kernel did you conducted your tests which you managed >> > to WA the problem with unsetting that bit? >> >> >> Hi Joseph, >> >> Really need your response here -- > > I'm sorry Or, I managed to miss your original request... sure.. it happens. >> 1. on which kernel did you manage to get along fine vxlan performance wise >> with this hack? > I was running 3.10.6. I see, will try it out, and just for getting closer to your env, what kernel where the guests running? was a bridge / ovs instance involved in the VM PV connectivity? >> 2. did the hack helped for both veth host traffic or only on PV VM traffic? > No, just VM. I haven't tried veth. I see, earlier I was somehow under the impression you noted the problem for veth too. > If you leave the DODGY bit, does your traffic get droped on Tx, after it > leaves vxlan and before it hits your driver, which is what I had seen. Is > that right? What I saw is that if I leave the DODGY bit set, practically things don't work at all, its not that some packets are dropped, was that what you saw? Or on your env only **some** or **few** packets were dropped each time but this killed the tcp session performance? Also, did you hack/modified the VM NIC MTU to take into the account the encapsulation overhead? > If you unset it, do you recover? let me redo this with your setting and see, please make sure to tell me what kernel the VM was running too (thanks!) > What is the output of your ethtool -k on the interface you are > transmitting from? will send you tomorrow, but this happens without offloads for encapsulated traffic. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-07 21:27 ` Or Gerlitz @ 2013-12-08 18:08 ` Joseph Gasparakis 2013-12-08 20:12 ` Or Gerlitz 0 siblings, 1 reply; 63+ messages in thread From: Joseph Gasparakis @ 2013-12-08 18:08 UTC (permalink / raw) To: Or Gerlitz Cc: Joseph Gasparakis, Or Gerlitz, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Sat, 7 Dec 2013, Or Gerlitz wrote: > On Fri, Dec 6, 2013, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: > > On Fri, 6 Dec 2013, Or Gerlitz wrote: > > > >> On 04/12/2013 11:41, Or Gerlitz wrote: > >> > On Wed, Dec 4, 2013 at 11:24 AM, Joseph > >> > Gasparakis<joseph.gasparakis@intel.com> wrote: > >> > > >And just for the record, > >> > > >you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was > >> > > >seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO > >> > > >support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests > >> > > before it. > >> > indeed, also, on what kernel did you conducted your tests which you managed > >> > to WA the problem with unsetting that bit? > >> > >> > >> Hi Joseph, > >> > >> Really need your response here -- > > > > I'm sorry Or, I managed to miss your original request... > > sure.. it happens. > > >> 1. on which kernel did you manage to get along fine vxlan performance wise > >> with this hack? > > > I was running 3.10.6. > > I see, will try it out, and just for getting closer to your env, what > kernel where the guests running? was a bridge / ovs instance involved > in the VM PV connectivity? > The VMs were running an old 2.6.32 kernel, although if I remember well I also tried a 3.x - sorry can't remember which one. The VMs were attached to a bridge, but I haven't noticed any packet loss there. > > >> 2. did the hack helped for both veth host traffic or only on PV VM traffic? > > > No, just VM. I haven't tried veth. > > I see, earlier I was somehow under the impression you noted the > problem for veth too. > > > If you leave the DODGY bit, does your traffic get droped on Tx, after it > > leaves vxlan and before it hits your driver, which is what I had seen. Is > > that right? > > What I saw is that if I leave the DODGY bit set, practically things > don't work at all, its not that some packets are dropped, was that > what you saw? > What I saw was gso packets badly segmented, causing many re-transmissions and dropping the performance to a few MB/s. > Or on your env only **some** or **few** packets were dropped each time > but this killed the tcp session performance? > > Also, did you hack/modified the VM NIC MTU to take into the account > the encapsulation overhead? > The virtio interfaces I used had MTU 1500, but the MTU of the physical NIC was increased to 1600. > > If you unset it, do you recover? > > let me redo this with your setting and see, please make sure to tell > me what kernel the VM was running too (thanks!) > > > What is the output of your ethtool -k on the interface you are > > transmitting from? > > will send you tomorrow, but this happens without offloads for > encapsulated traffic. > I have only noticed this with the offloads on. Turning off encapsuation TSO off, would simply make the gso's to get segmented in dev_hard_xmit() as expected. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-08 18:08 ` Joseph Gasparakis @ 2013-12-08 20:12 ` Or Gerlitz 0 siblings, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-08 20:12 UTC (permalink / raw) To: Joseph Gasparakis Cc: Or Gerlitz, Pravin B Shelar, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, Kirsher, Jeffrey T, John Fastabend On Sun, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote: >> What I saw is that if I leave the DODGY bit set, practically things >> don't work at all, its not that some packets are dropped, was that >> what you saw? > What I saw was gso packets badly segmented, causing many re-transmissions > and dropping the performance to a few MB/s. Yes, in my testbed upto about 400Mbs (b not B..., yes!) >> Also, did you hack/modified the VM NIC MTU to take into the account >> the encapsulation overhead? > The virtio interfaces I used had MTU 1500, but the MTU of the physical NIC > was increased to 1600. mmm, that's sort of equivalent, but zero touch VM wise, nice! > I have only noticed this with the offloads on. Turning off encapsuation > TSO off, would simply make the gso's to get segmented in dev_hard_xmit() > as expected. mmm, I am not sure this is the case with kernels > 3.10.x, but I'd like to double check that, basically, its possible that I didn't make sure to always have "proper" MTU at the VM @ all times. Also, did you see the unsimilarity between TX/RX which I reported earlier today, that is accelerated TX from single VM can go as far as > 30Gbs while RX to single VM or even multiple VMs doesn't go beyond 5-6Gbs probably as of the lack of GRO? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-06 10:30 ` Joseph Gasparakis 2013-12-07 21:27 ` Or Gerlitz @ 2013-12-08 15:21 ` Or Gerlitz 1 sibling, 0 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-08 15:21 UTC (permalink / raw) To: Joseph Gasparakis Cc: Pravin B Shelar, Or Gerlitz, Eric Dumazet, Jerry Chu, Eric Dumazet, Alexei Starovoitov, David Miller, netdev, jeffrey.t.kirsher, John Fastabend, Jerry Chu On 06/12/2013 12:30, Joseph Gasparakis wrote: > On Fri, 6 Dec 2013, Or Gerlitz wrote: > > >> 1. on which kernel did you manage to get along fine vxlan performance wise >> with this hack? >> > I was running 3.10.6. > >> 2. did the hack helped for both veth host traffic or only on PV VM traffic? >> > No, just VM. I haven't tried veth. > > If you leave the DODGY bit, does your traffic get droped on Tx, after it > leaves vxlan and before it hits your driver, which is what I had seen. Is > that right? > > If you unset it, do you recover? > > What is the output of your ethtool -k on the interface you are > transmitting from? > >> Currently it doesn't converge with 3.12.x or net.git, with veth/vxlan the >> DODGE bit isn't set when looking on the skb in the vxlan xmit time, so there's >> nothing for me to hack there. For VMs without unsetting the bit things don't >> really work, but unsetting it for itself so far didn't get me far performance >> wise. >> >> BTW guys, I saw the issues with both bridge/openvswitch configuration - seems >> that we might have here somehow large breakage of the system w.r.t vxlan >> traffic for rates that go over few Gbs -- so would love to get feedback of any >> kind from the people that were involved with vxlan over the last months/year. >> >> OK!! so finally I managed to get some hacked but stable ground to step on .... indeed with 3.10.X (I tried 3.10.19) if you 1. reduce the VM PV NIC MTU to account for the vxlan tunneling overhead (e.g to 1450 vs 1500) 2. unset the DODGY bit for GSO packets in the vxlan driver handle_offloads function --> You get sane vxlan performance when the VM xmits, without HW offloads I got up to 4-5 Gbs for single VM and with HW offloads > 30Gbs for single VM when the VM is sending to a peer hypervisor. On VM RX side, it doesn't go too much higher, e.g stays in the order of 3-4Gbs for single receiving VM, I am pretty sure this relates to the no GRO for vxlan which is pretty much terrible for VM traffic. So it seems the TODO here is the following: 1. manage to get the hack for vm-vxlan traffic to work on the net tree 2. fix the bug that make the hack necessary 3. find the problem with veth-vxlan traffic on the net tree 4. add GRO support for encapsulated/vxlan traffic Or. Or. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 15:05 vxlan/veth performance issues on net.git + latest kernels Or Gerlitz 2013-12-03 15:30 ` Eric Dumazet @ 2013-12-03 17:12 ` Eric Dumazet 2013-12-03 19:50 ` Or Gerlitz 1 sibling, 1 reply; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 17:12 UTC (permalink / raw) To: Or Gerlitz Cc: Eric Dumazet, Alexei Starovoitov, Pravin B Shelar, David Miller, netdev On Tue, 2013-12-03 at 17:05 +0200, Or Gerlitz wrote: > ---------------------- > > ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN > ifconfig vxlan42 192.168.42.144/24 up > > brctl addbr br-vx > ip link set br-vx up > > ifconfig br-vx 192.168.52.144/24 up > brctl addif br-vx vxlan42 > > ip link add type veth > brctl addif br-vx veth1 Aside question : If I do not have brctl on my host, how can I use "ip" command to perform this "brctl addif" action ? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 17:12 ` Eric Dumazet @ 2013-12-03 19:50 ` Or Gerlitz 2013-12-03 20:19 ` John Fastabend 2013-12-03 21:12 ` Eric Dumazet 0 siblings, 2 replies; 63+ messages in thread From: Or Gerlitz @ 2013-12-03 19:50 UTC (permalink / raw) To: Eric Dumazet; +Cc: Or Gerlitz, netdev On Tue, Dec 3, 2013 at 7:12 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> brctl addbr br-vx >> ip link set br-vx up >> ifconfig br-vx 192.168.52.144/24 up >> brctl addif br-vx vxlan42 >> ip link add type veth >> brctl addif br-vx veth1 > Aside question : If I do not have brctl on my host, > how can I use "ip" command to perform this "brctl addif" action ? you can get the sources from git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/bridge-utils.git or from http://sourceforge.net/projects/bridge/files/bridge/bridge-utils-1.5.tar.gz/download, should be trivial to configure and build, if I remember correct > > > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 19:50 ` Or Gerlitz @ 2013-12-03 20:19 ` John Fastabend 2013-12-03 21:12 ` Eric Dumazet 1 sibling, 0 replies; 63+ messages in thread From: John Fastabend @ 2013-12-03 20:19 UTC (permalink / raw) To: Or Gerlitz; +Cc: Eric Dumazet, Or Gerlitz, netdev On 12/3/2013 11:50 AM, Or Gerlitz wrote: > On Tue, Dec 3, 2013 at 7:12 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > >>> brctl addbr br-vx >>> ip link set br-vx up >>> ifconfig br-vx 192.168.52.144/24 up >>> brctl addif br-vx vxlan42 >>> ip link add type veth >>> brctl addif br-vx veth1 > >> Aside question : If I do not have brctl on my host, >> how can I use "ip" command to perform this "brctl addif" action ? > > you can get the sources from > git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/bridge-utils.git > or from http://sourceforge.net/projects/bridge/files/bridge/bridge-utils-1.5.tar.gz/download, > should be trivial to configure and build, if I remember correct > > The 'brctl addif' should be the same as setting the master. #ip link set dev veth1 master br-vs and then use nomaster to delete the interface. .John ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: vxlan/veth performance issues on net.git + latest kernels 2013-12-03 19:50 ` Or Gerlitz 2013-12-03 20:19 ` John Fastabend @ 2013-12-03 21:12 ` Eric Dumazet 1 sibling, 0 replies; 63+ messages in thread From: Eric Dumazet @ 2013-12-03 21:12 UTC (permalink / raw) To: Or Gerlitz; +Cc: Or Gerlitz, netdev On Tue, 2013-12-03 at 21:50 +0200, Or Gerlitz wrote: > you can get the sources from > git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/bridge-utils.git > or from http://sourceforge.net/projects/bridge/files/bridge/bridge-utils-1.5.tar.gz/download, > should be trivial to configure and build, if I remember correct Well, I did not elaborate. Lets say I have some hosts where I cannot install a new binary, for whatever reason ;) Thanks, John Fastabend replied to the question. ^ permalink raw reply [flat|nested] 63+ messages in thread
end of thread, other threads:[~2013-12-08 21:36 UTC | newest]
Thread overview: 63+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-03 15:05 vxlan/veth performance issues on net.git + latest kernels Or Gerlitz
2013-12-03 15:30 ` Eric Dumazet
2013-12-03 19:55 ` Or Gerlitz
2013-12-03 21:11 ` Joseph Gasparakis
2013-12-03 21:09 ` Or Gerlitz
2013-12-03 21:24 ` Eric Dumazet
2013-12-03 21:36 ` Or Gerlitz
2013-12-03 21:50 ` David Miller
2013-12-03 21:55 ` Eric Dumazet
2013-12-03 22:15 ` Or Gerlitz
2013-12-03 22:22 ` Or Gerlitz
2013-12-03 22:30 ` Hannes Frederic Sowa
2013-12-03 22:35 ` Or Gerlitz
2013-12-03 22:39 ` Hannes Frederic Sowa
2013-12-03 23:10 ` Or Gerlitz
2013-12-03 23:30 ` Or Gerlitz
2013-12-03 23:49 ` Hannes Frederic Sowa
2013-12-03 23:59 ` Eric Dumazet
2013-12-04 0:26 ` Alexei Starovoitov
2013-12-04 0:36 ` Eric Dumazet
2013-12-04 0:55 ` Alexei Starovoitov
2013-12-04 1:23 ` Eric Dumazet
2013-12-04 1:59 ` Alexei Starovoitov
2013-12-06 9:06 ` Or Gerlitz
2013-12-06 13:36 ` Eric Dumazet
2013-12-07 21:20 ` Or Gerlitz
2013-12-08 12:09 ` Or Gerlitz
2013-12-04 6:39 ` David Miller
2013-12-04 17:40 ` Eric Dumazet
2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet
2013-12-05 14:13 ` Hannes Frederic Sowa
2013-12-05 14:45 ` Eric Dumazet
2013-12-05 15:05 ` Eric Dumazet
2013-12-05 15:44 ` Hannes Frederic Sowa
2013-12-05 16:38 ` Eric Dumazet
2013-12-05 16:54 ` Hannes Frederic Sowa
2013-12-06 20:24 ` David Miller
2013-12-03 23:13 ` vxlan/veth performance issues on net.git + latest kernels Joseph Gasparakis
2013-12-03 23:09 ` Or Gerlitz
2013-12-04 0:35 ` Joseph Gasparakis
2013-12-04 0:34 ` Alexei Starovoitov
2013-12-04 1:29 ` Joseph Gasparakis
2013-12-04 1:18 ` Eric Dumazet
2013-12-04 0:44 ` Joseph Gasparakis
2013-12-04 8:35 ` Or Gerlitz
2013-12-04 9:24 ` Joseph Gasparakis
2013-12-04 9:41 ` Or Gerlitz
2013-12-04 15:20 ` Or Gerlitz
[not found] ` <52A197DF.5010806@mellanox.com>
2013-12-06 9:30 ` Or Gerlitz
2013-12-08 12:43 ` Mike Rapoport
2013-12-08 13:07 ` Or Gerlitz
2013-12-08 14:30 ` Mike Rapoport
2013-12-08 20:50 ` Eric Dumazet
2013-12-08 21:36 ` Eric Dumazet
2013-12-06 10:30 ` Joseph Gasparakis
2013-12-07 21:27 ` Or Gerlitz
2013-12-08 18:08 ` Joseph Gasparakis
2013-12-08 20:12 ` Or Gerlitz
2013-12-08 15:21 ` Or Gerlitz
2013-12-03 17:12 ` Eric Dumazet
2013-12-03 19:50 ` Or Gerlitz
2013-12-03 20:19 ` John Fastabend
2013-12-03 21:12 ` Eric Dumazet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).