* macvtap performance regression (bisected) between 3.13 and 3.14-rc1
@ 2014-02-27 20:52 Christian Borntraeger
2014-02-28 22:14 ` Vlad Yasevich
0 siblings, 1 reply; 8+ messages in thread
From: Christian Borntraeger @ 2014-02-27 20:52 UTC (permalink / raw)
To: Vlad Yasevich
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
Vlad,
commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.
causes a performance regression for iperf traffic between two KVM guests
on my s390 system. Both guests are connected via two macvtaps on the same OSA
network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
~4Gbit/sec
Latency seems to be unchanges (uperf 1byte ping pong).
According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
patch and ~ 40000 bytes without. So for some reason this patch causes the
network stack to do segmentation. (the guest kernel stays the same, only host
kernel is changed).
Any ideas?
Christian
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-02-27 20:52 macvtap performance regression (bisected) between 3.13 and 3.14-rc1 Christian Borntraeger
@ 2014-02-28 22:14 ` Vlad Yasevich
2014-03-01 11:15 ` Christian Borntraeger
0 siblings, 1 reply; 8+ messages in thread
From: Vlad Yasevich @ 2014-02-28 22:14 UTC (permalink / raw)
To: Christian Borntraeger
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
> Vlad,
>
> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
> macvtap: Add support of packet capture on macvtap device.
>
> causes a performance regression for iperf traffic between two KVM guests
> on my s390 system. Both guests are connected via two macvtaps on the same OSA
> network card.
> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
> ~4Gbit/sec
>
> Latency seems to be unchanges (uperf 1byte ping pong).
>
> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
> patch and ~ 40000 bytes without. So for some reason this patch causes the
> network stack to do segmentation. (the guest kernel stays the same, only host
> kernel is changed).
>
> Any ideas?
I am looking. It shouldn't cause addition segmentations and when I ran
netperf on the code I didn't see any difference in the throughput.
-vlad
>
>
> Christian
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-02-28 22:14 ` Vlad Yasevich
@ 2014-03-01 11:15 ` Christian Borntraeger
2014-03-01 19:27 ` Vlad Yasevich
0 siblings, 1 reply; 8+ messages in thread
From: Christian Borntraeger @ 2014-03-01 11:15 UTC (permalink / raw)
To: vyasevic
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 28/02/14 23:14, Vlad Yasevich wrote:
> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>> Vlad,
>>
>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>> macvtap: Add support of packet capture on macvtap device.
>>
>> causes a performance regression for iperf traffic between two KVM guests
>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>> network card.
>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>> ~4Gbit/sec
>>
>> Latency seems to be unchanges (uperf 1byte ping pong).
>>
>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>> network stack to do segmentation. (the guest kernel stays the same, only host
>> kernel is changed).
>>
>> Any ideas?
>
> I am looking. It shouldn't cause addition segmentations and when I ran
> netperf on the code I didn't see any difference in the throughput.
Dont know if the different bytes/packets ratio is really the reason or
just a side effect. As a hint: the underlying network device does not support
segmentation, but this should not matter for traffic between to guests.
Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
(macvtap: Perform GSO on forwarding path), the setup is basically the same.
Christian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-03-01 11:15 ` Christian Borntraeger
@ 2014-03-01 19:27 ` Vlad Yasevich
2014-03-02 1:21 ` Vlad Yasevich
2014-03-03 9:11 ` Christian Borntraeger
0 siblings, 2 replies; 8+ messages in thread
From: Vlad Yasevich @ 2014-03-01 19:27 UTC (permalink / raw)
To: Christian Borntraeger, vyasevic
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
> On 28/02/14 23:14, Vlad Yasevich wrote:
>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>> Vlad,
>>>
>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>> macvtap: Add support of packet capture on macvtap device.
>>>
>>> causes a performance regression for iperf traffic between two KVM guests
>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>> network card.
>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>> ~4Gbit/sec
>>>
>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>
>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>>> network stack to do segmentation. (the guest kernel stays the same, only host
>>> kernel is changed).
>>>
>>> Any ideas?
>>
>> I am looking. It shouldn't cause addition segmentations and when I ran
>> netperf on the code I didn't see any difference in the throughput.
>
> Dont know if the different bytes/packets ratio is really the reason or
> just a side effect. As a hint: the underlying network device does not support
> segmentation, but this should not matter for traffic between to guests.
Could you post 'ethtool -k' output for both lower-level device and the
macvtap device?
Thanks
-vlad
>
> Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
> (macvtap: Perform GSO on forwarding path), the setup is basically the same.
>
>
> Christian
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-03-01 19:27 ` Vlad Yasevich
@ 2014-03-02 1:21 ` Vlad Yasevich
2014-03-03 9:13 ` Christian Borntraeger
2014-03-03 9:11 ` Christian Borntraeger
1 sibling, 1 reply; 8+ messages in thread
From: Vlad Yasevich @ 2014-03-02 1:21 UTC (permalink / raw)
To: Christian Borntraeger, vyasevic
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>> Vlad,
>>>>
>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>> macvtap: Add support of packet capture on macvtap device.
>>>>
>>>> causes a performance regression for iperf traffic between two KVM guests
>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>> network card.
>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>> ~4Gbit/sec
>>>>
>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>
>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>>>> network stack to do segmentation. (the guest kernel stays the same, only host
>>>> kernel is changed).
>>>>
>>>> Any ideas?
>>>
>>> I am looking. It shouldn't cause addition segmentations and when I ran
>>> netperf on the code I didn't see any difference in the throughput.
>>
>> Dont know if the different bytes/packets ratio is really the reason or
>> just a side effect. As a hint: the underlying network device does not support
>> segmentation, but this should not matter for traffic between to guests.
>
> Could you post 'ethtool -k' output for both lower-level device and the
> macvtap device?
>
> Thanks
> -vlad
>
Ok. I think I see what's happening. Since you turn off offloads on
lower device, that's propagated to macvlan device. As a result, when
when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
lower level says it does support segmentation.
One way to fix this is to never disable offloads on macvlan. macvlan
will always try to use __dev_queue_xmit() with it's lower device, so any
segmentation can happen there.
-vlad
>>
>> Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8
>> (macvtap: Perform GSO on forwarding path), the setup is basically the same.
>>
>>
>> Christian
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-03-02 1:21 ` Vlad Yasevich
@ 2014-03-03 9:13 ` Christian Borntraeger
2014-03-03 19:36 ` Vlad Yasevich
0 siblings, 1 reply; 8+ messages in thread
From: Christian Borntraeger @ 2014-03-03 9:13 UTC (permalink / raw)
To: Vlad Yasevich, vyasevic
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 02/03/14 02:21, Vlad Yasevich wrote:
> On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
>> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>>> Vlad,
>>>>>
>>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>> macvtap: Add support of packet capture on macvtap device.
>>>>>
>>>>> causes a performance regression for iperf traffic between two KVM guests
>>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>>> network card.
>>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>>> ~4Gbit/sec
>>>>>
>>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>>
>>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>>>>> network stack to do segmentation. (the guest kernel stays the same, only host
>>>>> kernel is changed).
>>>>>
>>>>> Any ideas?
>>>>
>>>> I am looking. It shouldn't cause addition segmentations and when I ran
>>>> netperf on the code I didn't see any difference in the throughput.
>>>
>>> Dont know if the different bytes/packets ratio is really the reason or
>>> just a side effect. As a hint: the underlying network device does not support
>>> segmentation, but this should not matter for traffic between to guests.
>>
>> Could you post 'ethtool -k' output for both lower-level device and the
>> macvtap device?
>>
>> Thanks
>> -vlad
>>
>
> Ok. I think I see what's happening. Since you turn off offloads on
> lower device, that's propagated to macvlan device. As a result, when
> when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
> lower level says it does support segmentation.
>
> One way to fix this is to never disable offloads on macvlan. macvlan
> will always try to use __dev_queue_xmit() with it's lower device, so any
> segmentation can happen there.
If you have anything that I should test, let me know.
Christian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-03-03 9:13 ` Christian Borntraeger
@ 2014-03-03 19:36 ` Vlad Yasevich
0 siblings, 0 replies; 8+ messages in thread
From: Vlad Yasevich @ 2014-03-03 19:36 UTC (permalink / raw)
To: Christian Borntraeger, Vlad Yasevich
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 03/03/2014 04:13 AM, Christian Borntraeger wrote:
> On 02/03/14 02:21, Vlad Yasevich wrote:
>> On 03/01/2014 02:27 PM, Vlad Yasevich wrote:
>>> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>>>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>>>> Vlad,
>>>>>>
>>>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>> macvtap: Add support of packet capture on macvtap device.
>>>>>>
>>>>>> causes a performance regression for iperf traffic between two KVM guests
>>>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>>>> network card.
>>>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>>>> ~4Gbit/sec
>>>>>>
>>>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>>>
>>>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>>>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>>>>>> network stack to do segmentation. (the guest kernel stays the same, only host
>>>>>> kernel is changed).
>>>>>>
>>>>>> Any ideas?
>>>>>
>>>>> I am looking. It shouldn't cause addition segmentations and when I ran
>>>>> netperf on the code I didn't see any difference in the throughput.
>>>>
>>>> Dont know if the different bytes/packets ratio is really the reason or
>>>> just a side effect. As a hint: the underlying network device does not support
>>>> segmentation, but this should not matter for traffic between to guests.
>>>
>>> Could you post 'ethtool -k' output for both lower-level device and the
>>> macvtap device?
>>>
>>> Thanks
>>> -vlad
>>>
>>
>> Ok. I think I see what's happening. Since you turn off offloads on
>> lower device, that's propagated to macvlan device. As a result, when
>> when we call dev_queue_xmit on the vlan->dev, we end up segmenting since
>> lower level says it does support segmentation.
>>
>> One way to fix this is to never disable offloads on macvlan. macvlan
>> will always try to use __dev_queue_xmit() with it's lower device, so any
>> segmentation can happen there.
>
> If you have anything that I should test, let me know.
Hi Christian
Just sent out a patch to fix this. I tried it with namespaces and
kvm guests and it seems to restore performance for me.
Please give it a try.
Thanks
-vlad
>
> Christian
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
2014-03-01 19:27 ` Vlad Yasevich
2014-03-02 1:21 ` Vlad Yasevich
@ 2014-03-03 9:11 ` Christian Borntraeger
1 sibling, 0 replies; 8+ messages in thread
From: Christian Borntraeger @ 2014-03-03 9:11 UTC (permalink / raw)
To: Vlad Yasevich, vyasevic
Cc: David S. Miller, Jason Wang, Michael S. Tsirkin, netdev, KVM list,
Linux Kernel Mailing List
On 01/03/14 20:27, Vlad Yasevich wrote:
> On 03/01/2014 06:15 AM, Christian Borntraeger wrote:
>> On 28/02/14 23:14, Vlad Yasevich wrote:
>>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote:
>>>> Vlad,
>>>>
>>>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>> macvtap: Add support of packet capture on macvtap device.
>>>>
>>>> causes a performance regression for iperf traffic between two KVM guests
>>>> on my s390 system. Both guests are connected via two macvtaps on the same OSA
>>>> network card.
>>>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get
>>>> ~4Gbit/sec
>>>>
>>>> Latency seems to be unchanges (uperf 1byte ping pong).
>>>>
>>>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this
>>>> patch and ~ 40000 bytes without. So for some reason this patch causes the
>>>> network stack to do segmentation. (the guest kernel stays the same, only host
>>>> kernel is changed).
>>>>
>>>> Any ideas?
>>>
>>> I am looking. It shouldn't cause addition segmentations and when I ran
>>> netperf on the code I didn't see any difference in the throughput.
>>
>> Dont know if the different bytes/packets ratio is really the reason or
>> just a side effect. As a hint: the underlying network device does not support
>> segmentation, but this should not matter for traffic between to guests.
>
> Could you post 'ethtool -k' output for both lower-level device and the
> macvtap device?
Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off [fixed]
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [fixed]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
Features for macvtap1:
rx-checksumming: off [fixed]
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off [fixed]
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [fixed]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-03-03 19:36 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-27 20:52 macvtap performance regression (bisected) between 3.13 and 3.14-rc1 Christian Borntraeger
2014-02-28 22:14 ` Vlad Yasevich
2014-03-01 11:15 ` Christian Borntraeger
2014-03-01 19:27 ` Vlad Yasevich
2014-03-02 1:21 ` Vlad Yasevich
2014-03-03 9:13 ` Christian Borntraeger
2014-03-03 19:36 ` Vlad Yasevich
2014-03-03 9:11 ` Christian Borntraeger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).