Significant network performance difference when different NIC MQ handles the packet

Significant network performance difference when different NIC MQ handles the packet

[Zhangleiqiang (Trump)]

Hi, all:

	I have used pkt-gen to send udp packet (1400 bytes) from one Dom0 A to another Dom0 B each of which is connected by 10GE network. On the receive side (B)，different "ksoftirqd/x" processes will handle the packet during each testing because the nic has multiple queues. I find that if "ksoftirqd/0" handles the packet, the throughout can reach to about 7Gbps. However, when "ksoftirqd/2" handles the packet, the throughout can only reach to about 2Gbps, and for "ksoftirqd/3", the throughout can even as low as 1Gbps.

	I am wondering what the reason it is. Could anyone give me some hint about it ?

----------
zhangleiqiang (Trump)

Best Regards


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Significant network performance difference when different NIC MQ handles the packet

[Zoltan Kiss]

On 31/12/14 09:06, Zhangleiqiang (Trump) wrote:
> Hi, all:
>
> 	I have used pkt-gen to send udp packet (1400 bytes) from one Dom0 A to another Dom0 B each of which is connected by 10GE network. On the receive side (B)，different "ksoftirqd/x" processes will handle the packet during each testing because the nic has multiple queues. I find that if "ksoftirqd/0" handles the packet, the throughout can reach to about 7Gbps. However, when "ksoftirqd/2" handles the packet, the throughout can only reach to about 2Gbps, and for "ksoftirqd/3", the throughout can even as low as 1Gbps.
>
> 	I am wondering what the reason it is. Could anyone give me some hint about it ?
NUMA topology? Your card might use memory "close" to CPU core 0, but not 
to the others?

Zoltan

>
> ----------
> zhangleiqiang (Trump)
>
> Best Regards
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Significant network performance difference when different NIC MQ handles the packet

[Zhangleiqiang (Trump)]

> On 31/12/14 09:06, Zhangleiqiang (Trump) wrote:
> > Hi, all:
> >
> > 	I have used pkt-gen to send udp packet (1400 bytes) from one Dom0 A to
> another Dom0 B each of which is connected by 10GE network. On the receive
> side (B)，different "ksoftirqd/x" processes will handle the packet during each
> testing because the nic has multiple queues. I find that if "ksoftirqd/0" handles
> the packet, the throughout can reach to about 7Gbps. However, when
> "ksoftirqd/2" handles the packet, the throughout can only reach to about
> 2Gbps, and for "ksoftirqd/3", the throughout can even as low as 1Gbps.
> >
> > 	I am wondering what the reason it is. Could anyone give me some hint
> about it ?
> NUMA topology? Your card might use memory "close" to CPU core 0, but not
> to the others?

Thanks for your hint. 

I am not sure if the NUMA topology is the reason for it. Because my server has two nodes, each of which has 8 cpus. The result of "numctl --hardware" is as follows:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 32090 MB
node 0 free: 31640 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 32311 MB
node 1 free: 31643 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

I booted dom0 with "dom0_max_vcpus=6 dom0_vcpu_pin" command line. So I think "ksoftirqd/x" will running on cpu "x", e.g. ksoftirqd/0 and ksortirqd/2 are running on cpu0 and cpu2 respectively. If it is so, ksoftirqd/0 and ksortirqd/2 should have the "same distance" to the memory which NIC card uses.

> Zoltan
> 
> >
> > ----------
> > zhangleiqiang (Trump)
> >
> > Best Regards
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Significant network performance difference when different NIC MQ handles the packet

[Zoltan Kiss]

On 04/01/15 06:15, Zhangleiqiang (Trump) wrote:
>> On 31/12/14 09:06, Zhangleiqiang (Trump) wrote:
>>> Hi, all:
>>>
>>> 	I have used pkt-gen to send udp packet (1400 bytes) from one Dom0 A to
>> another Dom0 B each of which is connected by 10GE network. On the receive
>> side (B)，different "ksoftirqd/x" processes will handle the packet during each
>> testing because the nic has multiple queues. I find that if "ksoftirqd/0" handles
>> the packet, the throughout can reach to about 7Gbps. However, when
>> "ksoftirqd/2" handles the packet, the throughout can only reach to about
>> 2Gbps, and for "ksoftirqd/3", the throughout can even as low as 1Gbps.
>>>
>>> 	I am wondering what the reason it is. Could anyone give me some hint
>> about it ?
>> NUMA topology? Your card might use memory "close" to CPU core 0, but not
>> to the others?
>
> Thanks for your hint.
>
> I am not sure if the NUMA topology is the reason for it. Because my server has two nodes, each of which has 8 cpus. The result of "numctl --hardware" is as follows:
Can you send xl info -n, so we can see how Xen sees it?

>
> available: 2 nodes (0-1)
> node 0 cpus: 0 1 2 3 4 5 6 7
> node 0 size: 32090 MB
> node 0 free: 31640 MB
> node 1 cpus: 8 9 10 11 12 13 14 15
> node 1 size: 32311 MB
> node 1 free: 31643 MB
> node distances:
> node   0   1
>    0:  10  20
>    1:  20  10
>
> I booted dom0 with "dom0_max_vcpus=6 dom0_vcpu_pin" command line. So I think "ksoftirqd/x" will running on cpu "x", e.g. ksoftirqd/0 and ksortirqd/2 are running on cpu0 and cpu2 respectively. If it is so, ksoftirqd/0 and ksortirqd/2 should have the "same distance" to the memory which NIC card uses.

dom0_max_vcpus sets the number of VCPUs for Dom0, but it doesn't 
necessarily tell on which node they should run. Although AFAIK Xen tries 
to keep the vcpus on the same node.
Also, you should check the NUMA assignment of your NIC:

http://superuser.com/questions/701090/determine-pcie-device-numa-node

And check this as well if you haven't done so:

http://wiki.xen.org/wiki/Xen_on_NUMA_Machines


>
>> Zoltan
>>
>>>
>>> ----------
>>> zhangleiqiang (Trump)
>>>
>>> Best Regards
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel