* Performances are not scale with multiple ports
@ 2013-05-28 2:26 Shinae Woo
[not found] ` <CA+f=ZztBO=fTqSvj7n8SwKZUNz=fhOq1LcaFMJGJMRhy7xi41w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 8+ messages in thread
From: Shinae Woo @ 2013-05-28 2:26 UTC (permalink / raw)
To: dev-VfR2kkLFssw
[-- Attachment #1: Type: text/plain, Size: 4461 bytes --]
Hello, all.
I play the dpdk-1.2.3r1 with examples.
But I can not achieve the line-rate packet receive performance,
and the performance is not scale with multiple ports.
For example, in example l2fwd, I have tested two cases with 2 ports, and 4
ports,
using belowed command line each
./build/l2fwd -cf -n3 -- -p3
./build/l2fwd -cf -n3 -- -pf
But both cases, the aggregated performance are not scale.
== experiments environments ==
- Two Intel 82599 NICs (total 4 ports)
- Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores)
- 1024 * 2MB hugepages
- Linux 2.6.38-15-server
- Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps.
*1. Packet forwarding performance*
In 2 ports case, receive performance is 13Mpps,
In 4 ports case, not 26Mbps, only 16.8Mpps.
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 4292256
Packets received: 6517396
Packets dropped: 2224776
Statistics for port 1 ------------------------------
Packets sent: 4291840
Packets received: 6517044
Packets dropped: 2225556
Aggregate statistics ===============================
Total packets sent: 8584128
Total packets received: 13034472
Total packets dropped: 4450332
====================================================
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 1784064
Packets received: 2632700
Packets dropped: 848128
Statistics for port 1 ------------------------------
Packets sent: 1784104
Packets received: 2632196
Packets dropped: 848596
Statistics for port 2 ------------------------------
Packets sent: 3587616
Packets received: 5816344
Packets dropped: 2200176
Statistics for port 3 ------------------------------
Packets sent: 3587712
Packets received: 5787848
Packets dropped: 2228684
Aggregate statistics ===============================
Total packets sent: 10743560
Total packets received: 16869152
Total packets dropped: 6125608
====================================================
*2. Packet receiving performance*
I fix the codes for only receiving packets (not forwarding),
the performance is still not scalable as each 13.3Mpps, 18Mpps.
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 0
Packets received: 6678860
Packets dropped: 0
Statistics for port 1 ------------------------------
Packets sent: 0
Packets received: 6646120
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 0
Total packets received: 13325012
Total packets dropped: 0
====================================================
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent: 0
Packets received: 3129624
Packets dropped: 0
Statistics for port 1 ------------------------------
Packets sent: 0
Packets received: 3131292
Packets dropped: 0
Statistics for port 2 ------------------------------
Packets sent: 0
Packets received: 6260908
Packets dropped: 0
Statistics for port 3 ------------------------------
Packets sent: 0
Packets received: 6238764
Packets dropped: 0
Aggregate statistics ===============================
Total packets sent: 0
Total packets received: 18760640
Total packets dropped: 0
====================================================
The question is that
1. How I can achieve each port receiving full 14.88Mpps ?
What might be the bottleneck in current environment?
2. Why the performance using multiple ports is not scale?
I guess doubling ports shows the doubling the receiving performance,
but it shows not. I am curious about what is limiting the packet
receivng performance.
Thanks,
Shinae
[-- Attachment #2: Type: text/html, Size: 5153 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread[parent not found: <CA+f=ZztBO=fTqSvj7n8SwKZUNz=fhOq1LcaFMJGJMRhy7xi41w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Performances are not scale with multiple ports [not found] ` <CA+f=ZztBO=fTqSvj7n8SwKZUNz=fhOq1LcaFMJGJMRhy7xi41w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-05-28 2:30 ` Naoto MATSUMOTO [not found] ` <20130528113005.5E6D.C42C3789-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> 2013-05-28 9:22 ` Thomas Monjalon 1 sibling, 1 reply; 8+ messages in thread From: Naoto MATSUMOTO @ 2013-05-28 2:30 UTC (permalink / raw) To: Shinae Woo; +Cc: dev-VfR2kkLFssw FYI: Disruptive IP Networking with Intel DPDK on Linux http://slidesha.re/SeVFZo On Tue, 28 May 2013 11:26:30 +0900 Shinae Woo <shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Hello, all. > > I play the dpdk-1.2.3r1 with examples. > > But I can not achieve the line-rate packet receive performance, > and the performance is not scale with multiple ports. > > For example, in example l2fwd, I have tested two cases with 2 ports, and 4 > ports, > using belowed command line each > > ./build/l2fwd -cf -n3 -- -p3 > ./build/l2fwd -cf -n3 -- -pf > > But both cases, the aggregated performance are not scale. > > == experiments environments == > - Two Intel 82599 NICs (total 4 ports) > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores) > - 1024 * 2MB hugepages > - Linux 2.6.38-15-server > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps. > > *1. Packet forwarding performance* > > In 2 ports case, receive performance is 13Mpps, > In 4 ports case, not 26Mbps, only 16.8Mpps. > > Port statistics ==================================== > Statistics for port 0 ------------------------------ > Packets sent: 4292256 > Packets received: 6517396 > Packets dropped: 2224776 > Statistics for port 1 ------------------------------ > Packets sent: 4291840 > Packets received: 6517044 > Packets dropped: 2225556 > Aggregate statistics =============================== > Total packets sent: 8584128 > Total packets received: 13034472 > Total packets dropped: 4450332 > ==================================================== > > Port statistics ==================================== > Statistics for port 0 ------------------------------ > Packets sent: 1784064 > Packets received: 2632700 > Packets dropped: 848128 > Statistics for port 1 ------------------------------ > Packets sent: 1784104 > Packets received: 2632196 > Packets dropped: 848596 > Statistics for port 2 ------------------------------ > Packets sent: 3587616 > Packets received: 5816344 > Packets dropped: 2200176 > Statistics for port 3 ------------------------------ > Packets sent: 3587712 > Packets received: 5787848 > Packets dropped: 2228684 > Aggregate statistics =============================== > Total packets sent: 10743560 > Total packets received: 16869152 > Total packets dropped: 6125608 > ==================================================== > > *2. Packet receiving performance* > I fix the codes for only receiving packets (not forwarding), > the performance is still not scalable as each 13.3Mpps, 18Mpps. > > Port statistics ==================================== > Statistics for port 0 ------------------------------ > Packets sent: 0 > Packets received: 6678860 > Packets dropped: 0 > Statistics for port 1 ------------------------------ > Packets sent: 0 > Packets received: 6646120 > Packets dropped: 0 > Aggregate statistics =============================== > Total packets sent: 0 > Total packets received: 13325012 > Total packets dropped: 0 > ==================================================== > > Port statistics ==================================== > Statistics for port 0 ------------------------------ > Packets sent: 0 > Packets received: 3129624 > Packets dropped: 0 > Statistics for port 1 ------------------------------ > Packets sent: 0 > Packets received: 3131292 > Packets dropped: 0 > Statistics for port 2 ------------------------------ > Packets sent: 0 > Packets received: 6260908 > Packets dropped: 0 > Statistics for port 3 ------------------------------ > Packets sent: 0 > Packets received: 6238764 > Packets dropped: 0 > Aggregate statistics =============================== > Total packets sent: 0 > Total packets received: 18760640 > Total packets dropped: 0 > ==================================================== > > The question is that > 1. How I can achieve each port receiving full 14.88Mpps ? > What might be the bottleneck in current environment? > 2. Why the performance using multiple ports is not scale? > I guess doubling ports shows the doubling the receiving performance, > but it shows not. I am curious about what is limiting the packet > receivng performance. > > Thanks, > Shinae -- SAKURA Internet Inc. / Senior Researcher Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> SAKURA Internet Research Center <http://research.sakura.ad.jp/> ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <20130528113005.5E6D.C42C3789-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>]
* Re: Performances are not scale with multiple ports [not found] ` <20130528113005.5E6D.C42C3789-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> @ 2013-05-28 3:05 ` Shinae Woo [not found] ` <CA+f=Zzvyfgu8GcdP4R2W6qi0gfEpDhdEctHnhxmqYuTn8U+n3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 8+ messages in thread From: Shinae Woo @ 2013-05-28 3:05 UTC (permalink / raw) To: Naoto MATSUMOTO; +Cc: dev-VfR2kkLFssw [-- Attachment #1: Type: text/plain, Size: 6158 bytes --] Thanks for sharing Naoto. So in your experiments, the forwarding performance still does not reach the line rate. Your perf record shows that the cpu spend most of the time in polling for receiving packets, and no other heavy operation. Even though the application polling packets in its best, the forwarder miss some packets from elsewhere from the application-side. The dpdk document shows that 160Mpps forwarding performance in 2 sockets, but I can only reach the 13 Mpps in 2 ports. Even doubling the number of ports to 4 ports, the performance is still less than 17Mpps. I want to know where is the bottleneck lies in my environments, or how I can reprocuce the same performance as the dpdk published. Thank you, Shinae On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>wrote: > > FYI: Disruptive IP Networking with Intel DPDK on Linux > http://slidesha.re/SeVFZo > > > On Tue, 28 May 2013 11:26:30 +0900 > Shinae Woo <shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > Hello, all. > > > > I play the dpdk-1.2.3r1 with examples. > > > > But I can not achieve the line-rate packet receive performance, > > and the performance is not scale with multiple ports. > > > > For example, in example l2fwd, I have tested two cases with 2 ports, and > 4 > > ports, > > using belowed command line each > > > > ./build/l2fwd -cf -n3 -- -p3 > > ./build/l2fwd -cf -n3 -- -pf > > > > But both cases, the aggregated performance are not scale. > > > > == experiments environments == > > - Two Intel 82599 NICs (total 4 ports) > > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores) > > - 1024 * 2MB hugepages > > - Linux 2.6.38-15-server > > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps. > > > > *1. Packet forwarding performance* > > > > In 2 ports case, receive performance is 13Mpps, > > In 4 ports case, not 26Mbps, only 16.8Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 4292256 > > Packets received: 6517396 > > Packets dropped: 2224776 > > Statistics for port 1 ------------------------------ > > Packets sent: 4291840 > > Packets received: 6517044 > > Packets dropped: 2225556 > > Aggregate statistics =============================== > > Total packets sent: 8584128 > > Total packets received: 13034472 > > Total packets dropped: 4450332 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 1784064 > > Packets received: 2632700 > > Packets dropped: 848128 > > Statistics for port 1 ------------------------------ > > Packets sent: 1784104 > > Packets received: 2632196 > > Packets dropped: 848596 > > Statistics for port 2 ------------------------------ > > Packets sent: 3587616 > > Packets received: 5816344 > > Packets dropped: 2200176 > > Statistics for port 3 ------------------------------ > > Packets sent: 3587712 > > Packets received: 5787848 > > Packets dropped: 2228684 > > Aggregate statistics =============================== > > Total packets sent: 10743560 > > Total packets received: 16869152 > > Total packets dropped: 6125608 > > ==================================================== > > > > *2. Packet receiving performance* > > I fix the codes for only receiving packets (not forwarding), > > the performance is still not scalable as each 13.3Mpps, 18Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 6678860 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 6646120 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 13325012 > > Total packets dropped: 0 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 3129624 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 3131292 > > Packets dropped: 0 > > Statistics for port 2 ------------------------------ > > Packets sent: 0 > > Packets received: 6260908 > > Packets dropped: 0 > > Statistics for port 3 ------------------------------ > > Packets sent: 0 > > Packets received: 6238764 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 18760640 > > Total packets dropped: 0 > > ==================================================== > > > > The question is that > > 1. How I can achieve each port receiving full 14.88Mpps ? > > What might be the bottleneck in current environment? > > 2. Why the performance using multiple ports is not scale? > > I guess doubling ports shows the doubling the receiving performance, > > but it shows not. I am curious about what is limiting the packet > > receivng performance. > > > > Thanks, > > Shinae > > -- > SAKURA Internet Inc. / Senior Researcher > Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> > SAKURA Internet Research Center <http://research.sakura.ad.jp/> > > [-- Attachment #2: Type: text/html, Size: 7852 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <CA+f=Zzvyfgu8GcdP4R2W6qi0gfEpDhdEctHnhxmqYuTn8U+n3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Performances are not scale with multiple ports [not found] ` <CA+f=Zzvyfgu8GcdP4R2W6qi0gfEpDhdEctHnhxmqYuTn8U+n3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-05-28 3:15 ` Emre Eraltan [not found] ` <51A4214B.8040703-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> 2013-05-29 3:09 ` Naoto MATSUMOTO 1 sibling, 1 reply; 8+ messages in thread From: Emre Eraltan @ 2013-05-28 3:15 UTC (permalink / raw) To: Shinae Woo; +Cc: dev-VfR2kkLFssw [-- Attachment #1: Type: text/plain, Size: 7543 bytes --] Hello Shinae, Did you try to use the testpmd tool with multiple queues per port? It gives you more flexibility compared to l2fwd app. You need to trigger the RSS feature of the NIC by sending different streams (just by changing the destination port for instance or any information in the 5-tuple). This will load balance your packets among several cores so that you can probe multiple queues with different cores. Otherwise, you will use only one core (or thread if HT is enabled) per port for the RX side. Best Regards, Emre -- Emre ERALTAN 6WIND Field Application Engineer On 27/05/2013 20:05, Shinae Woo wrote: > Thanks for sharing Naoto. > > So in your experiments, the forwarding performance still does not > reach the line rate. > > Your perf record shows that the cpu spend most of the time in polling > for receiving packets, > and no other heavy operation. > Even though the application polling packets in its best, > the forwarder miss some packets from elsewhere from the application-side. > > The dpdk document shows that 160Mpps forwarding performance in 2 sockets, > but I can only reach the 13 Mpps in 2 ports. > Even doubling the number of ports to 4 ports, the performance is still > less than 17Mpps. > > I want to know where is the bottleneck lies in my environments, or > how I can reprocuce the same performance as the dpdk published. > > Thank you, > Shinae > > > > On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO > <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org <mailto:n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>> wrote: > > > FYI: Disruptive IP Networking with Intel DPDK on Linux > http://slidesha.re/SeVFZo > > > On Tue, 28 May 2013 11:26:30 +0900 > Shinae Woo <shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote: > > > Hello, all. > > > > I play the dpdk-1.2.3r1 with examples. > > > > But I can not achieve the line-rate packet receive performance, > > and the performance is not scale with multiple ports. > > > > For example, in example l2fwd, I have tested two cases with 2 > ports, and 4 > > ports, > > using belowed command line each > > > > ./build/l2fwd -cf -n3 -- -p3 > > ./build/l2fwd -cf -n3 -- -pf > > > > But both cases, the aggregated performance are not scale. > > > > == experiments environments == > > - Two Intel 82599 NICs (total 4 ports) > > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores) > > - 1024 * 2MB hugepages > > - Linux 2.6.38-15-server > > - Each ports receiving 10Gbps of traffic of 64 bytes packets, > 14.88Mpps. > > > > *1. Packet forwarding performance* > > > > In 2 ports case, receive performance is 13Mpps, > > In 4 ports case, not 26Mbps, only 16.8Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 4292256 > > Packets received: 6517396 > > Packets dropped: 2224776 > > Statistics for port 1 ------------------------------ > > Packets sent: 4291840 > > Packets received: 6517044 > > Packets dropped: 2225556 > > Aggregate statistics =============================== > > Total packets sent: 8584128 > > Total packets received: 13034472 > > Total packets dropped: 4450332 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 1784064 > > Packets received: 2632700 > > Packets dropped: 848128 > > Statistics for port 1 ------------------------------ > > Packets sent: 1784104 > > Packets received: 2632196 > > Packets dropped: 848596 > > Statistics for port 2 ------------------------------ > > Packets sent: 3587616 > > Packets received: 5816344 > > Packets dropped: 2200176 > > Statistics for port 3 ------------------------------ > > Packets sent: 3587712 > > Packets received: 5787848 > > Packets dropped: 2228684 > > Aggregate statistics =============================== > > Total packets sent: 10743560 > > Total packets received: 16869152 > > Total packets dropped: 6125608 > > ==================================================== > > > > *2. Packet receiving performance* > > I fix the codes for only receiving packets (not forwarding), > > the performance is still not scalable as each 13.3Mpps, 18Mpps. > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 6678860 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 6646120 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 13325012 > > Total packets dropped: 0 > > ==================================================== > > > > Port statistics ==================================== > > Statistics for port 0 ------------------------------ > > Packets sent: 0 > > Packets received: 3129624 > > Packets dropped: 0 > > Statistics for port 1 ------------------------------ > > Packets sent: 0 > > Packets received: 3131292 > > Packets dropped: 0 > > Statistics for port 2 ------------------------------ > > Packets sent: 0 > > Packets received: 6260908 > > Packets dropped: 0 > > Statistics for port 3 ------------------------------ > > Packets sent: 0 > > Packets received: 6238764 > > Packets dropped: 0 > > Aggregate statistics =============================== > > Total packets sent: 0 > > Total packets received: 18760640 > > Total packets dropped: 0 > > ==================================================== > > > > The question is that > > 1. How I can achieve each port receiving full 14.88Mpps ? > > What might be the bottleneck in current environment? > > 2. Why the performance using multiple ports is not scale? > > I guess doubling ports shows the doubling the receiving > performance, > > but it shows not. I am curious about what is limiting the packet > > receivng performance. > > > > Thanks, > > Shinae > > -- > SAKURA Internet Inc. / Senior Researcher > Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org > <mailto:n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>> > SAKURA Internet Research Center <http://research.sakura.ad.jp/> > > [-- Attachment #2: Type: text/html, Size: 14217 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <51A4214B.8040703-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>]
* Re: Performances are not scale with multiple ports [not found] ` <51A4214B.8040703-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> @ 2013-05-28 3:29 ` Stephen Hemminger [not found] ` <20130527202946.75184090-We1ePj4FEcvRI77zikRAJc56i+j3xesD0e7PPNI6Mm0@public.gmane.org> 0 siblings, 1 reply; 8+ messages in thread From: Stephen Hemminger @ 2013-05-28 3:29 UTC (permalink / raw) To: Emre Eraltan; +Cc: dev-VfR2kkLFssw On Mon, 27 May 2013 20:15:23 -0700 Emre Eraltan <emre.eraltan-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote: > Hello Shinae, > > Did you try to use the testpmd tool with multiple queues per port? It > gives you more flexibility compared to l2fwd app. > > You need to trigger the RSS feature of the NIC by sending different > streams (just by changing the destination port for instance or any > information in the 5-tuple). This will load balance your packets among > several cores so that you can probe multiple queues with different > cores. Otherwise, you will use only one core (or thread if HT is > enabled) per port for the RX side. > > Best Regards, > Emre > I wrote a partial baked version of pktgen for DPDK. Let me see if I can get ok to release and clean it up. Windriver has one as well, and they use it in their demo clases. Not sure why it never got included. ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <20130527202946.75184090-We1ePj4FEcvRI77zikRAJc56i+j3xesD0e7PPNI6Mm0@public.gmane.org>]
* Re: Performances are not scale with multiple ports [not found] ` <20130527202946.75184090-We1ePj4FEcvRI77zikRAJc56i+j3xesD0e7PPNI6Mm0@public.gmane.org> @ 2013-05-28 4:00 ` Shinae Woo 0 siblings, 0 replies; 8+ messages in thread From: Shinae Woo @ 2013-05-28 4:00 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev-VfR2kkLFssw [-- Attachment #1: Type: text/plain, Size: 1624 bytes --] Actually, I made my own packet reception application using DPDK to utilize multiple RSS which map each core (in my case, 12 queues for 12 cores). But it shows that multiple RSS shows, the reception performance is rather decreasing as below. # queue 1 2 3 4 5 6 Bandwidth (Gbps) 8.95 8.90 8.89 8.57 7.95 7.62 #queue 7 8 9 10 11 12 Bandwidth (Gbps) 7.18 7.13 7.12 7.04 7.58 6.80 I will try with testpmd to load balance again. I use the custom packet generator module over Packet I/O, but I'm looking forward to the pktgen for DPDK of Stephen's, too. Thanks, Shinae On Tue, May 28, 2013 at 12:29 PM, Stephen Hemminger < stephen-OTpzqLSitTUnbdJkjeBofR2eb7JE58TQ@public.gmane.org> wrote: > On Mon, 27 May 2013 20:15:23 -0700 > Emre Eraltan <emre.eraltan-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote: > > > Hello Shinae, > > > > Did you try to use the testpmd tool with multiple queues per port? It > > gives you more flexibility compared to l2fwd app. > > > > You need to trigger the RSS feature of the NIC by sending different > > streams (just by changing the destination port for instance or any > > information in the 5-tuple). This will load balance your packets among > > several cores so that you can probe multiple queues with different > > cores. Otherwise, you will use only one core (or thread if HT is > > enabled) per port for the RX side. > > > > Best Regards, > > Emre > > > > I wrote a partial baked version of pktgen for DPDK. Let me see if I can > get ok to release and clean it up. > > Windriver has one as well, and they use it in their demo clases. > Not sure why it never got included. > > [-- Attachment #2: Type: text/html, Size: 4129 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performances are not scale with multiple ports [not found] ` <CA+f=Zzvyfgu8GcdP4R2W6qi0gfEpDhdEctHnhxmqYuTn8U+n3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-05-28 3:15 ` Emre Eraltan @ 2013-05-29 3:09 ` Naoto MATSUMOTO 1 sibling, 0 replies; 8+ messages in thread From: Naoto MATSUMOTO @ 2013-05-29 3:09 UTC (permalink / raw) To: Shinae Woo; +Cc: dev-VfR2kkLFssw Hi shiane and all Emre and Stephen, thanks your suggest for us. We are not testing multi-port traffic gen using NETMAP though DPDK l3fw, but we understood some packet recive-side without NETMAP Linux. Please try it using stephen's pktgen if you have more traffic gen/recive server resources or Using IXIA. best regards, -- Naoto On Tue, 28 May 2013 12:05:50 +0900 Shinae Woo <shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Thanks for sharing Naoto. > > So in your experiments, the forwarding performance still does not reach the > line rate. > > Your perf record shows that the cpu spend most of the time in polling for > receiving packets, > and no other heavy operation. > Even though the application polling packets in its best, > the forwarder miss some packets from elsewhere from the application-side. > > The dpdk document shows that 160Mpps forwarding performance in 2 sockets, > but I can only reach the 13 Mpps in 2 ports. > Even doubling the number of ports to 4 ports, the performance is still less > than 17Mpps. > > I want to know where is the bottleneck lies in my environments, or > how I can reprocuce the same performance as the dpdk published. > > Thank you, > Shinae > > > > On Tue, May 28, 2013 at 11:30 AM, Naoto MATSUMOTO > <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>wrote: > > > > > FYI: Disruptive IP Networking with Intel DPDK on Linux > > http://slidesha.re/SeVFZo > > > > > > On Tue, 28 May 2013 11:26:30 +0900 > > Shinae Woo <shinae2012-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > > Hello, all. > > > > > > I play the dpdk-1.2.3r1 with examples. > > > > > > But I can not achieve the line-rate packet receive performance, > > > and the performance is not scale with multiple ports. > > > > > > For example, in example l2fwd, I have tested two cases with 2 ports, and > > 4 > > > ports, > > > using belowed command line each > > > > > > ./build/l2fwd -cf -n3 -- -p3 > > > ./build/l2fwd -cf -n3 -- -pf > > > > > > But both cases, the aggregated performance are not scale. > > > > > > == experiments environments == > > > - Two Intel 82599 NICs (total 4 ports) > > > - Intel Xeon X5690 @ 3.47GHz * 2 (total 12 cores) > > > - 1024 * 2MB hugepages > > > - Linux 2.6.38-15-server > > > - Each ports receiving 10Gbps of traffic of 64 bytes packets, 14.88Mpps. > > > > > > *1. Packet forwarding performance* > > > > > > In 2 ports case, receive performance is 13Mpps, > > > In 4 ports case, not 26Mbps, only 16.8Mpps. > > > > > > Port statistics ==================================== > > > Statistics for port 0 ------------------------------ > > > Packets sent: 4292256 > > > Packets received: 6517396 > > > Packets dropped: 2224776 > > > Statistics for port 1 ------------------------------ > > > Packets sent: 4291840 > > > Packets received: 6517044 > > > Packets dropped: 2225556 > > > Aggregate statistics =============================== > > > Total packets sent: 8584128 > > > Total packets received: 13034472 > > > Total packets dropped: 4450332 > > > ==================================================== > > > > > > Port statistics ==================================== > > > Statistics for port 0 ------------------------------ > > > Packets sent: 1784064 > > > Packets received: 2632700 > > > Packets dropped: 848128 > > > Statistics for port 1 ------------------------------ > > > Packets sent: 1784104 > > > Packets received: 2632196 > > > Packets dropped: 848596 > > > Statistics for port 2 ------------------------------ > > > Packets sent: 3587616 > > > Packets received: 5816344 > > > Packets dropped: 2200176 > > > Statistics for port 3 ------------------------------ > > > Packets sent: 3587712 > > > Packets received: 5787848 > > > Packets dropped: 2228684 > > > Aggregate statistics =============================== > > > Total packets sent: 10743560 > > > Total packets received: 16869152 > > > Total packets dropped: 6125608 > > > ==================================================== > > > > > > *2. Packet receiving performance* > > > I fix the codes for only receiving packets (not forwarding), > > > the performance is still not scalable as each 13.3Mpps, 18Mpps. > > > > > > Port statistics ==================================== > > > Statistics for port 0 ------------------------------ > > > Packets sent: 0 > > > Packets received: 6678860 > > > Packets dropped: 0 > > > Statistics for port 1 ------------------------------ > > > Packets sent: 0 > > > Packets received: 6646120 > > > Packets dropped: 0 > > > Aggregate statistics =============================== > > > Total packets sent: 0 > > > Total packets received: 13325012 > > > Total packets dropped: 0 > > > ==================================================== > > > > > > Port statistics ==================================== > > > Statistics for port 0 ------------------------------ > > > Packets sent: 0 > > > Packets received: 3129624 > > > Packets dropped: 0 > > > Statistics for port 1 ------------------------------ > > > Packets sent: 0 > > > Packets received: 3131292 > > > Packets dropped: 0 > > > Statistics for port 2 ------------------------------ > > > Packets sent: 0 > > > Packets received: 6260908 > > > Packets dropped: 0 > > > Statistics for port 3 ------------------------------ > > > Packets sent: 0 > > > Packets received: 6238764 > > > Packets dropped: 0 > > > Aggregate statistics =============================== > > > Total packets sent: 0 > > > Total packets received: 18760640 > > > Total packets dropped: 0 > > > ==================================================== > > > > > > The question is that > > > 1. How I can achieve each port receiving full 14.88Mpps ? > > > What might be the bottleneck in current environment? > > > 2. Why the performance using multiple ports is not scale? > > > I guess doubling ports shows the doubling the receiving performance, > > > but it shows not. I am curious about what is limiting the packet > > > receivng performance. > > > > > > Thanks, > > > Shinae > > > > -- > > SAKURA Internet Inc. / Senior Researcher > > Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> > > SAKURA Internet Research Center <http://research.sakura.ad.jp/> > > > > -- SAKURA Internet Inc. / Senior Researcher Naoto MATSUMOTO <n-matsumoto-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org> SAKURA Internet Research Center <http://research.sakura.ad.jp/> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performances are not scale with multiple ports [not found] ` <CA+f=ZztBO=fTqSvj7n8SwKZUNz=fhOq1LcaFMJGJMRhy7xi41w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-05-28 2:30 ` Naoto MATSUMOTO @ 2013-05-28 9:22 ` Thomas Monjalon 1 sibling, 0 replies; 8+ messages in thread From: Thomas Monjalon @ 2013-05-28 9:22 UTC (permalink / raw) To: Shinae Woo; +Cc: dev-VfR2kkLFssw Hello, 28/05/2013 04:26, Shinae Woo : > ./build/l2fwd -cf -n3 -- -p3 > ./build/l2fwd -cf -n3 -- -pf > > But both cases, the aggregated performance are not scale. [..] > The question is that > 1. How I can achieve each port receiving full 14.88Mpps ? > What might be the bottleneck in current environment? > 2. Why the performance using multiple ports is not scale? > I guess doubling ports shows the doubling the receiving performance, > but it shows not. I am curious about what is limiting the packet > receivng performance. In order to scale with line-rate performance, you probably have to adjust the configuration to your hardware. Several parameters should be considered: - PCIe bus (is it fast enough ?) - NUMA sockets (copy between NUMA nodes could slow forwarding) - CPU threads (classic SMP is better than hyperthreading) To make it short, you could check your PCI bus and your CPU configuration. For the latter, /proc/cpuinfo should provide enough information with fields "physical id" and "core id". -- Thomas ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-05-29 3:09 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-28 2:26 Performances are not scale with multiple ports Shinae Woo
[not found] ` <CA+f=ZztBO=fTqSvj7n8SwKZUNz=fhOq1LcaFMJGJMRhy7xi41w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-05-28 2:30 ` Naoto MATSUMOTO
[not found] ` <20130528113005.5E6D.C42C3789-R8x43lSm2JF5knmUyjT4kQ@public.gmane.org>
2013-05-28 3:05 ` Shinae Woo
[not found] ` <CA+f=Zzvyfgu8GcdP4R2W6qi0gfEpDhdEctHnhxmqYuTn8U+n3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-05-28 3:15 ` Emre Eraltan
[not found] ` <51A4214B.8040703-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>
2013-05-28 3:29 ` Stephen Hemminger
[not found] ` <20130527202946.75184090-We1ePj4FEcvRI77zikRAJc56i+j3xesD0e7PPNI6Mm0@public.gmane.org>
2013-05-28 4:00 ` Shinae Woo
2013-05-29 3:09 ` Naoto MATSUMOTO
2013-05-28 9:22 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).