From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Hemminger Subject: Re: Polling too often at lower packet rates? Date: Wed, 8 Apr 2015 11:00:58 -0700 Message-ID: References: <5D6C8629-393C-4195-8063-8168E206335B@arbor.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "dev-VfR2kkLFssw@public.gmane.org" To: Aaron Campbell Return-path: In-Reply-To: <5D6C8629-393C-4195-8063-8168E206335B-rd7evPjynkNeoWH0uzbU5w@public.gmane.org> List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces-VfR2kkLFssw@public.gmane.org Sender: "dev" We use adaptive polling loop similar to l3fwd-power example. See: http://video.fosdem.org/2015/devroom-network_management_and_sdn/ On Wed, Apr 8, 2015 at 9:35 AM, Aaron Campbell wrote: > Hi, > > I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps traffi= c > offered to only one of the 10G ports (the other 5 are unused). I also ha= ve > a program with a pretty standard looking DPDK receive loop, where it call= s > rte_eth_rx_burst() for each configured port. If I configure the loop to > read from all 6 ports, it can read the 1.23Mpps rate with no drops. If I > configure the loop to poll only 1 port (the ixgbe receiving the traffic),= I > lose about 1/3rd of the packets (i.e., the NIC drops ~400Kpps). > > Another data point is that if I configure the loop to read from 3 out of > the 6 ports, the drop rate is reduced to less than half (i.e., the NIC is > only dropping ~190Kpps now). So it seems that in this test, throughput > improves by adding NICs, not removing them, which is counter-intuitive. > Again, I get no drops when polling all 6 ports. Note, the burst size is = 32. > > I did find a reference to a similar issue in a recent paper ( > http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf= ), > Section III, which states: > > "The DPDK L2FWD application initially only managed to forward 13.8 Mpps i= n > the single direction test at the maximum CPU frequency, a similar result > can be found in [11]. Reducing the CPU frequency increased the throughput > to the expected value of 14.88 Mpps. Our investigation of this anomaly > revealed that the lack of any processing combined with the fast CPU cause= d > DPDK to poll the NIC too often. DPDK does not use interrupts, it utilizes= a > busy wait loop that polls the NIC until at least one packet is returned. > This resulted in a high poll rate which affected the throughput. We limit= ed > the poll rate to 500,000 poll operations per second (i.e., a batch size o= f > about 30 packets) and achieved line rate in the unidirectional test with > all frequencies. This effect was only observed with the X520 NIC, tests > with X540 NICs did not show this anomaly.=E2=80=9D > > Another reference, from this mailing list last year ( > http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html): > > "I suggest you to check average burst sizes on receive queues. Looks like > I stumbled upon a similar issue several times. If you are calling > rte_eth_rx_burst too frequently, NIC begins losing packets no matter how > many CPU horse power you have (more you have, more it loses, actually). I= n > my case this situation occured when average burst size is less than 20 > packets or so. I'm not sure what's the reason for this behavior, but I > observed it on several applications on Intel 82599 10Gb cards.=E2=80=9D > > So I=E2=80=99m wondering if anyone can explain at a lower level what happ= ens when > you poll =E2=80=9Ctoo often=E2=80=9D, and if there are any practical work= arounds. We=E2=80=99re > using this same program and DPDK version to process 10G line-rate in othe= r > scenarios, so I=E2=80=99m confident that the overall packet capture archi= tecture is > sound. > > -Aaron