From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zoltan Kiss Subject: Re: Calling rte_eth_rx_burst() multiple times Date: Thu, 15 Oct 2015 12:51:47 +0100 Message-ID: <561F9353.8080306@linaro.org> References: <561F64BA.2000502@ndsl.kaist.edu> <561F7E8C.1080508@linaro.org> <561F835F.3020300@ndsl.kaist.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable To: Younghwan Go , dev@dpdk.org Return-path: Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by dpdk.org (Postfix) with ESMTP id 774298E8D for ; Thu, 15 Oct 2015 13:51:47 +0200 (CEST) Received: by wicgb1 with SMTP id gb1so268770595wic.1 for ; Thu, 15 Oct 2015 04:51:47 -0700 (PDT) In-Reply-To: <561F835F.3020300@ndsl.kaist.edu> List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 15/10/15 11:43, Younghwan Go wrote: > Hi Zoltan, > > Thanks for the email. > > 2015-10-15 =EC=98=A4=ED=9B=84 7:23=EC=97=90 Zoltan Kiss =EC=9D=B4(=EA=B0= =80) =EC=93=B4 =EA=B8=80: >> >> >> On 15/10/15 09:32, Younghwan Go wrote: >>> Hi, >>> >>> I'm pretty new to playing with DPDK. I was trying to see if I can alw= ays >>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple time= s >>> on same pair (code shown below). I'm using DPDK-2.1.0 o= n 2 >>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel >>> 3.13.0-63-generic). >>> >>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40 >>> Gbps), I assumed rte_eth_rx_burst() would always receive some number = of >>> packets, eventually filling up MAX_BURST. But for multi-core case (4 >>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after so= me >>> time, causing all cores to be blocked forever. Analyzing the DPDK cod= e >>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside >>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" alway= s >>> returns 0 (where is this value set by the way?). >> >> I think it is set by the hardware. >> >>> >>> I didn't see this problem for single-core case, in which it returned >>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break = out >>> of while loop when I receive 0, I keep receiving packets in next >> queue> pairs. Does anyone know why this block might happen? Or am I n= ot >>> allowed to call rte_eth_rx_burst() multiple times on same >>> pair if I get 0? Any help will be great! Thank you! >> >> Although not mentioned in the documentation itself, as far as I know >> rte_eth_rx_burst() is not thread-safe. If you look in to receive >> functions, there are no locking anywhere. You should call it on >> separate queues from different threads, and configure e.g RSS to >> distribute the traffic by the hardware. > > I'm calling rte_eth_rx_burst() on separate queue ids for each thread. > I'm actually using lcore_id (=3D 0, 1, 2, 3 for 4 threads pinned to eac= h > separate CPU core) as queue_id. I also made sure that this problem is > not caused by threads conflicting by locking before calling > rte_eth_rx_burst(). > > For RSS, I configured with ETH_RSS_IP for load balancing traffic to eac= h > port and queue. But even if RSS wasn't set, shouldn't at least one core > be receiving packets? What I'm seeing is all threads getting stuck at > rte_eth_rx_burst() with return value of 0s indefinitely. > >>> >>> ---------------------------------------------------------------------= --- >>> int cnt =3D MAX_BURST; // MAX_BURST =3D 32 >>> int off =3D 0; >>> do { >>> ret =3D rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); Another thing which might cause your problem is that I don't see where=20 do you release the buffers after received into m_table. You need to call=20 rte_pktmbuf_free() on them at some point, otherwise your pool can get=20 depleted, and the receive function can't refill the descriptor rings. >>> if (ret =3D=3D 0) { >>> // don't break out but continue >>> } else if (ret > 0) { >>> off +=3D ret; >>> cnt -=3D ret; >>> } >>> } while (cnt > 0); >>> ---------------------------------------------------------------------= --- >>> >>> Best, >>> Younghwan > > Thanks, > Younghwan