From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vlad Zolotarov Subject: Re: i40e xmit path HW limitation Date: Thu, 30 Jul 2015 20:56:13 +0300 Message-ID: <55BA653D.5060109@cloudius-systems.com> References: <55BA3B5D.4020402@cloudius-systems.com> <55BA5468.80109@cloudius-systems.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Cc: "dev@dpdk.org" To: "Zhang, Helin" , "Ananyev, Konstantin" Return-path: Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) by dpdk.org (Postfix) with ESMTP id 7ECBDC6BC for ; Thu, 30 Jul 2015 19:56:16 +0200 (CEST) Received: by wibud3 with SMTP id ud3so30464269wib.1 for ; Thu, 30 Jul 2015 10:56:16 -0700 (PDT) In-Reply-To: List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 07/30/15 20:33, Zhang, Helin wrote: > >> -----Original Message----- >> From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com] >> Sent: Thursday, July 30, 2015 9:44 AM >> To: Zhang, Helin; Ananyev, Konstantin >> Cc: dev@dpdk.org >> Subject: Re: i40e xmit path HW limitation >> >> >> >> On 07/30/15 19:10, Zhang, Helin wrote: >>>> -----Original Message----- >>>> From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com] >>>> Sent: Thursday, July 30, 2015 7:58 AM >>>> To: dev@dpdk.org; Ananyev, Konstantin; Zhang, Helin >>>> Subject: RFC: i40e xmit path HW limitation >>>> >>>> Hi, Konstantin, Helin, >>>> there is a documented limitation of xl710 controllers (i40e driver) >>>> which is not handled in any way by a DPDK driver. >>>> From the datasheet chapter 8.4.1: >>>> >>>> "=E2=80=A2 A single transmit packet may span up to 8 buffers (up to = 8 data >>>> descriptors per packet including both the header and payload buffers= ). >>>> =E2=80=A2 The total number of data descriptors for the whole TSO (ex= plained >>>> later on in this chapter) is unlimited as long as each segment withi= n >>>> the TSO obeys the previous rule (up to 8 data descriptors per segmen= t >>>> for both the TSO header and the segment payload buffers)." >>> Yes, I remember the RX side just supports 5 segments per packet recei= ving. >>> But what's the possible issue you thought about? >> Note that it's a Tx size we are talking about. >> >> See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next = repo. >> If such a cluster arrives and you post it on the HW ring - HW will shu= t this HW ring >> down permanently. The application will see that it's ring is stuck. > That issue was because of using more than 8 descriptors for a packet fo= r TSO. There is no problem in transmitting the TSO packet with more than 8=20 fragments. On the opposite - one can't transmit a non-TSO packet with more than 8=20 fragments. One also can't transmit the TSO packet that would contain more than 8=20 fragments in a single TSO segment including the TSO headers. Pls., read the HW spec as I quoted above for more details. > >>>> This means that, for instance, long cluster with small fragments has= to be >>>> linearized before it may be placed on the HW ring. >>> What type of size of the small fragments? Basically 2KB is the defaul= t size of >> mbuf of most >>> example applications. 2KB x 8 is bigger than 1.5KB. So it is enough f= or the >> maximum >>> packet size we supported. >>> If 1KB mbuf is used, don't expect it can transmit more than 8KB size = of packet. >> I kinda lost u here. Again, we talk about the Tx side here and buffers >> are not obligatory completely filled. Namely there may be a cluster wi= th >> 15 fragments 100 bytes each. > The root cause is using more than 8 descriptors for a packet. That would be if u would like to SUPER simplify the HW limitation above.=20 In that case u would significantly limit the different packets that may=20 be sent without the linearization. > Linux driver can help > on reducing number of descriptors to be used by merging small size of p= ayload > together, right? > It is not for TSO, it is just for packet transmitting. 2 options in my = mind: > 1. Use should ensure it will not use more than 8 descriptors per packet= for transmitting. This requirement is too restricting. Pls., see above. > 2. DPDK driver should try to merge small packet together for such case,= like Linux kernel driver. > I prefer to use option 1, users should ensure that in the application o= r up layer software, > and keep the PMD driver as simple as possible. The above statement is super confusing: on the one hand u suggest the=20 DPDK driver to merge the small packet (fragments?) together (how?) and=20 then u immediately propose the user application to do that. Could u,=20 pls., clarify what exactly u suggest here? If that's to leave it to the application - note that it would demand=20 patching all existing DPDK applications that send TCP packets. > > But I have a thought that the maximum number of RX/TX descriptor should= be able to be > queried somewhere. There is no such thing as maximum number of Tx fragments in a TSO case.=20 It's only limited by the Tx ring size. > > Regards, > Helin >>>> In more standard environments like Linux or FreeBSD drivers the solu= tion is >>>> straight forward - call skb_linearize()/m_collapse() corresponding. >>>> In the non-conformist environment like DPDK life is not that easy - = there is no >>>> easy way to collapse the cluster into a linear buffer from inside th= e device >> driver >>>> since device driver doesn't allocate memory in a fast path and utili= zes the user >>>> allocated pools only. >>>> Here are two proposals for a solution: >>>> >>>> 1. We may provide a callback that would return a user TRUE if a g= ive >>>> cluster has to be linearized and it should always be called be= fore >>>> rte_eth_tx_burst(). Alternatively it may be called from inside= the >>>> rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return= some >>>> error code for a case when one of the clusters it's given has = to be >>>> linearized. >>>> 2. Another option is to allocate a mempool in the driver with the >>>> elements consuming a single page each (standard 2KB buffers wo= uld >>>> do). Number of elements in the pool should be as Tx ring lengt= h >>>> multiplied by "64KB/(linear data length of the buffer in the p= ool >>>> above)". Here I use 64KB as a maximum packet length and not ta= king >>>> into an account esoteric things like "Giant" TSO mentioned in = the >>>> spec above. Then we may actually go and linearize the cluster = if >>>> needed on top of the buffers from the pool above, post the buf= fer >>>> from the mempool above on the HW ring, link the original clust= er to >>>> that new cluster (using the private data) and release it when = the >>>> send is done. >>>> >>>> >>>> The first is a change in the API and would require from the applicat= ion some >>>> additional handling (linearization). The second would require some a= dditional >>>> memory but would keep all dirty details inside the driver and would = leave the >>>> rest of the code intact. >>>> >>>> Pls., comment. >>>> >>>> thanks, >>>> vlad >>>>