From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: Beyond DPDK 2.0 Date: Thu, 07 May 2015 19:05:28 +0300 Message-ID: <554B8D48.7010900@cloudius-systems.com> References: <26FA93C7ED1EAA44AB77D62FBE1D27BA54D1A917@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54D29B55@IRSMSX102.ger.corp.intel.com> <554B85D5.6010808@cloudius-systems.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-2; format=flowed Content-Transfer-Encoding: quoted-printable Cc: "dev-VfR2kkLFssw@public.gmane.org" To: "Wiles, Keith" , "O'Driscoll, Tim" Return-path: In-Reply-To: List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces-VfR2kkLFssw@public.gmane.org Sender: "dev" On 05/07/2015 06:49 PM, Wiles, Keith wrote: > > On 5/7/15, 8:33 AM, "Avi Kivity" wrote: > >> On 05/07/2015 06:27 PM, Wiles, Keith wrote: >>> On 5/7/15, 7:02 AM, "Avi Kivity" wrote: >>> >>>> On Wed, Apr 22, 2015 at 6:11 PM, O'Driscoll, Tim >>>> >>>> wrote: >>>> >>>>> Does anybody have any input or comments on this? >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: O'Driscoll, Tim >>>>>> Sent: Thursday, April 16, 2015 11:39 AM >>>>>> To: dev-VfR2kkLFssw@public.gmane.org >>>>>> Subject: Beyond DPDK 2.0 >>>>>> >>>>>> Following the launch of DPDK by Intel as an internal development >>>>>> project, the launch of dpdk.org by 6WIND in 2013, and the first DP= DK >>>>> RPM >>>>>> packages for Fedora in 2014, 6WIND, Red Hat and Intel would like t= o >>>>>> prepare for future releases after DPDK 2.0 by starting a discussio= n >>>>>> on >>>>>> its evolution. Anyone is welcome to join this initiative. >>>>>> >>>>>> Since then, the project has grown significantly: >>>>>> - The number of commits and mailing list posts has increased >>>>>> steadily. >>>>>> - Support has been added for a wide range of new NICs (Mellanox >>>>>> support submitted by 6WIND, Cisco VIC, Intel i40e and fm10k etc.). >>>>>> - DPDK is now supported on multiple architectures (IBM Power >>>>> support >>>>>> in DPDK 1.8, Tile support submitted by EZchip but not yet reviewed= or >>>>>> applied). >>>>>> >>>>>> While this is great progress, we need to make sure that the projec= t >>>>>> is >>>>>> structured in a way that enables it to continue to grow. To achiev= e >>>>>> this, 6WIND, Red Hat and Intel would like to start a discussion ab= out >>>>>> the future of the project, so that we can agree and establish >>>>> processes >>>>>> that satisfy the needs of the current and future DPDK community. >>>>>> >>>>>> We're very interested in hearing the views of everybody in the >>>>>> community. In addition to debate on the mailing list, we'll also >>>>>> schedule community calls to discuss this. >>>>>> >>>>>> >>>>>> Project Goals >>>>>> ------------- >>>>>> >>>>>> Some topics to be considered for the DPDK project include: >>>>>> - Project Charter: The charter of the DPDK project should be >>>>> clearly >>>>>> defined, and should explain the limits of DPDK (what it does and d= oes >>>>>> not cover). This does not mean that we would be stuck with a singu= lar >>>>>> charter for all time, but the direction and intent of the project >>>>> should >>>>>> be well understood. >>>> One problem we've seen with dpdk is that it is a framework, not a >>>> library: >>>> it wants to create threads, manage memory, and generally take over. >>>> This >>>> is a problem for us, as we are writing a framework (seastar, [1]) an= d >>>> need >>>> to create threads, manage memory, and generally take over ourselves. >>>> >>>> Perhaps dpdk can be split into two layers, a library layer that only >>>> provides mechanisms, and a framework layer that glues together those >>>> mechanisms and applies a policy, trading in generality for ease of u= se. >>> The DPDK system is somewhat divided now between the EAL, PMDS and >>> utility >>> functions like malloc/rings/=A9 >>> >>> The problem I see is the PMDs need a framework to be usable and the E= AL >>> plus the ethdev layers provide that support today. Setting up and >>> initializing the DPDK system is pretty clean just call the EAL init >>> routines along with the pool creates and the basic configs for the >>> PMDs/hardware. Once the system is inited one can create new threads a= nd >>> not requiring anyone to use DPDK launch routines. Maybe I am not >>> understanding your needs can you explain more? >> An initialization routine that accepts argc/argv can hardly be called >> clean. > You want a config file or structure initialization design? If that is t= he > case you can contribute that support as another way to initialize DPDK. A config file would be even worse. But we are discussing why=20 dpdk-as-a-framework is detrimental, not new ways for me to contribute. >> In seastar, we have our own malloc() (since seastar is sharded we can >> provide a faster thread-unsafe malloc implementation). We also have o= ur >> own threading, and since dpdk is an optional component in seastar, dpd= k >> support requires code duplication. > DPDK replies one the huge page support for allocation to get the > performance, do you also not require huge page support. Sorry, is this a question? Please rephrase. > The malloc system > in DPDK can be used as a replacement for the standard malloc if that wo= rks > for your needs. Also after DPDK inits you can use your own malloc and a= ny > other tools you want to use. How is memory partitioned between dpdk and my application? If I=20 underallocate dpdk memory, something bad will happen. If I overallocate=20 dpdk memory, then I am depriving my application of this memory. A=20 common pool means I do not overallocate or underallocate, but since dpdk=20 insists on managing its own pools, I can't do this. > I do not see a lot of duplicate code here > IMO. I guess if you are installing into a very small memory system then > yes it could be a problem, but DPDK is was not designed to run in a sys= tem > with limited memory. I am not talking about duplicate code, but about duplicate=20 functionality, done slightly differently. I want to use dpdk in the=20 same way as I use every other library, by calling its initialization=20 routine and then calling its functions. In this scenario, the library=20 is passive, only reacting to my calls. The way dpdk works now is=20 actively, taking over resources, creating thread, and calling my code=20 instead of the other way round. > >> I would like to launch my own threads, pin them where I like, and call >> PMD drivers to send and receive packets. Practically everything else >> that dpdk does gets in my way, including mbuf pools. I'd much prefer = to >> allocate mbufs myself. > You do not need to use the lauching of threads in the EAL and can suppl= y > your own, right? Right. > Regards, > ++Keith >> >>>> [1] http://seastar-project.org