Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions

[Thomas Monjalon <thomas@monjalon.net>]

08/11/2023 14:53, Dmitry Kozlyuk:
> 2023-11-07 23:03 (UTC+0530), rahul gupta:
> > > > From: Rahul Gupta <rahulgupt@linux.microsoft.com>
> > > > To: dev@dpdk.org,  thomas@monjalon.net
> > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org,  
> > > sujithsankar@microsoft.com,  sowmini.varadhan@microsoft.com,
> > > rahulrgupta27@gmail.com,  Rahul Gupta <rahulgupt@microsoft.com>,  Rahul
> > > Gupta <rahulgupt@linux.microsoft.com>  
> > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions
> > > > Date: Thu,  2 Nov 2023 11:19:24 -0700
> > > > X-Mailer: git-send-email 1.8.3.1
> > > >
> > > > From: Rahul Gupta <rahulgupt@microsoft.com>
> > > >
> > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create
> > > > which can consume a total time of 500-600 ms:
> > > > a) For many devices FLR may take a significant chunk of time
> > > >    (200-250 ms in our use-case), this FLR is triggered during device
> > > >    probe in rte_eal_init().
> > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for
> > > > applications that require huge memory.
> > > >
> > > > This cost is incurred on each restart (which happens in our use-case
> > > > during binary updates for servicing).
> > > > This patch provides an optimization using pthreads that appplications
> > > > can use and which can save 200-230ms.
> > > >
> > > > In this patch, rte_eal_init() is refactored into two parts-
> > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and
> > > >    mempool creation. So this code needs to be executed before any
> > > >    pthreads. Its named as rte_eal_init_setup()
> > > > b) 2nd part of code is independent code ie- it can execute in parallel
> > > >    to mempool creation in a pthread. Its named as rte_probe_and_ioctl().
> > > >
> > > > Existing applications require no changes unless they wish to leverage
> > > > the optimization.
> > > >
> > > > If the application wants to use pthread functionality, it should call-
> > > > a) rte_eal_init_setup() then create two or more pthreads-
> > > > b) in one pthread call- rte_probe_and_ioctl(),
> > > > c) second pthread call- rte_pktmbuf_pool_create()
> > > > d) (optional) Other pthreads for  any other independent function.
> > > >
> > > > Signed-off-by: Rahul Gupta <rahulgupt@linux.microsoft.com>  
> 
> I doubt that the new API is required.
> It is already possible to block all devices from automatic probing
> with EAL options and then probe explicitly in any threads desired.
> At the same time, this RFC shows a valuable optimization pattern,
> so maybe it is worth having in DPDK as an example.
> There are DPDK use cases when probing is completely unnecessary.

It seems here we want to do the device probing,
but start it in parallel of other tasks.

> Exposing the initialization process stages makes it harder to refactor
> and requires precise documentation of when and what is initialized
> (for example, in this RFC rte_eal_init_setup()
> does not make service core API usable yet).

Yes the init order is sensitive, that's why we have a big init function.
But in general I would agree to try splitting it with necessary warnings
and explanations.

> P. S. You may be also interested in using `--huge-unlink=never`
> to speed rte_pktmbuf_pool_create() during restarts:
> 
> https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3

Yes good tip :)