Re: A question about hugepage initialization time

[Neil Horman <nhorman-2XuSBdqkA4R54TAoqtyWWQ@public.gmane.org>]

On Wed, Dec 10, 2014 at 10:32:25AM +0000, Bruce Richardson wrote:
> On Tue, Dec 09, 2014 at 02:10:32PM -0800, Stephen Hemminger wrote:
> > On Tue, 9 Dec 2014 11:45:07 -0800
> > &rew <andras.kovacs-IzeFyvvaP7pWk0Htik3J/w@public.gmane.org> wrote:
> > 
> > > > Hey Folks,
> > > >
> > > > Our DPDK application deals with very large in memory data structures, and
> > > > can potentially use tens or even hundreds of gigabytes of hugepage memory.
> > > > During the course of development, we've noticed that as the number of huge
> > > > pages increases, the memory initialization time during EAL init gets to be
> > > > quite long, lasting several minutes at present.  The growth in init time
> > > > doesn't appear to be linear, which is concerning.
> > > >
> > > > This is a minor inconvenience for us and our customers, as memory
> > > > initialization makes our boot times a lot longer than it would otherwise
> > > > be.  Also, my experience has been that really long operations often are
> > > > hiding errors - what you think is merely a slow operation is actually a
> > > > timeout of some sort, often due to misconfiguration. This leads to two
> > > > questions:
> > > >
> > > > 1. Does the long initialization time suggest that there's an error
> > > > happening under the covers?
> > > > 2. If not, is there any simple way that we can shorten memory
> > > > initialization time?
> > > >
> > > > Thanks in advance for your insights.
> > > >
> > > > --
> > > > Matt Laswell
> > > > laswell-bIuJOMs36aleGPcbtGPokg@public.gmane.org
> > > > infinite io, inc.
> > > >
> > > 
> > > Hello,
> > > 
> > > please find some quick comments on the questions:
> > > 1.) By our experience long initialization time is normal in case of 
> > > large amount of memory. However this time depends on some things:
> > > - number of hugepages (pagefault handled by kernel is pretty expensive)
> > > - size of hugepages (memset at initialization)
> > > 
> > > 2.) Using 1G pages instead of 2M will reduce the initialization time 
> > > significantly. Using wmemset instead of memset adds an additional 20-30% 
> > > boost by our measurements. Or, just by touching the pages but not cleaning 
> > > them you can have still some more speedup. But in this case your layer or 
> > > the applications above need to do the cleanup at allocation time 
> > > (e.g. by using rte_zmalloc).
> > > 
> > > Cheers,
> > > &rew
> > 
> > I wonder if the whole rte_malloc code is even worth it with a modern kernel
> > with transparent huge pages? rte_malloc adds very little value and is less safe
> > and slower than glibc or other allocators. Plus you lose the ablilty to get
> > all the benefit out of valgrind or electric fence.
> 
> While I'd dearly love to not have our own custom malloc lib to maintain, for DPDK
> multiprocess, rte_malloc will be hard to replace as we would need a replacement
> solution that similarly guarantees that memory mapped in process A is also 
> available at the same address in process B. :-(
> 
Just out of curiosity, why even bother with multiprocess support?  What you're
talking about above is a multithread model, and your shoehorning multiple
processes into it.
Neil

> /Bruce
>