Re: [SPDK] OS memory allocation

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4074 bytes --]

Hi Paul

On Thu, 2018-07-12 at 17:43 +0000, Luse, Paul E wrote:
> Ben, isn't it also the case that the the spdk_malloc routines (I think all of
> them ) return mem from pre-allocated huge pages.  In cases where that's not
> required, there's no reason to use that scarce resource (depending on how you
> configured it) so it's better to use malloc/calloc.  Right?

Absolutely - spdk_malloc and its variants return DMA-safe and also potentially
cross-process shared memory. They are defined in include/spdk/env.h because
those types of memory allocations are not operations defined in the POSIX API.
DMA-safe memory is certainly scarce and often much less flexible in terms of
dynamic allocation, so we only allocate DMA-safe memory when we actually need to
perform DMA operations. For "regular" memory allocations, we just use POSIX
everywhere.

> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Thursday, July 12, 2018 10:41 AM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] OS memory allocation
> 
> Hi Leonid,
> 
> On Thu, 2018-07-12 at 07:10 +0000, Ravich, Leonid wrote:
> > Looks like SPDK uses OS dynamic memory  (malloc, zalloc ...) I am not 
> > understand why , It can use spdk_malloc/ spdk_zmalloc instead and be 
> > more "system friendly" .
> > 
> > If there is some good reason for such behavior I would like to suggest 
> > wrapping all the "raw"  linux system calls used by SPDK .
> > 
> > This way SPDK will be more flexible in different systems integrations .
> 
> 
> SPDK very intentionally depends on the POSIX API for a large number of
> operations. We've chosen to go this direction because SPDK is intended to run
> as regular a user space application and we want developers to be able to use
> all of the regular tools available in that environment. We're concerned that
> not allowing the use of POSIX APIs makes it much more difficult for people to
> jump in and contribute to the code, and also that it would be quite difficult
> to enforce in practice. Note that this is also the choice that DPDK has made.
> 
> We have taken some steps to support more exotic environments though. First,
> all POSIX include files are only included via include/spdk/stdinc.h, and we
> actually have tests to verify that this is the case. If you have a non-POSIX
> environment, you can swap out that header to include other implementations and
> map them to the POSIX calls we make.
> 
> SPDK also requires a number of operations that are outside of the purview of
> POSIX. These are all encapsulated by include/spdk/env.h. The implementation of
> that header is in lib/env_dpdk, and by default uses DPDK to implement all of
> those operations. However, this portion is designed to be swapped out. You can
> point SPDK at a different implementation of include/spdk/env.h through the
> configuration script:
> 
> ./configure --with-env=/path/to/your/lib.a
> 
> We further have a number of abstractions coming for threading that aren't
> quite done yet. SPDK is designed around cooperative multi-tasking and many
> locations in the code need to be able to send messages to other threads.
> However, SPDK doesn't want to force users into a particular green
> thread/cooperative multitasking/futures+promises framework. We're currently
> designing abstractions for "light weight" threads (without an associated
> implementation) that users should be able to map trivially to their particular
> framework.
> 
> I hope that helps clear things up. It's all about making SPDK as flexible as
> possible for as many people as we can, but also making the "simple" case as
> easy to program and use as possible.
> 
> Thanks,
> Ben
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk