Re: [SPDK] OS memory allocation

Re: [SPDK] OS memory allocation

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4074 bytes --]

Hi Paul

On Thu, 2018-07-12 at 17:43 +0000, Luse, Paul E wrote:
> Ben, isn't it also the case that the the spdk_malloc routines (I think all of
> them ) return mem from pre-allocated huge pages.  In cases where that's not
> required, there's no reason to use that scarce resource (depending on how you
> configured it) so it's better to use malloc/calloc.  Right?

Absolutely - spdk_malloc and its variants return DMA-safe and also potentially
cross-process shared memory. They are defined in include/spdk/env.h because
those types of memory allocations are not operations defined in the POSIX API.
DMA-safe memory is certainly scarce and often much less flexible in terms of
dynamic allocation, so we only allocate DMA-safe memory when we actually need to
perform DMA operations. For "regular" memory allocations, we just use POSIX
everywhere.

> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Thursday, July 12, 2018 10:41 AM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] OS memory allocation
> 
> Hi Leonid,
> 
> On Thu, 2018-07-12 at 07:10 +0000, Ravich, Leonid wrote:
> > Looks like SPDK uses OS dynamic memory  (malloc, zalloc ...) I am not 
> > understand why , It can use spdk_malloc/ spdk_zmalloc instead and be 
> > more "system friendly" .
> > 
> > If there is some good reason for such behavior I would like to suggest 
> > wrapping all the "raw"  linux system calls used by SPDK .
> > 
> > This way SPDK will be more flexible in different systems integrations .
> 
> 
> SPDK very intentionally depends on the POSIX API for a large number of
> operations. We've chosen to go this direction because SPDK is intended to run
> as regular a user space application and we want developers to be able to use
> all of the regular tools available in that environment. We're concerned that
> not allowing the use of POSIX APIs makes it much more difficult for people to
> jump in and contribute to the code, and also that it would be quite difficult
> to enforce in practice. Note that this is also the choice that DPDK has made.
> 
> We have taken some steps to support more exotic environments though. First,
> all POSIX include files are only included via include/spdk/stdinc.h, and we
> actually have tests to verify that this is the case. If you have a non-POSIX
> environment, you can swap out that header to include other implementations and
> map them to the POSIX calls we make.
> 
> SPDK also requires a number of operations that are outside of the purview of
> POSIX. These are all encapsulated by include/spdk/env.h. The implementation of
> that header is in lib/env_dpdk, and by default uses DPDK to implement all of
> those operations. However, this portion is designed to be swapped out. You can
> point SPDK at a different implementation of include/spdk/env.h through the
> configuration script:
> 
> ./configure --with-env=/path/to/your/lib.a
> 
> We further have a number of abstractions coming for threading that aren't
> quite done yet. SPDK is designed around cooperative multi-tasking and many
> locations in the code need to be able to send messages to other threads.
> However, SPDK doesn't want to force users into a particular green
> thread/cooperative multitasking/futures+promises framework. We're currently
> designing abstractions for "light weight" threads (without an associated
> implementation) that users should be able to map trivially to their particular
> framework.
> 
> I hope that helps clear things up. It's all about making SPDK as flexible as
> possible for as many people as we can, but also making the "simple" case as
> easy to program and use as possible.
> 
> Thanks,
> Ben
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] OS memory allocation

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 3210 bytes --]

Ben, isn't it also the case that the the spdk_malloc routines (I think all of them ) return mem from pre-allocated huge pages.  In cases where that's not required, there's no reason to use that scarce resource (depending on how you configured it) so it's better to use malloc/calloc.  Right?

Thx
Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Thursday, July 12, 2018 10:41 AM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] OS memory allocation

Hi Leonid,

On Thu, 2018-07-12 at 07:10 +0000, Ravich, Leonid wrote:
> Looks like SPDK uses OS dynamic memory  (malloc, zalloc ...) I am not 
> understand why , It can use spdk_malloc/ spdk_zmalloc instead and be 
> more "system friendly" .
> 
> If there is some good reason for such behavior I would like to suggest 
> wrapping all the "raw"  linux system calls used by SPDK .
> 
> This way SPDK will be more flexible in different systems integrations .


SPDK very intentionally depends on the POSIX API for a large number of operations. We've chosen to go this direction because SPDK is intended to run as regular a user space application and we want developers to be able to use all of the regular tools available in that environment. We're concerned that not allowing the use of POSIX APIs makes it much more difficult for people to jump in and contribute to the code, and also that it would be quite difficult to enforce in practice. Note that this is also the choice that DPDK has made.

We have taken some steps to support more exotic environments though. First, all POSIX include files are only included via include/spdk/stdinc.h, and we actually have tests to verify that this is the case. If you have a non-POSIX environment, you can swap out that header to include other implementations and map them to the POSIX calls we make.

SPDK also requires a number of operations that are outside of the purview of POSIX. These are all encapsulated by include/spdk/env.h. The implementation of that header is in lib/env_dpdk, and by default uses DPDK to implement all of those operations. However, this portion is designed to be swapped out. You can point SPDK at a different implementation of include/spdk/env.h through the configuration script:

./configure --with-env=/path/to/your/lib.a

We further have a number of abstractions coming for threading that aren't quite done yet. SPDK is designed around cooperative multi-tasking and many locations in the code need to be able to send messages to other threads. However, SPDK doesn't want to force users into a particular green thread/cooperative multitasking/futures+promises framework. We're currently designing abstractions for "light weight" threads (without an associated implementation) that users should be able to map trivially to their particular framework.

I hope that helps clear things up. It's all about making SPDK as flexible as possible for as many people as we can, but also making the "simple" case as easy to program and use as possible.

Thanks,
Ben
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] OS memory allocation

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

Hi Leonid,

On Thu, 2018-07-12 at 07:10 +0000, Ravich, Leonid wrote:
> Looks like SPDK uses OS dynamic memory  (malloc, zalloc ...) I am not
> understand why , 
> It can use spdk_malloc/ spdk_zmalloc instead and be more "system friendly" . 
> 
> If there is some good reason for such behavior I would like to suggest
> wrapping all the "raw"  linux system calls used by SPDK .
> 
> This way SPDK will be more flexible in different systems integrations .


SPDK very intentionally depends on the POSIX API for a large number of
operations. We've chosen to go this direction because SPDK is intended to run as
regular a user space application and we want developers to be able to use all of
the regular tools available in that environment. We're concerned that not
allowing the use of POSIX APIs makes it much more difficult for people to jump
in and contribute to the code, and also that it would be quite difficult to
enforce in practice. Note that this is also the choice that DPDK has made.

We have taken some steps to support more exotic environments though. First, all
POSIX include files are only included via include/spdk/stdinc.h, and we actually
have tests to verify that this is the case. If you have a non-POSIX environment,
you can swap out that header to include other implementations and map them to
the POSIX calls we make.

SPDK also requires a number of operations that are outside of the purview of
POSIX. These are all encapsulated by include/spdk/env.h. The implementation of
that header is in lib/env_dpdk, and by default uses DPDK to implement all of
those operations. However, this portion is designed to be swapped out. You can
point SPDK at a different implementation of include/spdk/env.h through the
configuration script:

./configure --with-env=/path/to/your/lib.a

We further have a number of abstractions coming for threading that aren't quite
done yet. SPDK is designed around cooperative multi-tasking and many locations
in the code need to be able to send messages to other threads. However, SPDK
doesn't want to force users into a particular green thread/cooperative
multitasking/futures+promises framework. We're currently designing abstractions
for "light weight" threads (without an associated implementation) that users
should be able to map trivially to their particular framework.

I hope that helps clear things up. It's all about making SPDK as flexible as
possible for as many people as we can, but also making the "simple" case as easy
to program and use as possible.

Thanks,
Ben

[SPDK] OS memory allocation

[Ravich, Leonid]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

Looks like SPDK uses OS dynamic memory  (malloc, zalloc ...) I am not understand why , 
It can use spdk_malloc/ spdk_zmalloc instead and be more "system friendly" . 

If there is some good reason for such behavior I would like to suggest wrapping all the "raw"  linux system calls used by SPDK .

This way SPDK will be more flexible in different systems integrations .

Leonid Ravich