[PATCH v2 0/6] Make VA reservation limits configurable

[Anatoly Burakov <anatoly.burakov@intel.com>]

Currently, the VA space reservation is governed by a combination of a few
values:

- Total max VA space (512G for most platforms, 1T for some, 2G for 32-bit) - Max
memory per memory type - Max pages per memory type

The "memory" type is defined as unique combination of NUMA node and page size.
The reason why there are two limits is because for large pages, having large
segment limit causes runaway multi-terabyte reservations, while for smaller
pages, having large memory limit causes hundreds of thousands of huge page
slots. The total maximum memory size was originally intended as a safeguard
against discontiguous NUMA nodes, but this has since been fixed by EAL API
explicitly supporting discontiguous NUMA nodes, so this is no longer a problem.

In addition to that, each memory type was split into multiple segment lists,
with the idea that it should be easier for a secondary process to reserve
multiple smaller chunks at discontiguous addresses than it is to reserve a large
single chunk of memory. It is unknown whether this actually makes a difference,
but what *is* known is that it's a source of additional complexity with memory
reservation, as well as a source of gratuitous memory reservation limits placed
on DPDK.

This patchset attempts to simplify and improve this situation in a few key
areas:

- Get rid of global memory limits

Total memory usage can, and should, scale with NUMA sockets, and so now it does.

- Get rid of multiple segment lists per memory type

This removes two config options, and makes the address space reservations a lot
simpler.

- Allocate all memory segment lists as one big blob of memory

This further simplifies address space reservations.

- Use memory size limits instead of segments limits

Despite smaller page sizes still needing limits on number of segments, they are
directly translated into memory size limits at init time, so that all limits the
VA space reservation ever sees are expressed in bytes, not segments. This
reduces complexity in how we manage the VA space reservations and work with our
limits.

- Do not use config constants directly

We switch to only invoking these constants once - at startup, when we are
discovering hugepage sizes available to the system. This allows us to be more
flexible in how we manage these limits.

- Add EAL command-line option to set per-page size limits

The final piece of the puzzle - the "more flexible in how we manage these
limits" part. This new parameter affords us more flexible VA space management,
including disabling specific page sizes entirely (by specifying 0 as the limit).
This allows increasing/decreasing VA space reservation limits without
recompiling DPDK.

v1 -> v2:
- Fix str_to_size not handling invalid input properly
- Move str_to_size autotests to string autotests file
- Fix hugepage file segment indexing to not use global constants
- More validation around VA reservation

Anatoly Burakov (6):
  eal: reject non-numeric input in str to size
  eal/memory: remove per-list segment and memory limits
  eal/memory: allocate all VA space in one go
  eal/memory: get rid of global VA space limits
  eal/memory: store default segment limits in config
  eal/memory: add page size VA limits EAL parameter

 app/test/test.c                               |   1 +
 app/test/test_eal_flags.c                     | 126 ++++++++++++
 app/test/test_malloc.c                        |  30 ---
 app/test/test_string_fns.c                    |  66 ++++++
 config/arm/meson.build                        |   1 -
 config/meson.build                            |   5 -
 config/rte_config.h                           |   2 -
 doc/guides/linux_gsg/linux_eal_parameters.rst |  13 ++
 .../prog_guide/env_abstraction_layer.rst      |  33 ++-
 lib/eal/common/eal_common_dynmem.c            | 192 ++++++++----------
 lib/eal/common/eal_common_memory.c            |  28 ++-
 lib/eal/common/eal_common_options.c           | 141 +++++++++++++
 lib/eal/common/eal_common_string_fns.c        |   4 +
 lib/eal/common/eal_filesystem.h               |  13 ++
 lib/eal/common/eal_internal_cfg.h             |   8 +
 lib/eal/common/eal_memcfg.h                   |   6 +
 lib/eal/common/eal_option_list.h              |   1 +
 lib/eal/common/eal_options.h                  |   1 +
 lib/eal/common/eal_private.h                  |  19 +-
 lib/eal/freebsd/eal.c                         |   6 +
 lib/eal/freebsd/eal_memory.c                  | 101 +++------
 lib/eal/linux/eal.c                           |   6 +
 lib/eal/linux/eal_memalloc.c                  |   4 +-
 lib/eal/linux/eal_memory.c                    | 170 ++++++++++------
 lib/eal/windows/eal.c                         |   6 +
 25 files changed, 687 insertions(+), 296 deletions(-)

-- 
2.47.3