Re: [SPDK] SPDK without hugepages?

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3714 bytes --]

On Wed, 2019-01-23 at 18:14 +0200, E.W.Z. wrote:
> As far as I understand DPDK is capable of doing both.
> 
> Nothing wrong with hugepages except the fact that I have to enable hugepages
> on host machines where my application is going to run. This is additional
> requirement for the host machine I would like to avoid.

This is certainly an area where some work could be put in. From an
architecture/hardware perspective, running without hugepages is possible in two
scenarios.

1) You aren't doing any direct DMA access (i.e. not using the NVMe or IOAT
drivers). This is actually the case for a number of SPDK users because they're
doing things like NVMe-oF (RDMA doesn't need hugepages) or iSCSI to virtual
disks on the back end.

2) You have an IOMMU enabled. This allows us to map arbitrary cpu virtual
addresses to I/O virtual addresses and use those instead of physical addresses
when programming the DMA engines. The IOMMU will correctly handle page table
changes.

I'd like SPDK to support running without hugepages in both of these cases. We
already support them, more or less, but the memory allocator in DPDK still
happens to allocate its memory from hugepages even though it doesn't strictly
need to. We had started investigating what it would take to change that some
time back, but since then DPDK has rewritten its memory allocator to support
dynamic memory allocation (partially at our request, and it's great!). If
someone were interested in resuming that investigation on the latest code
(basically, piping through DPDK's --no-huge option and confirming it behaves
correctly on the new DPDK memory allocator) that would be greatly appreciated. I
think supporting this would be a big step forward for SPDK.

Just as a clarification, it is not possible to support running the NVMe driver
without using either hugepages or an IOMMU. There are several other alternative
user space NVMe drivers floating around and from what I've seen they all make
this mistake (usually they mlock the memory, but that's not sufficient to
guarantee that the virtual to physical mapping never changes). This will
certainly result in silent data corruption. But at least on server platforms
with an IOMMU it would be really convenient to not require hugepages.

-Ben

> 
> 
> Sent from Mail for Windows 10
> 
> From: Andrey Kuzmin
> Sent: Wednesday, January 23, 2019 17:05
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] SPDK without hugepages?
> 
> On Wed, Jan 23, 2019 at 6:00 PM Ernest Zed <kreuzerkrieg(a)gmail.com> wrote:
> 
> > Looks like in the past there was an attempt to work without  hugepages, as
> > in `init.c` was a line of code like `args = spdk_push_arg(args, &argcount,
> > _sprintf_alloc("--no-huge"));` Looks like it didnt work and was removed.
> > So, is it possible?
> 
> 
> For that to work, an alternative to huge pages is needed that, at the very
> minimum, possesses the same basic qualities:
> 1) Never ever paged out,
> 2) Constant-time vtophys translation.
> 
> 
> > Any plans to revive this feature? I'm interested in
> > getting the NVMe Driver working without hugepages
> > 
> 
> And what's wrong with huge pages?
> 
> Regards,
> Andrey
> 
> 
> 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > 
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk