Re: [SPDK] FW: sharing of single NVMe device between spdk userspace and native kernel driver

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 9086 bytes --]

On Thu, 2016-07-14 at 21:34 +0000, Raj (Rajinikanth) Pandurangan wrote:
> Hi Ben,
> 
> Yes, I too agree that one of the most important requirement is to have a filesystem with SPDK.
> 
> What are all the known challenges to develop a filesystem with SPDK?  Are the interface/APIs
> provided by SPDK good enough?

SPDK provides only block-level access to storage devices today. At a minimum, a "filesystem" on top
of SPDK would need to provide a mechanism to dynamically allocate discontiguous physical blocks and
present them as a contiguous space that can be written to or read from in some unit (4k? 1 byte?).
I've been choosing to call that a "blob" to differentiate it from a file in the Unix sense, and I've
been calling the whole thing a "blobstore" as opposed to a filesystem. The blobstore would ensure
blobs are persistent and rediscoverable across reboots.

> 
> It would be good to list down the known challenges in the mailing-list, so that community may try
> to address/discuss about them?

Beyond the very basic requirements above, I think the additional requirements depend on the
application that is using it. Some applications can tolerate only being allowed to write and read in
sector-size chunks, for instance, which is important if the blobstore wishes to implement zero copy.
Other applications need finer granularity. Many databases don't need directories either - they can
live with a flat namespace in which to place their blobs. I think file-level permissions aren't
needed either.

Functional requirements aside, in order to get the best performance possible, the blobstore would need to be asynchronous, lockless, and polled mode. That's a real challenge due to shared metadata, although I have a number of ideas in this area.

> 
> Thanks,
> 
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Thursday, July 14, 2016 1:18 PM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] FW: sharing of single NVMe device between spdk userspace and native kernel
> driver
> 
> On Thu, 2016-07-14 at 11:59 -0700, txcy uio wrote:
> > > Thank you Ben for the detailed reply.
> > A filesystem which can make use of SPDK is precisely the requirement. 
> > Everything else is a way to get around that. In my specific use case I 
> > wish to have a single nvme device which will have a rootfs as well. So 
> > such a filesystem will need to handle that as well (probably I am being too ambitious here).
> 
> It won't ever be possible to use SPDK as the driver for your boot device. SR-IOV would let you
> share the device between the kernel (for booting) and your application, if and/or when that
> exists.
> 
> > The only other "filesystem" that I am aware of is Ceph's bluefs which 
> > is very minimal and specific to Rocksdb backend.
> 
> This is the only one that I'm aware of currently as well, and it has a number of features that
> make it not particularly suitable for use with SPDK (even though it does work with SPDK). The
> biggest problems are around synchronous I/O operations and lack of memory pre-registration,
> forcing copies on every I/O.
> 
> >  On a side note if I had more than one nvme device on a system , do 
> > all the nvme devices need to be unbound from the kernel driver?
> 
> Each NVMe device is independent. You can use some NVMe devices with SPDK and others with the
> kernel at the same time with no conflict. Our setup scripts do either bind or unbind all of them
> at once, but that's just for convenience.
> 
> > 
> > --Tyc
> >  
> > > On 7/14/16, 10:55 AM, "SPDK on behalf of Walker, Benjamin" 
> > > <spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
> > > 
> > > > 
> > > > 
> > > > On Wed, 2016-07-13 at 12:59 -0700, txcy uio wrote:
> > > > > Hello Ben
> > > > > 
> > > > > I have a use case where I want to attach one namespace of a nvme 
> > > > > device to spdk driver and
> > > use the
> > > > > other namespace as a kernel block device to create a regular 
> > > > > filesystem. Current
> > > implementation of
> > > > > spdk requires the device to be unbound completely from the native 
> > > > > kernel driver. I was
> > > wondering
> > > > > if this is at all possible and if yes can this be accomplished 
> > > > > with the current spdk implementation?
> > > > 
> > > > Your request is one we get every few days or so, and it is a perfectly reasonable thing to
> > > > ask.
> > > I
> > > > haven't written down my standard response on the mailing list yet, 
> > > > so I'm going to take this opportunity to lay out our position for all to see and discuss.
> > > > 
> > > > From a purely technical standpoint, it is impossible to both load 
> > > > the SPDK driver as it exists
> > > today
> > > > and the kernel driver against the same PCI device. The registers 
> > > > exposed by the PCI device
> > > contain
> > > > global state and so there can only be a single "owner". There is an 
> > > > established hardware
> > > mechanism
> > > > for creating multiple virtual PCI devices from a single physical 
> > > > devices that each can load
> > > their
> > > > own driver called SR-IOV. This is typically used by NICs today and 
> > > > I'm not aware of any NVMe
> > > SSDs
> > > > that support it currently. SR-IOV is the right solution for sharing 
> > > > the device like you outline
> > > in
> > > > the long term, though.
> > > > 
> > > > In the short term, it would be technically possible to create some 
> > > > kernel patches that add
> > > entries
> > > > to sysfs or provide ioctls that allow a user space process to claim 
> > > > an NVMe hardware queue for
> > > a
> > > > device that the kernel is managing. You could then run the SPDK 
> > > > driver's I/O path against that queue. Unfortunately, there are two 
> > > > insurmountable issues with this strategy. First, NVMe
> > > hardware
> > > > queues can write to any namespace on the device. Therefore, you 
> > > > couldn't enforce that the queue
> > > can
> > > > only write to the namespace you are intending. You couldn't even 
> > > > enforce that the queue is only
> > > used
> > > > for reads - you basically just have to trust the application to only do reasonable things.
> > > Second,
> > > > the device is owned by the kernel and therefore is not in an IOMMU 
> > > > protection domain with this strategy. The device can directly 
> > > > access the DMA engine, and with a small amount of work, you
> > > could
> > > > hijack that DMA engine to copy data to wherever you wanted on the 
> > > > system. For these two
> > > reasons,
> > > > patches of this nature would never be accepted into the mainline 
> > > > kernel. The SPDK team can't be
> > > in
> > > > the business of supporting patches that have been rejected by the kernel community.
> > > > 
> > > > Clearly, lots of people have requested to share a device between 
> > > > the kernel and SPDK, so I've
> > > been
> > > > trying to uncover all of the reasons they may want to do that. So 
> > > > far, in every case, it boils
> > > down
> > > > to not having a filesystem for use with SPDK. I'm hoping to steer 
> > > > the community to solve the
> > > problem
> > > > of not having a filesystem rather than trying to share the device. 
> > > > I'm not advocating for
> > > writing a
> > > > (mostly) POSIX compliant filesystem, but I do think there is a 
> > > > small core of functionality that
> > > most
> > > > databases or storage applications all require. These are things 
> > > > like allocating blocks into
> > > some
> > > > unit (I've been calling it a blob) that has a name and is 
> > > > persistent and rediscoverable across reboots. Writing this layer 
> > > > requires some serious thought - SPDK is fast in no small part
> > > because it
> > > > is purely asynchronous, polled, and lockless - so this layer would 
> > > > need to preserve those characteristics.
> > > > 
> > > > Sorry for the very long response, but I wanted to document my 
> > > > current thoughts on the mailing
> > > list
> > > > for all to see.
> > > > 
> > > > > 
> > > > > --Tyc
> > > > > 
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > 
> > 
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk