Re: [SPDK] Ceph/Bluestore SPDK based backend?

[Tobias Oberstein <tobias.oberstein at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3643 bytes --]

>> But what I couldn't find in above or on the net: is there a SPDK
>> backed
>> implementation of this new Bluestore OSD block device abstraction?
>>
>> Do you have a link for me? I really tried to find it ..
>
> Here is a link to the actual code:
>
> https://github.com/ceph/ceph/blob/master/src/os/bluestore/NVMEDevice.cc

Ahh. Thanks! That's a conclusive answer;) So it's definitely real.

Seems like the author of the code is the same as of the presentation I 
stumbled over.

>>> The impact to performance of Ceph was somewhat limited however.
>>> There are bottlenecks in the Ceph OSD.
>>
>> Ok=( Any public avail info on that?
>
> I don't have the actual numbers on hand, but it was a small improvement
> only. I'm speculating, but I can think of a number of problems in the
> above implementation that will limit performance. The biggest problem

Thanks for speculating! This is highly interesting.

> is that Ceph still relies on buffered I/O in a number of cases, but the
> SPDK implementation doesn't do any caching. Caching is of course the
> single most important aspect of storage performance. The above

I see. And then there is the question of where to cache (eg Ceph client 
caches or OSD side).

I am wondering how the SPDK iSCSI target approaches this. Does it 
contain it's own userspace block level caching?

> implementation also copies memory for every read and write into DMA-
> able buffers because Ceph doesn't allocate buffers from DMA-able memory

Uups.

How do I allocate DMA-able memory using SPDK/DPDK?

These are not talking about DMA

http://dpdk.org/doc/api/rte__malloc_8h.html
http://dpdk.org/doc/api/rte__mempool_8h.html

> by default. To fix that, Ceph would need to either make its memory
> manager pluggable as well, or just use SPDK/DPDK throughout for all
> data buffer allocations. Third, Ceph still does some blocking I/O in
> certain cases, and blocking I/O with SPDK, given there is no caching,
> is probably slower than the kernel.
>
>>
>> In general: having a SPDK+DPDK backed implementation of Ceph/OSD
>> seems
>> highly desirable with potentially big impact .. not?
>
> I think there is room to make it far faster than it is today using
> SPDK/DPDK, but it would take a much more dramatic set of changes to the

FWIW, I do think making Ceph block storage really fast would be a game 
changer. I recently benchmarked a data-warehouse box sitting on 8 Intel 
NVMes at 9.5 million random read IOPS - and I'd love to take that level 
of performance over to a solution that scales out. Ceph/RBD.

Using NVMe-oF and mdraid to combine into 1 block device on the 
data-warehouse host is the 2nd route I probably will have a chance to 
investigate - but this has a smell of "piecing together" and with lots 
of potential for things going wild when NVMe-oF targets 
disappear/reappear whereas Ceph/RBD was designed with that in mind.

> structure of the OSD to actually realize the benefit. The whole OSD
> would probably need to be rewritten to do one thread per core with
> message passing and entirely asynchronous network and storage stacks.
> That's effectively a brand new OSD.

I see. The bar indeed seems quite high.

The lack of (if I haven't missed it) a well defined and documented _wire 
protocol_ for talking to OSDs makes this a complete rewrite or an 
alternative complete OSD implementation even more unlikely.

Thanks alot for your detailed and informative response! This really 
helps me mapping out the options and perspectives for above mentioned 
data-warehouse user.

Cheers,
/Tobias