From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8793736651691237818==" MIME-Version: 1.0 From: Tobias Oberstein Subject: Re: [SPDK] Ceph/Bluestore SPDK based backend? Date: Tue, 07 Feb 2017 23:37:16 +0100 Message-ID: In-Reply-To: 1486493691.22338.1.camel@intel.com List-ID: To: spdk@lists.01.org --===============8793736651691237818== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable >> But what I couldn't find in above or on the net: is there a SPDK >> backed >> implementation of this new Bluestore OSD block device abstraction? >> >> Do you have a link for me? I really tried to find it .. > > Here is a link to the actual code: > > https://github.com/ceph/ceph/blob/master/src/os/bluestore/NVMEDevice.cc Ahh. Thanks! That's a conclusive answer;) So it's definitely real. Seems like the author of the code is the same as of the presentation I = stumbled over. >>> The impact to performance of Ceph was somewhat limited however. >>> There are bottlenecks in the Ceph OSD. >> >> Ok=3D( Any public avail info on that? > > I don't have the actual numbers on hand, but it was a small improvement > only. I'm speculating, but I can think of a number of problems in the > above implementation that will limit performance. The biggest problem Thanks for speculating! This is highly interesting. > is that Ceph still relies on buffered I/O in a number of cases, but the > SPDK implementation doesn't do any caching. Caching is of course the > single most important aspect of storage performance. The above I see. And then there is the question of where to cache (eg Ceph client = caches or OSD side). I am wondering how the SPDK iSCSI target approaches this. Does it = contain it's own userspace block level caching? > implementation also copies memory for every read and write into DMA- > able buffers because Ceph doesn't allocate buffers from DMA-able memory Uups. How do I allocate DMA-able memory using SPDK/DPDK? These are not talking about DMA http://dpdk.org/doc/api/rte__malloc_8h.html http://dpdk.org/doc/api/rte__mempool_8h.html > by default. To fix that, Ceph would need to either make its memory > manager pluggable as well, or just use SPDK/DPDK throughout for all > data buffer allocations. Third, Ceph still does some blocking I/O in > certain cases, and blocking I/O with SPDK, given there is no caching, > is probably slower than the kernel. > >> >> In general: having a SPDK+DPDK backed implementation of Ceph/OSD >> seems >> highly desirable with potentially big impact .. not? > > I think there is room to make it far faster than it is today using > SPDK/DPDK, but it would take a much more dramatic set of changes to the FWIW, I do think making Ceph block storage really fast would be a game = changer. I recently benchmarked a data-warehouse box sitting on 8 Intel = NVMes at 9.5 million random read IOPS - and I'd love to take that level = of performance over to a solution that scales out. Ceph/RBD. Using NVMe-oF and mdraid to combine into 1 block device on the = data-warehouse host is the 2nd route I probably will have a chance to = investigate - but this has a smell of "piecing together" and with lots = of potential for things going wild when NVMe-oF targets = disappear/reappear whereas Ceph/RBD was designed with that in mind. > structure of the OSD to actually realize the benefit. The whole OSD > would probably need to be rewritten to do one thread per core with > message passing and entirely asynchronous network and storage stacks. > That's effectively a brand new OSD. I see. The bar indeed seems quite high. The lack of (if I haven't missed it) a well defined and documented _wire = protocol_ for talking to OSDs makes this a complete rewrite or an = alternative complete OSD implementation even more unlikely. Thanks alot for your detailed and informative response! This really = helps me mapping out the options and perspectives for above mentioned = data-warehouse user. Cheers, /Tobias --===============8793736651691237818==--