From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54031) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UpzMK-0006ee-0I for qemu-devel@nongnu.org; Fri, 21 Jun 2013 07:18:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UpzMH-0004r5-IL for qemu-devel@nongnu.org; Fri, 21 Jun 2013 07:18:27 -0400 Received: from mail.avalus.com ([2001:41c8:10:1dd::10]:46705) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UpzMH-0004qr-CE for qemu-devel@nongnu.org; Fri, 21 Jun 2013 07:18:25 -0400 Date: Fri, 21 Jun 2013 12:18:11 +0100 From: Alex Bligh Message-ID: In-Reply-To: References: <7029962A8C6EFDBC98B51E44@nimrod.local> <20130411092548.GE8904@stefanha-thinkpad.redhat.com> <20130620094618.GC15672@stefanha-thinkpad.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Re: [Qemu-devel] Adding a persistent writeback cache to qemu Reply-To: Alex Bligh List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Sage Weil , Stefan Hajnoczi Cc: josh.durgin@inktank.com, qemu-devel@nongnu.org, Alex Bligh Sage, --On 20 June 2013 08:58:19 -0700 Sage Weil wrote: >> I'd like to hear from Ceph folks what their position on kernel rbd vs >> librados is. Why one do they recommend for QEMU guests and what are the >> pros/cons? > > I agree that a flashcache/bcache-like persistent cache would be a big win > for qemu + rbd users. Great. I think Stefan was really after testing my received wisdom that ceph+librbd will be greater performance than ceph+blkdev+kernelrbd (even without the persistent cache), and if so why. > There are few important issues with librbd vs kernel rbd: > > * librbd tends to get new features more quickly that the kernel rbd > (although now that layering has landed in 3.10 this will be less > painful than it was). > > * Using kernel rbd means users need bleeding edge kernels, a non-starter > for many orgs that are still running things like RHEL. Bug fixes are > difficult to roll out, etc. > > * librbd has an in-memory cache that behaves similar to an HDD's cache > (e.g., it forces writeback on flush). This improves performance > significantly for many workloads. Of course, having a bcache-like > layer mitigates this.. > > I'm not really sure what the best path forward is. Putting the > functionality in qemu would benefit lots of other storage backends, > putting it in librbd would capture various other librbd users (xen, tgt, > and future users like hyper-v), and using new kernels works today but > creates a lot of friction for operations. To be honest I'd not even thought of putting it in librbd (which might be simpler). I suspect it might be easier to get patches into librbd than into qemu, and that ensuring cache coherency might be simpler. If I get time to look at this, would you be interested in taking patches for this? -- Alex Bligh