[Qemu-devel] [PATCH] rbd: hook up cache options

[Qemu-devel] [PATCH] rbd: hook up cache options

[Josh Durgin]

Writeback caching was added in Ceph 0.46, and writethrough will be in
0.47. These are controlled by general config options, so there's no
need to check for librbd version.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
---
 block/rbd.c |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 1280d66..eebc334 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -476,6 +476,25 @@ static int qemu_rbd_open(BlockDriverState *bs, const char *filename, int flags)
         s->snap = g_strdup(snap_buf);
     }
 
+    /*
+     * Fallback to more conservative semantics if setting cache
+     * options fails. Ignore errors from setting rbd_cache because the
+     * only possible error is that the option does not exist, and
+     * librbd defaults to no caching. If write through caching cannot
+     * be set up, fall back to no caching.
+     */
+    if (flags & BDRV_O_NOCACHE) {
+        rados_conf_set(s->cluster, "rbd_cache", "false");
+    } else {
+        rados_conf_set(s->cluster, "rbd_cache", "true");
+        if (!(flags & BDRV_O_CACHE_WB)) {
+            r = rados_conf_set(s->cluster, "rbd_cache_max_dirty", "0");
+            if (r < 0) {
+                rados_conf_set(s->cluster, "rbd_cache", "false");
+            }
+        }
+    }
+
     if (strstr(conf, "conf=") == NULL) {
         /* try default location, but ignore failure */
         rados_conf_read_file(s->cluster, NULL);
-- 
1.7.5.4

Re: [Qemu-devel] [PATCH] rbd: hook up cache options

[Kevin Wolf]

Am 17.05.2012 22:42, schrieb Josh Durgin:
> Writeback caching was added in Ceph 0.46, and writethrough will be in
> 0.47. These are controlled by general config options, so there's no
> need to check for librbd version.
> 
> Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Thanks, applied to the block-next branch for 1.2.

Kevin

Re: [Qemu-devel] [PATCH] rbd: hook up cache options

[Paolo Bonzini]

Il 17/05/2012 22:42, Josh Durgin ha scritto:
> +     * Fallback to more conservative semantics if setting cache
> +     * options fails. Ignore errors from setting rbd_cache because the
> +     * only possible error is that the option does not exist, and
> +     * librbd defaults to no caching. If write through caching cannot
> +     * be set up, fall back to no caching.
> +     */
> +    if (flags & BDRV_O_NOCACHE) {
> +        rados_conf_set(s->cluster, "rbd_cache", "false");
> +    } else {
> +        rados_conf_set(s->cluster, "rbd_cache", "true");
> +        if (!(flags & BDRV_O_CACHE_WB)) {
> +            r = rados_conf_set(s->cluster, "rbd_cache_max_dirty", "0");
> +            if (r < 0) {
> +                rados_conf_set(s->cluster, "rbd_cache", "false");
> +            }
> +        }
> +    }

Last time I looked at ceph, rbd_flush was not a full flush of the cache;
it only ensured that the pending requests were sent.  So my questions are:

1) has this changed?  does rbd_flush now flush dirty items when
rbd_cache_max_dirty > 0?

2) should the usage of a cache be conditional on LIBRBD_VERSION_CODE >=
LIBRBD_VERSION(0, 1, 1)?

Thanks,

Paolo

Re: [Qemu-devel] [PATCH] rbd: hook up cache options

[Josh Durgin]

On 05/22/2012 02:18 AM, Paolo Bonzini wrote:
> Il 17/05/2012 22:42, Josh Durgin ha scritto:
>> +     * Fallback to more conservative semantics if setting cache
>> +     * options fails. Ignore errors from setting rbd_cache because the
>> +     * only possible error is that the option does not exist, and
>> +     * librbd defaults to no caching. If write through caching cannot
>> +     * be set up, fall back to no caching.
>> +     */
>> +    if (flags&  BDRV_O_NOCACHE) {
>> +        rados_conf_set(s->cluster, "rbd_cache", "false");
>> +    } else {
>> +        rados_conf_set(s->cluster, "rbd_cache", "true");
>> +        if (!(flags&  BDRV_O_CACHE_WB)) {
>> +            r = rados_conf_set(s->cluster, "rbd_cache_max_dirty", "0");
>> +            if (r<  0) {
>> +                rados_conf_set(s->cluster, "rbd_cache", "false");
>> +            }
>> +        }
>> +    }
>
> Last time I looked at ceph, rbd_flush was not a full flush of the cache;
> it only ensured that the pending requests were sent.  So my questions are:

I'm not sure which version you were looking at, but this hasn't been
the case since caching was implemented. I don't think it was ever the
case, actually. rbd_flush has always waited for pending I/Os to complete
(be on disk on all replicas), not just be in flight.

If you're interested in the current implementation, you can see:

src/librbd.cc: librbd::flush()

which goes into:

src/osdc/ObjectCacher.cc: ObjectCacher::commit_set()

or

src/librados/IoCtxImpl.cc: IoCtxImpl::flush_aio_writes()

> 1) has this changed?  does rbd_flush now flush dirty items when
> rbd_cache_max_dirty>  0?

The rbd_cache_* options did not exist before 0.46.

> 2) should the usage of a cache be conditional on LIBRBD_VERSION_CODE>=
> LIBRBD_VERSION(0, 1, 1)?

It doesn't matter if you use an older version because the non-existent
options don't have any effect.

Thanks,
Josh