[PATCH 0/4] rbd: handle images flattened while mapped

[PATCH 0/4] rbd: handle images flattened while mapped

[Alex Elder]

This series detects when a mapped clone image gets flattened,
and if any requests were in flight when that occurs, causes
them to get resubmitted.

					-Alex

[PATCH 1/4] rbd: detect when clone image is flattened
[PATCH 2/4] rbd: re-submit read request for flattened clone
[PATCH 3/4] rbd: re-submit write request for flattened clone
[PATCH 4/4] rbd: re-submit flattened write request (part 2)

[PATCH 1/4] rbd: detect when clone image is flattened

[Alex Elder]

A format 2 clone image can be the subject of a "flatten" operation,
during which all of its data gets "copied up" from its parent image,
leaving the image fully populated.  Once this is complete, the
clone's association with the parent is abolished.

Since this can occur when a clone is mapped, we need to detect when
it has occurred and handle it accordingly.  We know an image has
been flattened when we know it at one time had a parent, but we have
learned (via a "get_parent" object class method call) it no longer
has one.

There might be in-flight requests at the point we learn an image has
been flattened, so we can't simply clean up parent data structures
right away.  Instead, we'll drop the initial parent reference when
the parent has disappeared (rather than when the image gets
destroyed), which will allow the last in-flight reference to clean
things up when it's complete.

We leverage the fact that a zero parent overlap renders an image
effectively unlayered.  We set the overlap to 0 at the point we
detect the clone image has flattened, which allows the unlayered
behavior to take effect immediately, while keeping other parent
structures in place until in-flight requests to complete.

This and the next few patches resolve:
    http://tracker.ceph.com/issues/3763

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 71705f3..6cd08ba 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1927,6 +1927,11 @@ static void rbd_dev_parent_put(struct rbd_device
*rbd_dev)
  * If an image has a non-zero parent overlap, get a reference to its
  * parent.
  *
+ * We must get the reference before checking for the overlap to
+ * coordinate properly with zeroing the parent overlap in
+ * rbd_dev_v2_parent_info() when an image gets flattened.  We
+ * drop it again if there is no overlap.
+ *
  * Returns true if the rbd device has a parent with a non-zero
  * overlap and a reference for it was successfully taken, or
  * false otherwise.
@@ -3771,8 +3776,26 @@ static int rbd_dev_v2_parent_info(struct
rbd_device *rbd_dev)
 	end = reply_buf + ret;
 	ret = -ERANGE;
 	ceph_decode_64_safe(&p, end, parent_spec->pool_id, out_err);
-	if (parent_spec->pool_id == CEPH_NOPOOL)
+	if (parent_spec->pool_id == CEPH_NOPOOL) {
+		/*
+		 * Either the parent never existed, or we have
+		 * record of it but the image got flattened so it no
+		 * longer has a parent.  When the parent of a
+		 * layered image disappears we immediately set the
+		 * overlap to 0.  The effect of this is that all new
+		 * requests will be treated as if the image had no
+		 * parent.
+		 */
+		if (rbd_dev->parent_overlap) {
+			rbd_dev->parent_overlap = 0;
+			smp_mb();
+			rbd_dev_parent_put(rbd_dev);
+			pr_info("%s: clone image has been flattened\n",
+				rbd_dev->disk->disk_name);
+		}
+
 		goto out;	/* No parent?  No problem. */
+	}

 	/* The ceph file layout needs to fit pool id in 32 bits */

@@ -4632,7 +4655,10 @@ static void rbd_dev_unprobe(struct rbd_device
*rbd_dev)
 {
 	struct rbd_image_header	*header;

-	rbd_dev_parent_put(rbd_dev);
+	/* Drop parent reference unless it's already been done (or none) */
+
+	if (rbd_dev->parent_overlap)
+		rbd_dev_parent_put(rbd_dev);

 	/* Free dynamic fields from the header, then zero it out */

-- 
1.7.9.5

[PATCH 2/4] rbd: re-submit read request for flattened clone

[Alex Elder]

If a clone image gets flattened while a parent read request is
underway, the original rbd object request needs to be resubmitted.

The reason is that by the time we get the response to the parent
read request, the data read from the parent may be out of date.
In other words, we could see this sequence of events:

    rbd client                      parent image/osd
    ----------                      ----------------
    original object ENOENT;
        issue parent read
                                    respond to parent read
                                    child image flattened
    original image header refresh
             <--- original object written independently here
    parent read response received

Add code to rbd_img_parent_read_callback() to detect when a clone's
parent image has disappeared (as evidenced by its parent overlap
becoming 0), and re-submit the original read request in that case.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 6cd08ba..4dca3ce 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -2672,14 +2672,36 @@ static void rbd_img_parent_read_callback(struct
rbd_img_request *img_request)
 	struct rbd_obj_request *obj_request;
 	struct rbd_device *rbd_dev;
 	u64 obj_end;
+	u64 img_xferred;
+	int img_result;

 	rbd_assert(img_request_child_test(img_request));

+	/* First get what we need from the image request and release it */
+
 	obj_request = img_request->obj_request;
+	img_xferred = img_request->xferred;
+	img_result = img_request->result;
+	rbd_img_request_put(img_request);
+
+	/*
+	 * If the overlap has become 0 (most likely because the
+	 * image has been flattened) we need to re-submit the
+	 * original request.
+	 */
 	rbd_assert(obj_request);
 	rbd_assert(obj_request->img_request);
+	rbd_dev = obj_request->img_request->rbd_dev;
+	if (!rbd_dev->parent_overlap) {
+		struct ceph_osd_client *osdc;
+
+		osdc = &rbd_dev->rbd_client->client->osdc;
+		img_result = rbd_obj_request_submit(osdc, obj_request);
+		if (!img_result)
+			return;
+	}

-	obj_request->result = img_request->result;
+	obj_request->result = img_result;
 	if (obj_request->result)
 		goto out;

@@ -2692,7 +2714,6 @@ static void rbd_img_parent_read_callback(struct
rbd_img_request *img_request)
 	 */
 	rbd_assert(obj_request->img_offset < U64_MAX - obj_request->length);
 	obj_end = obj_request->img_offset + obj_request->length;
-	rbd_dev = obj_request->img_request->rbd_dev;
 	if (obj_end > rbd_dev->parent_overlap) {
 		u64 xferred = 0;

@@ -2700,12 +2721,11 @@ static void rbd_img_parent_read_callback(struct
rbd_img_request *img_request)
 			xferred = rbd_dev->parent_overlap -
 					obj_request->img_offset;

-		obj_request->xferred = min(img_request->xferred, xferred);
+		obj_request->xferred = min(img_xferred, xferred);
 	} else {
-		obj_request->xferred = img_request->xferred;
+		obj_request->xferred = img_xferred;
 	}
 out:
-	rbd_img_request_put(img_request);
 	rbd_img_obj_request_read_callback(obj_request);
 	rbd_obj_request_complete(obj_request);
 }
-- 
1.7.9.5

[PATCH 3/4] rbd: re-submit write request for flattened clone

[Alex Elder]

Add code to rbd_img_parent_read_full_callback() to detect when a
clone's parent image has disappeared, and re-submit the original
write request in that case.  (See the previous commit for more
reasoning about why this is appropriate.)

Rename some variables in rbd_img_obj_parent_read_full_callback()
to match the convention used in the previous patch.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   31 ++++++++++++++++++++++++-------
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 4dca3ce..dfa6bfc 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -2314,7 +2314,7 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)
 	struct ceph_osd_client *osdc;
 	struct rbd_device *rbd_dev;
 	struct page **pages;
-	int result;
+	int img_result;
 	u64 obj_size;
 	u64 offset;
 	u64 length;
@@ -2330,7 +2330,7 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)
 	orig_request = img_request->obj_request;
 	rbd_assert(orig_request != NULL);
 	rbd_assert(obj_request_type_valid(orig_request->type));
-	result = img_request->result;
+	img_result = img_request->result;
 	obj_size = img_request->length;
 	rbd_assert(obj_size == img_request->xferred);
 	rbd_img_request_put(img_request);
@@ -2340,7 +2340,24 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)
 	rbd_assert(rbd_dev);
 	rbd_assert(obj_size == (u64)1 << rbd_dev->header.obj_order);

-	if (result)
+	/*
+	 * If the overlap has become 0 (most likely because the
+	 * image has been flattened) we need to free the pages
+	 * and re-submit the original write request.
+	 */
+	if (!rbd_dev->parent_overlap) {
+		struct ceph_osd_client *osdc;
+		u32 page_count;
+
+		page_count = (u32)calc_pages_for(0, obj_size);
+		ceph_release_page_vector(pages, page_count);
+		osdc = &rbd_dev->rbd_client->client->osdc;
+		img_result = rbd_obj_request_submit(osdc, orig_request);
+		if (!img_result)
+			return;
+	}
+
+	if (img_result)
 		goto out_err;

 	/*
@@ -2349,7 +2366,7 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)
 	 * request.  Allocate the new copyup osd request for the
 	 * original request, and release the old one.
 	 */
-	result = -ENOMEM;
+	img_result = -ENOMEM;
 	osd_req = rbd_osd_req_create_copyup(orig_request);
 	if (!osd_req)
 		goto out_err;
@@ -2383,13 +2400,13 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)

 	orig_request->callback = rbd_img_obj_copyup_callback;
 	osdc = &rbd_dev->rbd_client->client->osdc;
-	result = rbd_obj_request_submit(osdc, orig_request);
-	if (!result)
+	img_result = rbd_obj_request_submit(osdc, orig_request);
+	if (!img_result)
 		return;
 out_err:
 	/* Record the error code and complete the request */

-	orig_request->result = result;
+	orig_request->result = img_result;
 	orig_request->xferred = 0;
 	obj_request_done_set(orig_request);
 	rbd_obj_request_complete(orig_request);
-- 
1.7.9.5

[PATCH 4/4] rbd: re-submit flattened write request (part 2)

[Alex Elder]

Add code to rbd_img_obj_exists_callback() to detect when a clone's
parent image has disappeared, and re-submit the original write
request in that case.

Kill off some redundant assertions.

This completes the resolution for:
    http://tracker.ceph.com/issues/3763

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index dfa6bfc..59bf75a 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -2508,6 +2508,7 @@ out_err:
 static void rbd_img_obj_exists_callback(struct rbd_obj_request
*obj_request)
 {
 	struct rbd_obj_request *orig_request;
+	struct rbd_device *rbd_dev;
 	int result;

 	rbd_assert(!obj_request_img_data_test(obj_request));
@@ -2530,8 +2531,20 @@ static void rbd_img_obj_exists_callback(struct
rbd_obj_request *obj_request)
 		obj_request->xferred, obj_request->length);
 	rbd_obj_request_put(obj_request);

-	rbd_assert(orig_request);
-	rbd_assert(orig_request->img_request);
+	/*
+	 * If the overlap has become 0 (most likely because the
+	 * image has been flattened) we need to free the pages
+	 * and re-submit the original write request.
+	 */
+	rbd_dev = orig_request->img_request->rbd_dev;
+	if (!rbd_dev->parent_overlap) {
+		struct ceph_osd_client *osdc;
+
+		osdc = &rbd_dev->rbd_client->client->osdc;
+		result = rbd_obj_request_submit(osdc, orig_request);
+		if (!result)
+			return;
+	}

 	/*
 	 * Our only purpose here is to determine whether the object
-- 
1.7.9.5

Re: [PATCH 0/4] rbd: handle images flattened while mapped

[Josh Durgin]

On 05/11/2013 10:45 AM, Alex Elder wrote:
> This series detects when a mapped clone image gets flattened,
> and if any requests were in flight when that occurs, causes
> them to get resubmitted.
>
> 					-Alex
>
> [PATCH 1/4] rbd: detect when clone image is flattened
> [PATCH 2/4] rbd: re-submit read request for flattened clone
> [PATCH 3/4] rbd: re-submit write request for flattened clone
> [PATCH 4/4] rbd: re-submit flattened write request (part 2)

These all look good.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>