[PATCH 00/20] libceph: per-op osd request data

[PATCH 00/20] libceph: per-op osd request data

[Alex Elder]

This series of patches arranges for each op in an osd request
to define its own incoming and outgoing data.  Doing so allows
a single request to contain muliple ops which provide outgoing
data, and even allows a single op to supply more than one item
of data.

A bunch of the patches here are fairly trivial cleanups, done
as I was figuring out how to get things done.  But in the end
the changes involved are significant so I tried to do them
in methodical steps.

These patches (and the one I just posted) are available in the
branch "review/wip-3761-1" in the ceph-client git repository.

					-Alex

[PATCH 01/20] rbd: define inbound data size for method ops
    Whoops, this one's not directly related to the rest, I
    intended to post this one by itself.

[PATCH 02/20] libceph: compute incoming bytes once
[PATCH 03/20] libceph: define osd data initialization helpers
[PATCH 04/20] libceph: define a few more helpers
[PATCH 05/20] libceph: define ceph_osd_data_length()
[PATCH 06/20] libceph: a few more osd data cleanups
    Simple cleanups, and code encapsulation.

[PATCH 07/20] rbd: define rbd_osd_req_format_op()
[PATCH 08/20] libceph: keep source rather than message osd op array
    Store the source op array in the osd request structure.

[PATCH 09/20] libceph: rename data out field in osd request op
[PATCH 10/20] libceph: add data pointers in osd op structures
[PATCH 11/20] libceph: specify osd op by index in request
    Start defining op data in the op structure itself.

[PATCH 12/20] rbd: don't set data in rbd_osd_req_format_op()
[PATCH 13/20] rbd: separate initialization of osd data
[PATCH 14/20] rbd: rearrange some code for consistency
    Changes to rbd so it doesn't assume single op per request.

[PATCH 15/20] libceph: format class info at init time
    Don't wait until formatting the op to lay out the request
    information (class, method, and data).  Define a new data
    item in the CALL op structure to represent this.

[PATCH 16/20] libceph: move ceph_osdc_build_request()
[PATCH 17/20] libceph: set message data when building osd request
[PATCH 18/20] libceph: combine initializing and setting osd data
[PATCH 19/20] libceph: set the data pointers when encoding ops
[PATCH 20/20] libceph: kill off osd request r_data_in and r_data_out
    These last five (four really) finally make the switch over
    to using data item descriptions present in the ops to
    dictate what's used in osd request and response messages.

[PATCH 01/20] rbd: define inbound data size for method ops

[Alex Elder]

When rbd creates an object request containing an object method call
operation it is passing 0 for the size.  I originally thought this
was because the length was not needed for method calls, but I think
it really should be supplied, to describe how much space is
available to receive response data.  So provide the supplied length.

This resolves:
    http://tracker.ceph.com/issues/4659

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 9fb51b5..5e579fa 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1839,12 +1839,11 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	int ret;

 	/*
-	 * Method calls are ultimately read operations but they
-	 * don't involve object data (so no offset or length).
-	 * The result should placed into the inbound buffer
-	 * provided.  They also supply outbound data--parameters for
-	 * the object method.  Currently if this is present it will
-	 * be a snapshot id.
+	 * Method calls are ultimately read operations.  The result
+	 * should placed into the inbound buffer provided.  They
+	 * also supply outbound data--parameters for the object
+	 * method.  Currently if this is present it will be a
+	 * snapshot id.
 	 */
 	page_count = (u32) calc_pages_for(0, inbound_size);
 	pages = ceph_alloc_page_vector(page_count, GFP_KERNEL);
@@ -1852,7 +1851,7 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 		return PTR_ERR(pages);

 	ret = -ENOMEM;
-	obj_request = rbd_obj_request_create(object_name, 0, 0,
+	obj_request = rbd_obj_request_create(object_name, 0, inbound_size,
 							OBJ_REQUEST_PAGES);
 	if (!obj_request)
 		goto out;
-- 
1.7.9.5

Re: [PATCH 01/20] rbd: define inbound data size for method ops

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:01 AM, Alex Elder wrote:
> When rbd creates an object request containing an object method call
> operation it is passing 0 for the size.  I originally thought this
> was because the length was not needed for method calls, but I think
> it really should be supplied, to describe how much space is
> available to receive response data.  So provide the supplied length.
>
> This resolves:
>      http://tracker.ceph.com/issues/4659
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c |   13 ++++++-------
>   1 file changed, 6 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 9fb51b5..5e579fa 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1839,12 +1839,11 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	int ret;
>
>   	/*
> -	 * Method calls are ultimately read operations but they
> -	 * don't involve object data (so no offset or length).
> -	 * The result should placed into the inbound buffer
> -	 * provided.  They also supply outbound data--parameters for
> -	 * the object method.  Currently if this is present it will
> -	 * be a snapshot id.
> +	 * Method calls are ultimately read operations.  The result
> +	 * should placed into the inbound buffer provided.  They
> +	 * also supply outbound data--parameters for the object
> +	 * method.  Currently if this is present it will be a
> +	 * snapshot id.
>   	 */
>   	page_count = (u32) calc_pages_for(0, inbound_size);
>   	pages = ceph_alloc_page_vector(page_count, GFP_KERNEL);
> @@ -1852,7 +1851,7 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   		return PTR_ERR(pages);
>
>   	ret = -ENOMEM;
> -	obj_request = rbd_obj_request_create(object_name, 0, 0,
> +	obj_request = rbd_obj_request_create(object_name, 0, inbound_size,
>   							OBJ_REQUEST_PAGES);
>   	if (!obj_request)
>   		goto out;
>

[PATCH 02/20] libceph: compute incoming bytes once

[Alex Elder]

This is a simple change, extracting the number of incoming data
bytes just once in handle_reply().

Signed-off-by: Alex Elder <elder@inktank.com>
---
 net/ceph/osd_client.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 426ca1f..1379b33 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -1293,6 +1293,7 @@ static void handle_reply(struct ceph_osd_client
*osdc, struct ceph_msg *msg,
 	u64 reassert_version;
 	u32 osdmap_epoch;
 	int already_completed;
+	u32 bytes;
 	int i;

 	tid = le64_to_cpu(msg->hdr.tid);
@@ -1347,9 +1348,10 @@ static void handle_reply(struct ceph_osd_client
*osdc, struct ceph_msg *msg,
 		payload_len += len;
 		p += sizeof(*op);
 	}
-	if (payload_len != le32_to_cpu(msg->hdr.data_len)) {
+	bytes = le32_to_cpu(msg->hdr.data_len);
+	if (payload_len != bytes) {
 		pr_warning("sum of op payload lens %d != data_len %d",
-			   payload_len, le32_to_cpu(msg->hdr.data_len));
+			   payload_len, bytes);
 		goto bad_put;
 	}

@@ -1359,10 +1361,8 @@ static void handle_reply(struct ceph_osd_client
*osdc, struct ceph_msg *msg,
 		req->r_reply_op_result[i] = ceph_decode_32(&p);

 	if (!req->r_got_reply) {
-		unsigned int bytes;

 		req->r_result = result;
-		bytes = le32_to_cpu(msg->hdr.data_len);
 		dout("handle_reply result %d bytes %d\n", req->r_result,
 		     bytes);
 		if (req->r_result == 0)
-- 
1.7.9.5

Re: [PATCH 02/20] libceph: compute incoming bytes once

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:01 AM, Alex Elder wrote:
> This is a simple change, extracting the number of incoming data
> bytes just once in handle_reply().
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   net/ceph/osd_client.c |    8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 426ca1f..1379b33 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -1293,6 +1293,7 @@ static void handle_reply(struct ceph_osd_client
> *osdc, struct ceph_msg *msg,
>   	u64 reassert_version;
>   	u32 osdmap_epoch;
>   	int already_completed;
> +	u32 bytes;
>   	int i;
>
>   	tid = le64_to_cpu(msg->hdr.tid);
> @@ -1347,9 +1348,10 @@ static void handle_reply(struct ceph_osd_client
> *osdc, struct ceph_msg *msg,
>   		payload_len += len;
>   		p += sizeof(*op);
>   	}
> -	if (payload_len != le32_to_cpu(msg->hdr.data_len)) {
> +	bytes = le32_to_cpu(msg->hdr.data_len);
> +	if (payload_len != bytes) {
>   		pr_warning("sum of op payload lens %d != data_len %d",
> -			   payload_len, le32_to_cpu(msg->hdr.data_len));
> +			   payload_len, bytes);
>   		goto bad_put;
>   	}
>
> @@ -1359,10 +1361,8 @@ static void handle_reply(struct ceph_osd_client
> *osdc, struct ceph_msg *msg,
>   		req->r_reply_op_result[i] = ceph_decode_32(&p);
>
>   	if (!req->r_got_reply) {
> -		unsigned int bytes;
>
>   		req->r_result = result;
> -		bytes = le32_to_cpu(msg->hdr.data_len);
>   		dout("handle_reply result %d bytes %d\n", req->r_result,
>   		     bytes);
>   		if (req->r_result == 0)
>

[PATCH 03/20] libceph: define osd data initialization helpers

[Alex Elder]

Define and use functions that encapsulate the initializion of a
ceph_osd_data structure.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c             |   14 ++++------
 fs/ceph/addr.c                  |   13 +++------
 fs/ceph/file.c                  |   10 +++----
 include/linux/ceph/osd_client.h |   11 ++++++++
 net/ceph/osd_client.c           |   55
+++++++++++++++++++++++++++------------
 5 files changed, 63 insertions(+), 40 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 5e579fa..a9d88a0 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1350,17 +1350,13 @@ static struct ceph_osd_request *rbd_osd_req_create(
 		break;		/* Nothing to do */
 	case OBJ_REQUEST_BIO:
 		rbd_assert(obj_request->bio_list != NULL);
-		osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
-		osd_data->bio = obj_request->bio_list;
-		osd_data->bio_length = obj_request->length;
+		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+					obj_request->length);
 		break;
 	case OBJ_REQUEST_PAGES:
-		osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-		osd_data->pages = obj_request->pages;
-		osd_data->length = obj_request->length;
-		osd_data->alignment = offset & ~PAGE_MASK;
-		osd_data->pages_from_pool = false;
-		osd_data->own_pages = false;
+		ceph_osd_data_pages_init(osd_data, obj_request->pages,
+				obj_request->length, offset & ~PAGE_MASK,
+				false, false);
 		break;
 	}

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index e0dd74c..8507389 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -342,10 +342,8 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 		}
 		pages[i] = page;
 	}
-	req->r_data_in.type = CEPH_OSD_DATA_TYPE_PAGES;
-	req->r_data_in.pages = pages;
-	req->r_data_in.length = len;
-	req->r_data_in.alignment = 0;
+	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+					false, false);
 	req->r_callback = finish_read;
 	req->r_inode = inode;

@@ -917,11 +915,8 @@ get_more_pages:
 		dout("writepages got %d pages at %llu~%llu\n",
 		     locked_pages, offset, len);

-		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
-		req->r_data_out.pages = pages;
-		req->r_data_out.length = len;
-		req->r_data_out.alignment = 0;
-		req->r_data_out.pages_from_pool = !!pool;
+		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
+						!!pool, false);

 		pages = NULL;	/* request message now owns the pages array */
 		pool = NULL;
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 66b8469..2f2d0a1 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -491,6 +491,7 @@ static ssize_t ceph_sync_write(struct file *file,
const char __user *data,
 	unsigned long buf_align;
 	int ret;
 	struct timespec mtime = CURRENT_TIME;
+	bool own_pages = false;

 	if (ceph_snap(file->f_dentry->d_inode) != CEPH_NOSNAP)
 		return -EROFS;
@@ -571,14 +572,11 @@ more:
 		if ((file->f_flags & O_SYNC) == 0) {
 			/* get a second commit callback */
 			req->r_safe_callback = sync_write_commit;
-			req->r_data_out.own_pages = 1;
+			own_pages = true;
 		}
 	}
-	req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
-	req->r_data_out.pages = pages;
-	req->r_data_out.length = len;
-	req->r_data_out.alignment = page_align;
-	req->r_inode = inode;
+	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+					false, own_pages);

 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
 	ceph_osdc_build_request(req, pos, num_ops, ops,
diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index 5ee1a37..af60dac 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -280,6 +280,17 @@ static inline void ceph_osdc_put_request(struct
ceph_osd_request *req)
 	kref_put(&req->r_kref, ceph_osdc_release_request);
 }

+extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+				     struct page **pages, u64 length,
+				     u32 alignment, bool pages_from_pool,
+				     bool own_pages);
+extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+					struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+				   struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
 extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
 				   struct ceph_osd_request *req,
 				   bool nofail);
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 1379b33..f8f8561 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -79,6 +79,38 @@ static int calc_layout(struct ceph_file_layout
*layout, u64 off, u64 *plen,
 	return 0;
 }

+void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+			struct page **pages, u64 length, u32 alignment,
+			bool pages_from_pool, bool own_pages)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
+	osd_data->pages = pages;
+	osd_data->length = length;
+	osd_data->alignment = alignment;
+	osd_data->pages_from_pool = pages_from_pool;
+	osd_data->own_pages = own_pages;
+}
+EXPORT_SYMBOL(ceph_osd_data_pages_init);
+
+void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+			struct ceph_pagelist *pagelist)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
+	osd_data->pagelist = pagelist;
+}
+EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
+
+#ifdef CONFIG_BLOCK
+void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+			struct bio *bio, size_t bio_length)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
+	osd_data->bio = bio;
+	osd_data->bio_length = bio_length;
+}
+EXPORT_SYMBOL(ceph_osd_data_bio_init);
+#endif /* CONFIG_BLOCK */
+
 /*
  * requests
  */
@@ -400,8 +432,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 		ceph_pagelist_append(pagelist, src->cls.indata,
 				     src->cls.indata_len);

-		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGELIST;
-		req->r_data_out.pagelist = pagelist;
+		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
 		out_data_len = pagelist->length;
 		break;
 	case CEPH_OSD_OP_STARTSYNC:
@@ -2056,7 +2087,6 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
 			struct page **pages, int num_pages, int page_align)
 {
 	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
 	struct ceph_osd_req_op op;
 	int rc = 0;

@@ -2071,14 +2101,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
*osdc,

 	/* it may be a short read due to an object boundary */

-	osd_data = &req->r_data_in;
-	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-	osd_data->pages = pages;
-	osd_data->length = *plen;
-	osd_data->alignment = page_align;
+	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
+				false, false);

 	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
-	     off, *plen, osd_data->length, page_align);
+	     off, *plen, *plen, page_align);

 	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);

@@ -2104,7 +2131,6 @@ int ceph_osdc_writepages(struct ceph_osd_client
*osdc, struct ceph_vino vino,
 			 struct page **pages, int num_pages)
 {
 	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
 	struct ceph_osd_req_op op;
 	int rc = 0;
 	int page_align = off & ~PAGE_MASK;
@@ -2119,12 +2145,9 @@ int ceph_osdc_writepages(struct ceph_osd_client
*osdc, struct ceph_vino vino,
 		return PTR_ERR(req);

 	/* it may be a short write due to an object boundary */
-	osd_data = &req->r_data_out;
-	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-	osd_data->pages = pages;
-	osd_data->length = len;
-	osd_data->alignment = page_align;
-	dout("writepages %llu~%llu (%llu bytes)\n", off, len, osd_data->length);
+	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+				false, false);
+	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

 	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);

-- 
1.7.9.5

Re: [PATCH 03/20] libceph: define osd data initialization helpers

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:02 AM, Alex Elder wrote:
> Define and use functions that encapsulate the initializion of a
> ceph_osd_data structure.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c             |   14 ++++------
>   fs/ceph/addr.c                  |   13 +++------
>   fs/ceph/file.c                  |   10 +++----
>   include/linux/ceph/osd_client.h |   11 ++++++++
>   net/ceph/osd_client.c           |   55
> +++++++++++++++++++++++++++------------
>   5 files changed, 63 insertions(+), 40 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 5e579fa..a9d88a0 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1350,17 +1350,13 @@ static struct ceph_osd_request *rbd_osd_req_create(
>   		break;		/* Nothing to do */
>   	case OBJ_REQUEST_BIO:
>   		rbd_assert(obj_request->bio_list != NULL);
> -		osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
> -		osd_data->bio = obj_request->bio_list;
> -		osd_data->bio_length = obj_request->length;
> +		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
> +					obj_request->length);
>   		break;
>   	case OBJ_REQUEST_PAGES:
> -		osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
> -		osd_data->pages = obj_request->pages;
> -		osd_data->length = obj_request->length;
> -		osd_data->alignment = offset & ~PAGE_MASK;
> -		osd_data->pages_from_pool = false;
> -		osd_data->own_pages = false;
> +		ceph_osd_data_pages_init(osd_data, obj_request->pages,
> +				obj_request->length, offset & ~PAGE_MASK,
> +				false, false);
>   		break;
>   	}
>
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index e0dd74c..8507389 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -342,10 +342,8 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   		}
>   		pages[i] = page;
>   	}
> -	req->r_data_in.type = CEPH_OSD_DATA_TYPE_PAGES;
> -	req->r_data_in.pages = pages;
> -	req->r_data_in.length = len;
> -	req->r_data_in.alignment = 0;
> +	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
> +					false, false);
>   	req->r_callback = finish_read;
>   	req->r_inode = inode;
>
> @@ -917,11 +915,8 @@ get_more_pages:
>   		dout("writepages got %d pages at %llu~%llu\n",
>   		     locked_pages, offset, len);
>
> -		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
> -		req->r_data_out.pages = pages;
> -		req->r_data_out.length = len;
> -		req->r_data_out.alignment = 0;
> -		req->r_data_out.pages_from_pool = !!pool;
> +		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
> +						!!pool, false);
>
>   		pages = NULL;	/* request message now owns the pages array */
>   		pool = NULL;
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 66b8469..2f2d0a1 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -491,6 +491,7 @@ static ssize_t ceph_sync_write(struct file *file,
> const char __user *data,
>   	unsigned long buf_align;
>   	int ret;
>   	struct timespec mtime = CURRENT_TIME;
> +	bool own_pages = false;
>
>   	if (ceph_snap(file->f_dentry->d_inode) != CEPH_NOSNAP)
>   		return -EROFS;
> @@ -571,14 +572,11 @@ more:
>   		if ((file->f_flags & O_SYNC) == 0) {
>   			/* get a second commit callback */
>   			req->r_safe_callback = sync_write_commit;
> -			req->r_data_out.own_pages = 1;
> +			own_pages = true;
>   		}
>   	}
> -	req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
> -	req->r_data_out.pages = pages;
> -	req->r_data_out.length = len;
> -	req->r_data_out.alignment = page_align;
> -	req->r_inode = inode;
> +	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
> +					false, own_pages);
>
>   	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
>   	ceph_osdc_build_request(req, pos, num_ops, ops,
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index 5ee1a37..af60dac 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -280,6 +280,17 @@ static inline void ceph_osdc_put_request(struct
> ceph_osd_request *req)
>   	kref_put(&req->r_kref, ceph_osdc_release_request);
>   }
>
> +extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
> +				     struct page **pages, u64 length,
> +				     u32 alignment, bool pages_from_pool,
> +				     bool own_pages);
> +extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
> +					struct ceph_pagelist *pagelist);
> +#ifdef CONFIG_BLOCK
> +extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
> +				   struct bio *bio, size_t bio_length);
> +#endif /* CONFIG_BLOCK */
> +
>   extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
>   				   struct ceph_osd_request *req,
>   				   bool nofail);
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 1379b33..f8f8561 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -79,6 +79,38 @@ static int calc_layout(struct ceph_file_layout
> *layout, u64 off, u64 *plen,
>   	return 0;
>   }
>
> +void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
> +			struct page **pages, u64 length, u32 alignment,
> +			bool pages_from_pool, bool own_pages)
> +{
> +	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
> +	osd_data->pages = pages;
> +	osd_data->length = length;
> +	osd_data->alignment = alignment;
> +	osd_data->pages_from_pool = pages_from_pool;
> +	osd_data->own_pages = own_pages;
> +}
> +EXPORT_SYMBOL(ceph_osd_data_pages_init);
> +
> +void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
> +			struct ceph_pagelist *pagelist)
> +{
> +	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
> +	osd_data->pagelist = pagelist;
> +}
> +EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
> +
> +#ifdef CONFIG_BLOCK
> +void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
> +			struct bio *bio, size_t bio_length)
> +{
> +	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
> +	osd_data->bio = bio;
> +	osd_data->bio_length = bio_length;
> +}
> +EXPORT_SYMBOL(ceph_osd_data_bio_init);
> +#endif /* CONFIG_BLOCK */
> +
>   /*
>    * requests
>    */
> @@ -400,8 +432,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   		ceph_pagelist_append(pagelist, src->cls.indata,
>   				     src->cls.indata_len);
>
> -		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGELIST;
> -		req->r_data_out.pagelist = pagelist;
> +		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
>   		out_data_len = pagelist->length;
>   		break;
>   	case CEPH_OSD_OP_STARTSYNC:
> @@ -2056,7 +2087,6 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
>   			struct page **pages, int num_pages, int page_align)
>   {
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_data *osd_data;
>   	struct ceph_osd_req_op op;
>   	int rc = 0;
>
> @@ -2071,14 +2101,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
> *osdc,
>
>   	/* it may be a short read due to an object boundary */
>
> -	osd_data = &req->r_data_in;
> -	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
> -	osd_data->pages = pages;
> -	osd_data->length = *plen;
> -	osd_data->alignment = page_align;
> +	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
> +				false, false);
>
>   	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
> -	     off, *plen, osd_data->length, page_align);
> +	     off, *plen, *plen, page_align);
>
>   	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
>
> @@ -2104,7 +2131,6 @@ int ceph_osdc_writepages(struct ceph_osd_client
> *osdc, struct ceph_vino vino,
>   			 struct page **pages, int num_pages)
>   {
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_data *osd_data;
>   	struct ceph_osd_req_op op;
>   	int rc = 0;
>   	int page_align = off & ~PAGE_MASK;
> @@ -2119,12 +2145,9 @@ int ceph_osdc_writepages(struct ceph_osd_client
> *osdc, struct ceph_vino vino,
>   		return PTR_ERR(req);
>
>   	/* it may be a short write due to an object boundary */
> -	osd_data = &req->r_data_out;
> -	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
> -	osd_data->pages = pages;
> -	osd_data->length = len;
> -	osd_data->alignment = page_align;
> -	dout("writepages %llu~%llu (%llu bytes)\n", off, len, osd_data->length);
> +	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
> +				false, false);
> +	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
>
>   	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
>

[PATCH 04/20] libceph: define a few more helpers

[Alex Elder]

Define ceph_osd_data_init() and ceph_osd_data_release() to clean up
a little code.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 net/ceph/osd_client.c |   44 ++++++++++++++++++++++++++------------------
 1 file changed, 26 insertions(+), 18 deletions(-)

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index f8f8561..df0f856 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -79,6 +79,12 @@ static int calc_layout(struct ceph_file_layout
*layout, u64 off, u64 *plen,
 	return 0;
 }

+static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
+{
+	memset(osd_data, 0, sizeof *osd_data);
+	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
+}
+
 void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
 			struct page **pages, u64 length, u32 alignment,
 			bool pages_from_pool, bool own_pages)
@@ -111,16 +117,28 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
*osd_data,
 EXPORT_SYMBOL(ceph_osd_data_bio_init);
 #endif /* CONFIG_BLOCK */

+static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
+{
+	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
+		return;
+
+	if (osd_data->own_pages) {
+		int num_pages;
+
+		num_pages = calc_pages_for((u64)osd_data->alignment,
+						(u64)osd_data->length);
+		ceph_release_page_vector(osd_data->pages, num_pages);
+	}
+}
+
 /*
  * requests
  */
 void ceph_osdc_release_request(struct kref *kref)
 {
-	int num_pages;
-	struct ceph_osd_request *req = container_of(kref,
-						    struct ceph_osd_request,
-						    r_kref);
+	struct ceph_osd_request *req;

+	req = container_of(kref, struct ceph_osd_request, r_kref);
 	if (req->r_request)
 		ceph_msg_put(req->r_request);
 	if (req->r_reply) {
@@ -128,18 +146,8 @@ void ceph_osdc_release_request(struct kref *kref)
 		ceph_msg_put(req->r_reply);
 	}

-	if (req->r_data_in.type == CEPH_OSD_DATA_TYPE_PAGES &&
-			req->r_data_in.own_pages) {
-		num_pages = calc_pages_for((u64)req->r_data_in.alignment,
-						(u64)req->r_data_in.length);
-		ceph_release_page_vector(req->r_data_in.pages, num_pages);
-	}
-	if (req->r_data_out.type == CEPH_OSD_DATA_TYPE_PAGES &&
-			req->r_data_out.own_pages) {
-		num_pages = calc_pages_for((u64)req->r_data_out.alignment,
-						(u64)req->r_data_out.length);
-		ceph_release_page_vector(req->r_data_out.pages, num_pages);
-	}
+	ceph_osd_data_release(&req->r_data_in);
+	ceph_osd_data_release(&req->r_data_out);

 	ceph_put_snap_context(req->r_snapc);
 	if (req->r_mempool)
@@ -203,8 +211,8 @@ struct ceph_osd_request
*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
 	}
 	req->r_reply = msg;

-	req->r_data_in.type = CEPH_OSD_DATA_TYPE_NONE;
-	req->r_data_out.type = CEPH_OSD_DATA_TYPE_NONE;
+	ceph_osd_data_init(&req->r_data_in);
+	ceph_osd_data_init(&req->r_data_out);

 	/* create request message; allow space for oid */
 	if (use_mempool)
-- 
1.7.9.5

Re: [PATCH 04/20] libceph: define a few more helpers

[Josh Durgin]

There's a sizeof without parentheses, but after fixing that:

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:02 AM, Alex Elder wrote:
> Define ceph_osd_data_init() and ceph_osd_data_release() to clean up
> a little code.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   net/ceph/osd_client.c |   44 ++++++++++++++++++++++++++------------------
>   1 file changed, 26 insertions(+), 18 deletions(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index f8f8561..df0f856 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -79,6 +79,12 @@ static int calc_layout(struct ceph_file_layout
> *layout, u64 off, u64 *plen,
>   	return 0;
>   }
>
> +static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
> +{
> +	memset(osd_data, 0, sizeof *osd_data);
> +	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
> +}
> +
>   void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
>   			struct page **pages, u64 length, u32 alignment,
>   			bool pages_from_pool, bool own_pages)
> @@ -111,16 +117,28 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
> *osd_data,
>   EXPORT_SYMBOL(ceph_osd_data_bio_init);
>   #endif /* CONFIG_BLOCK */
>
> +static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
> +{
> +	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
> +		return;
> +
> +	if (osd_data->own_pages) {
> +		int num_pages;
> +
> +		num_pages = calc_pages_for((u64)osd_data->alignment,
> +						(u64)osd_data->length);
> +		ceph_release_page_vector(osd_data->pages, num_pages);
> +	}
> +}
> +
>   /*
>    * requests
>    */
>   void ceph_osdc_release_request(struct kref *kref)
>   {
> -	int num_pages;
> -	struct ceph_osd_request *req = container_of(kref,
> -						    struct ceph_osd_request,
> -						    r_kref);
> +	struct ceph_osd_request *req;
>
> +	req = container_of(kref, struct ceph_osd_request, r_kref);
>   	if (req->r_request)
>   		ceph_msg_put(req->r_request);
>   	if (req->r_reply) {
> @@ -128,18 +146,8 @@ void ceph_osdc_release_request(struct kref *kref)
>   		ceph_msg_put(req->r_reply);
>   	}
>
> -	if (req->r_data_in.type == CEPH_OSD_DATA_TYPE_PAGES &&
> -			req->r_data_in.own_pages) {
> -		num_pages = calc_pages_for((u64)req->r_data_in.alignment,
> -						(u64)req->r_data_in.length);
> -		ceph_release_page_vector(req->r_data_in.pages, num_pages);
> -	}
> -	if (req->r_data_out.type == CEPH_OSD_DATA_TYPE_PAGES &&
> -			req->r_data_out.own_pages) {
> -		num_pages = calc_pages_for((u64)req->r_data_out.alignment,
> -						(u64)req->r_data_out.length);
> -		ceph_release_page_vector(req->r_data_out.pages, num_pages);
> -	}
> +	ceph_osd_data_release(&req->r_data_in);
> +	ceph_osd_data_release(&req->r_data_out);
>
>   	ceph_put_snap_context(req->r_snapc);
>   	if (req->r_mempool)
> @@ -203,8 +211,8 @@ struct ceph_osd_request
> *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
>   	}
>   	req->r_reply = msg;
>
> -	req->r_data_in.type = CEPH_OSD_DATA_TYPE_NONE;
> -	req->r_data_out.type = CEPH_OSD_DATA_TYPE_NONE;
> +	ceph_osd_data_init(&req->r_data_in);
> +	ceph_osd_data_init(&req->r_data_out);
>
>   	/* create request message; allow space for oid */
>   	if (use_mempool)
>

[PATCH 05/20] libceph: define ceph_osd_data_length()

[Alex Elder]

One more osd data helper, which returns the length of the
data item, regardless of its type.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 net/ceph/osd_client.c |   31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index df0f856..110cd83 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -117,6 +117,25 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
*osd_data,
 EXPORT_SYMBOL(ceph_osd_data_bio_init);
 #endif /* CONFIG_BLOCK */

+static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
+{
+	switch (osd_data->type) {
+	case CEPH_OSD_DATA_TYPE_NONE:
+		return 0;
+	case CEPH_OSD_DATA_TYPE_PAGES:
+		return osd_data->length;
+	case CEPH_OSD_DATA_TYPE_PAGELIST:
+		return (u64)osd_data->pagelist->length;
+#ifdef CONFIG_BLOCK
+	case CEPH_OSD_DATA_TYPE_BIO:
+		return (u64)osd_data->bio_length;
+#endif /* CONFIG_BLOCK */
+	default:
+		WARN(true, "unrecognized data type %d\n", (int)osd_data->type);
+		return 0;
+	}
+}
+
 static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
 {
 	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
@@ -1887,17 +1906,19 @@ bad:
 static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
 				struct ceph_osd_data *osd_data)
 {
+	u64 length = ceph_osd_data_length(osd_data);
+
 	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
-		BUG_ON(osd_data->length > (u64) SIZE_MAX);
-		if (osd_data->length)
+		BUG_ON(length > (u64) SIZE_MAX);
+		if (length)
 			ceph_msg_data_set_pages(msg, osd_data->pages,
-				osd_data->length, osd_data->alignment);
+					length, osd_data->alignment);
 	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
-		BUG_ON(!osd_data->pagelist->length);
+		BUG_ON(!length);
 		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
 #ifdef CONFIG_BLOCK
 	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
-		ceph_msg_data_set_bio(msg, osd_data->bio, osd_data->bio_length);
+		ceph_msg_data_set_bio(msg, osd_data->bio, length);
 #endif
 	} else {
 		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
-- 
1.7.9.5

Re: [PATCH 05/20] libceph: define ceph_osd_data_length()

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:02 AM, Alex Elder wrote:
> One more osd data helper, which returns the length of the
> data item, regardless of its type.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   net/ceph/osd_client.c |   31 ++++++++++++++++++++++++++-----
>   1 file changed, 26 insertions(+), 5 deletions(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index df0f856..110cd83 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -117,6 +117,25 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
> *osd_data,
>   EXPORT_SYMBOL(ceph_osd_data_bio_init);
>   #endif /* CONFIG_BLOCK */
>
> +static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
> +{
> +	switch (osd_data->type) {
> +	case CEPH_OSD_DATA_TYPE_NONE:
> +		return 0;
> +	case CEPH_OSD_DATA_TYPE_PAGES:
> +		return osd_data->length;
> +	case CEPH_OSD_DATA_TYPE_PAGELIST:
> +		return (u64)osd_data->pagelist->length;
> +#ifdef CONFIG_BLOCK
> +	case CEPH_OSD_DATA_TYPE_BIO:
> +		return (u64)osd_data->bio_length;
> +#endif /* CONFIG_BLOCK */
> +	default:
> +		WARN(true, "unrecognized data type %d\n", (int)osd_data->type);
> +		return 0;
> +	}
> +}
> +
>   static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
>   {
>   	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
> @@ -1887,17 +1906,19 @@ bad:
>   static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
>   				struct ceph_osd_data *osd_data)
>   {
> +	u64 length = ceph_osd_data_length(osd_data);
> +
>   	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
> -		BUG_ON(osd_data->length > (u64) SIZE_MAX);
> -		if (osd_data->length)
> +		BUG_ON(length > (u64) SIZE_MAX);
> +		if (length)
>   			ceph_msg_data_set_pages(msg, osd_data->pages,
> -				osd_data->length, osd_data->alignment);
> +					length, osd_data->alignment);
>   	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
> -		BUG_ON(!osd_data->pagelist->length);
> +		BUG_ON(!length);
>   		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
>   #ifdef CONFIG_BLOCK
>   	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
> -		ceph_msg_data_set_bio(msg, osd_data->bio, osd_data->bio_length);
> +		ceph_msg_data_set_bio(msg, osd_data->bio, length);
>   #endif
>   	} else {
>   		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
>

[PATCH 06/20] libceph: a few more osd data cleanups

[Alex Elder]

These are very small changes that make use osd_data local pointers
as shorthands for structures being operated on.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 fs/ceph/addr.c |   30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 8507389..c46b9d3 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -236,6 +236,7 @@ static int ceph_readpage(struct file *filp, struct
page *page)
 static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg)
 {
 	struct inode *inode = req->r_inode;
+	struct ceph_osd_data *osd_data;
 	int rc = req->r_result;
 	int bytes = le32_to_cpu(msg->hdr.data_len);
 	int num_pages;
@@ -244,11 +245,12 @@ static void finish_read(struct ceph_osd_request
*req, struct ceph_msg *msg)
 	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);

 	/* unlock all pages, zeroing any data we didn't read */
-	BUG_ON(req->r_data_in.type != CEPH_OSD_DATA_TYPE_PAGES);
-	num_pages = calc_pages_for((u64)req->r_data_in.alignment,
-					(u64)req->r_data_in.length);
+	osd_data = &req->r_data_in;
+	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+	num_pages = calc_pages_for((u64)osd_data->alignment,
+					(u64)osd_data->length);
 	for (i = 0; i < num_pages; i++) {
-		struct page *page = req->r_data_in.pages[i];
+		struct page *page = osd_data->pages[i];

 		if (bytes < (int)PAGE_CACHE_SIZE) {
 			/* zero (remainder of) page */
@@ -263,7 +265,7 @@ static void finish_read(struct ceph_osd_request
*req, struct ceph_msg *msg)
 		page_cache_release(page);
 		bytes -= PAGE_CACHE_SIZE;
 	}
-	kfree(req->r_data_in.pages);
+	kfree(osd_data->pages);
 }

 static void ceph_unlock_page_vector(struct page **pages, int num_pages)
@@ -557,6 +559,7 @@ static void writepages_finish(struct
ceph_osd_request *req,
 {
 	struct inode *inode = req->r_inode;
 	struct ceph_inode_info *ci = ceph_inode(inode);
+	struct ceph_osd_data *osd_data;
 	unsigned wrote;
 	struct page *page;
 	int num_pages;
@@ -569,9 +572,10 @@ static void writepages_finish(struct
ceph_osd_request *req,
 	long writeback_stat;
 	unsigned issued = ceph_caps_issued(ci);

-	BUG_ON(req->r_data_out.type != CEPH_OSD_DATA_TYPE_PAGES);
-	num_pages = calc_pages_for((u64)req->r_data_out.alignment,
-					(u64)req->r_data_out.length);
+	osd_data = &req->r_data_out;
+	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+	num_pages = calc_pages_for((u64)osd_data->alignment,
+					(u64)osd_data->length);
 	if (rc >= 0) {
 		/*
 		 * Assume we wrote the pages we originally sent.  The
@@ -589,7 +593,7 @@ static void writepages_finish(struct
ceph_osd_request *req,

 	/* clean all pages */
 	for (i = 0; i < num_pages; i++) {
-		page = req->r_data_out.pages[i];
+		page = osd_data->pages[i];
 		BUG_ON(!page);
 		WARN_ON(!PageUptodate(page));

@@ -620,12 +624,12 @@ static void writepages_finish(struct
ceph_osd_request *req,
 	dout("%p wrote+cleaned %d pages\n", inode, wrote);
 	ceph_put_wrbuffer_cap_refs(ci, num_pages, snapc);

-	ceph_release_pages(req->r_data_out.pages, num_pages);
-	if (req->r_data_out.pages_from_pool)
-		mempool_free(req->r_data_out.pages,
+	ceph_release_pages(osd_data->pages, num_pages);
+	if (osd_data->pages_from_pool)
+		mempool_free(osd_data->pages,
 			     ceph_sb_to_client(inode->i_sb)->wb_pagevec_pool);
 	else
-		kfree(req->r_data_out.pages);
+		kfree(osd_data->pages);
 	ceph_osdc_put_request(req);
 }

-- 
1.7.9.5

Re: [PATCH 06/20] libceph: a few more osd data cleanups

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:02 AM, Alex Elder wrote:
> These are very small changes that make use osd_data local pointers
> as shorthands for structures being operated on.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   fs/ceph/addr.c |   30 +++++++++++++++++-------------
>   1 file changed, 17 insertions(+), 13 deletions(-)
>
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index 8507389..c46b9d3 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -236,6 +236,7 @@ static int ceph_readpage(struct file *filp, struct
> page *page)
>   static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg)
>   {
>   	struct inode *inode = req->r_inode;
> +	struct ceph_osd_data *osd_data;
>   	int rc = req->r_result;
>   	int bytes = le32_to_cpu(msg->hdr.data_len);
>   	int num_pages;
> @@ -244,11 +245,12 @@ static void finish_read(struct ceph_osd_request
> *req, struct ceph_msg *msg)
>   	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);
>
>   	/* unlock all pages, zeroing any data we didn't read */
> -	BUG_ON(req->r_data_in.type != CEPH_OSD_DATA_TYPE_PAGES);
> -	num_pages = calc_pages_for((u64)req->r_data_in.alignment,
> -					(u64)req->r_data_in.length);
> +	osd_data = &req->r_data_in;
> +	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
> +	num_pages = calc_pages_for((u64)osd_data->alignment,
> +					(u64)osd_data->length);
>   	for (i = 0; i < num_pages; i++) {
> -		struct page *page = req->r_data_in.pages[i];
> +		struct page *page = osd_data->pages[i];
>
>   		if (bytes < (int)PAGE_CACHE_SIZE) {
>   			/* zero (remainder of) page */
> @@ -263,7 +265,7 @@ static void finish_read(struct ceph_osd_request
> *req, struct ceph_msg *msg)
>   		page_cache_release(page);
>   		bytes -= PAGE_CACHE_SIZE;
>   	}
> -	kfree(req->r_data_in.pages);
> +	kfree(osd_data->pages);
>   }
>
>   static void ceph_unlock_page_vector(struct page **pages, int num_pages)
> @@ -557,6 +559,7 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>   {
>   	struct inode *inode = req->r_inode;
>   	struct ceph_inode_info *ci = ceph_inode(inode);
> +	struct ceph_osd_data *osd_data;
>   	unsigned wrote;
>   	struct page *page;
>   	int num_pages;
> @@ -569,9 +572,10 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>   	long writeback_stat;
>   	unsigned issued = ceph_caps_issued(ci);
>
> -	BUG_ON(req->r_data_out.type != CEPH_OSD_DATA_TYPE_PAGES);
> -	num_pages = calc_pages_for((u64)req->r_data_out.alignment,
> -					(u64)req->r_data_out.length);
> +	osd_data = &req->r_data_out;
> +	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
> +	num_pages = calc_pages_for((u64)osd_data->alignment,
> +					(u64)osd_data->length);
>   	if (rc >= 0) {
>   		/*
>   		 * Assume we wrote the pages we originally sent.  The
> @@ -589,7 +593,7 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>
>   	/* clean all pages */
>   	for (i = 0; i < num_pages; i++) {
> -		page = req->r_data_out.pages[i];
> +		page = osd_data->pages[i];
>   		BUG_ON(!page);
>   		WARN_ON(!PageUptodate(page));
>
> @@ -620,12 +624,12 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>   	dout("%p wrote+cleaned %d pages\n", inode, wrote);
>   	ceph_put_wrbuffer_cap_refs(ci, num_pages, snapc);
>
> -	ceph_release_pages(req->r_data_out.pages, num_pages);
> -	if (req->r_data_out.pages_from_pool)
> -		mempool_free(req->r_data_out.pages,
> +	ceph_release_pages(osd_data->pages, num_pages);
> +	if (osd_data->pages_from_pool)
> +		mempool_free(osd_data->pages,
>   			     ceph_sb_to_client(inode->i_sb)->wb_pagevec_pool);
>   	else
> -		kfree(req->r_data_out.pages);
> +		kfree(osd_data->pages);
>   	ceph_osdc_put_request(req);
>   }
>

[PATCH 07/20] rbd: define rbd_osd_req_format_op()

[Alex Elder]

Define rbd_osd_req_format_op(), which encapsulates formatting
an osd op into an object request's osd request message.  Only
one op is supported right now.

Stop calling ceph_osdc_build_request() in rbd_osd_req_create().
Instead, call rbd_osd_req_format_op() in each of the callers of
rbd_osd_req_create().

This is to prepare for the next patch, in which the source ops for
an osd request will be held in the osd request itself.  Because of
that, we won't have the source op to work with until after the
request is created, so we can't format the op until then.

This an the next patch resolve:
    http://tracker.ceph.com/issues/4656

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   98
++++++++++++++++++++++++++++-----------------------
 1 file changed, 53 insertions(+), 45 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index a9d88a0..fc41675 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1311,29 +1311,47 @@ static void rbd_osd_req_callback(struct
ceph_osd_request *osd_req,
 		rbd_obj_request_complete(obj_request);
 }

+static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
+					bool write_request,
+					struct ceph_osd_req_op *op)
+{
+	struct rbd_img_request *img_request = obj_request->img_request;
+	struct ceph_snap_context *snapc = NULL;
+	u64 snap_id = CEPH_NOSNAP;
+	struct timespec *mtime = NULL;
+	struct timespec now;
+
+	rbd_assert(obj_request->osd_req != NULL);
+
+	if (write_request) {
+		now = CURRENT_TIME;
+		mtime = &now;
+		if (img_request)
+			snapc = img_request->snapc;
+	} else if (img_request) {
+		snap_id = img_request->snap_id;
+	}
+
+	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+			1, op, snapc, snap_id, mtime);
+}
+
 static struct ceph_osd_request *rbd_osd_req_create(
 					struct rbd_device *rbd_dev,
 					bool write_request,
-					struct rbd_obj_request *obj_request,
-					struct ceph_osd_req_op *op)
+					struct rbd_obj_request *obj_request)
 {
 	struct rbd_img_request *img_request = obj_request->img_request;
 	struct ceph_snap_context *snapc = NULL;
 	struct ceph_osd_client *osdc;
 	struct ceph_osd_request *osd_req;
 	struct ceph_osd_data *osd_data;
-	struct timespec now;
-	struct timespec *mtime;
-	u64 snap_id = CEPH_NOSNAP;
 	u64 offset = obj_request->offset;
-	u64 length = obj_request->length;

 	if (img_request) {
 		rbd_assert(img_request->write_request == write_request);
 		if (img_request->write_request)
 			snapc = img_request->snapc;
-		else
-			snap_id = img_request->snap_id;
 	}

 	/* Allocate and initialize the request, for the single op */
@@ -1360,16 +1378,10 @@ static struct ceph_osd_request *rbd_osd_req_create(
 		break;
 	}

-	if (write_request) {
+	if (write_request)
 		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
-		now = CURRENT_TIME;
-		mtime = &now;
-	} else {
+	else
 		osd_req->r_flags = CEPH_OSD_FLAG_READ;
-		mtime = NULL;	/* not needed for reads */
-		offset = 0;	/* These are not used... */
-		length = 0;	/* ...for osd read requests */
-	}

 	osd_req->r_callback = rbd_osd_req_callback;
 	osd_req->r_priv = obj_request;
@@ -1380,11 +1392,6 @@ static struct ceph_osd_request *rbd_osd_req_create(

 	osd_req->r_file_layout = rbd_dev->layout;	/* struct */

-	/* osd_req will get its own reference to snapc (if non-null) */
-
-	ceph_osdc_build_request(osd_req, offset, 1, op,
-				snapc, snap_id, mtime);
-
 	return osd_req;
 }

@@ -1538,6 +1545,7 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 	struct rbd_device *rbd_dev = img_request->rbd_dev;
 	struct rbd_obj_request *obj_request = NULL;
 	struct rbd_obj_request *next_obj_request;
+	bool write_request = img_request->write_request;
 	unsigned int bio_offset;
 	u64 image_offset;
 	u64 resid;
@@ -1545,8 +1553,7 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,

 	dout("%s: img %p bio %p\n", __func__, img_request, bio_list);

-	opcode = img_request->write_request ? CEPH_OSD_OP_WRITE
-					      : CEPH_OSD_OP_READ;
+	opcode = write_request ? CEPH_OSD_OP_WRITE : CEPH_OSD_OP_READ;
 	bio_offset = 0;
 	image_offset = img_request->offset;
 	rbd_assert(image_offset == bio_list->bi_sector << SECTOR_SHIFT);
@@ -1579,17 +1586,14 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		if (!obj_request->bio_list)
 			goto out_partial;

-		/*
-		 * Build up the op to use in building the osd
-		 * request.  Note that the contents of the op are
-		 * copied by rbd_osd_req_create().
-		 */
-		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
 		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
-						img_request->write_request,
-						obj_request, &op);
+						write_request, obj_request);
 		if (!obj_request->osd_req)
 			goto out_partial;
+
+		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
+		rbd_osd_req_format_op(obj_request, write_request, &op);
+
 		/* status and version are initially zero-filled */

 		rbd_img_obj_request_add(img_request, obj_request);
@@ -1699,12 +1703,13 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 		return -ENOMEM;

 	ret = -ENOMEM;
-	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
 	if (!obj_request->osd_req)
 		goto out;

+	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
 	osdc = &rbd_dev->rbd_client->client->osdc;
 	obj_request->callback = rbd_obj_request_put;
 	ret = rbd_obj_request_submit(osdc, obj_request);
@@ -1763,13 +1768,14 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 	if (!obj_request)
 		goto out_cancel;

+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true, obj_request);
+	if (!obj_request->osd_req)
+		goto out_cancel;
+
 	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
 				rbd_dev->watch_event->cookie,
 				rbd_dev->header.obj_version, start);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true,
-							obj_request, &op);
-	if (!obj_request->osd_req)
-		goto out_cancel;
+	rbd_osd_req_format_op(obj_request, true, &op);

 	if (start)
 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1855,13 +1861,14 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	obj_request->pages = pages;
 	obj_request->page_count = page_count;

-	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
-					outbound, outbound_size);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
 	if (!obj_request->osd_req)
 		goto out;

+	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+					outbound, outbound_size);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
@@ -2060,12 +2067,13 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	obj_request->pages = pages;
 	obj_request->page_count = page_count;

-	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
 	if (!obj_request->osd_req)
 		goto out;

+	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
-- 
1.7.9.5

Re: [PATCH 07/20] rbd: define rbd_osd_req_format_op()

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:03 AM, Alex Elder wrote:
> Define rbd_osd_req_format_op(), which encapsulates formatting
> an osd op into an object request's osd request message.  Only
> one op is supported right now.
>
> Stop calling ceph_osdc_build_request() in rbd_osd_req_create().
> Instead, call rbd_osd_req_format_op() in each of the callers of
> rbd_osd_req_create().
>
> This is to prepare for the next patch, in which the source ops for
> an osd request will be held in the osd request itself.  Because of
> that, we won't have the source op to work with until after the
> request is created, so we can't format the op until then.
>
> This an the next patch resolve:
>      http://tracker.ceph.com/issues/4656
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c |   98
> ++++++++++++++++++++++++++++-----------------------
>   1 file changed, 53 insertions(+), 45 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index a9d88a0..fc41675 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1311,29 +1311,47 @@ static void rbd_osd_req_callback(struct
> ceph_osd_request *osd_req,
>   		rbd_obj_request_complete(obj_request);
>   }
>
> +static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
> +					bool write_request,
> +					struct ceph_osd_req_op *op)
> +{
> +	struct rbd_img_request *img_request = obj_request->img_request;
> +	struct ceph_snap_context *snapc = NULL;
> +	u64 snap_id = CEPH_NOSNAP;
> +	struct timespec *mtime = NULL;
> +	struct timespec now;
> +
> +	rbd_assert(obj_request->osd_req != NULL);
> +
> +	if (write_request) {
> +		now = CURRENT_TIME;
> +		mtime = &now;
> +		if (img_request)
> +			snapc = img_request->snapc;
> +	} else if (img_request) {
> +		snap_id = img_request->snap_id;
> +	}
> +
> +	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
> +			1, op, snapc, snap_id, mtime);
> +}
> +
>   static struct ceph_osd_request *rbd_osd_req_create(
>   					struct rbd_device *rbd_dev,
>   					bool write_request,
> -					struct rbd_obj_request *obj_request,
> -					struct ceph_osd_req_op *op)
> +					struct rbd_obj_request *obj_request)
>   {
>   	struct rbd_img_request *img_request = obj_request->img_request;
>   	struct ceph_snap_context *snapc = NULL;
>   	struct ceph_osd_client *osdc;
>   	struct ceph_osd_request *osd_req;
>   	struct ceph_osd_data *osd_data;
> -	struct timespec now;
> -	struct timespec *mtime;
> -	u64 snap_id = CEPH_NOSNAP;
>   	u64 offset = obj_request->offset;
> -	u64 length = obj_request->length;
>
>   	if (img_request) {
>   		rbd_assert(img_request->write_request == write_request);
>   		if (img_request->write_request)
>   			snapc = img_request->snapc;
> -		else
> -			snap_id = img_request->snap_id;
>   	}
>
>   	/* Allocate and initialize the request, for the single op */
> @@ -1360,16 +1378,10 @@ static struct ceph_osd_request *rbd_osd_req_create(
>   		break;
>   	}
>
> -	if (write_request) {
> +	if (write_request)
>   		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
> -		now = CURRENT_TIME;
> -		mtime = &now;
> -	} else {
> +	else
>   		osd_req->r_flags = CEPH_OSD_FLAG_READ;
> -		mtime = NULL;	/* not needed for reads */
> -		offset = 0;	/* These are not used... */
> -		length = 0;	/* ...for osd read requests */
> -	}
>
>   	osd_req->r_callback = rbd_osd_req_callback;
>   	osd_req->r_priv = obj_request;
> @@ -1380,11 +1392,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
>
>   	osd_req->r_file_layout = rbd_dev->layout;	/* struct */
>
> -	/* osd_req will get its own reference to snapc (if non-null) */
> -
> -	ceph_osdc_build_request(osd_req, offset, 1, op,
> -				snapc, snap_id, mtime);
> -
>   	return osd_req;
>   }
>
> @@ -1538,6 +1545,7 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   	struct rbd_device *rbd_dev = img_request->rbd_dev;
>   	struct rbd_obj_request *obj_request = NULL;
>   	struct rbd_obj_request *next_obj_request;
> +	bool write_request = img_request->write_request;
>   	unsigned int bio_offset;
>   	u64 image_offset;
>   	u64 resid;
> @@ -1545,8 +1553,7 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>
>   	dout("%s: img %p bio %p\n", __func__, img_request, bio_list);
>
> -	opcode = img_request->write_request ? CEPH_OSD_OP_WRITE
> -					      : CEPH_OSD_OP_READ;
> +	opcode = write_request ? CEPH_OSD_OP_WRITE : CEPH_OSD_OP_READ;
>   	bio_offset = 0;
>   	image_offset = img_request->offset;
>   	rbd_assert(image_offset == bio_list->bi_sector << SECTOR_SHIFT);
> @@ -1579,17 +1586,14 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   		if (!obj_request->bio_list)
>   			goto out_partial;
>
> -		/*
> -		 * Build up the op to use in building the osd
> -		 * request.  Note that the contents of the op are
> -		 * copied by rbd_osd_req_create().
> -		 */
> -		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
>   		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
> -						img_request->write_request,
> -						obj_request, &op);
> +						write_request, obj_request);
>   		if (!obj_request->osd_req)
>   			goto out_partial;
> +
> +		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
> +		rbd_osd_req_format_op(obj_request, write_request, &op);
> +
>   		/* status and version are initially zero-filled */
>
>   		rbd_img_obj_request_add(img_request, obj_request);
> @@ -1699,12 +1703,13 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>   		return -ENOMEM;
>
>   	ret = -ENOMEM;
> -	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
> -	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
> -						obj_request, &op);
> +	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
>   	if (!obj_request->osd_req)
>   		goto out;
>
> +	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
> +	rbd_osd_req_format_op(obj_request, false, &op);
> +
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	obj_request->callback = rbd_obj_request_put;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> @@ -1763,13 +1768,14 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   	if (!obj_request)
>   		goto out_cancel;
>
> +	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true, obj_request);
> +	if (!obj_request->osd_req)
> +		goto out_cancel;
> +
>   	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
>   				rbd_dev->watch_event->cookie,
>   				rbd_dev->header.obj_version, start);
> -	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true,
> -							obj_request, &op);
> -	if (!obj_request->osd_req)
> -		goto out_cancel;
> +	rbd_osd_req_format_op(obj_request, true, &op);
>
>   	if (start)
>   		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
> @@ -1855,13 +1861,14 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	obj_request->pages = pages;
>   	obj_request->page_count = page_count;
>
> -	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
> -					outbound, outbound_size);
> -	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
> -						obj_request, &op);
> +	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
>   	if (!obj_request->osd_req)
>   		goto out;
>
> +	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
> +					outbound, outbound_size);
> +	rbd_osd_req_format_op(obj_request, false, &op);
> +
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
>   	if (ret)
> @@ -2060,12 +2067,13 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	obj_request->pages = pages;
>   	obj_request->page_count = page_count;
>
> -	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
> -	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
> -						obj_request, &op);
> +	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
>   	if (!obj_request->osd_req)
>   		goto out;
>
> +	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
> +	rbd_osd_req_format_op(obj_request, false, &op);
> +
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
>   	if (ret)
>

[PATCH 08/20] libceph: keep source rather than message osd op array

[Alex Elder]

An osd request keeps a pointer to the osd operations (ops) array
that it builds in its request message.

In order to allow each op in the array to have its own distinct
data, we will need to keep track of each op's data, and that
information does not go over the wire.

As long as we're tracking the data we might as well just track the
entire (source) op definition for each of the ops.  And if we're
doing that, we'll have no more need to keep a pointer to the
wire-encoded version.

This patch makes the array of source ops be kept with the osd
request structure, and uses that instead of the version encoded in
the message in places where that was previously used.  The array
will be embedded in the request structure, and the maximum number of
ops we ever actually use is currently 2.  So reduce CEPH_OSD_MAX_OP
to 2 to reduce the size of the structure.

The result of doing this sort of ripples back up, and as a result
various function parameters and local variables become unnecessary.

Make r_num_ops be unsigned, and move the definition of struct
ceph_osd_req_op earlier to ensure it's defined where needed.

It does not yet add per-op data, that's coming soon.

This resolves:
    http://tracker.ceph.com/issues/4656

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c             |   42 ++++++++++++-----------
 fs/ceph/addr.c                  |   21 +++++-------
 fs/ceph/file.c                  |    6 ++--
 include/linux/ceph/osd_client.h |   70
+++++++++++++++++++--------------------
 net/ceph/debugfs.c              |    4 +--
 net/ceph/osd_client.c           |   53 ++++++++++++++---------------
 6 files changed, 97 insertions(+), 99 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index fc41675..e6e2191 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1285,7 +1285,7 @@ static void rbd_osd_req_callback(struct
ceph_osd_request *osd_req,
 	 */
 	obj_request->xferred = osd_req->r_reply_op_len[0];
 	rbd_assert(obj_request->xferred < (u64) UINT_MAX);
-	opcode = osd_req->r_request_ops[0].op;
+	opcode = osd_req->r_ops[0].op;
 	switch (opcode) {
 	case CEPH_OSD_OP_READ:
 		rbd_osd_read_callback(obj_request);
@@ -1312,8 +1312,7 @@ static void rbd_osd_req_callback(struct
ceph_osd_request *osd_req,
 }

 static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
-					bool write_request,
-					struct ceph_osd_req_op *op)
+					bool write_request)
 {
 	struct rbd_img_request *img_request = obj_request->img_request;
 	struct ceph_snap_context *snapc = NULL;
@@ -1333,7 +1332,7 @@ static void rbd_osd_req_format_op(struct
rbd_obj_request *obj_request,
 	}

 	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
-			1, op, snapc, snap_id, mtime);
+			snapc, snap_id, mtime);
 }

 static struct ceph_osd_request *rbd_osd_req_create(
@@ -1562,7 +1561,7 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 	while (resid) {
 		const char *object_name;
 		unsigned int clone_size;
-		struct ceph_osd_req_op op;
+		struct ceph_osd_req_op *op;
 		u64 offset;
 		u64 length;

@@ -1591,8 +1590,9 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		if (!obj_request->osd_req)
 			goto out_partial;

-		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
-		rbd_osd_req_format_op(obj_request, write_request, &op);
+		op = &obj_request->osd_req->r_ops[0];
+		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+		rbd_osd_req_format_op(obj_request, write_request);

 		/* status and version are initially zero-filled */

@@ -1693,7 +1693,7 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 				   u64 ver, u64 notify_id)
 {
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
 	struct ceph_osd_client *osdc;
 	int ret;

@@ -1707,8 +1707,9 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	rbd_osd_req_format_op(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	obj_request->callback = rbd_obj_request_put;
@@ -1748,7 +1749,7 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 {
 	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
 	int ret;

 	rbd_assert(start ^ !!rbd_dev->watch_event);
@@ -1772,10 +1773,11 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 	if (!obj_request->osd_req)
 		goto out_cancel;

-	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
 				rbd_dev->watch_event->cookie,
 				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format_op(obj_request, true, &op);
+	rbd_osd_req_format_op(obj_request, true);

 	if (start)
 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1835,7 +1837,7 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 {
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_client *osdc;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
 	struct page **pages;
 	u32 page_count;
 	int ret;
@@ -1865,9 +1867,10 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
 					outbound, outbound_size);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	rbd_osd_req_format_op(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2045,8 +2048,8 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 				char *buf, u64 *version)

 {
-	struct ceph_osd_req_op op;
 	struct rbd_obj_request *obj_request;
+	struct ceph_osd_req_op *op;
 	struct ceph_osd_client *osdc;
 	struct page **pages = NULL;
 	u32 page_count;
@@ -2071,8 +2074,9 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	rbd_osd_req_format_op(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index c46b9d3..12bbba7 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -288,7 +288,6 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 	struct page *page = list_entry(page_list->prev, struct page, lru);
 	struct ceph_vino vino;
 	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
 	u64 off;
 	u64 len;
 	int i;
@@ -314,7 +313,7 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 	     off, len);
 	vino = ceph_vino(inode);
 	req = ceph_osdc_new_request(osdc, &ci->i_layout, vino, off, &len,
-				    1, &op, CEPH_OSD_OP_READ,
+				    1, CEPH_OSD_OP_READ,
 				    CEPH_OSD_FLAG_READ, NULL,
 				    ci->i_truncate_seq, ci->i_truncate_size,
 				    false);
@@ -349,7 +348,7 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 	req->r_callback = finish_read;
 	req->r_inode = inode;

-	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);

 	dout("start_read %p starting %p %lld~%lld\n", inode, req, off, len);
 	ret = ceph_osdc_start_request(osdc, req, false);
@@ -567,7 +566,7 @@ static void writepages_finish(struct
ceph_osd_request *req,
 	struct ceph_snap_context *snapc = req->r_snapc;
 	struct address_space *mapping = inode->i_mapping;
 	int rc = req->r_result;
-	u64 bytes = le64_to_cpu(req->r_request_ops[0].extent.length);
+	u64 bytes = req->r_ops[0].extent.length;
 	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
 	long writeback_stat;
 	unsigned issued = ceph_caps_issued(ci);
@@ -635,8 +634,7 @@ static void writepages_finish(struct
ceph_osd_request *req,

 static struct ceph_osd_request *
 ceph_writepages_osd_request(struct inode *inode, u64 offset, u64 *len,
-				struct ceph_snap_context *snapc,
-				int num_ops, struct ceph_osd_req_op *ops)
+				struct ceph_snap_context *snapc, int num_ops)
 {
 	struct ceph_fs_client *fsc;
 	struct ceph_inode_info *ci;
@@ -648,7 +646,7 @@ ceph_writepages_osd_request(struct inode *inode, u64
offset, u64 *len,
 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */

 	return ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
-			vino, offset, len, num_ops, ops, CEPH_OSD_OP_WRITE,
+			vino, offset, len, num_ops, CEPH_OSD_OP_WRITE,
 			CEPH_OSD_FLAG_WRITE|CEPH_OSD_FLAG_ONDISK,
 			snapc, ci->i_truncate_seq, ci->i_truncate_size, true);
 }
@@ -738,7 +736,6 @@ retry:
 	last_snapc = snapc;

 	while (!done && index <= end) {
-		struct ceph_osd_req_op ops[2];
 		int num_ops = do_sync ? 2 : 1;
 		struct ceph_vino vino;
 		unsigned i;
@@ -846,7 +843,7 @@ get_more_pages:
 				len = wsize;
 				req = ceph_writepages_osd_request(inode,
 							offset, &len, snapc,
-							num_ops, ops);
+							num_ops);

 				if (IS_ERR(req)) {
 					rc = PTR_ERR(req);
@@ -927,11 +924,11 @@ get_more_pages:

 		/* Update the write op length in case we changed it */

-		osd_req_op_extent_update(&ops[0], len);
+		osd_req_op_extent_update(&req->r_ops[0], len);

 		vino = ceph_vino(inode);
-		ceph_osdc_build_request(req, offset, num_ops, ops,
-					snapc, vino.snap, &inode->i_mtime);
+		ceph_osdc_build_request(req, offset, snapc, vino.snap,
+					&inode->i_mtime);

 		rc = ceph_osdc_start_request(&fsc->client->osdc, req, true);
 		BUG_ON(rc);
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 2f2d0a1..a1d85a8 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -478,7 +478,6 @@ static ssize_t ceph_sync_write(struct file *file,
const char __user *data,
 	struct ceph_snap_context *snapc;
 	struct ceph_vino vino;
 	struct ceph_osd_request *req;
-	struct ceph_osd_req_op ops[2];
 	int num_ops = 1;
 	struct page **pages;
 	int num_pages;
@@ -534,7 +533,7 @@ more:
 	snapc = ci->i_snap_realm->cached_context;
 	vino = ceph_vino(inode);
 	req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
-				    vino, pos, &len, num_ops, ops,
+				    vino, pos, &len, num_ops,
 				    CEPH_OSD_OP_WRITE, flags, snapc,
 				    ci->i_truncate_seq, ci->i_truncate_size,
 				    false);
@@ -579,8 +578,7 @@ more:
 					false, own_pages);

 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
-	ceph_osdc_build_request(req, pos, num_ops, ops,
-				snapc, vino.snap, &mtime);
+	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);

 	ret = ceph_osdc_start_request(&fsc->client->osdc, req, false);
 	if (!ret) {
diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index af60dac..f4c1a2a 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -48,7 +48,7 @@ struct ceph_osd {
 };


-#define CEPH_OSD_MAX_OP 10
+#define CEPH_OSD_MAX_OP	2

 enum ceph_osd_data_type {
 	CEPH_OSD_DATA_TYPE_NONE,
@@ -79,6 +79,34 @@ struct ceph_osd_data {
 	};
 };

+struct ceph_osd_req_op {
+	u16 op;           /* CEPH_OSD_OP_* */
+	u32 payload_len;
+	union {
+		struct {
+			u64 offset, length;
+			u64 truncate_size;
+			u32 truncate_seq;
+		} extent;
+		struct {
+			const char *class_name;
+			const char *method_name;
+			const void *indata;
+			u32 indata_len;
+			__u8 class_len;
+			__u8 method_len;
+			__u8 argc;
+		} cls;
+		struct {
+			u64 cookie;
+			u64 ver;
+			u32 prot_ver;
+			u32 timeout;
+			__u8 flag;
+		} watch;
+	};
+};
+
 /* an in-flight request */
 struct ceph_osd_request {
 	u64             r_tid;              /* unique for this client */
@@ -95,10 +123,11 @@ struct ceph_osd_request {
 	struct ceph_msg  *r_request, *r_reply;
 	int               r_flags;     /* any additional flags for the osd */
 	u32               r_sent;      /* >0 if r_request is sending/sent */
-	int               r_num_ops;

-	/* encoded message content */
-	struct ceph_osd_op *r_request_ops;
+	/* request osd ops array  */
+	unsigned int		r_num_ops;
+	struct ceph_osd_req_op	r_ops[CEPH_OSD_MAX_OP];
+
 	/* these are updated on each send */
 	__le32           *r_request_osdmap_epoch;
 	__le32           *r_request_flags;
@@ -193,34 +222,6 @@ struct ceph_osd_client {
 	struct workqueue_struct	*notify_wq;
 };

-struct ceph_osd_req_op {
-	u16 op;           /* CEPH_OSD_OP_* */
-	u32 payload_len;
-	union {
-		struct {
-			u64 offset, length;
-			u64 truncate_size;
-			u32 truncate_seq;
-		} extent;
-		struct {
-			const char *class_name;
-			const char *method_name;
-			const void *indata;
-			u32 indata_len;
-			__u8 class_len;
-			__u8 method_len;
-			__u8 argc;
-		} cls;
-		struct {
-			u64 cookie;
-			u64 ver;
-			u32 prot_ver;
-			u32 timeout;
-			__u8 flag;
-		} watch;
-	};
-};
-
 extern int ceph_osdc_init(struct ceph_osd_client *osdc,
 			  struct ceph_client *client);
 extern void ceph_osdc_stop(struct ceph_osd_client *osdc);
@@ -249,8 +250,6 @@ extern struct ceph_osd_request
*ceph_osdc_alloc_request(struct ceph_osd_client *
 					       gfp_t gfp_flags);

 extern void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
-				    unsigned int num_ops,
-				    struct ceph_osd_req_op *src_ops,
 				    struct ceph_snap_context *snapc,
 				    u64 snap_id,
 				    struct timespec *mtime);
@@ -259,8 +258,7 @@ extern struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *,
 				      struct ceph_file_layout *layout,
 				      struct ceph_vino vino,
 				      u64 offset, u64 *len,
-				      int num_ops, struct ceph_osd_req_op *ops,
-				      int opcode, int flags,
+				      int num_ops, int opcode, int flags,
 				      struct ceph_snap_context *snapc,
 				      u32 truncate_seq, u64 truncate_size,
 				      bool use_mempool);
diff --git a/net/ceph/debugfs.c b/net/ceph/debugfs.c
index 00d051f..83661cd 100644
--- a/net/ceph/debugfs.c
+++ b/net/ceph/debugfs.c
@@ -123,8 +123,8 @@ static int osdc_show(struct seq_file *s, void *pp)
 	mutex_lock(&osdc->request_mutex);
 	for (p = rb_first(&osdc->requests); p; p = rb_next(p)) {
 		struct ceph_osd_request *req;
+		unsigned int i;
 		int opcode;
-		int i;

 		req = rb_entry(p, struct ceph_osd_request, r_node);

@@ -142,7 +142,7 @@ static int osdc_show(struct seq_file *s, void *pp)
 			seq_printf(s, "\t");

 		for (i = 0; i < req->r_num_ops; i++) {
-			opcode = le16_to_cpu(req->r_request_ops[i].op);
+			opcode = req->r_ops[i].op;
 			seq_printf(s, "\t%s", ceph_osd_op_name(opcode));
 		}

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 110cd83..ed4493d 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -186,6 +186,9 @@ struct ceph_osd_request
*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
 	struct ceph_msg *msg;
 	size_t msg_size;

+	BUILD_BUG_ON(CEPH_OSD_MAX_OP > U16_MAX);
+	BUG_ON(num_ops > CEPH_OSD_MAX_OP);
+
 	msg_size = 4 + 4 + 8 + 8 + 4+8;
 	msg_size += 2 + 4 + 8 + 4 + 4; /* oloc */
 	msg_size += 1 + 8 + 4 + 4;     /* pg_t */
@@ -207,6 +210,7 @@ struct ceph_osd_request
*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,

 	req->r_osdc = osdc;
 	req->r_mempool = use_mempool;
+	req->r_num_ops = num_ops;

 	kref_init(&req->r_kref);
 	init_completion(&req->r_completion);
@@ -418,12 +422,14 @@ void osd_req_op_watch_init(struct ceph_osd_req_op
*op, u16 opcode,
 EXPORT_SYMBOL(osd_req_op_watch_init);

 static u64 osd_req_encode_op(struct ceph_osd_request *req,
-			      struct ceph_osd_op *dst,
-			      struct ceph_osd_req_op *src)
+			      struct ceph_osd_op *dst, unsigned int which)
 {
+	struct ceph_osd_req_op *src;
 	u64 out_data_len = 0;
 	struct ceph_pagelist *pagelist;

+	BUG_ON(which >= req->r_num_ops);
+	src = &req->r_ops[which];
 	if (WARN_ON(!osd_req_opcode_valid(src->op))) {
 		pr_err("unrecognized osd opcode %d\n", src->op);

@@ -487,21 +493,17 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
  * build new request AND message
  *
  */
-void ceph_osdc_build_request(struct ceph_osd_request *req,
-			     u64 off, unsigned int num_ops,
-			     struct ceph_osd_req_op *src_ops,
-			     struct ceph_snap_context *snapc, u64 snap_id,
-			     struct timespec *mtime)
+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+				struct ceph_snap_context *snapc, u64 snap_id,
+				struct timespec *mtime)
 {
 	struct ceph_msg *msg = req->r_request;
-	struct ceph_osd_req_op *src_op;
 	void *p;
 	size_t msg_size;
 	int flags = req->r_flags;
 	u64 data_len;
-	int i;
+	unsigned int i;

-	req->r_num_ops = num_ops;
 	req->r_snapid = snap_id;
 	req->r_snapc = ceph_get_snap_context(snapc);

@@ -541,12 +543,10 @@ void ceph_osdc_build_request(struct
ceph_osd_request *req,
 	p += req->r_oid_len;

 	/* ops--can imply data */
-	ceph_encode_16(&p, num_ops);
-	src_op = src_ops;
-	req->r_request_ops = p;
+	ceph_encode_16(&p, (u16)req->r_num_ops);
 	data_len = 0;
-	for (i = 0; i < num_ops; i++, src_op++) {
-		data_len += osd_req_encode_op(req, p, src_op);
+	for (i = 0; i < req->r_num_ops; i++) {
+		data_len += osd_req_encode_op(req, p, i);
 		p += sizeof(struct ceph_osd_op);
 	}

@@ -602,7 +602,6 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					       struct ceph_file_layout *layout,
 					       struct ceph_vino vino,
 					       u64 off, u64 *plen, int num_ops,
-					       struct ceph_osd_req_op *ops,
 					       int opcode, int flags,
 					       struct ceph_snap_context *snapc,
 					       u32 truncate_seq,
@@ -610,6 +609,7 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					       bool use_mempool)
 {
 	struct ceph_osd_request *req;
+	struct ceph_osd_req_op *op;
 	u64 objnum = 0;
 	u64 objoff = 0;
 	u64 objlen = 0;
@@ -623,6 +623,7 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					GFP_NOFS);
 	if (!req)
 		return ERR_PTR(-ENOMEM);
+
 	req->r_flags = flags;

 	/* calculate max write size */
@@ -642,7 +643,8 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 			truncate_size = object_size;
 	}

-	osd_req_op_extent_init(&ops[0], opcode, objoff, objlen,
+	op = &req->r_ops[0];
+	osd_req_op_extent_init(op, opcode, objoff, objlen,
 				truncate_size, truncate_seq);
 	/*
 	 * A second op in the ops array means the caller wants to
@@ -650,7 +652,7 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 	 * osd will flush data quickly.
 	 */
 	if (num_ops > 1)
-		osd_req_op_init(&ops[1], CEPH_OSD_OP_STARTSYNC);
+		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);

 	req->r_file_layout = *layout;  /* keep a copy */

@@ -1342,7 +1344,8 @@ static void handle_reply(struct ceph_osd_client
*osdc, struct ceph_msg *msg,
 	struct ceph_osd_request *req;
 	u64 tid;
 	int object_len;
-	int numops, payload_len, flags;
+	unsigned int numops;
+	int payload_len, flags;
 	s32 result;
 	s32 retry_attempt;
 	struct ceph_pg pg;
@@ -1352,7 +1355,7 @@ static void handle_reply(struct ceph_osd_client
*osdc, struct ceph_msg *msg,
 	u32 osdmap_epoch;
 	int already_completed;
 	u32 bytes;
-	int i;
+	unsigned int i;

 	tid = le64_to_cpu(msg->hdr.tid);
 	dout("handle_reply %p tid %llu\n", msg, tid);
@@ -2116,12 +2119,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
*osdc,
 			struct page **pages, int num_pages, int page_align)
 {
 	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
 	int rc = 0;

 	dout("readpages on ino %llx.%llx on %llu~%llu\n", vino.ino,
 	     vino.snap, off, *plen);
-	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1, &op,
+	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1,
 				    CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ,
 				    NULL, truncate_seq, truncate_size,
 				    false);
@@ -2136,7 +2138,7 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
 	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
 	     off, *plen, *plen, page_align);

-	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);

 	rc = ceph_osdc_start_request(osdc, req, false);
 	if (!rc)
@@ -2160,12 +2162,11 @@ int ceph_osdc_writepages(struct ceph_osd_client
*osdc, struct ceph_vino vino,
 			 struct page **pages, int num_pages)
 {
 	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
 	int rc = 0;
 	int page_align = off & ~PAGE_MASK;

 	BUG_ON(vino.snap != CEPH_NOSNAP);	/* snapshots aren't writeable */
-	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1, &op,
+	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1,
 				    CEPH_OSD_OP_WRITE,
 				    CEPH_OSD_FLAG_ONDISK | CEPH_OSD_FLAG_WRITE,
 				    snapc, truncate_seq, truncate_size,
@@ -2178,7 +2179,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
*osdc, struct ceph_vino vino,
 				false, false);
 	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

-	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
+	ceph_osdc_build_request(req, off, snapc, CEPH_NOSNAP, mtime);

 	rc = ceph_osdc_start_request(osdc, req, true);
 	if (!rc)
-- 
1.7.9.5

Re: [PATCH 08/20] libceph: keep source rather than message osd op array

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:03 AM, Alex Elder wrote:
> An osd request keeps a pointer to the osd operations (ops) array
> that it builds in its request message.
>
> In order to allow each op in the array to have its own distinct
> data, we will need to keep track of each op's data, and that
> information does not go over the wire.
>
> As long as we're tracking the data we might as well just track the
> entire (source) op definition for each of the ops.  And if we're
> doing that, we'll have no more need to keep a pointer to the
> wire-encoded version.
>
> This patch makes the array of source ops be kept with the osd
> request structure, and uses that instead of the version encoded in
> the message in places where that was previously used.  The array
> will be embedded in the request structure, and the maximum number of
> ops we ever actually use is currently 2.  So reduce CEPH_OSD_MAX_OP
> to 2 to reduce the size of the structure.
>
> The result of doing this sort of ripples back up, and as a result
> various function parameters and local variables become unnecessary.
>
> Make r_num_ops be unsigned, and move the definition of struct
> ceph_osd_req_op earlier to ensure it's defined where needed.
>
> It does not yet add per-op data, that's coming soon.
>
> This resolves:
>      http://tracker.ceph.com/issues/4656
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c             |   42 ++++++++++++-----------
>   fs/ceph/addr.c                  |   21 +++++-------
>   fs/ceph/file.c                  |    6 ++--
>   include/linux/ceph/osd_client.h |   70
> +++++++++++++++++++--------------------
>   net/ceph/debugfs.c              |    4 +--
>   net/ceph/osd_client.c           |   53 ++++++++++++++---------------
>   6 files changed, 97 insertions(+), 99 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index fc41675..e6e2191 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1285,7 +1285,7 @@ static void rbd_osd_req_callback(struct
> ceph_osd_request *osd_req,
>   	 */
>   	obj_request->xferred = osd_req->r_reply_op_len[0];
>   	rbd_assert(obj_request->xferred < (u64) UINT_MAX);
> -	opcode = osd_req->r_request_ops[0].op;
> +	opcode = osd_req->r_ops[0].op;
>   	switch (opcode) {
>   	case CEPH_OSD_OP_READ:
>   		rbd_osd_read_callback(obj_request);
> @@ -1312,8 +1312,7 @@ static void rbd_osd_req_callback(struct
> ceph_osd_request *osd_req,
>   }
>
>   static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
> -					bool write_request,
> -					struct ceph_osd_req_op *op)
> +					bool write_request)
>   {
>   	struct rbd_img_request *img_request = obj_request->img_request;
>   	struct ceph_snap_context *snapc = NULL;
> @@ -1333,7 +1332,7 @@ static void rbd_osd_req_format_op(struct
> rbd_obj_request *obj_request,
>   	}
>
>   	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
> -			1, op, snapc, snap_id, mtime);
> +			snapc, snap_id, mtime);
>   }
>
>   static struct ceph_osd_request *rbd_osd_req_create(
> @@ -1562,7 +1561,7 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   	while (resid) {
>   		const char *object_name;
>   		unsigned int clone_size;
> -		struct ceph_osd_req_op op;
> +		struct ceph_osd_req_op *op;
>   		u64 offset;
>   		u64 length;
>
> @@ -1591,8 +1590,9 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   		if (!obj_request->osd_req)
>   			goto out_partial;
>
> -		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
> -		rbd_osd_req_format_op(obj_request, write_request, &op);
> +		op = &obj_request->osd_req->r_ops[0];
> +		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
> +		rbd_osd_req_format_op(obj_request, write_request);
>
>   		/* status and version are initially zero-filled */
>
> @@ -1693,7 +1693,7 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>   				   u64 ver, u64 notify_id)
>   {
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_req_op op;
> +	struct ceph_osd_req_op *op;
>   	struct ceph_osd_client *osdc;
>   	int ret;
>
> @@ -1707,8 +1707,9 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
> -	rbd_osd_req_format_op(obj_request, false, &op);
> +	op = &obj_request->osd_req->r_ops[0];
> +	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
> +	rbd_osd_req_format_op(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	obj_request->callback = rbd_obj_request_put;
> @@ -1748,7 +1749,7 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   {
>   	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_req_op op;
> +	struct ceph_osd_req_op *op;
>   	int ret;
>
>   	rbd_assert(start ^ !!rbd_dev->watch_event);
> @@ -1772,10 +1773,11 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   	if (!obj_request->osd_req)
>   		goto out_cancel;
>
> -	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
> +	op = &obj_request->osd_req->r_ops[0];
> +	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
>   				rbd_dev->watch_event->cookie,
>   				rbd_dev->header.obj_version, start);
> -	rbd_osd_req_format_op(obj_request, true, &op);
> +	rbd_osd_req_format_op(obj_request, true);
>
>   	if (start)
>   		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
> @@ -1835,7 +1837,7 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   {
>   	struct rbd_obj_request *obj_request;
>   	struct ceph_osd_client *osdc;
> -	struct ceph_osd_req_op op;
> +	struct ceph_osd_req_op *op;
>   	struct page **pages;
>   	u32 page_count;
>   	int ret;
> @@ -1865,9 +1867,10 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
> +	op = &obj_request->osd_req->r_ops[0];
> +	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
>   					outbound, outbound_size);
> -	rbd_osd_req_format_op(obj_request, false, &op);
> +	rbd_osd_req_format_op(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> @@ -2045,8 +2048,8 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   				char *buf, u64 *version)
>
>   {
> -	struct ceph_osd_req_op op;
>   	struct rbd_obj_request *obj_request;
> +	struct ceph_osd_req_op *op;
>   	struct ceph_osd_client *osdc;
>   	struct page **pages = NULL;
>   	u32 page_count;
> @@ -2071,8 +2074,9 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
> -	rbd_osd_req_format_op(obj_request, false, &op);
> +	op = &obj_request->osd_req->r_ops[0];
> +	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
> +	rbd_osd_req_format_op(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index c46b9d3..12bbba7 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -288,7 +288,6 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   	struct page *page = list_entry(page_list->prev, struct page, lru);
>   	struct ceph_vino vino;
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_req_op op;
>   	u64 off;
>   	u64 len;
>   	int i;
> @@ -314,7 +313,7 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   	     off, len);
>   	vino = ceph_vino(inode);
>   	req = ceph_osdc_new_request(osdc, &ci->i_layout, vino, off, &len,
> -				    1, &op, CEPH_OSD_OP_READ,
> +				    1, CEPH_OSD_OP_READ,
>   				    CEPH_OSD_FLAG_READ, NULL,
>   				    ci->i_truncate_seq, ci->i_truncate_size,
>   				    false);
> @@ -349,7 +348,7 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   	req->r_callback = finish_read;
>   	req->r_inode = inode;
>
> -	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
> +	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);
>
>   	dout("start_read %p starting %p %lld~%lld\n", inode, req, off, len);
>   	ret = ceph_osdc_start_request(osdc, req, false);
> @@ -567,7 +566,7 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>   	struct ceph_snap_context *snapc = req->r_snapc;
>   	struct address_space *mapping = inode->i_mapping;
>   	int rc = req->r_result;
> -	u64 bytes = le64_to_cpu(req->r_request_ops[0].extent.length);
> +	u64 bytes = req->r_ops[0].extent.length;
>   	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
>   	long writeback_stat;
>   	unsigned issued = ceph_caps_issued(ci);
> @@ -635,8 +634,7 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>
>   static struct ceph_osd_request *
>   ceph_writepages_osd_request(struct inode *inode, u64 offset, u64 *len,
> -				struct ceph_snap_context *snapc,
> -				int num_ops, struct ceph_osd_req_op *ops)
> +				struct ceph_snap_context *snapc, int num_ops)
>   {
>   	struct ceph_fs_client *fsc;
>   	struct ceph_inode_info *ci;
> @@ -648,7 +646,7 @@ ceph_writepages_osd_request(struct inode *inode, u64
> offset, u64 *len,
>   	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
>
>   	return ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
> -			vino, offset, len, num_ops, ops, CEPH_OSD_OP_WRITE,
> +			vino, offset, len, num_ops, CEPH_OSD_OP_WRITE,
>   			CEPH_OSD_FLAG_WRITE|CEPH_OSD_FLAG_ONDISK,
>   			snapc, ci->i_truncate_seq, ci->i_truncate_size, true);
>   }
> @@ -738,7 +736,6 @@ retry:
>   	last_snapc = snapc;
>
>   	while (!done && index <= end) {
> -		struct ceph_osd_req_op ops[2];
>   		int num_ops = do_sync ? 2 : 1;
>   		struct ceph_vino vino;
>   		unsigned i;
> @@ -846,7 +843,7 @@ get_more_pages:
>   				len = wsize;
>   				req = ceph_writepages_osd_request(inode,
>   							offset, &len, snapc,
> -							num_ops, ops);
> +							num_ops);
>
>   				if (IS_ERR(req)) {
>   					rc = PTR_ERR(req);
> @@ -927,11 +924,11 @@ get_more_pages:
>
>   		/* Update the write op length in case we changed it */
>
> -		osd_req_op_extent_update(&ops[0], len);
> +		osd_req_op_extent_update(&req->r_ops[0], len);
>
>   		vino = ceph_vino(inode);
> -		ceph_osdc_build_request(req, offset, num_ops, ops,
> -					snapc, vino.snap, &inode->i_mtime);
> +		ceph_osdc_build_request(req, offset, snapc, vino.snap,
> +					&inode->i_mtime);
>
>   		rc = ceph_osdc_start_request(&fsc->client->osdc, req, true);
>   		BUG_ON(rc);
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 2f2d0a1..a1d85a8 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -478,7 +478,6 @@ static ssize_t ceph_sync_write(struct file *file,
> const char __user *data,
>   	struct ceph_snap_context *snapc;
>   	struct ceph_vino vino;
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_req_op ops[2];
>   	int num_ops = 1;
>   	struct page **pages;
>   	int num_pages;
> @@ -534,7 +533,7 @@ more:
>   	snapc = ci->i_snap_realm->cached_context;
>   	vino = ceph_vino(inode);
>   	req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
> -				    vino, pos, &len, num_ops, ops,
> +				    vino, pos, &len, num_ops,
>   				    CEPH_OSD_OP_WRITE, flags, snapc,
>   				    ci->i_truncate_seq, ci->i_truncate_size,
>   				    false);
> @@ -579,8 +578,7 @@ more:
>   					false, own_pages);
>
>   	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
> -	ceph_osdc_build_request(req, pos, num_ops, ops,
> -				snapc, vino.snap, &mtime);
> +	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
>
>   	ret = ceph_osdc_start_request(&fsc->client->osdc, req, false);
>   	if (!ret) {
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index af60dac..f4c1a2a 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -48,7 +48,7 @@ struct ceph_osd {
>   };
>
>
> -#define CEPH_OSD_MAX_OP 10
> +#define CEPH_OSD_MAX_OP	2
>
>   enum ceph_osd_data_type {
>   	CEPH_OSD_DATA_TYPE_NONE,
> @@ -79,6 +79,34 @@ struct ceph_osd_data {
>   	};
>   };
>
> +struct ceph_osd_req_op {
> +	u16 op;           /* CEPH_OSD_OP_* */
> +	u32 payload_len;
> +	union {
> +		struct {
> +			u64 offset, length;
> +			u64 truncate_size;
> +			u32 truncate_seq;
> +		} extent;
> +		struct {
> +			const char *class_name;
> +			const char *method_name;
> +			const void *indata;
> +			u32 indata_len;
> +			__u8 class_len;
> +			__u8 method_len;
> +			__u8 argc;
> +		} cls;
> +		struct {
> +			u64 cookie;
> +			u64 ver;
> +			u32 prot_ver;
> +			u32 timeout;
> +			__u8 flag;
> +		} watch;
> +	};
> +};
> +
>   /* an in-flight request */
>   struct ceph_osd_request {
>   	u64             r_tid;              /* unique for this client */
> @@ -95,10 +123,11 @@ struct ceph_osd_request {
>   	struct ceph_msg  *r_request, *r_reply;
>   	int               r_flags;     /* any additional flags for the osd */
>   	u32               r_sent;      /* >0 if r_request is sending/sent */
> -	int               r_num_ops;
>
> -	/* encoded message content */
> -	struct ceph_osd_op *r_request_ops;
> +	/* request osd ops array  */
> +	unsigned int		r_num_ops;
> +	struct ceph_osd_req_op	r_ops[CEPH_OSD_MAX_OP];
> +
>   	/* these are updated on each send */
>   	__le32           *r_request_osdmap_epoch;
>   	__le32           *r_request_flags;
> @@ -193,34 +222,6 @@ struct ceph_osd_client {
>   	struct workqueue_struct	*notify_wq;
>   };
>
> -struct ceph_osd_req_op {
> -	u16 op;           /* CEPH_OSD_OP_* */
> -	u32 payload_len;
> -	union {
> -		struct {
> -			u64 offset, length;
> -			u64 truncate_size;
> -			u32 truncate_seq;
> -		} extent;
> -		struct {
> -			const char *class_name;
> -			const char *method_name;
> -			const void *indata;
> -			u32 indata_len;
> -			__u8 class_len;
> -			__u8 method_len;
> -			__u8 argc;
> -		} cls;
> -		struct {
> -			u64 cookie;
> -			u64 ver;
> -			u32 prot_ver;
> -			u32 timeout;
> -			__u8 flag;
> -		} watch;
> -	};
> -};
> -
>   extern int ceph_osdc_init(struct ceph_osd_client *osdc,
>   			  struct ceph_client *client);
>   extern void ceph_osdc_stop(struct ceph_osd_client *osdc);
> @@ -249,8 +250,6 @@ extern struct ceph_osd_request
> *ceph_osdc_alloc_request(struct ceph_osd_client *
>   					       gfp_t gfp_flags);
>
>   extern void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
> -				    unsigned int num_ops,
> -				    struct ceph_osd_req_op *src_ops,
>   				    struct ceph_snap_context *snapc,
>   				    u64 snap_id,
>   				    struct timespec *mtime);
> @@ -259,8 +258,7 @@ extern struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *,
>   				      struct ceph_file_layout *layout,
>   				      struct ceph_vino vino,
>   				      u64 offset, u64 *len,
> -				      int num_ops, struct ceph_osd_req_op *ops,
> -				      int opcode, int flags,
> +				      int num_ops, int opcode, int flags,
>   				      struct ceph_snap_context *snapc,
>   				      u32 truncate_seq, u64 truncate_size,
>   				      bool use_mempool);
> diff --git a/net/ceph/debugfs.c b/net/ceph/debugfs.c
> index 00d051f..83661cd 100644
> --- a/net/ceph/debugfs.c
> +++ b/net/ceph/debugfs.c
> @@ -123,8 +123,8 @@ static int osdc_show(struct seq_file *s, void *pp)
>   	mutex_lock(&osdc->request_mutex);
>   	for (p = rb_first(&osdc->requests); p; p = rb_next(p)) {
>   		struct ceph_osd_request *req;
> +		unsigned int i;
>   		int opcode;
> -		int i;
>
>   		req = rb_entry(p, struct ceph_osd_request, r_node);
>
> @@ -142,7 +142,7 @@ static int osdc_show(struct seq_file *s, void *pp)
>   			seq_printf(s, "\t");
>
>   		for (i = 0; i < req->r_num_ops; i++) {
> -			opcode = le16_to_cpu(req->r_request_ops[i].op);
> +			opcode = req->r_ops[i].op;
>   			seq_printf(s, "\t%s", ceph_osd_op_name(opcode));
>   		}
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 110cd83..ed4493d 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -186,6 +186,9 @@ struct ceph_osd_request
> *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
>   	struct ceph_msg *msg;
>   	size_t msg_size;
>
> +	BUILD_BUG_ON(CEPH_OSD_MAX_OP > U16_MAX);
> +	BUG_ON(num_ops > CEPH_OSD_MAX_OP);
> +
>   	msg_size = 4 + 4 + 8 + 8 + 4+8;
>   	msg_size += 2 + 4 + 8 + 4 + 4; /* oloc */
>   	msg_size += 1 + 8 + 4 + 4;     /* pg_t */
> @@ -207,6 +210,7 @@ struct ceph_osd_request
> *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
>
>   	req->r_osdc = osdc;
>   	req->r_mempool = use_mempool;
> +	req->r_num_ops = num_ops;
>
>   	kref_init(&req->r_kref);
>   	init_completion(&req->r_completion);
> @@ -418,12 +422,14 @@ void osd_req_op_watch_init(struct ceph_osd_req_op
> *op, u16 opcode,
>   EXPORT_SYMBOL(osd_req_op_watch_init);
>
>   static u64 osd_req_encode_op(struct ceph_osd_request *req,
> -			      struct ceph_osd_op *dst,
> -			      struct ceph_osd_req_op *src)
> +			      struct ceph_osd_op *dst, unsigned int which)
>   {
> +	struct ceph_osd_req_op *src;
>   	u64 out_data_len = 0;
>   	struct ceph_pagelist *pagelist;
>
> +	BUG_ON(which >= req->r_num_ops);
> +	src = &req->r_ops[which];
>   	if (WARN_ON(!osd_req_opcode_valid(src->op))) {
>   		pr_err("unrecognized osd opcode %d\n", src->op);
>
> @@ -487,21 +493,17 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>    * build new request AND message
>    *
>    */
> -void ceph_osdc_build_request(struct ceph_osd_request *req,
> -			     u64 off, unsigned int num_ops,
> -			     struct ceph_osd_req_op *src_ops,
> -			     struct ceph_snap_context *snapc, u64 snap_id,
> -			     struct timespec *mtime)
> +void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
> +				struct ceph_snap_context *snapc, u64 snap_id,
> +				struct timespec *mtime)
>   {
>   	struct ceph_msg *msg = req->r_request;
> -	struct ceph_osd_req_op *src_op;
>   	void *p;
>   	size_t msg_size;
>   	int flags = req->r_flags;
>   	u64 data_len;
> -	int i;
> +	unsigned int i;
>
> -	req->r_num_ops = num_ops;
>   	req->r_snapid = snap_id;
>   	req->r_snapc = ceph_get_snap_context(snapc);
>
> @@ -541,12 +543,10 @@ void ceph_osdc_build_request(struct
> ceph_osd_request *req,
>   	p += req->r_oid_len;
>
>   	/* ops--can imply data */
> -	ceph_encode_16(&p, num_ops);
> -	src_op = src_ops;
> -	req->r_request_ops = p;
> +	ceph_encode_16(&p, (u16)req->r_num_ops);
>   	data_len = 0;
> -	for (i = 0; i < num_ops; i++, src_op++) {
> -		data_len += osd_req_encode_op(req, p, src_op);
> +	for (i = 0; i < req->r_num_ops; i++) {
> +		data_len += osd_req_encode_op(req, p, i);
>   		p += sizeof(struct ceph_osd_op);
>   	}
>
> @@ -602,7 +602,6 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					       struct ceph_file_layout *layout,
>   					       struct ceph_vino vino,
>   					       u64 off, u64 *plen, int num_ops,
> -					       struct ceph_osd_req_op *ops,
>   					       int opcode, int flags,
>   					       struct ceph_snap_context *snapc,
>   					       u32 truncate_seq,
> @@ -610,6 +609,7 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					       bool use_mempool)
>   {
>   	struct ceph_osd_request *req;
> +	struct ceph_osd_req_op *op;
>   	u64 objnum = 0;
>   	u64 objoff = 0;
>   	u64 objlen = 0;
> @@ -623,6 +623,7 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					GFP_NOFS);
>   	if (!req)
>   		return ERR_PTR(-ENOMEM);
> +
>   	req->r_flags = flags;
>
>   	/* calculate max write size */
> @@ -642,7 +643,8 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   			truncate_size = object_size;
>   	}
>
> -	osd_req_op_extent_init(&ops[0], opcode, objoff, objlen,
> +	op = &req->r_ops[0];
> +	osd_req_op_extent_init(op, opcode, objoff, objlen,
>   				truncate_size, truncate_seq);
>   	/*
>   	 * A second op in the ops array means the caller wants to
> @@ -650,7 +652,7 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   	 * osd will flush data quickly.
>   	 */
>   	if (num_ops > 1)
> -		osd_req_op_init(&ops[1], CEPH_OSD_OP_STARTSYNC);
> +		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
>
>   	req->r_file_layout = *layout;  /* keep a copy */
>
> @@ -1342,7 +1344,8 @@ static void handle_reply(struct ceph_osd_client
> *osdc, struct ceph_msg *msg,
>   	struct ceph_osd_request *req;
>   	u64 tid;
>   	int object_len;
> -	int numops, payload_len, flags;
> +	unsigned int numops;
> +	int payload_len, flags;
>   	s32 result;
>   	s32 retry_attempt;
>   	struct ceph_pg pg;
> @@ -1352,7 +1355,7 @@ static void handle_reply(struct ceph_osd_client
> *osdc, struct ceph_msg *msg,
>   	u32 osdmap_epoch;
>   	int already_completed;
>   	u32 bytes;
> -	int i;
> +	unsigned int i;
>
>   	tid = le64_to_cpu(msg->hdr.tid);
>   	dout("handle_reply %p tid %llu\n", msg, tid);
> @@ -2116,12 +2119,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
> *osdc,
>   			struct page **pages, int num_pages, int page_align)
>   {
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_req_op op;
>   	int rc = 0;
>
>   	dout("readpages on ino %llx.%llx on %llu~%llu\n", vino.ino,
>   	     vino.snap, off, *plen);
> -	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1, &op,
> +	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1,
>   				    CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ,
>   				    NULL, truncate_seq, truncate_size,
>   				    false);
> @@ -2136,7 +2138,7 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
>   	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
>   	     off, *plen, *plen, page_align);
>
> -	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
> +	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);
>
>   	rc = ceph_osdc_start_request(osdc, req, false);
>   	if (!rc)
> @@ -2160,12 +2162,11 @@ int ceph_osdc_writepages(struct ceph_osd_client
> *osdc, struct ceph_vino vino,
>   			 struct page **pages, int num_pages)
>   {
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_req_op op;
>   	int rc = 0;
>   	int page_align = off & ~PAGE_MASK;
>
>   	BUG_ON(vino.snap != CEPH_NOSNAP);	/* snapshots aren't writeable */
> -	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1, &op,
> +	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1,
>   				    CEPH_OSD_OP_WRITE,
>   				    CEPH_OSD_FLAG_ONDISK | CEPH_OSD_FLAG_WRITE,
>   				    snapc, truncate_seq, truncate_size,
> @@ -2178,7 +2179,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
> *osdc, struct ceph_vino vino,
>   				false, false);
>   	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
>
> -	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
> +	ceph_osdc_build_request(req, off, snapc, CEPH_NOSNAP, mtime);
>
>   	rc = ceph_osdc_start_request(osdc, req, true);
>   	if (!rc)
>

[PATCH 09/20] libceph: rename data out field in osd request op

[Alex Elder]

There are fields "indata" and "indata_len" defined the ceph osd
request op structure.  The "in" part is with from the point of view
of the osd server, but is a little confusing here on the client
side.  Change their names to use "request" instead of "in" to
indicate that it defines data provided with the request (as opposed
the data returned in the response).

Rename the local variable in osd_req_encode_op() to match.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 include/linux/ceph/osd_client.h |    4 ++--
 net/ceph/osd_client.c           |   18 +++++++++---------
 2 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index f4c1a2a..a9c4089 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -91,8 +91,8 @@ struct ceph_osd_req_op {
 		struct {
 			const char *class_name;
 			const char *method_name;
-			const void *indata;
-			u32 indata_len;
+			const void *request_data;
+			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index ed4493d..e74e454 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -395,9 +395,9 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op,
u16 opcode,
 	op->cls.method_len = size;
 	payload_len += size;

-	op->cls.indata = request_data;
+	op->cls.request_data = request_data;
 	BUG_ON(request_data_size > (size_t) U32_MAX);
-	op->cls.indata_len = (u32) request_data_size;
+	op->cls.request_data_len = (u32) request_data_size;
 	payload_len += request_data_size;

 	op->cls.argc = 0;	/* currently unused */
@@ -425,7 +425,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 			      struct ceph_osd_op *dst, unsigned int which)
 {
 	struct ceph_osd_req_op *src;
-	u64 out_data_len = 0;
+	u64 request_data_len = 0;
 	struct ceph_pagelist *pagelist;

 	BUG_ON(which >= req->r_num_ops);
@@ -442,7 +442,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 	case CEPH_OSD_OP_READ:
 	case CEPH_OSD_OP_WRITE:
 		if (src->op == CEPH_OSD_OP_WRITE)
-			out_data_len = src->extent.length;
+			request_data_len = src->extent.length;
 		dst->extent.offset = cpu_to_le64(src->extent.offset);
 		dst->extent.length = cpu_to_le64(src->extent.length);
 		dst->extent.truncate_size =
@@ -457,16 +457,16 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,

 		dst->cls.class_len = src->cls.class_len;
 		dst->cls.method_len = src->cls.method_len;
-		dst->cls.indata_len = cpu_to_le32(src->cls.indata_len);
+		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
 		ceph_pagelist_append(pagelist, src->cls.class_name,
 				     src->cls.class_len);
 		ceph_pagelist_append(pagelist, src->cls.method_name,
 				     src->cls.method_len);
-		ceph_pagelist_append(pagelist, src->cls.indata,
-				     src->cls.indata_len);
+		ceph_pagelist_append(pagelist, src->cls.request_data,
+				     src->cls.request_data_len);

 		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
-		out_data_len = pagelist->length;
+		request_data_len = pagelist->length;
 		break;
 	case CEPH_OSD_OP_STARTSYNC:
 		break;
@@ -486,7 +486,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 	dst->op = cpu_to_le16(src->op);
 	dst->payload_len = cpu_to_le32(src->payload_len);

-	return out_data_len;
+	return request_data_len;
 }

 /*
-- 
1.7.9.5

Re: [PATCH 09/20] libceph: rename data out field in osd request op

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:03 AM, Alex Elder wrote:
> There are fields "indata" and "indata_len" defined the ceph osd
> request op structure.  The "in" part is with from the point of view
> of the osd server, but is a little confusing here on the client
> side.  Change their names to use "request" instead of "in" to
> indicate that it defines data provided with the request (as opposed
> the data returned in the response).
>
> Rename the local variable in osd_req_encode_op() to match.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   include/linux/ceph/osd_client.h |    4 ++--
>   net/ceph/osd_client.c           |   18 +++++++++---------
>   2 files changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index f4c1a2a..a9c4089 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -91,8 +91,8 @@ struct ceph_osd_req_op {
>   		struct {
>   			const char *class_name;
>   			const char *method_name;
> -			const void *indata;
> -			u32 indata_len;
> +			const void *request_data;
> +			u32 request_data_len;
>   			__u8 class_len;
>   			__u8 method_len;
>   			__u8 argc;
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index ed4493d..e74e454 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -395,9 +395,9 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op,
> u16 opcode,
>   	op->cls.method_len = size;
>   	payload_len += size;
>
> -	op->cls.indata = request_data;
> +	op->cls.request_data = request_data;
>   	BUG_ON(request_data_size > (size_t) U32_MAX);
> -	op->cls.indata_len = (u32) request_data_size;
> +	op->cls.request_data_len = (u32) request_data_size;
>   	payload_len += request_data_size;
>
>   	op->cls.argc = 0;	/* currently unused */
> @@ -425,7 +425,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   			      struct ceph_osd_op *dst, unsigned int which)
>   {
>   	struct ceph_osd_req_op *src;
> -	u64 out_data_len = 0;
> +	u64 request_data_len = 0;
>   	struct ceph_pagelist *pagelist;
>
>   	BUG_ON(which >= req->r_num_ops);
> @@ -442,7 +442,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   	case CEPH_OSD_OP_READ:
>   	case CEPH_OSD_OP_WRITE:
>   		if (src->op == CEPH_OSD_OP_WRITE)
> -			out_data_len = src->extent.length;
> +			request_data_len = src->extent.length;
>   		dst->extent.offset = cpu_to_le64(src->extent.offset);
>   		dst->extent.length = cpu_to_le64(src->extent.length);
>   		dst->extent.truncate_size =
> @@ -457,16 +457,16 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>
>   		dst->cls.class_len = src->cls.class_len;
>   		dst->cls.method_len = src->cls.method_len;
> -		dst->cls.indata_len = cpu_to_le32(src->cls.indata_len);
> +		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
>   		ceph_pagelist_append(pagelist, src->cls.class_name,
>   				     src->cls.class_len);
>   		ceph_pagelist_append(pagelist, src->cls.method_name,
>   				     src->cls.method_len);
> -		ceph_pagelist_append(pagelist, src->cls.indata,
> -				     src->cls.indata_len);
> +		ceph_pagelist_append(pagelist, src->cls.request_data,
> +				     src->cls.request_data_len);
>
>   		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
> -		out_data_len = pagelist->length;
> +		request_data_len = pagelist->length;
>   		break;
>   	case CEPH_OSD_OP_STARTSYNC:
>   		break;
> @@ -486,7 +486,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   	dst->op = cpu_to_le16(src->op);
>   	dst->payload_len = cpu_to_le32(src->payload_len);
>
> -	return out_data_len;
> +	return request_data_len;
>   }
>
>   /*
>

[PATCH 10/20] libceph: add data pointers in osd op structures

[Alex Elder]

An extent type osd operation currently implies that there will
be corresponding data supplied in the data portion of the request
(for write) or response (for read) message.  Similarly, an osd class
method operation implies a data item will be supplied to receive
the response data from the operation.

Add a ceph_osd_data pointer to each of those structures, and assign
it to point to eithre the incoming or the outgoing data structure in
the osd message.  The data is not always available when an op is
initially set up, so add two new functions to allow setting them
after the op has been initialized.

Begin to make use of the data item pointer available in the osd
operation rather than the request data in or out structure in
places where it's convenient.  Add some assertions to verify
pointers are always set the way they're expected to be.

This is a sort of stepping stone toward really moving the data
into the osd request ops, to allow for some validation before
making that jump.

This is the first in a series of patches that resolve:
    http://tracker.ceph.com/issues/4657

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c             |   24 ++++++++++++++++++++----
 fs/ceph/addr.c                  |    8 +++++---
 fs/ceph/file.c                  |    5 +++--
 include/linux/ceph/osd_client.h |    6 ++++++
 net/ceph/osd_client.c           |   26 +++++++++++++++++++++++++-
 5 files changed, 59 insertions(+), 10 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index e6e2191..59f9db1 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1315,23 +1315,39 @@ static void rbd_osd_req_format_op(struct
rbd_obj_request *obj_request,
 					bool write_request)
 {
 	struct rbd_img_request *img_request = obj_request->img_request;
+	struct ceph_osd_request *osd_req = obj_request->osd_req;
+	struct ceph_osd_data *osd_data = NULL;
 	struct ceph_snap_context *snapc = NULL;
 	u64 snap_id = CEPH_NOSNAP;
 	struct timespec *mtime = NULL;
 	struct timespec now;

-	rbd_assert(obj_request->osd_req != NULL);
+	rbd_assert(osd_req != NULL);

 	if (write_request) {
+		osd_data = &osd_req->r_data_out;
 		now = CURRENT_TIME;
 		mtime = &now;
 		if (img_request)
 			snapc = img_request->snapc;
-	} else if (img_request) {
-		snap_id = img_request->snap_id;
+	} else {
+		osd_data = &osd_req->r_data_in;
+		if (img_request)
+			snap_id = img_request->snap_id;
 	}
+	if (obj_request->type != OBJ_REQUEST_NODATA) {
+		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];

-	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+		/*
+		 * If it has data, it's either a object class method
+		 * call (cls) or it's an extent operation.
+		 */
+		if (op->op == CEPH_OSD_OP_CALL)
+			osd_req_op_cls_response_data(op, osd_data);
+		else
+			osd_req_op_extent_osd_data(op, osd_data);
+	}
+	ceph_osdc_build_request(osd_req, obj_request->offset,
 			snapc, snap_id, mtime);
 }

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 12bbba7..3caadb0 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -343,7 +343,8 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 		}
 		pages[i] = page;
 	}
-	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
+	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
 					false, false);
 	req->r_callback = finish_read;
 	req->r_inode = inode;
@@ -916,8 +917,9 @@ get_more_pages:
 		dout("writepages got %d pages at %llu~%llu\n",
 		     locked_pages, offset, len);

-		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
-						!!pool, false);
+		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
+						len, 0, !!pool, false);

 		pages = NULL;	/* request message now owns the pages array */
 		pool = NULL;
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index a1d85a8..66d0938 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -574,8 +574,9 @@ more:
 			own_pages = true;
 		}
 	}
-	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
-					false, own_pages);
+	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+					page_align, false, own_pages);

 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
 	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index a9c4089..ae51935 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -87,12 +87,14 @@ struct ceph_osd_req_op {
 			u64 offset, length;
 			u64 truncate_size;
 			u32 truncate_seq;
+			struct ceph_osd_data *osd_data;
 		} extent;
 		struct {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
 			u32 request_data_len;
+			struct ceph_osd_data *response_data;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
@@ -236,10 +238,14 @@ extern void osd_req_op_extent_init(struct
ceph_osd_req_op *op, u16 opcode,
 					u64 offset, u64 length,
 					u64 truncate_size, u32 truncate_seq);
 extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+					struct ceph_osd_data *osd_data);
 extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
 					const char *class, const char *method,
 					const void *request_data,
 					size_t request_data_size);
+extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+					struct ceph_osd_data *response_data);
 extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
 					u64 cookie, u64 version, int flag);

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index e74e454..2a14187 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -372,6 +372,13 @@ void osd_req_op_extent_update(struct
ceph_osd_req_op *op, u64 length)
 }
 EXPORT_SYMBOL(osd_req_op_extent_update);

+void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+				struct ceph_osd_data *osd_data)
+{
+	op->extent.osd_data = osd_data;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
 void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
 			const char *class, const char *method,
 			const void *request_data, size_t request_data_size)
@@ -406,6 +413,13 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
*op, u16 opcode,
 }
 EXPORT_SYMBOL(osd_req_op_cls_init);

+void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+				struct ceph_osd_data *response_data)
+{
+	op->cls.response_data = response_data;
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data);
+
 void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
 				u64 cookie, u64 version, int flag)
 {
@@ -449,6 +463,10 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
 			cpu_to_le64(src->extent.truncate_size);
 		dst->extent.truncate_seq =
 			cpu_to_le32(src->extent.truncate_seq);
+		if (src->op == CEPH_OSD_OP_WRITE)
+			WARN_ON(src->extent.osd_data != &req->r_data_out);
+		else
+			WARN_ON(src->extent.osd_data != &req->r_data_in);
 		break;
 	case CEPH_OSD_OP_CALL:
 		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
@@ -464,8 +482,9 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 				     src->cls.method_len);
 		ceph_pagelist_append(pagelist, src->cls.request_data,
 				     src->cls.request_data_len);
-
 		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
+
+		WARN_ON(src->cls.response_data != &req->r_data_in);
 		request_data_len = pagelist->length;
 		break;
 	case CEPH_OSD_OP_STARTSYNC:
@@ -609,6 +628,7 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					       bool use_mempool)
 {
 	struct ceph_osd_request *req;
+	struct ceph_osd_data *osd_data;
 	struct ceph_osd_req_op *op;
 	u64 objnum = 0;
 	u64 objoff = 0;
@@ -623,6 +643,8 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					GFP_NOFS);
 	if (!req)
 		return ERR_PTR(-ENOMEM);
+	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
+					       : &req->r_data_in;

 	req->r_flags = flags;

@@ -646,6 +668,8 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 	op = &req->r_ops[0];
 	osd_req_op_extent_init(op, opcode, objoff, objlen,
 				truncate_size, truncate_seq);
+	osd_req_op_extent_osd_data(op, osd_data);
+
 	/*
 	 * A second op in the ops array means the caller wants to
 	 * also issue a include a 'startsync' command so that the
-- 
1.7.9.5

Re: [PATCH 10/20] libceph: add data pointers in osd op structures

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:03 AM, Alex Elder wrote:
> An extent type osd operation currently implies that there will
> be corresponding data supplied in the data portion of the request
> (for write) or response (for read) message.  Similarly, an osd class
> method operation implies a data item will be supplied to receive
> the response data from the operation.
>
> Add a ceph_osd_data pointer to each of those structures, and assign
> it to point to eithre the incoming or the outgoing data structure in
> the osd message.  The data is not always available when an op is
> initially set up, so add two new functions to allow setting them
> after the op has been initialized.
>
> Begin to make use of the data item pointer available in the osd
> operation rather than the request data in or out structure in
> places where it's convenient.  Add some assertions to verify
> pointers are always set the way they're expected to be.
>
> This is a sort of stepping stone toward really moving the data
> into the osd request ops, to allow for some validation before
> making that jump.
>
> This is the first in a series of patches that resolve:
>      http://tracker.ceph.com/issues/4657
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c             |   24 ++++++++++++++++++++----
>   fs/ceph/addr.c                  |    8 +++++---
>   fs/ceph/file.c                  |    5 +++--
>   include/linux/ceph/osd_client.h |    6 ++++++
>   net/ceph/osd_client.c           |   26 +++++++++++++++++++++++++-
>   5 files changed, 59 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index e6e2191..59f9db1 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1315,23 +1315,39 @@ static void rbd_osd_req_format_op(struct
> rbd_obj_request *obj_request,
>   					bool write_request)
>   {
>   	struct rbd_img_request *img_request = obj_request->img_request;
> +	struct ceph_osd_request *osd_req = obj_request->osd_req;
> +	struct ceph_osd_data *osd_data = NULL;
>   	struct ceph_snap_context *snapc = NULL;
>   	u64 snap_id = CEPH_NOSNAP;
>   	struct timespec *mtime = NULL;
>   	struct timespec now;
>
> -	rbd_assert(obj_request->osd_req != NULL);
> +	rbd_assert(osd_req != NULL);
>
>   	if (write_request) {
> +		osd_data = &osd_req->r_data_out;
>   		now = CURRENT_TIME;
>   		mtime = &now;
>   		if (img_request)
>   			snapc = img_request->snapc;
> -	} else if (img_request) {
> -		snap_id = img_request->snap_id;
> +	} else {
> +		osd_data = &osd_req->r_data_in;
> +		if (img_request)
> +			snap_id = img_request->snap_id;
>   	}
> +	if (obj_request->type != OBJ_REQUEST_NODATA) {
> +		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
>
> -	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
> +		/*
> +		 * If it has data, it's either a object class method
> +		 * call (cls) or it's an extent operation.
> +		 */
> +		if (op->op == CEPH_OSD_OP_CALL)
> +			osd_req_op_cls_response_data(op, osd_data);
> +		else
> +			osd_req_op_extent_osd_data(op, osd_data);
> +	}
> +	ceph_osdc_build_request(osd_req, obj_request->offset,
>   			snapc, snap_id, mtime);
>   }
>
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index 12bbba7..3caadb0 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -343,7 +343,8 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   		}
>   		pages[i] = page;
>   	}
> -	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
> +	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
> +	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
>   					false, false);
>   	req->r_callback = finish_read;
>   	req->r_inode = inode;
> @@ -916,8 +917,9 @@ get_more_pages:
>   		dout("writepages got %d pages at %llu~%llu\n",
>   		     locked_pages, offset, len);
>
> -		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
> -						!!pool, false);
> +		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
> +		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
> +						len, 0, !!pool, false);
>
>   		pages = NULL;	/* request message now owns the pages array */
>   		pool = NULL;
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index a1d85a8..66d0938 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -574,8 +574,9 @@ more:
>   			own_pages = true;
>   		}
>   	}
> -	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
> -					false, own_pages);
> +	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
> +	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
> +					page_align, false, own_pages);
>
>   	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
>   	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index a9c4089..ae51935 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -87,12 +87,14 @@ struct ceph_osd_req_op {
>   			u64 offset, length;
>   			u64 truncate_size;
>   			u32 truncate_seq;
> +			struct ceph_osd_data *osd_data;
>   		} extent;
>   		struct {
>   			const char *class_name;
>   			const char *method_name;
>   			const void *request_data;
>   			u32 request_data_len;
> +			struct ceph_osd_data *response_data;
>   			__u8 class_len;
>   			__u8 method_len;
>   			__u8 argc;
> @@ -236,10 +238,14 @@ extern void osd_req_op_extent_init(struct
> ceph_osd_req_op *op, u16 opcode,
>   					u64 offset, u64 length,
>   					u64 truncate_size, u32 truncate_seq);
>   extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
> length);
> +extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
> +					struct ceph_osd_data *osd_data);
>   extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
>   					const char *class, const char *method,
>   					const void *request_data,
>   					size_t request_data_size);
> +extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
> +					struct ceph_osd_data *response_data);
>   extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
>   					u64 cookie, u64 version, int flag);
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index e74e454..2a14187 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -372,6 +372,13 @@ void osd_req_op_extent_update(struct
> ceph_osd_req_op *op, u64 length)
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_update);
>
> +void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
> +				struct ceph_osd_data *osd_data)
> +{
> +	op->extent.osd_data = osd_data;
> +}
> +EXPORT_SYMBOL(osd_req_op_extent_osd_data);
> +
>   void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
>   			const char *class, const char *method,
>   			const void *request_data, size_t request_data_size)
> @@ -406,6 +413,13 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
> *op, u16 opcode,
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_init);
>
> +void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
> +				struct ceph_osd_data *response_data)
> +{
> +	op->cls.response_data = response_data;
> +}
> +EXPORT_SYMBOL(osd_req_op_cls_response_data);
> +
>   void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
>   				u64 cookie, u64 version, int flag)
>   {
> @@ -449,6 +463,10 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>   			cpu_to_le64(src->extent.truncate_size);
>   		dst->extent.truncate_seq =
>   			cpu_to_le32(src->extent.truncate_seq);
> +		if (src->op == CEPH_OSD_OP_WRITE)
> +			WARN_ON(src->extent.osd_data != &req->r_data_out);
> +		else
> +			WARN_ON(src->extent.osd_data != &req->r_data_in);
>   		break;
>   	case CEPH_OSD_OP_CALL:
>   		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
> @@ -464,8 +482,9 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   				     src->cls.method_len);
>   		ceph_pagelist_append(pagelist, src->cls.request_data,
>   				     src->cls.request_data_len);
> -
>   		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
> +
> +		WARN_ON(src->cls.response_data != &req->r_data_in);
>   		request_data_len = pagelist->length;
>   		break;
>   	case CEPH_OSD_OP_STARTSYNC:
> @@ -609,6 +628,7 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					       bool use_mempool)
>   {
>   	struct ceph_osd_request *req;
> +	struct ceph_osd_data *osd_data;
>   	struct ceph_osd_req_op *op;
>   	u64 objnum = 0;
>   	u64 objoff = 0;
> @@ -623,6 +643,8 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					GFP_NOFS);
>   	if (!req)
>   		return ERR_PTR(-ENOMEM);
> +	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
> +					       : &req->r_data_in;
>
>   	req->r_flags = flags;
>
> @@ -646,6 +668,8 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   	op = &req->r_ops[0];
>   	osd_req_op_extent_init(op, opcode, objoff, objlen,
>   				truncate_size, truncate_seq);
> +	osd_req_op_extent_osd_data(op, osd_data);
> +
>   	/*
>   	 * A second op in the ops array means the caller wants to
>   	 * also issue a include a 'startsync' command so that the
>

[PATCH 11/20] libceph: specify osd op by index in request

[Alex Elder]

An osd request now holds all of its source op structures, and every
place that initializes one of these is in fact initializing one
of the entries in the the osd request's array.

So rather than supplying the address of the op to initialize, have
caller specify the osd request and an indication of which op it
would like to initialize.  This better hides the details the
op structure (and faciltates moving the data pointers they use).

Since osd_req_op_init() is a common routine, and it's not used
outside the osd client code, give it static scope.  Also make
it return the address of the specified op (so all the other
init routines don't have to repeat that code).

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c             |   35 +++++++++------------
 fs/ceph/addr.c                  |    2 +-
 include/linux/ceph/osd_client.h |   19 +++++++-----
 net/ceph/osd_client.c           |   64
++++++++++++++++++++++++---------------
 4 files changed, 67 insertions(+), 53 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 59f9db1..7a62327 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1336,16 +1336,17 @@ static void rbd_osd_req_format_op(struct
rbd_obj_request *obj_request,
 			snap_id = img_request->snap_id;
 	}
 	if (obj_request->type != OBJ_REQUEST_NODATA) {
-		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
-
 		/*
 		 * If it has data, it's either a object class method
 		 * call (cls) or it's an extent operation.
 		 */
-		if (op->op == CEPH_OSD_OP_CALL)
-			osd_req_op_cls_response_data(op, osd_data);
+		/* XXX This use of the ops array goes away in the next patch */
+		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
+			osd_req_op_cls_response_data(obj_request->osd_req, 0,
+						osd_data);
 		else
-			osd_req_op_extent_osd_data(op, osd_data);
+			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
+						osd_data);
 	}
 	ceph_osdc_build_request(osd_req, obj_request->offset,
 			snapc, snap_id, mtime);
@@ -1577,7 +1578,6 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 	while (resid) {
 		const char *object_name;
 		unsigned int clone_size;
-		struct ceph_osd_req_op *op;
 		u64 offset;
 		u64 length;

@@ -1606,8 +1606,8 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		if (!obj_request->osd_req)
 			goto out_partial;

-		op = &obj_request->osd_req->r_ops[0];
-		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+		osd_req_op_extent_init(obj_request->osd_req, 0,
+					opcode, offset, length, 0, 0);
 		rbd_osd_req_format_op(obj_request, write_request);

 		/* status and version are initially zero-filled */
@@ -1709,7 +1709,6 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 				   u64 ver, u64 notify_id)
 {
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
 	struct ceph_osd_client *osdc;
 	int ret;

@@ -1723,8 +1722,8 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
+					notify_id, ver, 0);
 	rbd_osd_req_format_op(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
@@ -1765,7 +1764,6 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 {
 	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
 	int ret;

 	rbd_assert(start ^ !!rbd_dev->watch_event);
@@ -1789,8 +1787,7 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 	if (!obj_request->osd_req)
 		goto out_cancel;

-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
 				rbd_dev->watch_event->cookie,
 				rbd_dev->header.obj_version, start);
 	rbd_osd_req_format_op(obj_request, true);
@@ -1853,7 +1850,6 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 {
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_client *osdc;
-	struct ceph_osd_req_op *op;
 	struct page **pages;
 	u32 page_count;
 	int ret;
@@ -1883,8 +1879,8 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
+	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
+					class_name, method_name,
 					outbound, outbound_size);
 	rbd_osd_req_format_op(obj_request, false);

@@ -2065,7 +2061,6 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,

 {
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
 	struct ceph_osd_client *osdc;
 	struct page **pages = NULL;
 	u32 page_count;
@@ -2090,8 +2085,8 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
+					offset, length, 0, 0);
 	rbd_osd_req_format_op(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 3caadb0..dd5d263 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -926,7 +926,7 @@ get_more_pages:

 		/* Update the write op length in case we changed it */

-		osd_req_op_extent_update(&req->r_ops[0], len);
+		osd_req_op_extent_update(req, 0, len);

 		vino = ceph_vino(inode);
 		ceph_osdc_build_request(req, offset, snapc, vino.snap,
diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index ae51935..144d57c 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -233,20 +233,25 @@ extern void ceph_osdc_handle_reply(struct
ceph_osd_client *osdc,
 extern void ceph_osdc_handle_map(struct ceph_osd_client *osdc,
 				 struct ceph_msg *msg);

-extern void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode);
-extern void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
 					u64 offset, u64 length,
 					u64 truncate_size, u32 truncate_seq);
-extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+					unsigned int which, u64 length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+					unsigned int which,
 					struct ceph_osd_data *osd_data);
-extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
 					const char *class, const char *method,
 					const void *request_data,
 					size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+					unsigned int which,
 					struct ceph_osd_data *response_data);
-extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
 					u64 cookie, u64 version, int flag);

 extern struct ceph_osd_request *ceph_osdc_alloc_request(struct
ceph_osd_client *osdc,
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 2a14187..e698ccf 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -329,25 +329,32 @@ static bool osd_req_opcode_valid(u16 opcode)
  * other information associated with them.  It also serves as a
  * common init routine for all the other init functions, below.
  */
-void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode)
+static struct ceph_osd_req_op *
+osd_req_op_init(struct ceph_osd_request *osd_req, unsigned int which,
+				u16 opcode)
 {
+	struct ceph_osd_req_op *op;
+
+	BUG_ON(which >= osd_req->r_num_ops);
 	BUG_ON(!osd_req_opcode_valid(opcode));

+	op = &osd_req->r_ops[which];
 	memset(op, 0, sizeof (*op));
-
 	op->op = opcode;
+
+	return op;
 }

-void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
+void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+				unsigned int which, u16 opcode,
 				u64 offset, u64 length,
 				u64 truncate_size, u32 truncate_seq)
 {
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
 	size_t payload_len = 0;

 	BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE);

-	osd_req_op_init(op, opcode);
-
 	op->extent.offset = offset;
 	op->extent.length = length;
 	op->extent.truncate_size = truncate_size;
@@ -359,9 +366,15 @@ void osd_req_op_extent_init(struct ceph_osd_req_op
*op, u16 opcode,
 }
 EXPORT_SYMBOL(osd_req_op_extent_init);

-void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length)
+void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+				unsigned int which, u64 length)
 {
-	u64 previous = op->extent.length;
+	struct ceph_osd_req_op *op;
+	u64 previous;
+
+	BUG_ON(which >= osd_req->r_num_ops);
+	op = &osd_req->r_ops[which];
+	previous = op->extent.length;

 	if (length == previous)
 		return;		/* Nothing to do */
@@ -372,24 +385,25 @@ void osd_req_op_extent_update(struct
ceph_osd_req_op *op, u64 length)
 }
 EXPORT_SYMBOL(osd_req_op_extent_update);

-void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+				unsigned int which,
 				struct ceph_osd_data *osd_data)
 {
-	op->extent.osd_data = osd_data;
+	BUG_ON(which >= osd_req->r_num_ops);
+	osd_req->r_ops[which].extent.osd_data = osd_data;
 }
 EXPORT_SYMBOL(osd_req_op_extent_osd_data);

-void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
-			const char *class, const char *method,
+void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
which,
+			u16 opcode, const char *class, const char *method,
 			const void *request_data, size_t request_data_size)
 {
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
 	size_t payload_len = 0;
 	size_t size;

 	BUG_ON(opcode != CEPH_OSD_OP_CALL);

-	osd_req_op_init(op, opcode);
-
 	op->cls.class_name = class;
 	size = strlen(class);
 	BUG_ON(size > (size_t) U8_MAX);
@@ -412,26 +426,28 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
*op, u16 opcode,
 	op->payload_len = payload_len;
 }
 EXPORT_SYMBOL(osd_req_op_cls_init);
-
-void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+				unsigned int which,
 				struct ceph_osd_data *response_data)
 {
-	op->cls.response_data = response_data;
+	BUG_ON(which >= osd_req->r_num_ops);
+	osd_req->r_ops[which].cls.response_data = response_data;
 }
 EXPORT_SYMBOL(osd_req_op_cls_response_data);

-void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
+void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+				unsigned int which, u16 opcode,
 				u64 cookie, u64 version, int flag)
 {
-	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);

-	osd_req_op_init(op, opcode);
+	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);

 	op->watch.cookie = cookie;
 	/* op->watch.ver = version; */	/* XXX 3847 */
 	op->watch.ver = cpu_to_le64(version);
 	if (opcode == CEPH_OSD_OP_WATCH && flag)
-		op->watch.flag = (u8) 1;
+		op->watch.flag = (u8)1;
 }
 EXPORT_SYMBOL(osd_req_op_watch_init);

@@ -629,7 +645,6 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 {
 	struct ceph_osd_request *req;
 	struct ceph_osd_data *osd_data;
-	struct ceph_osd_req_op *op;
 	u64 objnum = 0;
 	u64 objoff = 0;
 	u64 objlen = 0;
@@ -665,10 +680,9 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 			truncate_size = object_size;
 	}

-	op = &req->r_ops[0];
-	osd_req_op_extent_init(op, opcode, objoff, objlen,
+	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
 				truncate_size, truncate_seq);
-	osd_req_op_extent_osd_data(op, osd_data);
+	osd_req_op_extent_osd_data(req, 0, osd_data);

 	/*
 	 * A second op in the ops array means the caller wants to
@@ -676,7 +690,7 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 	 * osd will flush data quickly.
 	 */
 	if (num_ops > 1)
-		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
+		osd_req_op_init(req, 1, CEPH_OSD_OP_STARTSYNC);

 	req->r_file_layout = *layout;  /* keep a copy */

-- 
1.7.9.5

Re: [PATCH 11/20] libceph: specify osd op by index in request

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:04 AM, Alex Elder wrote:
> An osd request now holds all of its source op structures, and every
> place that initializes one of these is in fact initializing one
> of the entries in the the osd request's array.
>
> So rather than supplying the address of the op to initialize, have
> caller specify the osd request and an indication of which op it
> would like to initialize.  This better hides the details the
> op structure (and faciltates moving the data pointers they use).
>
> Since osd_req_op_init() is a common routine, and it's not used
> outside the osd client code, give it static scope.  Also make
> it return the address of the specified op (so all the other
> init routines don't have to repeat that code).
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c             |   35 +++++++++------------
>   fs/ceph/addr.c                  |    2 +-
>   include/linux/ceph/osd_client.h |   19 +++++++-----
>   net/ceph/osd_client.c           |   64
> ++++++++++++++++++++++++---------------
>   4 files changed, 67 insertions(+), 53 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 59f9db1..7a62327 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1336,16 +1336,17 @@ static void rbd_osd_req_format_op(struct
> rbd_obj_request *obj_request,
>   			snap_id = img_request->snap_id;
>   	}
>   	if (obj_request->type != OBJ_REQUEST_NODATA) {
> -		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
> -
>   		/*
>   		 * If it has data, it's either a object class method
>   		 * call (cls) or it's an extent operation.
>   		 */
> -		if (op->op == CEPH_OSD_OP_CALL)
> -			osd_req_op_cls_response_data(op, osd_data);
> +		/* XXX This use of the ops array goes away in the next patch */
> +		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
> +			osd_req_op_cls_response_data(obj_request->osd_req, 0,
> +						osd_data);
>   		else
> -			osd_req_op_extent_osd_data(op, osd_data);
> +			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
> +						osd_data);
>   	}
>   	ceph_osdc_build_request(osd_req, obj_request->offset,
>   			snapc, snap_id, mtime);
> @@ -1577,7 +1578,6 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   	while (resid) {
>   		const char *object_name;
>   		unsigned int clone_size;
> -		struct ceph_osd_req_op *op;
>   		u64 offset;
>   		u64 length;
>
> @@ -1606,8 +1606,8 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   		if (!obj_request->osd_req)
>   			goto out_partial;
>
> -		op = &obj_request->osd_req->r_ops[0];
> -		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
> +		osd_req_op_extent_init(obj_request->osd_req, 0,
> +					opcode, offset, length, 0, 0);
>   		rbd_osd_req_format_op(obj_request, write_request);
>
>   		/* status and version are initially zero-filled */
> @@ -1709,7 +1709,6 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>   				   u64 ver, u64 notify_id)
>   {
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_req_op *op;
>   	struct ceph_osd_client *osdc;
>   	int ret;
>
> @@ -1723,8 +1722,8 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	op = &obj_request->osd_req->r_ops[0];
> -	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
> +	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
> +					notify_id, ver, 0);
>   	rbd_osd_req_format_op(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
> @@ -1765,7 +1764,6 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   {
>   	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_req_op *op;
>   	int ret;
>
>   	rbd_assert(start ^ !!rbd_dev->watch_event);
> @@ -1789,8 +1787,7 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   	if (!obj_request->osd_req)
>   		goto out_cancel;
>
> -	op = &obj_request->osd_req->r_ops[0];
> -	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
> +	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
>   				rbd_dev->watch_event->cookie,
>   				rbd_dev->header.obj_version, start);
>   	rbd_osd_req_format_op(obj_request, true);
> @@ -1853,7 +1850,6 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   {
>   	struct rbd_obj_request *obj_request;
>   	struct ceph_osd_client *osdc;
> -	struct ceph_osd_req_op *op;
>   	struct page **pages;
>   	u32 page_count;
>   	int ret;
> @@ -1883,8 +1879,8 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	op = &obj_request->osd_req->r_ops[0];
> -	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
> +	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
> +					class_name, method_name,
>   					outbound, outbound_size);
>   	rbd_osd_req_format_op(obj_request, false);
>
> @@ -2065,7 +2061,6 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>
>   {
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_req_op *op;
>   	struct ceph_osd_client *osdc;
>   	struct page **pages = NULL;
>   	u32 page_count;
> @@ -2090,8 +2085,8 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	op = &obj_request->osd_req->r_ops[0];
> -	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
> +	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
> +					offset, length, 0, 0);
>   	rbd_osd_req_format_op(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index 3caadb0..dd5d263 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -926,7 +926,7 @@ get_more_pages:
>
>   		/* Update the write op length in case we changed it */
>
> -		osd_req_op_extent_update(&req->r_ops[0], len);
> +		osd_req_op_extent_update(req, 0, len);
>
>   		vino = ceph_vino(inode);
>   		ceph_osdc_build_request(req, offset, snapc, vino.snap,
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index ae51935..144d57c 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -233,20 +233,25 @@ extern void ceph_osdc_handle_reply(struct
> ceph_osd_client *osdc,
>   extern void ceph_osdc_handle_map(struct ceph_osd_client *osdc,
>   				 struct ceph_msg *msg);
>
> -extern void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode);
> -extern void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
> +extern void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
> +					unsigned int which, u16 opcode,
>   					u64 offset, u64 length,
>   					u64 truncate_size, u32 truncate_seq);
> -extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
> length);
> -extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
> +extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
> +					unsigned int which, u64 length);
> +extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
> +					unsigned int which,
>   					struct ceph_osd_data *osd_data);
> -extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
> +extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
> +					unsigned int which, u16 opcode,
>   					const char *class, const char *method,
>   					const void *request_data,
>   					size_t request_data_size);
> -extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
> +extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
> +					unsigned int which,
>   					struct ceph_osd_data *response_data);
> -extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
> +extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
> +					unsigned int which, u16 opcode,
>   					u64 cookie, u64 version, int flag);
>
>   extern struct ceph_osd_request *ceph_osdc_alloc_request(struct
> ceph_osd_client *osdc,
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 2a14187..e698ccf 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -329,25 +329,32 @@ static bool osd_req_opcode_valid(u16 opcode)
>    * other information associated with them.  It also serves as a
>    * common init routine for all the other init functions, below.
>    */
> -void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode)
> +static struct ceph_osd_req_op *
> +osd_req_op_init(struct ceph_osd_request *osd_req, unsigned int which,
> +				u16 opcode)
>   {
> +	struct ceph_osd_req_op *op;
> +
> +	BUG_ON(which >= osd_req->r_num_ops);
>   	BUG_ON(!osd_req_opcode_valid(opcode));
>
> +	op = &osd_req->r_ops[which];
>   	memset(op, 0, sizeof (*op));
> -
>   	op->op = opcode;
> +
> +	return op;
>   }
>
> -void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
> +void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
> +				unsigned int which, u16 opcode,
>   				u64 offset, u64 length,
>   				u64 truncate_size, u32 truncate_seq)
>   {
> +	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
>   	size_t payload_len = 0;
>
>   	BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE);
>
> -	osd_req_op_init(op, opcode);
> -
>   	op->extent.offset = offset;
>   	op->extent.length = length;
>   	op->extent.truncate_size = truncate_size;
> @@ -359,9 +366,15 @@ void osd_req_op_extent_init(struct ceph_osd_req_op
> *op, u16 opcode,
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_init);
>
> -void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length)
> +void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
> +				unsigned int which, u64 length)
>   {
> -	u64 previous = op->extent.length;
> +	struct ceph_osd_req_op *op;
> +	u64 previous;
> +
> +	BUG_ON(which >= osd_req->r_num_ops);
> +	op = &osd_req->r_ops[which];
> +	previous = op->extent.length;
>
>   	if (length == previous)
>   		return;		/* Nothing to do */
> @@ -372,24 +385,25 @@ void osd_req_op_extent_update(struct
> ceph_osd_req_op *op, u64 length)
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_update);
>
> -void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
> +void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
> +				unsigned int which,
>   				struct ceph_osd_data *osd_data)
>   {
> -	op->extent.osd_data = osd_data;
> +	BUG_ON(which >= osd_req->r_num_ops);
> +	osd_req->r_ops[which].extent.osd_data = osd_data;
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_osd_data);
>
> -void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
> -			const char *class, const char *method,
> +void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
> which,
> +			u16 opcode, const char *class, const char *method,
>   			const void *request_data, size_t request_data_size)
>   {
> +	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
>   	size_t payload_len = 0;
>   	size_t size;
>
>   	BUG_ON(opcode != CEPH_OSD_OP_CALL);
>
> -	osd_req_op_init(op, opcode);
> -
>   	op->cls.class_name = class;
>   	size = strlen(class);
>   	BUG_ON(size > (size_t) U8_MAX);
> @@ -412,26 +426,28 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
> *op, u16 opcode,
>   	op->payload_len = payload_len;
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_init);
> -
> -void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
> +void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
> +				unsigned int which,
>   				struct ceph_osd_data *response_data)
>   {
> -	op->cls.response_data = response_data;
> +	BUG_ON(which >= osd_req->r_num_ops);
> +	osd_req->r_ops[which].cls.response_data = response_data;
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_response_data);
>
> -void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
> +void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
> +				unsigned int which, u16 opcode,
>   				u64 cookie, u64 version, int flag)
>   {
> -	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
> +	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
>
> -	osd_req_op_init(op, opcode);
> +	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
>
>   	op->watch.cookie = cookie;
>   	/* op->watch.ver = version; */	/* XXX 3847 */
>   	op->watch.ver = cpu_to_le64(version);
>   	if (opcode == CEPH_OSD_OP_WATCH && flag)
> -		op->watch.flag = (u8) 1;
> +		op->watch.flag = (u8)1;
>   }
>   EXPORT_SYMBOL(osd_req_op_watch_init);
>
> @@ -629,7 +645,6 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   {
>   	struct ceph_osd_request *req;
>   	struct ceph_osd_data *osd_data;
> -	struct ceph_osd_req_op *op;
>   	u64 objnum = 0;
>   	u64 objoff = 0;
>   	u64 objlen = 0;
> @@ -665,10 +680,9 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   			truncate_size = object_size;
>   	}
>
> -	op = &req->r_ops[0];
> -	osd_req_op_extent_init(op, opcode, objoff, objlen,
> +	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
>   				truncate_size, truncate_seq);
> -	osd_req_op_extent_osd_data(op, osd_data);
> +	osd_req_op_extent_osd_data(req, 0, osd_data);
>
>   	/*
>   	 * A second op in the ops array means the caller wants to
> @@ -676,7 +690,7 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   	 * osd will flush data quickly.
>   	 */
>   	if (num_ops > 1)
> -		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
> +		osd_req_op_init(req, 1, CEPH_OSD_OP_STARTSYNC);
>
>   	req->r_file_layout = *layout;  /* keep a copy */
>

[PATCH 12/20] rbd: don't set data in rbd_osd_req_format_op()

[Alex Elder]

Currently an object request has its osd request's data field set in
rbd_osd_req_format_op().  That assumes a single osd op per object
request, and that won't be the case for long.

Move the code that sets this out and into the caller.

Rename rbd_osd_req_format_op() to be just rbd_osd_req_format(),
removing the notion that it's doing anything op-specific.

This and the next patch resolve:
    http://tracker.ceph.com/issues/4658

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   55
+++++++++++++++++++++++----------------------------
 1 file changed, 25 insertions(+), 30 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 7a62327..3b90283 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1311,12 +1311,11 @@ static void rbd_osd_req_callback(struct
ceph_osd_request *osd_req,
 		rbd_obj_request_complete(obj_request);
 }

-static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
+static void rbd_osd_req_format(struct rbd_obj_request *obj_request,
 					bool write_request)
 {
 	struct rbd_img_request *img_request = obj_request->img_request;
 	struct ceph_osd_request *osd_req = obj_request->osd_req;
-	struct ceph_osd_data *osd_data = NULL;
 	struct ceph_snap_context *snapc = NULL;
 	u64 snap_id = CEPH_NOSNAP;
 	struct timespec *mtime = NULL;
@@ -1325,28 +1324,12 @@ static void rbd_osd_req_format_op(struct
rbd_obj_request *obj_request,
 	rbd_assert(osd_req != NULL);

 	if (write_request) {
-		osd_data = &osd_req->r_data_out;
 		now = CURRENT_TIME;
 		mtime = &now;
 		if (img_request)
 			snapc = img_request->snapc;
-	} else {
-		osd_data = &osd_req->r_data_in;
-		if (img_request)
-			snap_id = img_request->snap_id;
-	}
-	if (obj_request->type != OBJ_REQUEST_NODATA) {
-		/*
-		 * If it has data, it's either a object class method
-		 * call (cls) or it's an extent operation.
-		 */
-		/* XXX This use of the ops array goes away in the next patch */
-		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
-			osd_req_op_cls_response_data(obj_request->osd_req, 0,
-						osd_data);
-		else
-			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
-						osd_data);
+	} else if (img_request) {
+		snap_id = img_request->snap_id;
 	}
 	ceph_osdc_build_request(osd_req, obj_request->offset,
 			snapc, snap_id, mtime);
@@ -1576,6 +1559,8 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 	resid = img_request->length;
 	rbd_assert(resid > 0);
 	while (resid) {
+		struct ceph_osd_request *osd_req;
+		struct ceph_osd_data *osd_data;
 		const char *object_name;
 		unsigned int clone_size;
 		u64 offset;
@@ -1601,14 +1586,18 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		if (!obj_request->bio_list)
 			goto out_partial;

-		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
-						write_request, obj_request);
-		if (!obj_request->osd_req)
+		osd_req = rbd_osd_req_create(rbd_dev, write_request,
+						obj_request);
+		if (!osd_req)
 			goto out_partial;
+		obj_request->osd_req = osd_req;

-		osd_req_op_extent_init(obj_request->osd_req, 0,
-					opcode, offset, length, 0, 0);
-		rbd_osd_req_format_op(obj_request, write_request);
+		osd_data = write_request ? &osd_req->r_data_out
+					 : &osd_req->r_data_in;
+		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
+						0, 0);
+		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+		rbd_osd_req_format(obj_request, write_request);

 		/* status and version are initially zero-filled */

@@ -1724,7 +1713,7 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,

 	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
 					notify_id, ver, 0);
-	rbd_osd_req_format_op(obj_request, false);
+	rbd_osd_req_format(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	obj_request->callback = rbd_obj_request_put;
@@ -1790,7 +1779,7 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
 				rbd_dev->watch_event->cookie,
 				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format_op(obj_request, true);
+	rbd_osd_req_format(obj_request, true);

 	if (start)
 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1849,6 +1838,7 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 			     u64 *version)
 {
 	struct rbd_obj_request *obj_request;
+	struct ceph_osd_data *osd_data;
 	struct ceph_osd_client *osdc;
 	struct page **pages;
 	u32 page_count;
@@ -1879,10 +1869,12 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

+	osd_data = &obj_request->osd_req->r_data_in;
 	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
 					class_name, method_name,
 					outbound, outbound_size);
-	rbd_osd_req_format_op(obj_request, false);
+	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
+	rbd_osd_req_format(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2061,6 +2053,7 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,

 {
 	struct rbd_obj_request *obj_request;
+	struct ceph_osd_data *osd_data;
 	struct ceph_osd_client *osdc;
 	struct page **pages = NULL;
 	u32 page_count;
@@ -2085,9 +2078,11 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

+	osd_data = &obj_request->osd_req->r_data_in;
 	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
 					offset, length, 0, 0);
-	rbd_osd_req_format_op(obj_request, false);
+	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
+	rbd_osd_req_format(obj_request, false);

 	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
-- 
1.7.9.5

Re: [PATCH 12/20] rbd: don't set data in rbd_osd_req_format_op()

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:04 AM, Alex Elder wrote:
> Currently an object request has its osd request's data field set in
> rbd_osd_req_format_op().  That assumes a single osd op per object
> request, and that won't be the case for long.
>
> Move the code that sets this out and into the caller.
>
> Rename rbd_osd_req_format_op() to be just rbd_osd_req_format(),
> removing the notion that it's doing anything op-specific.
>
> This and the next patch resolve:
>      http://tracker.ceph.com/issues/4658
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c |   55
> +++++++++++++++++++++++----------------------------
>   1 file changed, 25 insertions(+), 30 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 7a62327..3b90283 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1311,12 +1311,11 @@ static void rbd_osd_req_callback(struct
> ceph_osd_request *osd_req,
>   		rbd_obj_request_complete(obj_request);
>   }
>
> -static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
> +static void rbd_osd_req_format(struct rbd_obj_request *obj_request,
>   					bool write_request)
>   {
>   	struct rbd_img_request *img_request = obj_request->img_request;
>   	struct ceph_osd_request *osd_req = obj_request->osd_req;
> -	struct ceph_osd_data *osd_data = NULL;
>   	struct ceph_snap_context *snapc = NULL;
>   	u64 snap_id = CEPH_NOSNAP;
>   	struct timespec *mtime = NULL;
> @@ -1325,28 +1324,12 @@ static void rbd_osd_req_format_op(struct
> rbd_obj_request *obj_request,
>   	rbd_assert(osd_req != NULL);
>
>   	if (write_request) {
> -		osd_data = &osd_req->r_data_out;
>   		now = CURRENT_TIME;
>   		mtime = &now;
>   		if (img_request)
>   			snapc = img_request->snapc;
> -	} else {
> -		osd_data = &osd_req->r_data_in;
> -		if (img_request)
> -			snap_id = img_request->snap_id;
> -	}
> -	if (obj_request->type != OBJ_REQUEST_NODATA) {
> -		/*
> -		 * If it has data, it's either a object class method
> -		 * call (cls) or it's an extent operation.
> -		 */
> -		/* XXX This use of the ops array goes away in the next patch */
> -		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
> -			osd_req_op_cls_response_data(obj_request->osd_req, 0,
> -						osd_data);
> -		else
> -			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
> -						osd_data);
> +	} else if (img_request) {
> +		snap_id = img_request->snap_id;
>   	}
>   	ceph_osdc_build_request(osd_req, obj_request->offset,
>   			snapc, snap_id, mtime);
> @@ -1576,6 +1559,8 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   	resid = img_request->length;
>   	rbd_assert(resid > 0);
>   	while (resid) {
> +		struct ceph_osd_request *osd_req;
> +		struct ceph_osd_data *osd_data;
>   		const char *object_name;
>   		unsigned int clone_size;
>   		u64 offset;
> @@ -1601,14 +1586,18 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   		if (!obj_request->bio_list)
>   			goto out_partial;
>
> -		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
> -						write_request, obj_request);
> -		if (!obj_request->osd_req)
> +		osd_req = rbd_osd_req_create(rbd_dev, write_request,
> +						obj_request);
> +		if (!osd_req)
>   			goto out_partial;
> +		obj_request->osd_req = osd_req;
>
> -		osd_req_op_extent_init(obj_request->osd_req, 0,
> -					opcode, offset, length, 0, 0);
> -		rbd_osd_req_format_op(obj_request, write_request);
> +		osd_data = write_request ? &osd_req->r_data_out
> +					 : &osd_req->r_data_in;
> +		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
> +						0, 0);
> +		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
> +		rbd_osd_req_format(obj_request, write_request);
>
>   		/* status and version are initially zero-filled */
>
> @@ -1724,7 +1713,7 @@ static int rbd_obj_notify_ack(struct rbd_device
> *rbd_dev,
>
>   	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
>   					notify_id, ver, 0);
> -	rbd_osd_req_format_op(obj_request, false);
> +	rbd_osd_req_format(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	obj_request->callback = rbd_obj_request_put;
> @@ -1790,7 +1779,7 @@ static int rbd_dev_header_watch_sync(struct
> rbd_device *rbd_dev, int start)
>   	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
>   				rbd_dev->watch_event->cookie,
>   				rbd_dev->header.obj_version, start);
> -	rbd_osd_req_format_op(obj_request, true);
> +	rbd_osd_req_format(obj_request, true);
>
>   	if (start)
>   		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
> @@ -1849,6 +1838,7 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   			     u64 *version)
>   {
>   	struct rbd_obj_request *obj_request;
> +	struct ceph_osd_data *osd_data;
>   	struct ceph_osd_client *osdc;
>   	struct page **pages;
>   	u32 page_count;
> @@ -1879,10 +1869,12 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> +	osd_data = &obj_request->osd_req->r_data_in;
>   	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
>   					class_name, method_name,
>   					outbound, outbound_size);
> -	rbd_osd_req_format_op(obj_request, false);
> +	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
> +	rbd_osd_req_format(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> @@ -2061,6 +2053,7 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>
>   {
>   	struct rbd_obj_request *obj_request;
> +	struct ceph_osd_data *osd_data;
>   	struct ceph_osd_client *osdc;
>   	struct page **pages = NULL;
>   	u32 page_count;
> @@ -2085,9 +2078,11 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> +	osd_data = &obj_request->osd_req->r_data_in;
>   	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
>   					offset, length, 0, 0);
> -	rbd_osd_req_format_op(obj_request, false);
> +	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
> +	rbd_osd_req_format(obj_request, false);
>
>   	osdc = &rbd_dev->rbd_client->client->osdc;
>   	ret = rbd_obj_request_submit(osdc, obj_request);
>

[PATCH 13/20] rbd: separate initialization of osd data

[Alex Elder]

The osd data for a request is currently initialized inside
rbd_osd_req_create(), but that assumes an object request's data
belongs in the osd request's data in or data out field.

There are only three places where requests with data are set up, and
it turns out it's easier to call just the osd data init routines
directly there rather than handling it in rbd_osd_req_create().

(The real motivation here is moving toward getting rid of the
osd request in and out data fields.)

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |   27 ++++++++-------------------
 1 file changed, 8 insertions(+), 19 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 3b90283..1cd776b 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1344,8 +1344,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
 	struct ceph_snap_context *snapc = NULL;
 	struct ceph_osd_client *osdc;
 	struct ceph_osd_request *osd_req;
-	struct ceph_osd_data *osd_data;
-	u64 offset = obj_request->offset;

 	if (img_request) {
 		rbd_assert(img_request->write_request == write_request);
@@ -1359,23 +1357,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
 	osd_req = ceph_osdc_alloc_request(osdc, snapc, 1, false, GFP_ATOMIC);
 	if (!osd_req)
 		return NULL;	/* ENOMEM */
-	osd_data = write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
-
-	rbd_assert(obj_request_type_valid(obj_request->type));
-	switch (obj_request->type) {
-	case OBJ_REQUEST_NODATA:
-		break;		/* Nothing to do */
-	case OBJ_REQUEST_BIO:
-		rbd_assert(obj_request->bio_list != NULL);
-		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
-					obj_request->length);
-		break;
-	case OBJ_REQUEST_PAGES:
-		ceph_osd_data_pages_init(osd_data, obj_request->pages,
-				obj_request->length, offset & ~PAGE_MASK,
-				false, false);
-		break;
-	}

 	if (write_request)
 		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
@@ -1596,6 +1577,8 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 					 : &osd_req->r_data_in;
 		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
 						0, 0);
+		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+					obj_request->length);
 		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
 		rbd_osd_req_format(obj_request, write_request);

@@ -1873,6 +1856,8 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
 					class_name, method_name,
 					outbound, outbound_size);
+	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+					false, false);
 	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

@@ -2081,6 +2066,10 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	osd_data = &obj_request->osd_req->r_data_in;
 	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
 					offset, length, 0, 0);
+	ceph_osd_data_pages_init(osd_data, obj_request->pages,
+					obj_request->length,
+					obj_request->offset & ~PAGE_MASK,
+					false, false);
 	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

-- 
1.7.9.5

Re: [PATCH 13/20] rbd: separate initialization of osd data

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:04 AM, Alex Elder wrote:
> The osd data for a request is currently initialized inside
> rbd_osd_req_create(), but that assumes an object request's data
> belongs in the osd request's data in or data out field.
>
> There are only three places where requests with data are set up, and
> it turns out it's easier to call just the osd data init routines
> directly there rather than handling it in rbd_osd_req_create().
>
> (The real motivation here is moving toward getting rid of the
> osd request in and out data fields.)
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c |   27 ++++++++-------------------
>   1 file changed, 8 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 3b90283..1cd776b 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1344,8 +1344,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
>   	struct ceph_snap_context *snapc = NULL;
>   	struct ceph_osd_client *osdc;
>   	struct ceph_osd_request *osd_req;
> -	struct ceph_osd_data *osd_data;
> -	u64 offset = obj_request->offset;
>
>   	if (img_request) {
>   		rbd_assert(img_request->write_request == write_request);
> @@ -1359,23 +1357,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
>   	osd_req = ceph_osdc_alloc_request(osdc, snapc, 1, false, GFP_ATOMIC);
>   	if (!osd_req)
>   		return NULL;	/* ENOMEM */
> -	osd_data = write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
> -
> -	rbd_assert(obj_request_type_valid(obj_request->type));
> -	switch (obj_request->type) {
> -	case OBJ_REQUEST_NODATA:
> -		break;		/* Nothing to do */
> -	case OBJ_REQUEST_BIO:
> -		rbd_assert(obj_request->bio_list != NULL);
> -		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
> -					obj_request->length);
> -		break;
> -	case OBJ_REQUEST_PAGES:
> -		ceph_osd_data_pages_init(osd_data, obj_request->pages,
> -				obj_request->length, offset & ~PAGE_MASK,
> -				false, false);
> -		break;
> -	}
>
>   	if (write_request)
>   		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
> @@ -1596,6 +1577,8 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   					 : &osd_req->r_data_in;
>   		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
>   						0, 0);
> +		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
> +					obj_request->length);
>   		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
>   		rbd_osd_req_format(obj_request, write_request);
>
> @@ -1873,6 +1856,8 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
>   					class_name, method_name,
>   					outbound, outbound_size);
> +	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
> +					false, false);
>   	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
>   	rbd_osd_req_format(obj_request, false);
>
> @@ -2081,6 +2066,10 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	osd_data = &obj_request->osd_req->r_data_in;
>   	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
>   					offset, length, 0, 0);
> +	ceph_osd_data_pages_init(osd_data, obj_request->pages,
> +					obj_request->length,
> +					obj_request->offset & ~PAGE_MASK,
> +					false, false);
>   	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
>   	rbd_osd_req_format(obj_request, false);
>

[PATCH 14/20] rbd: rearrange some code for consistency

[Alex Elder]

This patch just trivially moves around some code for consistency.

In preparation for initializing osd request data fields in
ceph_osdc_build_request(), I wanted to verify that rbd did in fact
call that immediatley before it called ceph_osdc_start_request().
It was true (although image requests are built in a group and then
started as a group).  But I made the changes here just to make
it more obvious, by making all of the calls follow a common
sequence:
	osd_req_op_<optype>_init();
	ceph_osd_data_<type>_init()
	osd_req_op_<optype>_<datafield>()
	rbd_osd_req_format()
	...
	ret = rbd_obj_request_submit()

I moved the initialization of the callback for image object requests
into rbd_img_request_fill_bio(), again, for consistency.  To avoid
a forward reference, I moved the definition of rbd_img_obj_callback()
up in the file.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c |  129
+++++++++++++++++++++++++--------------------------
 1 file changed, 63 insertions(+), 66 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 1cd776b..3e8e6d5 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1519,6 +1519,57 @@ static void rbd_img_request_destroy(struct kref
*kref)
 	kfree(img_request);
 }

+static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
+{
+	struct rbd_img_request *img_request;
+	u32 which = obj_request->which;
+	bool more = true;
+
+	img_request = obj_request->img_request;
+
+	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
+	rbd_assert(img_request != NULL);
+	rbd_assert(img_request->rq != NULL);
+	rbd_assert(img_request->obj_request_count > 0);
+	rbd_assert(which != BAD_WHICH);
+	rbd_assert(which < img_request->obj_request_count);
+	rbd_assert(which >= img_request->next_completion);
+
+	spin_lock_irq(&img_request->completion_lock);
+	if (which != img_request->next_completion)
+		goto out;
+
+	for_each_obj_request_from(img_request, obj_request) {
+		unsigned int xferred;
+		int result;
+
+		rbd_assert(more);
+		rbd_assert(which < img_request->obj_request_count);
+
+		if (!obj_request_done_test(obj_request))
+			break;
+
+		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
+		xferred = (unsigned int) obj_request->xferred;
+		result = (int) obj_request->result;
+		if (result)
+			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
+				img_request->write_request ? "write" : "read",
+				result, xferred);
+
+		more = blk_end_request(img_request->rq, result, xferred);
+		which++;
+	}
+
+	rbd_assert(more ^ (which == img_request->obj_request_count));
+	img_request->next_completion = which;
+out:
+	spin_unlock_irq(&img_request->completion_lock);
+
+	if (!more)
+		rbd_img_request_complete(img_request);
+}
+
 static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
 					struct bio *bio_list)
 {
@@ -1572,6 +1623,7 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		if (!osd_req)
 			goto out_partial;
 		obj_request->osd_req = osd_req;
+		obj_request->callback = rbd_img_obj_callback;

 		osd_data = write_request ? &osd_req->r_data_out
 					 : &osd_req->r_data_in;
@@ -1582,8 +1634,6 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
 		rbd_osd_req_format(obj_request, write_request);

-		/* status and version are initially zero-filled */
-
 		rbd_img_obj_request_add(img_request, obj_request);

 		image_offset += length;
@@ -1601,57 +1651,6 @@ out_unwind:
 	return -ENOMEM;
 }

-static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
-{
-	struct rbd_img_request *img_request;
-	u32 which = obj_request->which;
-	bool more = true;
-
-	img_request = obj_request->img_request;
-
-	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
-	rbd_assert(img_request != NULL);
-	rbd_assert(img_request->rq != NULL);
-	rbd_assert(img_request->obj_request_count > 0);
-	rbd_assert(which != BAD_WHICH);
-	rbd_assert(which < img_request->obj_request_count);
-	rbd_assert(which >= img_request->next_completion);
-
-	spin_lock_irq(&img_request->completion_lock);
-	if (which != img_request->next_completion)
-		goto out;
-
-	for_each_obj_request_from(img_request, obj_request) {
-		unsigned int xferred;
-		int result;
-
-		rbd_assert(more);
-		rbd_assert(which < img_request->obj_request_count);
-
-		if (!obj_request_done_test(obj_request))
-			break;
-
-		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
-		xferred = (unsigned int) obj_request->xferred;
-		result = (int) obj_request->result;
-		if (result)
-			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
-				img_request->write_request ? "write" : "read",
-				result, xferred);
-
-		more = blk_end_request(img_request->rq, result, xferred);
-		which++;
-	}
-
-	rbd_assert(more ^ (which == img_request->obj_request_count));
-	img_request->next_completion = which;
-out:
-	spin_unlock_irq(&img_request->completion_lock);
-
-	if (!more)
-		rbd_img_request_complete(img_request);
-}
-
 static int rbd_img_request_submit(struct rbd_img_request *img_request)
 {
 	struct rbd_device *rbd_dev = img_request->rbd_dev;
@@ -1662,7 +1661,6 @@ static int rbd_img_request_submit(struct
rbd_img_request *img_request)
 	for_each_obj_request(img_request, obj_request) {
 		int ret;

-		obj_request->callback = rbd_img_obj_callback;
 		ret = rbd_obj_request_submit(osdc, obj_request);
 		if (ret)
 			return ret;
@@ -1681,8 +1679,9 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 				   u64 ver, u64 notify_id)
 {
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_client *osdc;
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	int ret;
+	osdc = &rbd_dev->rbd_client->client->osdc;

 	obj_request = rbd_obj_request_create(rbd_dev->header_name, 0, 0,
 							OBJ_REQUEST_NODATA);
@@ -1693,13 +1692,12 @@ static int rbd_obj_notify_ack(struct rbd_device
*rbd_dev,
 	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
 	if (!obj_request->osd_req)
 		goto out;
+	obj_request->callback = rbd_obj_request_put;

 	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
 					notify_id, ver, 0);
 	rbd_osd_req_format(obj_request, false);

-	osdc = &rbd_dev->rbd_client->client->osdc;
-	obj_request->callback = rbd_obj_request_put;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 out:
 	if (ret)
@@ -1759,16 +1757,17 @@ static int rbd_dev_header_watch_sync(struct
rbd_device *rbd_dev, int start)
 	if (!obj_request->osd_req)
 		goto out_cancel;

-	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
-				rbd_dev->watch_event->cookie,
-				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format(obj_request, true);
-
 	if (start)
 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
 	else
 		ceph_osdc_unregister_linger_request(osdc,
 					rbd_dev->watch_request->osd_req);
+
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
+				rbd_dev->watch_event->cookie,
+				rbd_dev->header.obj_version, start);
+	rbd_osd_req_format(obj_request, true);
+
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
 		goto out_cancel;
@@ -1820,9 +1819,9 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 			     size_t inbound_size,
 			     u64 *version)
 {
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_data *osd_data;
-	struct ceph_osd_client *osdc;
 	struct page **pages;
 	u32 page_count;
 	int ret;
@@ -1861,7 +1860,6 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

-	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
 		goto out;
@@ -2037,9 +2035,9 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 				char *buf, u64 *version)

 {
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_data *osd_data;
-	struct ceph_osd_client *osdc;
 	struct page **pages = NULL;
 	u32 page_count;
 	size_t size;
@@ -2073,7 +2071,6 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

-	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
 		goto out;
-- 
1.7.9.5

Re: [PATCH 14/20] rbd: rearrange some code for consistency

[Josh Durgin]

Just a couple nits.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Alex Elder <elder@inktank.com> wrote:
>This patch just trivially moves around some code for consistency.
>
>In preparation for initializing osd request data fields in
>ceph_osdc_build_request(), I wanted to verify that rbd did in fact
>call that immediatley before it called ceph_osdc_start_request().

typo in immediately

>It was true (although image requests are built in a group and then
>started as a group).  But I made the changes here just to make
>it more obvious, by making all of the calls follow a common
>sequence:
>	osd_req_op_<optype>_init();
>	ceph_osd_data_<type>_init()
>	osd_req_op_<optype>_<datafield>()
>	rbd_osd_req_format()
>	...
>	ret = rbd_obj_request_submit()
>
>I moved the initialization of the callback for image object requests
>into rbd_img_request_fill_bio(), again, for consistency.  To avoid
>a forward reference, I moved the definition of rbd_img_obj_callback()
>up in the file.
>
>Signed-off-by: Alex Elder <elder@inktank.com>
>---
> drivers/block/rbd.c |  129
>+++++++++++++++++++++++++--------------------------
> 1 file changed, 63 insertions(+), 66 deletions(-)
>
>diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
>index 1cd776b..3e8e6d5 100644
>--- a/drivers/block/rbd.c
>+++ b/drivers/block/rbd.c
>@@ -1519,6 +1519,57 @@ static void rbd_img_request_destroy(struct kref
>*kref)
> 	kfree(img_request);
> }
>
>+static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
>+{
>+	struct rbd_img_request *img_request;
>+	u32 which = obj_request->which;
>+	bool more = true;
>+
>+	img_request = obj_request->img_request;
>+
>+	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
>+	rbd_assert(img_request != NULL);
>+	rbd_assert(img_request->rq != NULL);
>+	rbd_assert(img_request->obj_request_count > 0);
>+	rbd_assert(which != BAD_WHICH);
>+	rbd_assert(which < img_request->obj_request_count);
>+	rbd_assert(which >= img_request->next_completion);
>+
>+	spin_lock_irq(&img_request->completion_lock);
>+	if (which != img_request->next_completion)
>+		goto out;
>+
>+	for_each_obj_request_from(img_request, obj_request) {
>+		unsigned int xferred;
>+		int result;
>+
>+		rbd_assert(more);
>+		rbd_assert(which < img_request->obj_request_count);
>+
>+		if (!obj_request_done_test(obj_request))
>+			break;
>+
>+		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
>+		xferred = (unsigned int) obj_request->xferred;
>+		result = (int) obj_request->result;
>+		if (result)
>+			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
>+				img_request->write_request ? "write" : "read",
>+				result, xferred);
>+
>+		more = blk_end_request(img_request->rq, result, xferred);
>+		which++;
>+	}
>+
>+	rbd_assert(more ^ (which == img_request->obj_request_count));
>+	img_request->next_completion = which;
>+out:
>+	spin_unlock_irq(&img_request->completion_lock);
>+
>+	if (!more)
>+		rbd_img_request_complete(img_request);
>+}
>+
>static int rbd_img_request_fill_bio(struct rbd_img_request
>*img_request,
> 					struct bio *bio_list)
> {
>@@ -1572,6 +1623,7 @@ static int rbd_img_request_fill_bio(struct
>rbd_img_request *img_request,
> 		if (!osd_req)
> 			goto out_partial;
> 		obj_request->osd_req = osd_req;
>+		obj_request->callback = rbd_img_obj_callback;
>
> 		osd_data = write_request ? &osd_req->r_data_out
> 					 : &osd_req->r_data_in;
>@@ -1582,8 +1634,6 @@ static int rbd_img_request_fill_bio(struct
>rbd_img_request *img_request,
> 		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
> 		rbd_osd_req_format(obj_request, write_request);
>
>-		/* status and version are initially zero-filled */
>-
> 		rbd_img_obj_request_add(img_request, obj_request);
>
> 		image_offset += length;
>@@ -1601,57 +1651,6 @@ out_unwind:
> 	return -ENOMEM;
> }
>
>-static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
>-{
>-	struct rbd_img_request *img_request;
>-	u32 which = obj_request->which;
>-	bool more = true;
>-
>-	img_request = obj_request->img_request;
>-
>-	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
>-	rbd_assert(img_request != NULL);
>-	rbd_assert(img_request->rq != NULL);
>-	rbd_assert(img_request->obj_request_count > 0);
>-	rbd_assert(which != BAD_WHICH);
>-	rbd_assert(which < img_request->obj_request_count);
>-	rbd_assert(which >= img_request->next_completion);
>-
>-	spin_lock_irq(&img_request->completion_lock);
>-	if (which != img_request->next_completion)
>-		goto out;
>-
>-	for_each_obj_request_from(img_request, obj_request) {
>-		unsigned int xferred;
>-		int result;
>-
>-		rbd_assert(more);
>-		rbd_assert(which < img_request->obj_request_count);
>-
>-		if (!obj_request_done_test(obj_request))
>-			break;
>-
>-		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
>-		xferred = (unsigned int) obj_request->xferred;
>-		result = (int) obj_request->result;
>-		if (result)
>-			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
>-				img_request->write_request ? "write" : "read",
>-				result, xferred);
>-
>-		more = blk_end_request(img_request->rq, result, xferred);
>-		which++;
>-	}
>-
>-	rbd_assert(more ^ (which == img_request->obj_request_count));
>-	img_request->next_completion = which;
>-out:
>-	spin_unlock_irq(&img_request->completion_lock);
>-
>-	if (!more)
>-		rbd_img_request_complete(img_request);
>-}
>-
> static int rbd_img_request_submit(struct rbd_img_request *img_request)
> {
> 	struct rbd_device *rbd_dev = img_request->rbd_dev;
>@@ -1662,7 +1661,6 @@ static int rbd_img_request_submit(struct
>rbd_img_request *img_request)
> 	for_each_obj_request(img_request, obj_request) {
> 		int ret;
>
>-		obj_request->callback = rbd_img_obj_callback;
> 		ret = rbd_obj_request_submit(osdc, obj_request);
> 		if (ret)
> 			return ret;
>@@ -1681,8 +1679,9 @@ static int rbd_obj_notify_ack(struct rbd_device
>*rbd_dev,
> 				   u64 ver, u64 notify_id)
> {
> 	struct rbd_obj_request *obj_request;
>-	struct ceph_osd_client *osdc;
>+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
> 	int ret;
>+	osdc = &rbd_dev->rbd_client->client->osdc;

This is the same as two lines up.

>
> 	obj_request = rbd_obj_request_create(rbd_dev->header_name, 0, 0,
> 							OBJ_REQUEST_NODATA);
>@@ -1693,13 +1692,12 @@ static int rbd_obj_notify_ack(struct rbd_device
>*rbd_dev,
>	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
>obj_request);
> 	if (!obj_request->osd_req)
> 		goto out;
>+	obj_request->callback = rbd_obj_request_put;
>
>	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
> 					notify_id, ver, 0);
> 	rbd_osd_req_format(obj_request, false);
>
>-	osdc = &rbd_dev->rbd_client->client->osdc;
>-	obj_request->callback = rbd_obj_request_put;
> 	ret = rbd_obj_request_submit(osdc, obj_request);
> out:
> 	if (ret)
>@@ -1759,16 +1757,17 @@ static int rbd_dev_header_watch_sync(struct
>rbd_device *rbd_dev, int start)
> 	if (!obj_request->osd_req)
> 		goto out_cancel;
>
>-	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
>-				rbd_dev->watch_event->cookie,
>-				rbd_dev->header.obj_version, start);
>-	rbd_osd_req_format(obj_request, true);
>-
> 	if (start)
> 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
> 	else
> 		ceph_osdc_unregister_linger_request(osdc,
> 					rbd_dev->watch_request->osd_req);
>+
>+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
>+				rbd_dev->watch_event->cookie,
>+				rbd_dev->header.obj_version, start);
>+	rbd_osd_req_format(obj_request, true);
>+
> 	ret = rbd_obj_request_submit(osdc, obj_request);
> 	if (ret)
> 		goto out_cancel;
>@@ -1820,9 +1819,9 @@ static int rbd_obj_method_sync(struct rbd_device
>*rbd_dev,
> 			     size_t inbound_size,
> 			     u64 *version)
> {
>+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
> 	struct rbd_obj_request *obj_request;
> 	struct ceph_osd_data *osd_data;
>-	struct ceph_osd_client *osdc;
> 	struct page **pages;
> 	u32 page_count;
> 	int ret;
>@@ -1861,7 +1860,6 @@ static int rbd_obj_method_sync(struct rbd_device
>*rbd_dev,
> 	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
> 	rbd_osd_req_format(obj_request, false);
>
>-	osdc = &rbd_dev->rbd_client->client->osdc;
> 	ret = rbd_obj_request_submit(osdc, obj_request);
> 	if (ret)
> 		goto out;
>@@ -2037,9 +2035,9 @@ static int rbd_obj_read_sync(struct rbd_device
>*rbd_dev,
> 				char *buf, u64 *version)
>
> {
>+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
> 	struct rbd_obj_request *obj_request;
> 	struct ceph_osd_data *osd_data;
>-	struct ceph_osd_client *osdc;
> 	struct page **pages = NULL;
> 	u32 page_count;
> 	size_t size;
>@@ -2073,7 +2071,6 @@ static int rbd_obj_read_sync(struct rbd_device
>*rbd_dev,
> 	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
> 	rbd_osd_req_format(obj_request, false);
>
>-	osdc = &rbd_dev->rbd_client->client->osdc;
> 	ret = rbd_obj_request_submit(osdc, obj_request);
> 	if (ret)
> 		goto out;

[PATCH 15/20] libceph: format class info at init time

[Alex Elder]

An object class method is formatted using a pagelist which contains
the class name, the method name, and the data concatenated into an
osd request's outbound data.

Currently when a class op is initialized in osd_req_op_cls_init(),
the lengths of and pointers to these three items are recorded.
Later, when the op is getting formatted into the request message, a
new pagelist is created and that is when these items get copied into
the pagelist.

This patch makes it so the pagelist to hold these items is created
when the op is initialized instead.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 include/linux/ceph/osd_client.h |    3 ++-
 net/ceph/osd_client.c           |   29 +++++++++++++++--------------
 2 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index 144d57c..71c4157 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -93,8 +93,9 @@ struct ceph_osd_req_op {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
-			u32 request_data_len;
+			struct ceph_osd_data *request_info;
 			struct ceph_osd_data *response_data;
+			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index e698ccf..6c6d7b8 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -399,28 +399,39 @@ void osd_req_op_cls_init(struct ceph_osd_request
*osd_req, unsigned int which,
 			const void *request_data, size_t request_data_size)
 {
 	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
+	struct ceph_pagelist *pagelist;
 	size_t payload_len = 0;
 	size_t size;

 	BUG_ON(opcode != CEPH_OSD_OP_CALL);

+	pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
+	BUG_ON(!pagelist);
+	ceph_pagelist_init(pagelist);
+
 	op->cls.class_name = class;
 	size = strlen(class);
 	BUG_ON(size > (size_t) U8_MAX);
 	op->cls.class_len = size;
+	ceph_pagelist_append(pagelist, class, size);
 	payload_len += size;

 	op->cls.method_name = method;
 	size = strlen(method);
 	BUG_ON(size > (size_t) U8_MAX);
 	op->cls.method_len = size;
+	ceph_pagelist_append(pagelist, method, size);
 	payload_len += size;

 	op->cls.request_data = request_data;
 	BUG_ON(request_data_size > (size_t) U32_MAX);
 	op->cls.request_data_len = (u32) request_data_size;
+	ceph_pagelist_append(pagelist, request_data, request_data_size);
 	payload_len += request_data_size;

+	op->cls.request_info = &osd_req->r_data_out;
+	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+
 	op->cls.argc = 0;	/* currently unused */

 	op->payload_len = payload_len;
@@ -456,7 +467,6 @@ static u64 osd_req_encode_op(struct ceph_osd_request
*req,
 {
 	struct ceph_osd_req_op *src;
 	u64 request_data_len = 0;
-	struct ceph_pagelist *pagelist;

 	BUG_ON(which >= req->r_num_ops);
 	src = &req->r_ops[which];
@@ -485,23 +495,14 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
 			WARN_ON(src->extent.osd_data != &req->r_data_in);
 		break;
 	case CEPH_OSD_OP_CALL:
-		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
-		BUG_ON(!pagelist);
-		ceph_pagelist_init(pagelist);
-
 		dst->cls.class_len = src->cls.class_len;
 		dst->cls.method_len = src->cls.method_len;
 		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
-		ceph_pagelist_append(pagelist, src->cls.class_name,
-				     src->cls.class_len);
-		ceph_pagelist_append(pagelist, src->cls.method_name,
-				     src->cls.method_len);
-		ceph_pagelist_append(pagelist, src->cls.request_data,
-				     src->cls.request_data_len);
-		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
-
 		WARN_ON(src->cls.response_data != &req->r_data_in);
-		request_data_len = pagelist->length;
+		WARN_ON(src->cls.request_info != &req->r_data_out);
+		BUG_ON(src->cls.request_info->type !=
+					CEPH_OSD_DATA_TYPE_PAGELIST);
+		request_data_len = src->cls.request_info->pagelist->length;
 		break;
 	case CEPH_OSD_OP_STARTSYNC:
 		break;
-- 
1.7.9.5

Re: [PATCH 15/20] libceph: format class info at init time

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:05 AM, Alex Elder wrote:
> An object class method is formatted using a pagelist which contains
> the class name, the method name, and the data concatenated into an
> osd request's outbound data.
>
> Currently when a class op is initialized in osd_req_op_cls_init(),
> the lengths of and pointers to these three items are recorded.
> Later, when the op is getting formatted into the request message, a
> new pagelist is created and that is when these items get copied into
> the pagelist.
>
> This patch makes it so the pagelist to hold these items is created
> when the op is initialized instead.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   include/linux/ceph/osd_client.h |    3 ++-
>   net/ceph/osd_client.c           |   29 +++++++++++++++--------------
>   2 files changed, 17 insertions(+), 15 deletions(-)
>
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index 144d57c..71c4157 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -93,8 +93,9 @@ struct ceph_osd_req_op {
>   			const char *class_name;
>   			const char *method_name;
>   			const void *request_data;
> -			u32 request_data_len;
> +			struct ceph_osd_data *request_info;
>   			struct ceph_osd_data *response_data;
> +			u32 request_data_len;
>   			__u8 class_len;
>   			__u8 method_len;
>   			__u8 argc;
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index e698ccf..6c6d7b8 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -399,28 +399,39 @@ void osd_req_op_cls_init(struct ceph_osd_request
> *osd_req, unsigned int which,
>   			const void *request_data, size_t request_data_size)
>   {
>   	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
> +	struct ceph_pagelist *pagelist;
>   	size_t payload_len = 0;
>   	size_t size;
>
>   	BUG_ON(opcode != CEPH_OSD_OP_CALL);
>
> +	pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
> +	BUG_ON(!pagelist);
> +	ceph_pagelist_init(pagelist);
> +
>   	op->cls.class_name = class;
>   	size = strlen(class);
>   	BUG_ON(size > (size_t) U8_MAX);
>   	op->cls.class_len = size;
> +	ceph_pagelist_append(pagelist, class, size);
>   	payload_len += size;
>
>   	op->cls.method_name = method;
>   	size = strlen(method);
>   	BUG_ON(size > (size_t) U8_MAX);
>   	op->cls.method_len = size;
> +	ceph_pagelist_append(pagelist, method, size);
>   	payload_len += size;
>
>   	op->cls.request_data = request_data;
>   	BUG_ON(request_data_size > (size_t) U32_MAX);
>   	op->cls.request_data_len = (u32) request_data_size;
> +	ceph_pagelist_append(pagelist, request_data, request_data_size);
>   	payload_len += request_data_size;
>
> +	op->cls.request_info = &osd_req->r_data_out;
> +	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
> +
>   	op->cls.argc = 0;	/* currently unused */
>
>   	op->payload_len = payload_len;
> @@ -456,7 +467,6 @@ static u64 osd_req_encode_op(struct ceph_osd_request
> *req,
>   {
>   	struct ceph_osd_req_op *src;
>   	u64 request_data_len = 0;
> -	struct ceph_pagelist *pagelist;
>
>   	BUG_ON(which >= req->r_num_ops);
>   	src = &req->r_ops[which];
> @@ -485,23 +495,14 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>   			WARN_ON(src->extent.osd_data != &req->r_data_in);
>   		break;
>   	case CEPH_OSD_OP_CALL:
> -		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
> -		BUG_ON(!pagelist);
> -		ceph_pagelist_init(pagelist);
> -
>   		dst->cls.class_len = src->cls.class_len;
>   		dst->cls.method_len = src->cls.method_len;
>   		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
> -		ceph_pagelist_append(pagelist, src->cls.class_name,
> -				     src->cls.class_len);
> -		ceph_pagelist_append(pagelist, src->cls.method_name,
> -				     src->cls.method_len);
> -		ceph_pagelist_append(pagelist, src->cls.request_data,
> -				     src->cls.request_data_len);
> -		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
> -
>   		WARN_ON(src->cls.response_data != &req->r_data_in);
> -		request_data_len = pagelist->length;
> +		WARN_ON(src->cls.request_info != &req->r_data_out);
> +		BUG_ON(src->cls.request_info->type !=
> +					CEPH_OSD_DATA_TYPE_PAGELIST);
> +		request_data_len = src->cls.request_info->pagelist->length;
>   		break;
>   	case CEPH_OSD_OP_STARTSYNC:
>   		break;
>

[PATCH 16/20] libceph: move ceph_osdc_build_request()

[Alex Elder]

This simply moves ceph_osdc_build_request() later in its source
file without any change.  Done as a separate patch to facilitate
review of the change in the next patch.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 net/ceph/osd_client.c |  196
++++++++++++++++++++++++-------------------------
 1 file changed, 98 insertions(+), 98 deletions(-)

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 6c6d7b8..f80984e 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -526,104 +526,6 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
 }

 /*
- * build new request AND message
- *
- */
-void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
-				struct ceph_snap_context *snapc, u64 snap_id,
-				struct timespec *mtime)
-{
-	struct ceph_msg *msg = req->r_request;
-	void *p;
-	size_t msg_size;
-	int flags = req->r_flags;
-	u64 data_len;
-	unsigned int i;
-
-	req->r_snapid = snap_id;
-	req->r_snapc = ceph_get_snap_context(snapc);
-
-	/* encode request */
-	msg->hdr.version = cpu_to_le16(4);
-
-	p = msg->front.iov_base;
-	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
-	req->r_request_osdmap_epoch = p;
-	p += 4;
-	req->r_request_flags = p;
-	p += 4;
-	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
-		ceph_encode_timespec(p, mtime);
-	p += sizeof(struct ceph_timespec);
-	req->r_request_reassert_version = p;
-	p += sizeof(struct ceph_eversion); /* will get filled in */
-
-	/* oloc */
-	ceph_encode_8(&p, 4);
-	ceph_encode_8(&p, 4);
-	ceph_encode_32(&p, 8 + 4 + 4);
-	req->r_request_pool = p;
-	p += 8;
-	ceph_encode_32(&p, -1);  /* preferred */
-	ceph_encode_32(&p, 0);   /* key len */
-
-	ceph_encode_8(&p, 1);
-	req->r_request_pgid = p;
-	p += 8 + 4;
-	ceph_encode_32(&p, -1);  /* preferred */
-
-	/* oid */
-	ceph_encode_32(&p, req->r_oid_len);
-	memcpy(p, req->r_oid, req->r_oid_len);
-	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
-	p += req->r_oid_len;
-
-	/* ops--can imply data */
-	ceph_encode_16(&p, (u16)req->r_num_ops);
-	data_len = 0;
-	for (i = 0; i < req->r_num_ops; i++) {
-		data_len += osd_req_encode_op(req, p, i);
-		p += sizeof(struct ceph_osd_op);
-	}
-
-	/* snaps */
-	ceph_encode_64(&p, req->r_snapid);
-	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
-	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
-	if (req->r_snapc) {
-		for (i = 0; i < snapc->num_snaps; i++) {
-			ceph_encode_64(&p, req->r_snapc->snaps[i]);
-		}
-	}
-
-	req->r_request_attempts = p;
-	p += 4;
-
-	/* data */
-	if (flags & CEPH_OSD_FLAG_WRITE) {
-		u16 data_off;
-
-		/*
-		 * The header "data_off" is a hint to the receiver
-		 * allowing it to align received data into its
-		 * buffers such that there's no need to re-copy
-		 * it before writing it to disk (direct I/O).
-		 */
-		data_off = (u16) (off & 0xffff);
-		req->r_request->hdr.data_off = cpu_to_le16(data_off);
-	}
-	req->r_request->hdr.data_len = cpu_to_le32(data_len);
-
-	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
-	msg_size = p - msg->front.iov_base;
-	msg->front.iov_len = msg_size;
-	msg->hdr.front_len = cpu_to_le32(msg_size);
-
-	dout("build_request msg_size was %d\n", (int)msg_size);
-}
-EXPORT_SYMBOL(ceph_osdc_build_request);
-
-/*
  * build new request AND message, calculate layout, and adjust file
  * extent as needed.
  *
@@ -1968,6 +1870,104 @@ static void ceph_osdc_msg_data_set(struct
ceph_msg *msg,
 }

 /*
+ * build new request AND message
+ *
+ */
+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+				struct ceph_snap_context *snapc, u64 snap_id,
+				struct timespec *mtime)
+{
+	struct ceph_msg *msg = req->r_request;
+	void *p;
+	size_t msg_size;
+	int flags = req->r_flags;
+	u64 data_len;
+	unsigned int i;
+
+	req->r_snapid = snap_id;
+	req->r_snapc = ceph_get_snap_context(snapc);
+
+	/* encode request */
+	msg->hdr.version = cpu_to_le16(4);
+
+	p = msg->front.iov_base;
+	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
+	req->r_request_osdmap_epoch = p;
+	p += 4;
+	req->r_request_flags = p;
+	p += 4;
+	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
+		ceph_encode_timespec(p, mtime);
+	p += sizeof(struct ceph_timespec);
+	req->r_request_reassert_version = p;
+	p += sizeof(struct ceph_eversion); /* will get filled in */
+
+	/* oloc */
+	ceph_encode_8(&p, 4);
+	ceph_encode_8(&p, 4);
+	ceph_encode_32(&p, 8 + 4 + 4);
+	req->r_request_pool = p;
+	p += 8;
+	ceph_encode_32(&p, -1);  /* preferred */
+	ceph_encode_32(&p, 0);   /* key len */
+
+	ceph_encode_8(&p, 1);
+	req->r_request_pgid = p;
+	p += 8 + 4;
+	ceph_encode_32(&p, -1);  /* preferred */
+
+	/* oid */
+	ceph_encode_32(&p, req->r_oid_len);
+	memcpy(p, req->r_oid, req->r_oid_len);
+	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
+	p += req->r_oid_len;
+
+	/* ops--can imply data */
+	ceph_encode_16(&p, (u16)req->r_num_ops);
+	data_len = 0;
+	for (i = 0; i < req->r_num_ops; i++) {
+		data_len += osd_req_encode_op(req, p, i);
+		p += sizeof(struct ceph_osd_op);
+	}
+
+	/* snaps */
+	ceph_encode_64(&p, req->r_snapid);
+	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
+	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
+	if (req->r_snapc) {
+		for (i = 0; i < snapc->num_snaps; i++) {
+			ceph_encode_64(&p, req->r_snapc->snaps[i]);
+		}
+	}
+
+	req->r_request_attempts = p;
+	p += 4;
+
+	/* data */
+	if (flags & CEPH_OSD_FLAG_WRITE) {
+		u16 data_off;
+
+		/*
+		 * The header "data_off" is a hint to the receiver
+		 * allowing it to align received data into its
+		 * buffers such that there's no need to re-copy
+		 * it before writing it to disk (direct I/O).
+		 */
+		data_off = (u16) (off & 0xffff);
+		req->r_request->hdr.data_off = cpu_to_le16(data_off);
+	}
+	req->r_request->hdr.data_len = cpu_to_le32(data_len);
+
+	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
+	msg_size = p - msg->front.iov_base;
+	msg->front.iov_len = msg_size;
+	msg->hdr.front_len = cpu_to_le32(msg_size);
+
+	dout("build_request msg_size was %d\n", (int)msg_size);
+}
+EXPORT_SYMBOL(ceph_osdc_build_request);
+
+/*
  * Register request, send initial attempt.
  */
 int ceph_osdc_start_request(struct ceph_osd_client *osdc,
-- 
1.7.9.5

Re: [PATCH 16/20] libceph: move ceph_osdc_build_request()

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:05 AM, Alex Elder wrote:
> This simply moves ceph_osdc_build_request() later in its source
> file without any change.  Done as a separate patch to facilitate
> review of the change in the next patch.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   net/ceph/osd_client.c |  196
> ++++++++++++++++++++++++-------------------------
>   1 file changed, 98 insertions(+), 98 deletions(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 6c6d7b8..f80984e 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -526,104 +526,6 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>   }
>
>   /*
> - * build new request AND message
> - *
> - */
> -void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
> -				struct ceph_snap_context *snapc, u64 snap_id,
> -				struct timespec *mtime)
> -{
> -	struct ceph_msg *msg = req->r_request;
> -	void *p;
> -	size_t msg_size;
> -	int flags = req->r_flags;
> -	u64 data_len;
> -	unsigned int i;
> -
> -	req->r_snapid = snap_id;
> -	req->r_snapc = ceph_get_snap_context(snapc);
> -
> -	/* encode request */
> -	msg->hdr.version = cpu_to_le16(4);
> -
> -	p = msg->front.iov_base;
> -	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
> -	req->r_request_osdmap_epoch = p;
> -	p += 4;
> -	req->r_request_flags = p;
> -	p += 4;
> -	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
> -		ceph_encode_timespec(p, mtime);
> -	p += sizeof(struct ceph_timespec);
> -	req->r_request_reassert_version = p;
> -	p += sizeof(struct ceph_eversion); /* will get filled in */
> -
> -	/* oloc */
> -	ceph_encode_8(&p, 4);
> -	ceph_encode_8(&p, 4);
> -	ceph_encode_32(&p, 8 + 4 + 4);
> -	req->r_request_pool = p;
> -	p += 8;
> -	ceph_encode_32(&p, -1);  /* preferred */
> -	ceph_encode_32(&p, 0);   /* key len */
> -
> -	ceph_encode_8(&p, 1);
> -	req->r_request_pgid = p;
> -	p += 8 + 4;
> -	ceph_encode_32(&p, -1);  /* preferred */
> -
> -	/* oid */
> -	ceph_encode_32(&p, req->r_oid_len);
> -	memcpy(p, req->r_oid, req->r_oid_len);
> -	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
> -	p += req->r_oid_len;
> -
> -	/* ops--can imply data */
> -	ceph_encode_16(&p, (u16)req->r_num_ops);
> -	data_len = 0;
> -	for (i = 0; i < req->r_num_ops; i++) {
> -		data_len += osd_req_encode_op(req, p, i);
> -		p += sizeof(struct ceph_osd_op);
> -	}
> -
> -	/* snaps */
> -	ceph_encode_64(&p, req->r_snapid);
> -	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
> -	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
> -	if (req->r_snapc) {
> -		for (i = 0; i < snapc->num_snaps; i++) {
> -			ceph_encode_64(&p, req->r_snapc->snaps[i]);
> -		}
> -	}
> -
> -	req->r_request_attempts = p;
> -	p += 4;
> -
> -	/* data */
> -	if (flags & CEPH_OSD_FLAG_WRITE) {
> -		u16 data_off;
> -
> -		/*
> -		 * The header "data_off" is a hint to the receiver
> -		 * allowing it to align received data into its
> -		 * buffers such that there's no need to re-copy
> -		 * it before writing it to disk (direct I/O).
> -		 */
> -		data_off = (u16) (off & 0xffff);
> -		req->r_request->hdr.data_off = cpu_to_le16(data_off);
> -	}
> -	req->r_request->hdr.data_len = cpu_to_le32(data_len);
> -
> -	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
> -	msg_size = p - msg->front.iov_base;
> -	msg->front.iov_len = msg_size;
> -	msg->hdr.front_len = cpu_to_le32(msg_size);
> -
> -	dout("build_request msg_size was %d\n", (int)msg_size);
> -}
> -EXPORT_SYMBOL(ceph_osdc_build_request);
> -
> -/*
>    * build new request AND message, calculate layout, and adjust file
>    * extent as needed.
>    *
> @@ -1968,6 +1870,104 @@ static void ceph_osdc_msg_data_set(struct
> ceph_msg *msg,
>   }
>
>   /*
> + * build new request AND message
> + *
> + */
> +void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
> +				struct ceph_snap_context *snapc, u64 snap_id,
> +				struct timespec *mtime)
> +{
> +	struct ceph_msg *msg = req->r_request;
> +	void *p;
> +	size_t msg_size;
> +	int flags = req->r_flags;
> +	u64 data_len;
> +	unsigned int i;
> +
> +	req->r_snapid = snap_id;
> +	req->r_snapc = ceph_get_snap_context(snapc);
> +
> +	/* encode request */
> +	msg->hdr.version = cpu_to_le16(4);
> +
> +	p = msg->front.iov_base;
> +	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
> +	req->r_request_osdmap_epoch = p;
> +	p += 4;
> +	req->r_request_flags = p;
> +	p += 4;
> +	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
> +		ceph_encode_timespec(p, mtime);
> +	p += sizeof(struct ceph_timespec);
> +	req->r_request_reassert_version = p;
> +	p += sizeof(struct ceph_eversion); /* will get filled in */
> +
> +	/* oloc */
> +	ceph_encode_8(&p, 4);
> +	ceph_encode_8(&p, 4);
> +	ceph_encode_32(&p, 8 + 4 + 4);
> +	req->r_request_pool = p;
> +	p += 8;
> +	ceph_encode_32(&p, -1);  /* preferred */
> +	ceph_encode_32(&p, 0);   /* key len */
> +
> +	ceph_encode_8(&p, 1);
> +	req->r_request_pgid = p;
> +	p += 8 + 4;
> +	ceph_encode_32(&p, -1);  /* preferred */
> +
> +	/* oid */
> +	ceph_encode_32(&p, req->r_oid_len);
> +	memcpy(p, req->r_oid, req->r_oid_len);
> +	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
> +	p += req->r_oid_len;
> +
> +	/* ops--can imply data */
> +	ceph_encode_16(&p, (u16)req->r_num_ops);
> +	data_len = 0;
> +	for (i = 0; i < req->r_num_ops; i++) {
> +		data_len += osd_req_encode_op(req, p, i);
> +		p += sizeof(struct ceph_osd_op);
> +	}
> +
> +	/* snaps */
> +	ceph_encode_64(&p, req->r_snapid);
> +	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
> +	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
> +	if (req->r_snapc) {
> +		for (i = 0; i < snapc->num_snaps; i++) {
> +			ceph_encode_64(&p, req->r_snapc->snaps[i]);
> +		}
> +	}
> +
> +	req->r_request_attempts = p;
> +	p += 4;
> +
> +	/* data */
> +	if (flags & CEPH_OSD_FLAG_WRITE) {
> +		u16 data_off;
> +
> +		/*
> +		 * The header "data_off" is a hint to the receiver
> +		 * allowing it to align received data into its
> +		 * buffers such that there's no need to re-copy
> +		 * it before writing it to disk (direct I/O).
> +		 */
> +		data_off = (u16) (off & 0xffff);
> +		req->r_request->hdr.data_off = cpu_to_le16(data_off);
> +	}
> +	req->r_request->hdr.data_len = cpu_to_le32(data_len);
> +
> +	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
> +	msg_size = p - msg->front.iov_base;
> +	msg->front.iov_len = msg_size;
> +	msg->hdr.front_len = cpu_to_le32(msg_size);
> +
> +	dout("build_request msg_size was %d\n", (int)msg_size);
> +}
> +EXPORT_SYMBOL(ceph_osdc_build_request);
> +
> +/*
>    * Register request, send initial attempt.
>    */
>   int ceph_osdc_start_request(struct ceph_osd_client *osdc,
>

[PATCH 17/20] libceph: set message data when building osd request

[Alex Elder]

All calls of ceph_osdc_start_request() are preceded (in the case of
rbd, almost) immediately by a call to ceph_osdc_build_request().

Move the build calls at the top of ceph_osdc_start_request() out of
there and into the ceph_osdc_build_request().  Nothing prevents
moving these calls to the top of ceph_osdc_build_request(), either
(and we're going to want them there in the next patch) so put them
at the top.

This and the next patch are related to:
    http://tracker.ceph.com/issues/4657

Signed-off-by: Alex Elder <elder@inktank.com>
---
 net/ceph/osd_client.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index f80984e..40466ab 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -1884,6 +1884,11 @@ void ceph_osdc_build_request(struct
ceph_osd_request *req, u64 off,
 	u64 data_len;
 	unsigned int i;

+	/* Set up response incoming data and request outgoing data fields */
+
+	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
+	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
+
 	req->r_snapid = snap_id;
 	req->r_snapc = ceph_get_snap_context(snapc);

@@ -1976,11 +1981,6 @@ int ceph_osdc_start_request(struct
ceph_osd_client *osdc,
 {
 	int rc = 0;

-	/* Set up response incoming data and request outgoing data fields */
-
-	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
-	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
 	down_read(&osdc->map_sem);
 	mutex_lock(&osdc->request_mutex);
 	__register_request(osdc, req);
-- 
1.7.9.5

Re: [PATCH 17/20] libceph: set message data when building osd request

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:05 AM, Alex Elder wrote:
> All calls of ceph_osdc_start_request() are preceded (in the case of
> rbd, almost) immediately by a call to ceph_osdc_build_request().
>
> Move the build calls at the top of ceph_osdc_start_request() out of
> there and into the ceph_osdc_build_request().  Nothing prevents
> moving these calls to the top of ceph_osdc_build_request(), either
> (and we're going to want them there in the next patch) so put them
> at the top.
>
> This and the next patch are related to:
>      http://tracker.ceph.com/issues/4657
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   net/ceph/osd_client.c |   10 +++++-----
>   1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index f80984e..40466ab 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -1884,6 +1884,11 @@ void ceph_osdc_build_request(struct
> ceph_osd_request *req, u64 off,
>   	u64 data_len;
>   	unsigned int i;
>
> +	/* Set up response incoming data and request outgoing data fields */
> +
> +	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
> +	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
> +
>   	req->r_snapid = snap_id;
>   	req->r_snapc = ceph_get_snap_context(snapc);
>
> @@ -1976,11 +1981,6 @@ int ceph_osdc_start_request(struct
> ceph_osd_client *osdc,
>   {
>   	int rc = 0;
>
> -	/* Set up response incoming data and request outgoing data fields */
> -
> -	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
> -	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
> -
>   	down_read(&osdc->map_sem);
>   	mutex_lock(&osdc->request_mutex);
>   	__register_request(osdc, req);
>

[PATCH 18/20] libceph: combine initializing and setting osd data

[Alex Elder]

This ends up being a rather large patch but what it's doing is
somewhat straightforward.

Basically, this is replacing two calls with one.  The first of the
two calls is initializing a struct ceph_osd_data with data (either a
page array, a page list, or a bio list); the second is setting an
osd request op so it associates that data with one of the op's
parameters.  In place of those two will be a single function that
initializes the op directly.

That means we sort of fan out a set of the needed functions:
    - extent ops with pages data
    - extent ops with pagelist data
    - extent ops with bio list data
and
    - class ops with page data for receiving a response

We also have define another one, but it's only used internally:
    - class ops with pagelist data for request parameters

Note that we *still* haven't gotten rid of the osd request's
r_data_in and r_data_out fields.  All the osd ops refer to them for
their data.  For now, these data fields are pointers assigned to the
appropriate r_data_* field when these new functions are called.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 drivers/block/rbd.c             |   20 ++---
 fs/ceph/addr.c                  |   12 ++-
 fs/ceph/file.c                  |    3 +-
 include/linux/ceph/osd_client.h |   43 +++++++----
 net/ceph/osd_client.c           |  157
++++++++++++++++++++++++++++++---------
 5 files changed, 162 insertions(+), 73 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 3e8e6d5..7db244e 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1592,7 +1592,6 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 	rbd_assert(resid > 0);
 	while (resid) {
 		struct ceph_osd_request *osd_req;
-		struct ceph_osd_data *osd_data;
 		const char *object_name;
 		unsigned int clone_size;
 		u64 offset;
@@ -1625,13 +1624,10 @@ static int rbd_img_request_fill_bio(struct
rbd_img_request *img_request,
 		obj_request->osd_req = osd_req;
 		obj_request->callback = rbd_img_obj_callback;

-		osd_data = write_request ? &osd_req->r_data_out
-					 : &osd_req->r_data_in;
 		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
 						0, 0);
-		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
-					obj_request->length);
-		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+		osd_req_op_extent_osd_data_bio(osd_req, 0, write_request,
+				obj_request->bio_list, obj_request->length);
 		rbd_osd_req_format(obj_request, write_request);

 		rbd_img_obj_request_add(img_request, obj_request);
@@ -1821,7 +1817,6 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 {
 	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_data *osd_data;
 	struct page **pages;
 	u32 page_count;
 	int ret;
@@ -1851,13 +1846,12 @@ static int rbd_obj_method_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	osd_data = &obj_request->osd_req->r_data_in;
 	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
 					class_name, method_name,
 					outbound, outbound_size);
-	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+	osd_req_op_cls_response_data_pages(obj_request->osd_req, 0,
+					obj_request->pages, 0, 0,
 					false, false);
-	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

 	ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2037,7 +2031,6 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 {
 	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_data *osd_data;
 	struct page **pages = NULL;
 	u32 page_count;
 	size_t size;
@@ -2061,14 +2054,13 @@ static int rbd_obj_read_sync(struct rbd_device
*rbd_dev,
 	if (!obj_request->osd_req)
 		goto out;

-	osd_data = &obj_request->osd_req->r_data_in;
 	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
 					offset, length, 0, 0);
-	ceph_osd_data_pages_init(osd_data, obj_request->pages,
+	osd_req_op_extent_osd_data_pages(obj_request->osd_req, 0, false,
+					obj_request->pages,
 					obj_request->length,
 					obj_request->offset & ~PAGE_MASK,
 					false, false);
-	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);

 	ret = rbd_obj_request_submit(osdc, obj_request);
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index dd5d263..068d2c8 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -245,7 +245,7 @@ static void finish_read(struct ceph_osd_request
*req, struct ceph_msg *msg)
 	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);

 	/* unlock all pages, zeroing any data we didn't read */
-	osd_data = &req->r_data_in;
+	osd_data = osd_req_op_extent_osd_data(req, 0, false);
 	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
 	num_pages = calc_pages_for((u64)osd_data->alignment,
 					(u64)osd_data->length);
@@ -343,8 +343,7 @@ static int start_read(struct inode *inode, struct
list_head *page_list, int max)
 		}
 		pages[i] = page;
 	}
-	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
-	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
+	osd_req_op_extent_osd_data_pages(req, 0, false, pages, len, 0,
 					false, false);
 	req->r_callback = finish_read;
 	req->r_inode = inode;
@@ -572,7 +571,7 @@ static void writepages_finish(struct
ceph_osd_request *req,
 	long writeback_stat;
 	unsigned issued = ceph_caps_issued(ci);

-	osd_data = &req->r_data_out;
+	osd_data = osd_req_op_extent_osd_data(req, 0, true);
 	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
 	num_pages = calc_pages_for((u64)osd_data->alignment,
 					(u64)osd_data->length);
@@ -917,9 +916,8 @@ get_more_pages:
 		dout("writepages got %d pages at %llu~%llu\n",
 		     locked_pages, offset, len);

-		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
-		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
-						len, 0, !!pool, false);
+		osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, 0,
+							!!pool, false);

 		pages = NULL;	/* request message now owns the pages array */
 		pool = NULL;
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 66d0938..b7e6caa 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -574,8 +574,7 @@ more:
 			own_pages = true;
 		}
 	}
-	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
-	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len,
 					page_align, false, own_pages);

 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index 71c4157..f8a00b4 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -240,17 +240,39 @@ extern void osd_req_op_extent_init(struct
ceph_osd_request *osd_req,
 					u64 truncate_size, u32 truncate_seq);
 extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
 					unsigned int which, u64 length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+
+extern struct ceph_osd_data *osd_req_op_extent_osd_data(
+					struct ceph_osd_request *osd_req,
+					unsigned int which, bool write_request);
+extern struct ceph_osd_data *osd_req_op_cls_response_data(
+					struct ceph_osd_request *osd_req,
+					unsigned int which);
+
+extern void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct page **pages, u64 length,
+					u32 alignment, bool pages_from_pool,
+					bool own_pages);
+extern void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
+extern void osd_req_op_cls_response_data_pages(struct ceph_osd_request *,
 					unsigned int which,
-					struct ceph_osd_data *osd_data);
+					struct page **pages, u64 length,
+					u32 alignment, bool pages_from_pool,
+					bool own_pages);
+
 extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
 					unsigned int which, u16 opcode,
 					const char *class, const char *method,
 					const void *request_data,
 					size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
-					unsigned int which,
-					struct ceph_osd_data *response_data);
 extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
 					unsigned int which, u16 opcode,
 					u64 cookie, u64 version, int flag);
@@ -290,17 +312,6 @@ static inline void ceph_osdc_put_request(struct
ceph_osd_request *req)
 	kref_put(&req->r_kref, ceph_osdc_release_request);
 }

-extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
-				     struct page **pages, u64 length,
-				     u32 alignment, bool pages_from_pool,
-				     bool own_pages);
-extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
-					struct ceph_pagelist *pagelist);
-#ifdef CONFIG_BLOCK
-extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
-				   struct bio *bio, size_t bio_length);
-#endif /* CONFIG_BLOCK */
-
 extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
 				   struct ceph_osd_request *req,
 				   bool nofail);
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 40466ab..86cb524 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -1,3 +1,4 @@
+
 #include <linux/ceph/ceph_debug.h>

 #include <linux/module.h>
@@ -81,11 +82,11 @@ static int calc_layout(struct ceph_file_layout
*layout, u64 off, u64 *plen,

 static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
 {
-	memset(osd_data, 0, sizeof *osd_data);
+	memset(osd_data, 0, sizeof (*osd_data));
 	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
 }

-void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
 			struct page **pages, u64 length, u32 alignment,
 			bool pages_from_pool, bool own_pages)
 {
@@ -96,27 +97,131 @@ void ceph_osd_data_pages_init(struct ceph_osd_data
*osd_data,
 	osd_data->pages_from_pool = pages_from_pool;
 	osd_data->own_pages = own_pages;
 }
-EXPORT_SYMBOL(ceph_osd_data_pages_init);

-void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
 			struct ceph_pagelist *pagelist)
 {
 	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
 	osd_data->pagelist = pagelist;
 }
-EXPORT_SYMBOL(ceph_osd_data_pagelist_init);

 #ifdef CONFIG_BLOCK
-void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
 			struct bio *bio, size_t bio_length)
 {
 	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
 	osd_data->bio = bio;
 	osd_data->bio_length = bio_length;
 }
-EXPORT_SYMBOL(ceph_osd_data_bio_init);
 #endif /* CONFIG_BLOCK */

+struct ceph_osd_data *
+osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].extent.osd_data; */
+	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
+struct ceph_osd_data *
+osd_req_op_cls_request_info(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].cls.request_info; */
+	return &osd_req->r_data_out;	/* Request data is outgoing */
+}
+EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */
+
+struct ceph_osd_data *
+osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].cls.response_data; */
+	return &osd_req->r_data_in;	/* Response data is incoming */
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */
+
+void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct page **pages, u64 length, u32 alignment,
+			bool pages_from_pool, bool own_pages)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+				pages_from_pool, own_pages);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
+
+void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct ceph_pagelist *pagelist)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
+
+#ifdef CONFIG_BLOCK
+void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct bio *bio, size_t bio_length)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_bio_init(osd_data, bio, bio_length);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
+#endif /* CONFIG_BLOCK */
+
+static void osd_req_op_cls_request_info_pagelist(
+			struct ceph_osd_request *osd_req,
+			unsigned int which, struct ceph_pagelist *pagelist)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_cls_request_info(osd_req, which);
+	ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+	osd_req->r_ops[which].cls.request_info =
+		osd_req_op_cls_request_info(osd_req, which);
+}
+
+void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
+			unsigned int which, struct page **pages, u64 length,
+			u32 alignment, bool pages_from_pool, bool own_pages)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_cls_response_data(osd_req, which);
+	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+				pages_from_pool, own_pages);
+
+	osd_req->r_ops[which].cls.response_data =
+		osd_req_op_cls_response_data(osd_req, which);
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
+
 static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
 {
 	switch (osd_data->type) {
@@ -385,15 +490,6 @@ void osd_req_op_extent_update(struct
ceph_osd_request *osd_req,
 }
 EXPORT_SYMBOL(osd_req_op_extent_update);

-void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
-				unsigned int which,
-				struct ceph_osd_data *osd_data)
-{
-	BUG_ON(which >= osd_req->r_num_ops);
-	osd_req->r_ops[which].extent.osd_data = osd_data;
-}
-EXPORT_SYMBOL(osd_req_op_extent_osd_data);
-
 void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
which,
 			u16 opcode, const char *class, const char *method,
 			const void *request_data, size_t request_data_size)
@@ -429,22 +525,13 @@ void osd_req_op_cls_init(struct ceph_osd_request
*osd_req, unsigned int which,
 	ceph_pagelist_append(pagelist, request_data, request_data_size);
 	payload_len += request_data_size;

-	op->cls.request_info = &osd_req->r_data_out;
-	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+	osd_req_op_cls_request_info_pagelist(osd_req, which, pagelist);

 	op->cls.argc = 0;	/* currently unused */

 	op->payload_len = payload_len;
 }
 EXPORT_SYMBOL(osd_req_op_cls_init);
-void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
-				unsigned int which,
-				struct ceph_osd_data *response_data)
-{
-	BUG_ON(which >= osd_req->r_num_ops);
-	osd_req->r_ops[which].cls.response_data = response_data;
-}
-EXPORT_SYMBOL(osd_req_op_cls_response_data);

 void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
 				unsigned int which, u16 opcode,
@@ -547,7 +634,6 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					       bool use_mempool)
 {
 	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
 	u64 objnum = 0;
 	u64 objoff = 0;
 	u64 objlen = 0;
@@ -561,8 +647,6 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,
 					GFP_NOFS);
 	if (!req)
 		return ERR_PTR(-ENOMEM);
-	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
-					       : &req->r_data_in;

 	req->r_flags = flags;

@@ -585,7 +669,6 @@ struct ceph_osd_request
*ceph_osdc_new_request(struct ceph_osd_client *osdc,

 	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
 				truncate_size, truncate_seq);
-	osd_req_op_extent_osd_data(req, 0, osd_data);

 	/*
 	 * A second op in the ops array means the caller wants to
@@ -2171,8 +2254,8 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,

 	/* it may be a short read due to an object boundary */

-	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
-				false, false);
+	osd_req_op_extent_osd_data_pages(req, 0, false,
+				pages, *plen, page_align, false, false);

 	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
 	     off, *plen, *plen, page_align);
@@ -2214,7 +2297,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
*osdc, struct ceph_vino vino,
 		return PTR_ERR(req);

 	/* it may be a short write due to an object boundary */
-	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, page_align,
 				false, false);
 	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

@@ -2308,8 +2391,14 @@ static struct ceph_msg *get_reply(struct
ceph_connection *con,
 	m = ceph_msg_get(req->r_reply);

 	if (data_len > 0) {
-		struct ceph_osd_data *osd_data = &req->r_data_in;
+		struct ceph_osd_data *osd_data;

+		/*
+		 * XXX This is assuming there is only one op containing
+		 * XXX page data.  Probably OK for reads, but this
+		 * XXX ought to be done more generally.
+		 */
+		osd_data = osd_req_op_extent_osd_data(req, 0, false);
 		if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
 			if (osd_data->pages &&
 				unlikely(osd_data->length < data_len)) {
-- 
1.7.9.5

Re: [PATCH 18/20] libceph: combine initializing and setting osd data

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:05 AM, Alex Elder wrote:
> This ends up being a rather large patch but what it's doing is
> somewhat straightforward.
>
> Basically, this is replacing two calls with one.  The first of the
> two calls is initializing a struct ceph_osd_data with data (either a
> page array, a page list, or a bio list); the second is setting an
> osd request op so it associates that data with one of the op's
> parameters.  In place of those two will be a single function that
> initializes the op directly.
>
> That means we sort of fan out a set of the needed functions:
>      - extent ops with pages data
>      - extent ops with pagelist data
>      - extent ops with bio list data
> and
>      - class ops with page data for receiving a response
>
> We also have define another one, but it's only used internally:
>      - class ops with pagelist data for request parameters
>
> Note that we *still* haven't gotten rid of the osd request's
> r_data_in and r_data_out fields.  All the osd ops refer to them for
> their data.  For now, these data fields are pointers assigned to the
> appropriate r_data_* field when these new functions are called.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   drivers/block/rbd.c             |   20 ++---
>   fs/ceph/addr.c                  |   12 ++-
>   fs/ceph/file.c                  |    3 +-
>   include/linux/ceph/osd_client.h |   43 +++++++----
>   net/ceph/osd_client.c           |  157
> ++++++++++++++++++++++++++++++---------
>   5 files changed, 162 insertions(+), 73 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 3e8e6d5..7db244e 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -1592,7 +1592,6 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   	rbd_assert(resid > 0);
>   	while (resid) {
>   		struct ceph_osd_request *osd_req;
> -		struct ceph_osd_data *osd_data;
>   		const char *object_name;
>   		unsigned int clone_size;
>   		u64 offset;
> @@ -1625,13 +1624,10 @@ static int rbd_img_request_fill_bio(struct
> rbd_img_request *img_request,
>   		obj_request->osd_req = osd_req;
>   		obj_request->callback = rbd_img_obj_callback;
>
> -		osd_data = write_request ? &osd_req->r_data_out
> -					 : &osd_req->r_data_in;
>   		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
>   						0, 0);
> -		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
> -					obj_request->length);
> -		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
> +		osd_req_op_extent_osd_data_bio(osd_req, 0, write_request,
> +				obj_request->bio_list, obj_request->length);
>   		rbd_osd_req_format(obj_request, write_request);
>
>   		rbd_img_obj_request_add(img_request, obj_request);
> @@ -1821,7 +1817,6 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   {
>   	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_data *osd_data;
>   	struct page **pages;
>   	u32 page_count;
>   	int ret;
> @@ -1851,13 +1846,12 @@ static int rbd_obj_method_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	osd_data = &obj_request->osd_req->r_data_in;
>   	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
>   					class_name, method_name,
>   					outbound, outbound_size);
> -	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
> +	osd_req_op_cls_response_data_pages(obj_request->osd_req, 0,
> +					obj_request->pages, 0, 0,
>   					false, false);
> -	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
>   	rbd_osd_req_format(obj_request, false);
>
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> @@ -2037,7 +2031,6 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   {
>   	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
>   	struct rbd_obj_request *obj_request;
> -	struct ceph_osd_data *osd_data;
>   	struct page **pages = NULL;
>   	u32 page_count;
>   	size_t size;
> @@ -2061,14 +2054,13 @@ static int rbd_obj_read_sync(struct rbd_device
> *rbd_dev,
>   	if (!obj_request->osd_req)
>   		goto out;
>
> -	osd_data = &obj_request->osd_req->r_data_in;
>   	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
>   					offset, length, 0, 0);
> -	ceph_osd_data_pages_init(osd_data, obj_request->pages,
> +	osd_req_op_extent_osd_data_pages(obj_request->osd_req, 0, false,
> +					obj_request->pages,
>   					obj_request->length,
>   					obj_request->offset & ~PAGE_MASK,
>   					false, false);
> -	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
>   	rbd_osd_req_format(obj_request, false);
>
>   	ret = rbd_obj_request_submit(osdc, obj_request);
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index dd5d263..068d2c8 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -245,7 +245,7 @@ static void finish_read(struct ceph_osd_request
> *req, struct ceph_msg *msg)
>   	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);
>
>   	/* unlock all pages, zeroing any data we didn't read */
> -	osd_data = &req->r_data_in;
> +	osd_data = osd_req_op_extent_osd_data(req, 0, false);
>   	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
>   	num_pages = calc_pages_for((u64)osd_data->alignment,
>   					(u64)osd_data->length);
> @@ -343,8 +343,7 @@ static int start_read(struct inode *inode, struct
> list_head *page_list, int max)
>   		}
>   		pages[i] = page;
>   	}
> -	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
> -	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
> +	osd_req_op_extent_osd_data_pages(req, 0, false, pages, len, 0,
>   					false, false);
>   	req->r_callback = finish_read;
>   	req->r_inode = inode;
> @@ -572,7 +571,7 @@ static void writepages_finish(struct
> ceph_osd_request *req,
>   	long writeback_stat;
>   	unsigned issued = ceph_caps_issued(ci);
>
> -	osd_data = &req->r_data_out;
> +	osd_data = osd_req_op_extent_osd_data(req, 0, true);
>   	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
>   	num_pages = calc_pages_for((u64)osd_data->alignment,
>   					(u64)osd_data->length);
> @@ -917,9 +916,8 @@ get_more_pages:
>   		dout("writepages got %d pages at %llu~%llu\n",
>   		     locked_pages, offset, len);
>
> -		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
> -		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
> -						len, 0, !!pool, false);
> +		osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, 0,
> +							!!pool, false);
>
>   		pages = NULL;	/* request message now owns the pages array */
>   		pool = NULL;
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 66d0938..b7e6caa 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -574,8 +574,7 @@ more:
>   			own_pages = true;
>   		}
>   	}
> -	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
> -	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
> +	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len,
>   					page_align, false, own_pages);
>
>   	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index 71c4157..f8a00b4 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -240,17 +240,39 @@ extern void osd_req_op_extent_init(struct
> ceph_osd_request *osd_req,
>   					u64 truncate_size, u32 truncate_seq);
>   extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
>   					unsigned int which, u64 length);
> -extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
> +
> +extern struct ceph_osd_data *osd_req_op_extent_osd_data(
> +					struct ceph_osd_request *osd_req,
> +					unsigned int which, bool write_request);
> +extern struct ceph_osd_data *osd_req_op_cls_response_data(
> +					struct ceph_osd_request *osd_req,
> +					unsigned int which);
> +
> +extern void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *,
> +					unsigned int which, bool write_request,
> +					struct page **pages, u64 length,
> +					u32 alignment, bool pages_from_pool,
> +					bool own_pages);
> +extern void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *,
> +					unsigned int which, bool write_request,
> +					struct ceph_pagelist *pagelist);
> +#ifdef CONFIG_BLOCK
> +extern void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *,
> +					unsigned int which, bool write_request,
> +					struct bio *bio, size_t bio_length);
> +#endif /* CONFIG_BLOCK */
> +
> +extern void osd_req_op_cls_response_data_pages(struct ceph_osd_request *,
>   					unsigned int which,
> -					struct ceph_osd_data *osd_data);
> +					struct page **pages, u64 length,
> +					u32 alignment, bool pages_from_pool,
> +					bool own_pages);
> +
>   extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
>   					unsigned int which, u16 opcode,
>   					const char *class, const char *method,
>   					const void *request_data,
>   					size_t request_data_size);
> -extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
> -					unsigned int which,
> -					struct ceph_osd_data *response_data);
>   extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
>   					unsigned int which, u16 opcode,
>   					u64 cookie, u64 version, int flag);
> @@ -290,17 +312,6 @@ static inline void ceph_osdc_put_request(struct
> ceph_osd_request *req)
>   	kref_put(&req->r_kref, ceph_osdc_release_request);
>   }
>
> -extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
> -				     struct page **pages, u64 length,
> -				     u32 alignment, bool pages_from_pool,
> -				     bool own_pages);
> -extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
> -					struct ceph_pagelist *pagelist);
> -#ifdef CONFIG_BLOCK
> -extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
> -				   struct bio *bio, size_t bio_length);
> -#endif /* CONFIG_BLOCK */
> -
>   extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
>   				   struct ceph_osd_request *req,
>   				   bool nofail);
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 40466ab..86cb524 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -1,3 +1,4 @@
> +
>   #include <linux/ceph/ceph_debug.h>
>
>   #include <linux/module.h>
> @@ -81,11 +82,11 @@ static int calc_layout(struct ceph_file_layout
> *layout, u64 off, u64 *plen,
>
>   static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
>   {
> -	memset(osd_data, 0, sizeof *osd_data);
> +	memset(osd_data, 0, sizeof (*osd_data));
>   	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
>   }
>
> -void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
> +static void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
>   			struct page **pages, u64 length, u32 alignment,
>   			bool pages_from_pool, bool own_pages)
>   {
> @@ -96,27 +97,131 @@ void ceph_osd_data_pages_init(struct ceph_osd_data
> *osd_data,
>   	osd_data->pages_from_pool = pages_from_pool;
>   	osd_data->own_pages = own_pages;
>   }
> -EXPORT_SYMBOL(ceph_osd_data_pages_init);
>
> -void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
> +static void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
>   			struct ceph_pagelist *pagelist)
>   {
>   	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
>   	osd_data->pagelist = pagelist;
>   }
> -EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
>
>   #ifdef CONFIG_BLOCK
> -void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
> +static void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
>   			struct bio *bio, size_t bio_length)
>   {
>   	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
>   	osd_data->bio = bio;
>   	osd_data->bio_length = bio_length;
>   }
> -EXPORT_SYMBOL(ceph_osd_data_bio_init);
>   #endif /* CONFIG_BLOCK */
>
> +struct ceph_osd_data *
> +osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
> +			unsigned int which, bool write_request)
> +{
> +	BUG_ON(which >= osd_req->r_num_ops);
> +
> +	/* return &osd_req->r_ops[which].extent.osd_data; */
> +	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
> +}
> +EXPORT_SYMBOL(osd_req_op_extent_osd_data);
> +
> +struct ceph_osd_data *
> +osd_req_op_cls_request_info(struct ceph_osd_request *osd_req,
> +			unsigned int which)
> +{
> +	BUG_ON(which >= osd_req->r_num_ops);
> +
> +	/* return &osd_req->r_ops[which].cls.request_info; */
> +	return &osd_req->r_data_out;	/* Request data is outgoing */
> +}
> +EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */
> +
> +struct ceph_osd_data *
> +osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
> +			unsigned int which)
> +{
> +	BUG_ON(which >= osd_req->r_num_ops);
> +
> +	/* return &osd_req->r_ops[which].cls.response_data; */
> +	return &osd_req->r_data_in;	/* Response data is incoming */
> +}
> +EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */
> +
> +void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *osd_req,
> +			unsigned int which, bool write_request,
> +			struct page **pages, u64 length, u32 alignment,
> +			bool pages_from_pool, bool own_pages)
> +{
> +	struct ceph_osd_data *osd_data;
> +
> +	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
> +	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
> +				pages_from_pool, own_pages);
> +
> +	osd_req->r_ops[which].extent.osd_data =
> +		osd_req_op_extent_osd_data(osd_req, which, write_request);
> +}
> +EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
> +
> +void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *osd_req,
> +			unsigned int which, bool write_request,
> +			struct ceph_pagelist *pagelist)
> +{
> +	struct ceph_osd_data *osd_data;
> +
> +	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
> +	ceph_osd_data_pagelist_init(osd_data, pagelist);
> +
> +	osd_req->r_ops[which].extent.osd_data =
> +		osd_req_op_extent_osd_data(osd_req, which, write_request);
> +}
> +EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
> +
> +#ifdef CONFIG_BLOCK
> +void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *osd_req,
> +			unsigned int which, bool write_request,
> +			struct bio *bio, size_t bio_length)
> +{
> +	struct ceph_osd_data *osd_data;
> +
> +	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
> +	ceph_osd_data_bio_init(osd_data, bio, bio_length);
> +
> +	osd_req->r_ops[which].extent.osd_data =
> +		osd_req_op_extent_osd_data(osd_req, which, write_request);
> +}
> +EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
> +#endif /* CONFIG_BLOCK */
> +
> +static void osd_req_op_cls_request_info_pagelist(
> +			struct ceph_osd_request *osd_req,
> +			unsigned int which, struct ceph_pagelist *pagelist)
> +{
> +	struct ceph_osd_data *osd_data;
> +
> +	osd_data = osd_req_op_cls_request_info(osd_req, which);
> +	ceph_osd_data_pagelist_init(osd_data, pagelist);
> +
> +	osd_req->r_ops[which].cls.request_info =
> +		osd_req_op_cls_request_info(osd_req, which);
> +}
> +
> +void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
> +			unsigned int which, struct page **pages, u64 length,
> +			u32 alignment, bool pages_from_pool, bool own_pages)
> +{
> +	struct ceph_osd_data *osd_data;
> +
> +	osd_data = osd_req_op_cls_response_data(osd_req, which);
> +	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
> +				pages_from_pool, own_pages);
> +
> +	osd_req->r_ops[which].cls.response_data =
> +		osd_req_op_cls_response_data(osd_req, which);
> +}
> +EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
> +
>   static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
>   {
>   	switch (osd_data->type) {
> @@ -385,15 +490,6 @@ void osd_req_op_extent_update(struct
> ceph_osd_request *osd_req,
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_update);
>
> -void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
> -				unsigned int which,
> -				struct ceph_osd_data *osd_data)
> -{
> -	BUG_ON(which >= osd_req->r_num_ops);
> -	osd_req->r_ops[which].extent.osd_data = osd_data;
> -}
> -EXPORT_SYMBOL(osd_req_op_extent_osd_data);
> -
>   void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
> which,
>   			u16 opcode, const char *class, const char *method,
>   			const void *request_data, size_t request_data_size)
> @@ -429,22 +525,13 @@ void osd_req_op_cls_init(struct ceph_osd_request
> *osd_req, unsigned int which,
>   	ceph_pagelist_append(pagelist, request_data, request_data_size);
>   	payload_len += request_data_size;
>
> -	op->cls.request_info = &osd_req->r_data_out;
> -	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
> +	osd_req_op_cls_request_info_pagelist(osd_req, which, pagelist);
>
>   	op->cls.argc = 0;	/* currently unused */
>
>   	op->payload_len = payload_len;
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_init);
> -void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
> -				unsigned int which,
> -				struct ceph_osd_data *response_data)
> -{
> -	BUG_ON(which >= osd_req->r_num_ops);
> -	osd_req->r_ops[which].cls.response_data = response_data;
> -}
> -EXPORT_SYMBOL(osd_req_op_cls_response_data);
>
>   void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
>   				unsigned int which, u16 opcode,
> @@ -547,7 +634,6 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					       bool use_mempool)
>   {
>   	struct ceph_osd_request *req;
> -	struct ceph_osd_data *osd_data;
>   	u64 objnum = 0;
>   	u64 objoff = 0;
>   	u64 objlen = 0;
> @@ -561,8 +647,6 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>   					GFP_NOFS);
>   	if (!req)
>   		return ERR_PTR(-ENOMEM);
> -	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
> -					       : &req->r_data_in;
>
>   	req->r_flags = flags;
>
> @@ -585,7 +669,6 @@ struct ceph_osd_request
> *ceph_osdc_new_request(struct ceph_osd_client *osdc,
>
>   	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
>   				truncate_size, truncate_seq);
> -	osd_req_op_extent_osd_data(req, 0, osd_data);
>
>   	/*
>   	 * A second op in the ops array means the caller wants to
> @@ -2171,8 +2254,8 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
>
>   	/* it may be a short read due to an object boundary */
>
> -	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
> -				false, false);
> +	osd_req_op_extent_osd_data_pages(req, 0, false,
> +				pages, *plen, page_align, false, false);
>
>   	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
>   	     off, *plen, *plen, page_align);
> @@ -2214,7 +2297,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
> *osdc, struct ceph_vino vino,
>   		return PTR_ERR(req);
>
>   	/* it may be a short write due to an object boundary */
> -	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
> +	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, page_align,
>   				false, false);
>   	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
>
> @@ -2308,8 +2391,14 @@ static struct ceph_msg *get_reply(struct
> ceph_connection *con,
>   	m = ceph_msg_get(req->r_reply);
>
>   	if (data_len > 0) {
> -		struct ceph_osd_data *osd_data = &req->r_data_in;
> +		struct ceph_osd_data *osd_data;
>
> +		/*
> +		 * XXX This is assuming there is only one op containing
> +		 * XXX page data.  Probably OK for reads, but this
> +		 * XXX ought to be done more generally.
> +		 */
> +		osd_data = osd_req_op_extent_osd_data(req, 0, false);
>   		if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
>   			if (osd_data->pages &&
>   				unlikely(osd_data->length < data_len)) {
>

[PATCH 19/20] libceph: set the data pointers when encoding ops

[Alex Elder]

Still using the osd request r_data_in and r_data_out pointer, but
we're basically only referring to it via the data pointers in the
osd ops.  And we're transferring that information to the request
or reply message only when the op indicates it's needed, in
osd_req_encode_op().

To avoid a forward reference, ceph_osdc_msg_data_set() was moved up
in the file.

Don't bother calling ceph_osd_data_init(), in ceph_osd_alloc(),
because the ops array will already be zeroed anyway.

Signed-off-by: Alex Elder <elder@inktank.com>
---
 include/linux/ceph/osd_client.h |    2 +-
 net/ceph/osd_client.c           |   63
+++++++++++++++++++--------------------
 2 files changed, 32 insertions(+), 33 deletions(-)

diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index f8a00b4..dd4ca4ba 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -51,7 +51,7 @@ struct ceph_osd {
 #define CEPH_OSD_MAX_OP	2

 enum ceph_osd_data_type {
-	CEPH_OSD_DATA_TYPE_NONE,
+	CEPH_OSD_DATA_TYPE_NONE = 0,
 	CEPH_OSD_DATA_TYPE_PAGES,
 	CEPH_OSD_DATA_TYPE_PAGELIST,
 #ifdef CONFIG_BLOCK
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 86cb524..cc4003f 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -339,9 +339,6 @@ struct ceph_osd_request
*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
 	}
 	req->r_reply = msg;

-	ceph_osd_data_init(&req->r_data_in);
-	ceph_osd_data_init(&req->r_data_out);
-
 	/* create request message; allow space for oid */
 	if (use_mempool)
 		msg = ceph_msgpool_get(&osdc->msgpool_op, 0);
@@ -549,6 +546,28 @@ void osd_req_op_watch_init(struct ceph_osd_request
*osd_req,
 }
 EXPORT_SYMBOL(osd_req_op_watch_init);

+static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
+				struct ceph_osd_data *osd_data)
+{
+	u64 length = ceph_osd_data_length(osd_data);
+
+	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
+		BUG_ON(length > (u64) SIZE_MAX);
+		if (length)
+			ceph_msg_data_set_pages(msg, osd_data->pages,
+					length, osd_data->alignment);
+	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
+		BUG_ON(!length);
+		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
+#ifdef CONFIG_BLOCK
+	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
+		ceph_msg_data_set_bio(msg, osd_data->bio, length);
+#endif
+	} else {
+		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
+	}
+}
+
 static u64 osd_req_encode_op(struct ceph_osd_request *req,
 			      struct ceph_osd_op *dst, unsigned int which)
 {
@@ -576,17 +595,24 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
 			cpu_to_le64(src->extent.truncate_size);
 		dst->extent.truncate_seq =
 			cpu_to_le32(src->extent.truncate_seq);
-		if (src->op == CEPH_OSD_OP_WRITE)
+		if (src->op == CEPH_OSD_OP_WRITE) {
 			WARN_ON(src->extent.osd_data != &req->r_data_out);
-		else
+			ceph_osdc_msg_data_set(req->r_request,
+						src->extent.osd_data);
+		} else {
 			WARN_ON(src->extent.osd_data != &req->r_data_in);
+			ceph_osdc_msg_data_set(req->r_reply,
+						src->extent.osd_data);
+		}
 		break;
 	case CEPH_OSD_OP_CALL:
 		dst->cls.class_len = src->cls.class_len;
 		dst->cls.method_len = src->cls.method_len;
 		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
 		WARN_ON(src->cls.response_data != &req->r_data_in);
+		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
 		WARN_ON(src->cls.request_info != &req->r_data_out);
+		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
 		BUG_ON(src->cls.request_info->type !=
 					CEPH_OSD_DATA_TYPE_PAGELIST);
 		request_data_len = src->cls.request_info->pagelist->length;
@@ -1930,28 +1956,6 @@ bad:
 	return;
 }

-static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
-				struct ceph_osd_data *osd_data)
-{
-	u64 length = ceph_osd_data_length(osd_data);
-
-	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
-		BUG_ON(length > (u64) SIZE_MAX);
-		if (length)
-			ceph_msg_data_set_pages(msg, osd_data->pages,
-					length, osd_data->alignment);
-	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
-		BUG_ON(!length);
-		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
-#ifdef CONFIG_BLOCK
-	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
-		ceph_msg_data_set_bio(msg, osd_data->bio, length);
-#endif
-	} else {
-		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
-	}
-}
-
 /*
  * build new request AND message
  *
@@ -1967,11 +1971,6 @@ void ceph_osdc_build_request(struct
ceph_osd_request *req, u64 off,
 	u64 data_len;
 	unsigned int i;

-	/* Set up response incoming data and request outgoing data fields */
-
-	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
-	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
 	req->r_snapid = snap_id;
 	req->r_snapc = ceph_get_snap_context(snapc);

-- 
1.7.9.5

Re: [PATCH 19/20] libceph: set the data pointers when encoding ops

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:06 AM, Alex Elder wrote:
> Still using the osd request r_data_in and r_data_out pointer, but
> we're basically only referring to it via the data pointers in the
> osd ops.  And we're transferring that information to the request
> or reply message only when the op indicates it's needed, in
> osd_req_encode_op().
>
> To avoid a forward reference, ceph_osdc_msg_data_set() was moved up
> in the file.
>
> Don't bother calling ceph_osd_data_init(), in ceph_osd_alloc(),
> because the ops array will already be zeroed anyway.
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   include/linux/ceph/osd_client.h |    2 +-
>   net/ceph/osd_client.c           |   63
> +++++++++++++++++++--------------------
>   2 files changed, 32 insertions(+), 33 deletions(-)
>
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index f8a00b4..dd4ca4ba 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -51,7 +51,7 @@ struct ceph_osd {
>   #define CEPH_OSD_MAX_OP	2
>
>   enum ceph_osd_data_type {
> -	CEPH_OSD_DATA_TYPE_NONE,
> +	CEPH_OSD_DATA_TYPE_NONE = 0,
>   	CEPH_OSD_DATA_TYPE_PAGES,
>   	CEPH_OSD_DATA_TYPE_PAGELIST,
>   #ifdef CONFIG_BLOCK
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 86cb524..cc4003f 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -339,9 +339,6 @@ struct ceph_osd_request
> *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
>   	}
>   	req->r_reply = msg;
>
> -	ceph_osd_data_init(&req->r_data_in);
> -	ceph_osd_data_init(&req->r_data_out);
> -
>   	/* create request message; allow space for oid */
>   	if (use_mempool)
>   		msg = ceph_msgpool_get(&osdc->msgpool_op, 0);
> @@ -549,6 +546,28 @@ void osd_req_op_watch_init(struct ceph_osd_request
> *osd_req,
>   }
>   EXPORT_SYMBOL(osd_req_op_watch_init);
>
> +static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
> +				struct ceph_osd_data *osd_data)
> +{
> +	u64 length = ceph_osd_data_length(osd_data);
> +
> +	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
> +		BUG_ON(length > (u64) SIZE_MAX);
> +		if (length)
> +			ceph_msg_data_set_pages(msg, osd_data->pages,
> +					length, osd_data->alignment);
> +	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
> +		BUG_ON(!length);
> +		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
> +#ifdef CONFIG_BLOCK
> +	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
> +		ceph_msg_data_set_bio(msg, osd_data->bio, length);
> +#endif
> +	} else {
> +		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
> +	}
> +}
> +
>   static u64 osd_req_encode_op(struct ceph_osd_request *req,
>   			      struct ceph_osd_op *dst, unsigned int which)
>   {
> @@ -576,17 +595,24 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>   			cpu_to_le64(src->extent.truncate_size);
>   		dst->extent.truncate_seq =
>   			cpu_to_le32(src->extent.truncate_seq);
> -		if (src->op == CEPH_OSD_OP_WRITE)
> +		if (src->op == CEPH_OSD_OP_WRITE) {
>   			WARN_ON(src->extent.osd_data != &req->r_data_out);
> -		else
> +			ceph_osdc_msg_data_set(req->r_request,
> +						src->extent.osd_data);
> +		} else {
>   			WARN_ON(src->extent.osd_data != &req->r_data_in);
> +			ceph_osdc_msg_data_set(req->r_reply,
> +						src->extent.osd_data);
> +		}
>   		break;
>   	case CEPH_OSD_OP_CALL:
>   		dst->cls.class_len = src->cls.class_len;
>   		dst->cls.method_len = src->cls.method_len;
>   		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
>   		WARN_ON(src->cls.response_data != &req->r_data_in);
> +		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
>   		WARN_ON(src->cls.request_info != &req->r_data_out);
> +		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
>   		BUG_ON(src->cls.request_info->type !=
>   					CEPH_OSD_DATA_TYPE_PAGELIST);
>   		request_data_len = src->cls.request_info->pagelist->length;
> @@ -1930,28 +1956,6 @@ bad:
>   	return;
>   }
>
> -static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
> -				struct ceph_osd_data *osd_data)
> -{
> -	u64 length = ceph_osd_data_length(osd_data);
> -
> -	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
> -		BUG_ON(length > (u64) SIZE_MAX);
> -		if (length)
> -			ceph_msg_data_set_pages(msg, osd_data->pages,
> -					length, osd_data->alignment);
> -	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
> -		BUG_ON(!length);
> -		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
> -#ifdef CONFIG_BLOCK
> -	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
> -		ceph_msg_data_set_bio(msg, osd_data->bio, length);
> -#endif
> -	} else {
> -		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
> -	}
> -}
> -
>   /*
>    * build new request AND message
>    *
> @@ -1967,11 +1971,6 @@ void ceph_osdc_build_request(struct
> ceph_osd_request *req, u64 off,
>   	u64 data_len;
>   	unsigned int i;
>
> -	/* Set up response incoming data and request outgoing data fields */
> -
> -	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
> -	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
> -
>   	req->r_snapid = snap_id;
>   	req->r_snapc = ceph_get_snap_context(snapc);
>

[PATCH 20/20] libceph: kill off osd request r_data_in and r_data_out

[Alex Elder]

Finally!  Convert the osd op data pointers into real structures, and
make the switch over to using them instead of having all ops share
the in and/or out data structures in the osd request.

Set up a new function to traverse the set of ops and release any
data associated with them (pages).

This and the patches leading up to it resolve:
    http://tracker.ceph.com/issues/4657

Signed-off-by: Alex Elder <elder@inktank.com>
---
 include/linux/ceph/osd_client.h |    9 ++---
 net/ceph/osd_client.c           |   79
+++++++++++++++++++--------------------
 2 files changed, 42 insertions(+), 46 deletions(-)

diff --git a/include/linux/ceph/osd_client.h
b/include/linux/ceph/osd_client.h
index dd4ca4ba..4ec46c0 100644
--- a/include/linux/ceph/osd_client.h
+++ b/include/linux/ceph/osd_client.h
@@ -87,14 +87,14 @@ struct ceph_osd_req_op {
 			u64 offset, length;
 			u64 truncate_size;
 			u32 truncate_seq;
-			struct ceph_osd_data *osd_data;
+			struct ceph_osd_data osd_data;
 		} extent;
 		struct {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
-			struct ceph_osd_data *request_info;
-			struct ceph_osd_data *response_data;
+			struct ceph_osd_data request_info;
+			struct ceph_osd_data response_data;
 			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
@@ -164,9 +164,6 @@ struct ceph_osd_request {

 	struct ceph_file_layout r_file_layout;
 	struct ceph_snap_context *r_snapc;    /* snap context for writes */
-
-	struct ceph_osd_data r_data_in;
-	struct ceph_osd_data r_data_out;
 };

 struct ceph_osd_event {
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index cc4003f..2562e4e 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -121,8 +121,7 @@ osd_req_op_extent_osd_data(struct ceph_osd_request
*osd_req,
 {
 	BUG_ON(which >= osd_req->r_num_ops);

-	/* return &osd_req->r_ops[which].extent.osd_data; */
-	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+	return &osd_req->r_ops[which].extent.osd_data;
 }
 EXPORT_SYMBOL(osd_req_op_extent_osd_data);

@@ -132,8 +131,7 @@ osd_req_op_cls_request_info(struct ceph_osd_request
*osd_req,
 {
 	BUG_ON(which >= osd_req->r_num_ops);

-	/* return &osd_req->r_ops[which].cls.request_info; */
-	return &osd_req->r_data_out;	/* Request data is outgoing */
+	return &osd_req->r_ops[which].cls.request_info;
 }
 EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */

@@ -143,8 +141,7 @@ osd_req_op_cls_response_data(struct ceph_osd_request
*osd_req,
 {
 	BUG_ON(which >= osd_req->r_num_ops);

-	/* return &osd_req->r_ops[which].cls.response_data; */
-	return &osd_req->r_data_in;	/* Response data is incoming */
+	return &osd_req->r_ops[which].cls.response_data;
 }
 EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */

@@ -158,9 +155,6 @@ void osd_req_op_extent_osd_data_pages(struct
ceph_osd_request *osd_req,
 	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
 	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
 				pages_from_pool, own_pages);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
 }
 EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);

@@ -172,9 +166,6 @@ void osd_req_op_extent_osd_data_pagelist(struct
ceph_osd_request *osd_req,

 	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
 	ceph_osd_data_pagelist_init(osd_data, pagelist);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
 }
 EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);

@@ -187,9 +178,6 @@ void osd_req_op_extent_osd_data_bio(struct
ceph_osd_request *osd_req,

 	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
 	ceph_osd_data_bio_init(osd_data, bio, bio_length);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
 }
 EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
 #endif /* CONFIG_BLOCK */
@@ -202,9 +190,6 @@ static void osd_req_op_cls_request_info_pagelist(

 	osd_data = osd_req_op_cls_request_info(osd_req, which);
 	ceph_osd_data_pagelist_init(osd_data, pagelist);
-
-	osd_req->r_ops[which].cls.request_info =
-		osd_req_op_cls_request_info(osd_req, which);
 }

 void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
@@ -216,9 +201,6 @@ void osd_req_op_cls_response_data_pages(struct
ceph_osd_request *osd_req,
 	osd_data = osd_req_op_cls_response_data(osd_req, which);
 	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
 				pages_from_pool, own_pages);
-
-	osd_req->r_ops[which].cls.response_data =
-		osd_req_op_cls_response_data(osd_req, which);
 }
 EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);

@@ -241,18 +223,39 @@ static u64 ceph_osd_data_length(struct
ceph_osd_data *osd_data)
 	}
 }

+
 static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
 {
-	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
-		return;
-
-	if (osd_data->own_pages) {
+	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES && osd_data->own_pages) {
 		int num_pages;

 		num_pages = calc_pages_for((u64)osd_data->alignment,
 						(u64)osd_data->length);
 		ceph_release_page_vector(osd_data->pages, num_pages);
 	}
+	ceph_osd_data_init(osd_data);
+}
+
+static void osd_req_op_data_release(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	struct ceph_osd_req_op *op;
+
+	BUG_ON(which >= osd_req->r_num_ops);
+	op = &osd_req->r_ops[which];
+
+	switch (op->op) {
+	case CEPH_OSD_OP_READ:
+	case CEPH_OSD_OP_WRITE:
+		ceph_osd_data_release(&op->extent.osd_data);
+		break;
+	case CEPH_OSD_OP_CALL:
+		ceph_osd_data_release(&op->cls.request_info);
+		ceph_osd_data_release(&op->cls.response_data);
+		break;
+	default:
+		break;
+	}
 }

 /*
@@ -261,6 +264,7 @@ static void ceph_osd_data_release(struct
ceph_osd_data *osd_data)
 void ceph_osdc_release_request(struct kref *kref)
 {
 	struct ceph_osd_request *req;
+	unsigned int which;

 	req = container_of(kref, struct ceph_osd_request, r_kref);
 	if (req->r_request)
@@ -270,8 +274,8 @@ void ceph_osdc_release_request(struct kref *kref)
 		ceph_msg_put(req->r_reply);
 	}

-	ceph_osd_data_release(&req->r_data_in);
-	ceph_osd_data_release(&req->r_data_out);
+	for (which = 0; which < req->r_num_ops; which++)
+		osd_req_op_data_release(req, which);

 	ceph_put_snap_context(req->r_snapc);
 	if (req->r_mempool)
@@ -595,27 +599,22 @@ static u64 osd_req_encode_op(struct
ceph_osd_request *req,
 			cpu_to_le64(src->extent.truncate_size);
 		dst->extent.truncate_seq =
 			cpu_to_le32(src->extent.truncate_seq);
-		if (src->op == CEPH_OSD_OP_WRITE) {
-			WARN_ON(src->extent.osd_data != &req->r_data_out);
+		if (src->op == CEPH_OSD_OP_WRITE)
 			ceph_osdc_msg_data_set(req->r_request,
-						src->extent.osd_data);
-		} else {
-			WARN_ON(src->extent.osd_data != &req->r_data_in);
+						&src->extent.osd_data);
+		else
 			ceph_osdc_msg_data_set(req->r_reply,
-						src->extent.osd_data);
-		}
+						&src->extent.osd_data);
 		break;
 	case CEPH_OSD_OP_CALL:
 		dst->cls.class_len = src->cls.class_len;
 		dst->cls.method_len = src->cls.method_len;
 		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
-		WARN_ON(src->cls.response_data != &req->r_data_in);
-		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
-		WARN_ON(src->cls.request_info != &req->r_data_out);
-		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
-		BUG_ON(src->cls.request_info->type !=
+		ceph_osdc_msg_data_set(req->r_reply, &src->cls.response_data);
+		ceph_osdc_msg_data_set(req->r_request, &src->cls.request_info);
+		BUG_ON(src->cls.request_info.type !=
 					CEPH_OSD_DATA_TYPE_PAGELIST);
-		request_data_len = src->cls.request_info->pagelist->length;
+		request_data_len = src->cls.request_info.pagelist->length;
 		break;
 	case CEPH_OSD_OP_STARTSYNC:
 		break;
-- 
1.7.9.5

Re: [PATCH 20/20] libceph: kill off osd request r_data_in and r_data_out

[Josh Durgin]

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

On 04/05/2013 07:06 AM, Alex Elder wrote:
> Finally!  Convert the osd op data pointers into real structures, and
> make the switch over to using them instead of having all ops share
> the in and/or out data structures in the osd request.
>
> Set up a new function to traverse the set of ops and release any
> data associated with them (pages).
>
> This and the patches leading up to it resolve:
>      http://tracker.ceph.com/issues/4657
>
> Signed-off-by: Alex Elder <elder@inktank.com>
> ---
>   include/linux/ceph/osd_client.h |    9 ++---
>   net/ceph/osd_client.c           |   79
> +++++++++++++++++++--------------------
>   2 files changed, 42 insertions(+), 46 deletions(-)
>
> diff --git a/include/linux/ceph/osd_client.h
> b/include/linux/ceph/osd_client.h
> index dd4ca4ba..4ec46c0 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -87,14 +87,14 @@ struct ceph_osd_req_op {
>   			u64 offset, length;
>   			u64 truncate_size;
>   			u32 truncate_seq;
> -			struct ceph_osd_data *osd_data;
> +			struct ceph_osd_data osd_data;
>   		} extent;
>   		struct {
>   			const char *class_name;
>   			const char *method_name;
>   			const void *request_data;
> -			struct ceph_osd_data *request_info;
> -			struct ceph_osd_data *response_data;
> +			struct ceph_osd_data request_info;
> +			struct ceph_osd_data response_data;
>   			u32 request_data_len;
>   			__u8 class_len;
>   			__u8 method_len;
> @@ -164,9 +164,6 @@ struct ceph_osd_request {
>
>   	struct ceph_file_layout r_file_layout;
>   	struct ceph_snap_context *r_snapc;    /* snap context for writes */
> -
> -	struct ceph_osd_data r_data_in;
> -	struct ceph_osd_data r_data_out;
>   };
>
>   struct ceph_osd_event {
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index cc4003f..2562e4e 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -121,8 +121,7 @@ osd_req_op_extent_osd_data(struct ceph_osd_request
> *osd_req,
>   {
>   	BUG_ON(which >= osd_req->r_num_ops);
>
> -	/* return &osd_req->r_ops[which].extent.osd_data; */
> -	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
> +	return &osd_req->r_ops[which].extent.osd_data;
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_osd_data);
>
> @@ -132,8 +131,7 @@ osd_req_op_cls_request_info(struct ceph_osd_request
> *osd_req,
>   {
>   	BUG_ON(which >= osd_req->r_num_ops);
>
> -	/* return &osd_req->r_ops[which].cls.request_info; */
> -	return &osd_req->r_data_out;	/* Request data is outgoing */
> +	return &osd_req->r_ops[which].cls.request_info;
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */
>
> @@ -143,8 +141,7 @@ osd_req_op_cls_response_data(struct ceph_osd_request
> *osd_req,
>   {
>   	BUG_ON(which >= osd_req->r_num_ops);
>
> -	/* return &osd_req->r_ops[which].cls.response_data; */
> -	return &osd_req->r_data_in;	/* Response data is incoming */
> +	return &osd_req->r_ops[which].cls.response_data;
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */
>
> @@ -158,9 +155,6 @@ void osd_req_op_extent_osd_data_pages(struct
> ceph_osd_request *osd_req,
>   	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
>   	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
>   				pages_from_pool, own_pages);
> -
> -	osd_req->r_ops[which].extent.osd_data =
> -		osd_req_op_extent_osd_data(osd_req, which, write_request);
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
>
> @@ -172,9 +166,6 @@ void osd_req_op_extent_osd_data_pagelist(struct
> ceph_osd_request *osd_req,
>
>   	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
>   	ceph_osd_data_pagelist_init(osd_data, pagelist);
> -
> -	osd_req->r_ops[which].extent.osd_data =
> -		osd_req_op_extent_osd_data(osd_req, which, write_request);
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
>
> @@ -187,9 +178,6 @@ void osd_req_op_extent_osd_data_bio(struct
> ceph_osd_request *osd_req,
>
>   	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
>   	ceph_osd_data_bio_init(osd_data, bio, bio_length);
> -
> -	osd_req->r_ops[which].extent.osd_data =
> -		osd_req_op_extent_osd_data(osd_req, which, write_request);
>   }
>   EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
>   #endif /* CONFIG_BLOCK */
> @@ -202,9 +190,6 @@ static void osd_req_op_cls_request_info_pagelist(
>
>   	osd_data = osd_req_op_cls_request_info(osd_req, which);
>   	ceph_osd_data_pagelist_init(osd_data, pagelist);
> -
> -	osd_req->r_ops[which].cls.request_info =
> -		osd_req_op_cls_request_info(osd_req, which);
>   }
>
>   void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
> @@ -216,9 +201,6 @@ void osd_req_op_cls_response_data_pages(struct
> ceph_osd_request *osd_req,
>   	osd_data = osd_req_op_cls_response_data(osd_req, which);
>   	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
>   				pages_from_pool, own_pages);
> -
> -	osd_req->r_ops[which].cls.response_data =
> -		osd_req_op_cls_response_data(osd_req, which);
>   }
>   EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
>
> @@ -241,18 +223,39 @@ static u64 ceph_osd_data_length(struct
> ceph_osd_data *osd_data)
>   	}
>   }
>
> +
>   static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
>   {
> -	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
> -		return;
> -
> -	if (osd_data->own_pages) {
> +	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES && osd_data->own_pages) {
>   		int num_pages;
>
>   		num_pages = calc_pages_for((u64)osd_data->alignment,
>   						(u64)osd_data->length);
>   		ceph_release_page_vector(osd_data->pages, num_pages);
>   	}
> +	ceph_osd_data_init(osd_data);
> +}
> +
> +static void osd_req_op_data_release(struct ceph_osd_request *osd_req,
> +			unsigned int which)
> +{
> +	struct ceph_osd_req_op *op;
> +
> +	BUG_ON(which >= osd_req->r_num_ops);
> +	op = &osd_req->r_ops[which];
> +
> +	switch (op->op) {
> +	case CEPH_OSD_OP_READ:
> +	case CEPH_OSD_OP_WRITE:
> +		ceph_osd_data_release(&op->extent.osd_data);
> +		break;
> +	case CEPH_OSD_OP_CALL:
> +		ceph_osd_data_release(&op->cls.request_info);
> +		ceph_osd_data_release(&op->cls.response_data);
> +		break;
> +	default:
> +		break;
> +	}
>   }
>
>   /*
> @@ -261,6 +264,7 @@ static void ceph_osd_data_release(struct
> ceph_osd_data *osd_data)
>   void ceph_osdc_release_request(struct kref *kref)
>   {
>   	struct ceph_osd_request *req;
> +	unsigned int which;
>
>   	req = container_of(kref, struct ceph_osd_request, r_kref);
>   	if (req->r_request)
> @@ -270,8 +274,8 @@ void ceph_osdc_release_request(struct kref *kref)
>   		ceph_msg_put(req->r_reply);
>   	}
>
> -	ceph_osd_data_release(&req->r_data_in);
> -	ceph_osd_data_release(&req->r_data_out);
> +	for (which = 0; which < req->r_num_ops; which++)
> +		osd_req_op_data_release(req, which);
>
>   	ceph_put_snap_context(req->r_snapc);
>   	if (req->r_mempool)
> @@ -595,27 +599,22 @@ static u64 osd_req_encode_op(struct
> ceph_osd_request *req,
>   			cpu_to_le64(src->extent.truncate_size);
>   		dst->extent.truncate_seq =
>   			cpu_to_le32(src->extent.truncate_seq);
> -		if (src->op == CEPH_OSD_OP_WRITE) {
> -			WARN_ON(src->extent.osd_data != &req->r_data_out);
> +		if (src->op == CEPH_OSD_OP_WRITE)
>   			ceph_osdc_msg_data_set(req->r_request,
> -						src->extent.osd_data);
> -		} else {
> -			WARN_ON(src->extent.osd_data != &req->r_data_in);
> +						&src->extent.osd_data);
> +		else
>   			ceph_osdc_msg_data_set(req->r_reply,
> -						src->extent.osd_data);
> -		}
> +						&src->extent.osd_data);
>   		break;
>   	case CEPH_OSD_OP_CALL:
>   		dst->cls.class_len = src->cls.class_len;
>   		dst->cls.method_len = src->cls.method_len;
>   		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
> -		WARN_ON(src->cls.response_data != &req->r_data_in);
> -		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
> -		WARN_ON(src->cls.request_info != &req->r_data_out);
> -		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
> -		BUG_ON(src->cls.request_info->type !=
> +		ceph_osdc_msg_data_set(req->r_reply, &src->cls.response_data);
> +		ceph_osdc_msg_data_set(req->r_request, &src->cls.request_info);
> +		BUG_ON(src->cls.request_info.type !=
>   					CEPH_OSD_DATA_TYPE_PAGELIST);
> -		request_data_len = src->cls.request_info->pagelist->length;
> +		request_data_len = src->cls.request_info.pagelist->length;
>   		break;
>   	case CEPH_OSD_OP_STARTSYNC:
>   		break;
>