[RFD] [PATCHSETS 0/2 0/2 0/5] Remove of blk_rq_append_bio usage

[RFD] [PATCHSETS 0/2 0/2 0/5] Remove of blk_rq_append_bio usage

[Boaz Harrosh]

Hi all

In an effort to remove blk_rq_append_bio() and only use well published
block API, I would like to propose the below solution for OSD and exofs.

Current discussion has two sides:
* blk_make_request():
   exofs and other non-block-based filesystems can use a bio, in which
   case a well established block API should allow of allocating a request
   given a bio.

* blk_rq_map_pages():
   Let filesystems submit an array of page-pointers, and refrain from using
   bio(s). bio(s) is a privilege of block-based filesystems only.

The first solution is trivial, robust, safe with minimum changes and impact.
It will allow any future directions easily.

I have also implemented a second approach with the use of a new
blk_rq_map_pages. It works, all exofs tests pass successfully just
as with today's code. But I hate it because:
* Jeff Garzik is working on a block-over-osd-object block-device.
  I'm not sure of his plans, but sitting under block-layer at it's
  prepare_fn routine, the most trivial solution is to pass the bio
  as is to libosd. Otherwise he will need to copy and recopy all these
  pages information.
  Jeff please comment.
* Use of the in-kernel raid engines.
  Current dm-raid engines are heavily bio based. If we want to tap into-that
  (And we must) from exofs and pNFS-objects-layout we have no choice but to
  use bio(s) and be able to submit bios.
* For any none trivial usage, even today. bio is more stable, robust
  faster. and less code since it is already there.

Here is some code:
[PATCHSET 0/2] Allow append of kernel pointers to request
    [PATCH 1/4] allow blk_rq_map_kern to append to requests
    [PATCH 2/4] libosd: Use new blk_rq_map_kern

    These two patches are not disputable and safe and should go in as is
    even to 2.6.30. They remove one usage of blk_rq_append_bio from the
    osd_initiator.

[PROPOSAL_1 0/2] blk_make_request()
    [PATCH 3/4] [RFC] Export new blk_make_request() which takes bio and returns request
    [PATCH 4/4] libosd: Use of new blk_make_request

    This is option one above. Trivial, cosmetic, no core changes.

[PROPOSAL_2 0/5] blk_rq_map_pages()
    [RFC 1/5] blk_rq_map_pages() new API                                        
    [RFC 2/5] libosd: Rename osd_req_write/read to osd_req_write/read_old       
    [RFC 3/5] libosd: No bio for you
    [RFC 4/5] exofs: No bio for you
    [RFC 5/5] libosd: Remove deprecated bio based API

    This is option two above. Needs lots of core changes to exofs.
    After patch 5/5 code runs well, all tests pass. There is 
    runtime bisectability problem, the exofs will not run between
    [3/5]-[4/5].

Please advise
Boaz

[PATCH A 1/2] allow blk_rq_map_kern to append to requests

[Boaz Harrosh]

From: James Bottomley <James.Bottomley@HansenPartnership.com>

Use blk_rq_append_bio() internally instead of blk_rq_bio_prep()
so blk_rq_map_kern can be called multiple times, to map multiple
buffers.

This is in the effort to un-export blk_rq_append_bio()

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 block/blk-map.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/block/blk-map.c b/block/blk-map.c
index f103729..ada399e 100644
--- a/block/blk-map.c
+++ b/block/blk-map.c
@@ -282,7 +282,8 @@ EXPORT_SYMBOL(blk_rq_unmap_user);
  *
  * Description:
  *    Data will be mapped directly if possible. Otherwise a bounce
- *    buffer is used.
+ *    buffer is used. Can be called multiple times to append multiple
+ *    buffers.
  */
 int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf,
 		    unsigned int len, gfp_t gfp_mask)
@@ -290,6 +291,7 @@ int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf,
 	int reading = rq_data_dir(rq) == READ;
 	int do_copy = 0;
 	struct bio *bio;
+	int ret;
 
 	if (len > (q->max_hw_sectors << 9))
 		return -EINVAL;
@@ -311,7 +313,13 @@ int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf,
 	if (do_copy)
 		rq->cmd_flags |= REQ_COPY_USER;
 
-	blk_rq_bio_prep(q, rq, bio);
+	ret = blk_rq_append_bio(q, rq, bio);
+	if (unlikely(ret)) {
+		/* request is too big */
+		bio_put(bio);
+		return ret;
+	}
+
 	blk_queue_bounce(q, &rq->bio);
 	rq->buffer = rq->data = NULL;
 	return 0;
-- 
1.6.2.1

[PATCH A 2/2] libosd: Use new blk_rq_map_kern

[Boaz Harrosh]

Now that blk_rq_map_kern will append the buffer onto the
request we can use it easily for adding extra segments
(eg. attributes)

This patch is dependent on a block layer patch titled:
   [BLOCK] allow blk_rq_map_kern to append to requests

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 drivers/scsi/osd/osd_initiator.c |   24 ++----------------------
 1 files changed, 2 insertions(+), 22 deletions(-)

diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index 6c921ce..ccfd347 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -834,26 +834,6 @@ int osd_req_add_set_attr_list(struct osd_request *or,
 }
 EXPORT_SYMBOL(osd_req_add_set_attr_list);
 
-static int _append_map_kern(struct request *req,
-	void *buff, unsigned len, gfp_t flags)
-{
-	struct bio *bio;
-	int ret;
-
-	bio = bio_map_kern(req->q, buff, len, flags);
-	if (IS_ERR(bio)) {
-		OSD_ERR("Failed bio_map_kern(%p, %d) => %ld\n", buff, len,
-			PTR_ERR(bio));
-		return PTR_ERR(bio);
-	}
-	ret = blk_rq_append_bio(req->q, req, bio);
-	if (ret) {
-		OSD_ERR("Failed blk_rq_append_bio(%p) => %d\n", bio, ret);
-		bio_put(bio);
-	}
-	return ret;
-}
-
 static int _req_append_segment(struct osd_request *or,
 	unsigned padding, struct _osd_req_data_segment *seg,
 	struct _osd_req_data_segment *last_seg, struct _osd_io_info *io)
@@ -869,14 +849,14 @@ static int _req_append_segment(struct osd_request *or,
 		else
 			pad_buff = io->pad_buff;
 
-		ret = _append_map_kern(io->req, pad_buff, padding,
+		ret = blk_rq_map_kern(io->req->q, io->req, pad_buff, padding,
 				       or->alloc_flags);
 		if (ret)
 			return ret;
 		io->total_bytes += padding;
 	}
 
-	ret = _append_map_kern(io->req, seg->buff, seg->total_bytes,
+	ret = blk_rq_map_kern(io->req->q, io->req, seg->buff, seg->total_bytes,
 			       or->alloc_flags);
 	if (ret)
 		return ret;
-- 
1.6.2.1

[RFC B 1/2] Export new blk_make_request() which takes bio and returns request

[Boaz Harrosh]

given a struct bio allocates a new request. This is the parallel of
generic_make_request for BLOCK_PC commands users.

The new API is:
+struct request *blk_make_request(struct request_queue *q, struct bio *bio,
+				 gfp_t gfp_mask);

This is in the effort of un-exporting blk_rq_append_bio()

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 block/blk-core.c       |   13 +++++++++++++
 include/linux/blkdev.h |    5 ++++-
 2 files changed, 17 insertions(+), 1 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index 9ee243e..7136c11 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -903,6 +903,19 @@ struct request *blk_get_request(struct request_queue *q, int rw, gfp_t gfp_mask)
 }
 EXPORT_SYMBOL(blk_get_request);
 
+struct request *blk_make_request(struct request_queue *q, struct bio *bio,
+				 gfp_t gfp_mask)
+{
+	struct request *rq = blk_get_request(q, bio_data_dir(bio), gfp_mask);
+
+	if (rq) {
+		blk_rq_bio_prep(q, rq, bio);
+		blk_queue_bounce(q, &rq->bio);
+	}
+	return rq;
+}
+EXPORT_SYMBOL(blk_make_request);
+
 /**
  * blk_start_queueing - initiate dispatch of requests to device
  * @q:		request queue to kick into gear
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 465d6ba..22dcdde 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -729,6 +729,8 @@ extern void blk_rq_init(struct request_queue *q, struct request *rq);
 extern void blk_put_request(struct request *);
 extern void __blk_put_request(struct request_queue *, struct request *);
 extern struct request *blk_get_request(struct request_queue *, int, gfp_t);
+extern struct request *blk_make_request(struct request_queue *, struct bio *,
+					gfp_t);
 extern void blk_insert_request(struct request_queue *, struct request *, int, void *);
 extern void blk_requeue_request(struct request_queue *, struct request *);
 extern int blk_rq_check_limits(struct request_queue *q, struct request *rq);

[RFC B 2/2] libosd: Use of new blk_make_request

[Boaz Harrosh]

Use new blk_make_request() to allocate a request from bio
and avoid using deprecated blk_rq_append_bio().

This patch is dependent on a block layer patch titled:
    [BLOCK] Export new blk_make_request() which takes bio \
            and returns request

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 drivers/scsi/osd/osd_initiator.c |   33 +++++++++++----------------------
 1 files changed, 11 insertions(+), 22 deletions(-)

diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index ccfd347..66c3e92 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -1208,6 +1208,14 @@ static int _osd_req_finalize_data_integrity(struct osd_request *or,
 /*
  * osd_finalize_request and helpers
  */
+struct request *_make_request(struct request_queue *q, bool has_write,
+			      struct bio *bio, gfp_t flags)
+{
+	if (bio)
+		return blk_make_request(q, bio, flags);
+	else
+		return blk_get_request(q, has_write ? WRITE : READ, flags);
+}
 
 static int _init_blk_request(struct osd_request *or,
 	bool has_in, bool has_out)
@@ -1218,7 +1226,8 @@ static int _init_blk_request(struct osd_request *or,
 	struct request *req;
 	int ret = -ENOMEM;
 
-	req = blk_get_request(q, has_out, flags);
+	req = _make_request(q, has_out, has_out ? or->out.bio : or->in.bio,
+			    flags);
 	if (!req)
 		goto out;
 
@@ -1233,7 +1242,7 @@ static int _init_blk_request(struct osd_request *or,
 		or->out.req = req;
 		if (has_in) {
 			/* allocate bidi request */
-			req = blk_get_request(q, READ, flags);
+			req = _make_request(q, false, or->in.bio, flags);
 			if (!req) {
 				OSD_DEBUG("blk_get_request for bidi failed\n");
 				goto out;
@@ -1279,26 +1288,6 @@ int osd_finalize_request(struct osd_request *or,
 		return ret;
 	}
 
-	if (or->out.bio) {
-		ret = blk_rq_append_bio(or->request->q, or->out.req,
-					or->out.bio);
-		if (ret) {
-			OSD_DEBUG("blk_rq_append_bio out failed\n");
-			return ret;
-		}
-		OSD_DEBUG("out bytes=%llu (bytes_req=%u)\n",
-			_LLU(or->out.total_bytes), or->out.req->data_len);
-	}
-	if (or->in.bio) {
-		ret = blk_rq_append_bio(or->request->q, or->in.req, or->in.bio);
-		if (ret) {
-			OSD_DEBUG("blk_rq_append_bio in failed\n");
-			return ret;
-		}
-		OSD_DEBUG("in bytes=%llu (bytes_req=%u)\n",
-			_LLU(or->in.total_bytes), or->in.req->data_len);
-	}
-
 	or->out.pad_buff = sg_out_pad_buffer;
 	or->in.pad_buff = sg_in_pad_buffer;
 
-- 
1.6.2.1

[RFC C 1/5] blk_rq_map_pages() new API

[Boaz Harrosh]

Define a new API for mapping array of pages into a request.

Internally the bio is created using the
rq_map_data + null_mapped + NULL_buffer mode of
bio_copy_user(). The bio is only referenced once
and there is no unmap function to call.

TODO:
   out of the 7 users of blk_rq_map_user + rq_map_data, 6 of them
   are null_mapped + NULL_buffer and could be converted to this
   API.

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 block/blk-map.c        |   54 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/blkdev.h |    3 ++
 2 files changed, 57 insertions(+), 0 deletions(-)

diff --git a/block/blk-map.c b/block/blk-map.c
index ada399e..60a7a0b 100644
--- a/block/blk-map.c
+++ b/block/blk-map.c
@@ -325,3 +325,57 @@ int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf,
 	return 0;
 }
 EXPORT_SYMBOL(blk_rq_map_kern);
+
+/**
+ * blk_rq_map_pages - map pages array to a request, for REQ_TYPE_BLOCK_PC usage
+ * @q:		request queue where request should be inserted
+ * @rq:		request to fill
+ * @pages:	an array of struct page pointers to map
+ * @nr_pages:	count of valid pages at @pages
+ * @offset:	Offset into first page to start from (Must be blk_rq_aligned())
+ * @len:	length of user data to map
+ * @gfp_mask:	memory allocation flags
+ *
+ * Description:
+ *    Data will be mapped directly if possible. Otherwise a bounce
+ *    buffer is used. Can be called multiple times to append multiple
+ *    page arrays. No need to call an unmap function, and bio is only
+ *    referenced once. (Pages are of zero order)
+ */
+int blk_rq_map_pages(struct request_queue *q, struct request *rq,
+		    struct page **pages, unsigned nr_pages,
+		    unsigned long offset, unsigned len, gfp_t gfp_mask)
+{
+	struct bio *bio;
+	int reading, ret;
+	struct rq_map_data map_data = {
+		.pages = pages,
+		.nr_entries = nr_pages,
+		.offset = offset,
+		.null_mapped = 1,
+		.page_order = 0,
+	};
+
+	reading = rq_data_dir(rq) == READ;
+
+	/* The name is confusing, but bio_copy_user+map_data+NULL_buffer
+         * Actually just adds the pages to the bio, no copy is performed.
+	 */
+	bio = bio_copy_user(q, &map_data, 0, len, reading, gfp_mask);
+
+	if (IS_ERR(bio))
+		return PTR_ERR(bio);
+
+	bio->bi_flags |= (1 << BIO_NULL_MAPPED);
+
+	/* We might still need to bounce if memory mask does not match */
+	blk_queue_bounce(q, &bio);
+
+	ret = blk_rq_append_bio(q, rq, bio);
+	if (likely(!ret))
+		return 0;
+
+	bio_put(bio);
+	return ret;
+}
+EXPORT_SYMBOL(blk_rq_map_pages);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 22dcdde..81b681c 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -784,6 +784,9 @@ extern int blk_rq_map_user(struct request_queue *, struct request *,
 extern int blk_rq_unmap_user(struct bio *);
 extern int blk_rq_map_kern(struct request_queue *, struct request *, void *,
 			   unsigned int, gfp_t);
+extern int blk_rq_map_pages(struct request_queue *, struct request *,
+			    struct page **, unsigned, unsigned long offset,
+			    unsigned , gfp_t);
 extern int blk_rq_map_user_iov(struct request_queue *, struct request *,
 			       struct rq_map_data *, struct sg_iovec *, int,
 			       unsigned int, gfp_t);
-- 
1.6.2.1

[RFC C 2/5] libosd: Rename osd_req_write/read to osd_req_write/read_old

[Boaz Harrosh]

In an attempt to not use bio(s) at the initiator, we want
to convert to something else (array of page pointers).
This patch renames the current bio API to xxx_old.

The next patch will implement the new API.
After all Kernel users (exofs) will convert to the new API
The old can be removed.

The API is marked __deprecated and will spew warnings until
conversion is done.

Note that this patch patches all users (exofs)

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 drivers/scsi/osd/osd_initiator.c |    4 ++++
 fs/exofs/common.h                |    4 ++++
 include/scsi/osd_initiator.h     |    4 ++--
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index ccfd347..8885524 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -52,6 +52,10 @@
 #    define __unused			__attribute__((unused))
 #endif
 
+/* FIXME: Temporarly until next patch */
+#define osd_req_write osd_req_write_old
+#define osd_req_read  osd_req_read_old
+
 enum { OSD_REQ_RETRIES = 1 };
 
 MODULE_AUTHOR("Boaz Harrosh <bharrosh@panasas.com>");
diff --git a/fs/exofs/common.h b/fs/exofs/common.h
index 8a56338..f02cb4c 100644
--- a/fs/exofs/common.h
+++ b/fs/exofs/common.h
@@ -44,6 +44,10 @@
 #include <scsi/osd_initiator.h>
 #include <scsi/osd_sec.h>
 
+/* FIXME: Temporarly until exofs tree syncs with scsi-misc's osd_initiator */
+#define osd_req_write osd_req_write_old
+#define osd_req_read  osd_req_read_old
+
 /****************************************************************************
  * Object ID related defines
  * NOTE: inode# = object ID - EXOFS_OBJ_OFF
diff --git a/include/scsi/osd_initiator.h b/include/scsi/osd_initiator.h
index 6132790..7a317aa 100644
--- a/include/scsi/osd_initiator.h
+++ b/include/scsi/osd_initiator.h
@@ -362,7 +362,7 @@ void osd_req_set_member_attrs(struct osd_request *or, ...);/* V2-only NI */
 void osd_req_create_object(struct osd_request *or, struct osd_obj_id *);
 void osd_req_remove_object(struct osd_request *or, struct osd_obj_id *);
 
-void osd_req_write(struct osd_request *or,
+void __deprecated osd_req_write_old(struct osd_request *or,
 	const struct osd_obj_id *, struct bio *data_out, u64 offset);
 int osd_req_write_kern(struct osd_request *or,
 	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len);
@@ -379,7 +379,7 @@ void osd_req_flush_object(struct osd_request *or,
 	const struct osd_obj_id *, enum osd_options_flush_scope_values,
 	/*V2*/ u64 offset, /*V2*/ u64 len);
 
-void osd_req_read(struct osd_request *or,
+void __deprecated osd_req_read_old(struct osd_request *or,
 	const struct osd_obj_id *, struct bio *data_in, u64 offset);
 int osd_req_read_kern(struct osd_request *or,
 	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len);
-- 
1.6.2.1

[RFC C 3/5] libosd: No bio for you

[Boaz Harrosh]

Change osd_initiator API to not be bio based.
The new API will receive an array of pages to
write/read from. The pages info is held in a new
struct osd_page_array.

At osd_finalize_request the code uses the new
blk_rq_map_pages() API to build the request for
submission.

this patch is dependent on a block layer patch titled:
   [BLOCK] blk_rq_map_pages() new API

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 drivers/scsi/osd/osd_initiator.c |  109 +++++++++++++++++++++-----------------
 include/scsi/osd_initiator.h     |   31 +++++++++--
 2 files changed, 88 insertions(+), 52 deletions(-)

diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index 8885524..d18402c 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -52,10 +52,6 @@
 #    define __unused			__attribute__((unused))
 #endif
 
-/* FIXME: Temporarly until next patch */
-#define osd_req_write osd_req_write_old
-#define osd_req_read  osd_req_read_old
-
 enum { OSD_REQ_RETRIES = 1 };
 
 MODULE_AUTHOR("Boaz Harrosh <bharrosh@panasas.com>");
@@ -585,25 +581,15 @@ static int _osd_req_list_objects(struct osd_request *or,
 	__be16 action, const struct osd_obj_id *obj, osd_id initial_id,
 	struct osd_obj_id_list *list, unsigned nelem)
 {
-	struct request_queue *q = or->osd_dev->scsi_device->request_queue;
 	u64 len = nelem * sizeof(osd_id) + sizeof(*list);
-	struct bio *bio;
 
 	_osd_req_encode_common(or, action, obj, (u64)initial_id, len);
 
 	if (list->list_identifier)
 		_osd_req_encode_olist(or, list);
 
-	WARN_ON(or->in.bio);
-	bio = bio_map_kern(q, list, len, or->alloc_flags);
-	if (!bio) {
-		OSD_ERR("!!! Failed to allocate list_objects BIO\n");
-		return -ENOMEM;
-	}
-
-	bio->bi_rw &= ~(1 << BIO_RW);
-	or->in.bio = bio;
-	or->in.total_bytes = bio->bi_size;
+	opa_map_kern(&or->in.opa, list, len);
+	or->in.total_bytes = len;
 	return 0;
 }
 
@@ -693,27 +679,35 @@ EXPORT_SYMBOL(osd_req_remove_object);
 	struct osd_obj_id *first, struct osd_obj_id_list *list, unsigned nelem);
 */
 
-void osd_req_write(struct osd_request *or,
+void osd_req_write_old(struct osd_request *or,
 	const struct osd_obj_id *obj, struct bio *bio, u64 offset)
 {
-	_osd_req_encode_common(or, OSD_ACT_WRITE, obj, offset, bio->bi_size);
-	WARN_ON(or->out.bio || or->out.total_bytes);
-	bio->bi_rw |= (1 << BIO_RW);
-	or->out.bio = bio;
-	or->out.total_bytes = bio->bi_size;
+	BUG_ON(1);
+}
+EXPORT_SYMBOL(osd_req_write_old);
+
+void osd_req_write(struct osd_request *or,
+	const struct osd_obj_id *obj, const struct osd_pages_array *opa,
+	u64 offset)
+{
+	_osd_req_encode_common(or, OSD_ACT_WRITE, obj, offset, opa->length);
+
+	WARN_ON(or->out.total_bytes);
+	WARN_ON(or->out.opa.length);
+	or->out.opa = *opa;
+	or->out.total_bytes = opa->length;
 }
 EXPORT_SYMBOL(osd_req_write);
 
 int osd_req_write_kern(struct osd_request *or,
-	const struct osd_obj_id *obj, u64 offset, void* buff, u64 len)
+	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len)
 {
-	struct request_queue *req_q = or->osd_dev->scsi_device->request_queue;
-	struct bio *bio = bio_map_kern(req_q, buff, len, GFP_KERNEL);
+	struct osd_pages_array opa;
 
-	if (!bio)
-		return -ENOMEM;
+	memset(&opa, 0, sizeof(opa));
+	opa_map_kern(&opa, buff, len);
 
-	osd_req_write(or, obj, bio, offset);
+	osd_req_write(or, obj, &opa, offset);
 	return 0;
 }
 EXPORT_SYMBOL(osd_req_write_kern);
@@ -742,27 +736,35 @@ void osd_req_flush_object(struct osd_request *or,
 }
 EXPORT_SYMBOL(osd_req_flush_object);
 
-void osd_req_read(struct osd_request *or,
+void osd_req_read_old(struct osd_request *or,
 	const struct osd_obj_id *obj, struct bio *bio, u64 offset)
 {
-	_osd_req_encode_common(or, OSD_ACT_READ, obj, offset, bio->bi_size);
-	WARN_ON(or->in.bio || or->in.total_bytes);
-	bio->bi_rw &= ~(1 << BIO_RW);
-	or->in.bio = bio;
-	or->in.total_bytes = bio->bi_size;
+	BUG_ON(1);
+}
+EXPORT_SYMBOL(osd_req_read_old);
+
+void osd_req_read(struct osd_request *or,
+	const struct osd_obj_id *obj, const struct osd_pages_array *opa,
+	u64 offset)
+{
+	_osd_req_encode_common(or, OSD_ACT_READ, obj, offset, opa->length);
+
+	WARN_ON(or->in.total_bytes);
+	WARN_ON(or->in.opa.length);
+	or->in.opa = *opa;
+	or->in.total_bytes = opa->length;
 }
 EXPORT_SYMBOL(osd_req_read);
 
 int osd_req_read_kern(struct osd_request *or,
-	const struct osd_obj_id *obj, u64 offset, void* buff, u64 len)
+	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len)
 {
-	struct request_queue *req_q = or->osd_dev->scsi_device->request_queue;
-	struct bio *bio = bio_map_kern(req_q, buff, len, GFP_KERNEL);
+	struct osd_pages_array opa;
 
-	if (!bio)
-		return -ENOMEM;
+	memset(&opa, 0, sizeof(opa));
+	opa_map_kern(&opa, buff, len);
 
-	osd_req_read(or, obj, bio, offset);
+	osd_req_read(or, obj, &opa, offset);
 	return 0;
 }
 EXPORT_SYMBOL(osd_req_read_kern);
@@ -1168,7 +1170,7 @@ static int _osd_req_finalize_data_integrity(struct osd_request *or,
 		unsigned pad;
 
 		or->out_data_integ.data_bytes = cpu_to_be64(
-			or->out.bio ? or->out.bio->bi_size : 0);
+			or->out.opa.length ?: 0);
 		or->out_data_integ.set_attributes_bytes = cpu_to_be64(
 			or->set_attr.total_bytes);
 		or->out_data_integ.get_attributes_bytes = cpu_to_be64(
@@ -1255,6 +1257,18 @@ out:
 	return ret;
 }
 
+static int _rq_map_opa(struct osd_request *or, struct _osd_io_info *io)
+{
+	if (io->opa.pages)
+		return blk_rq_map_pages(io->req->q, io->req, io->opa.pages,
+					io->opa.nr_pages, io->opa.offset,
+					io->opa.length, or->alloc_flags);
+	else
+		return blk_rq_map_kern(io->req->q, io->req,
+				       (void *)io->opa.offset, io->opa.length,
+				       or->alloc_flags);
+}
+
 int osd_finalize_request(struct osd_request *or,
 	u8 options, const void *cap, const u8 *cap_key)
 {
@@ -1273,8 +1287,8 @@ int osd_finalize_request(struct osd_request *or,
 
 	osd_set_caps(&or->cdb, cap);
 
-	has_in = or->in.bio || or->get_attr.total_bytes;
-	has_out = or->out.bio || or->set_attr.total_bytes ||
+	has_in = or->in.opa.length || or->get_attr.total_bytes;
+	has_out = or->out.opa.length || or->set_attr.total_bytes ||
 		or->enc_get_attr.total_bytes;
 
 	ret = _init_blk_request(or, has_in, has_out);
@@ -1283,9 +1297,8 @@ int osd_finalize_request(struct osd_request *or,
 		return ret;
 	}
 
-	if (or->out.bio) {
-		ret = blk_rq_append_bio(or->request->q, or->out.req,
-					or->out.bio);
+	if (or->out.opa.length) {
+		ret = _rq_map_opa(or, &or->out);
 		if (ret) {
 			OSD_DEBUG("blk_rq_append_bio out failed\n");
 			return ret;
@@ -1293,8 +1306,8 @@ int osd_finalize_request(struct osd_request *or,
 		OSD_DEBUG("out bytes=%llu (bytes_req=%u)\n",
 			_LLU(or->out.total_bytes), or->out.req->data_len);
 	}
-	if (or->in.bio) {
-		ret = blk_rq_append_bio(or->request->q, or->in.req, or->in.bio);
+	if (or->in.opa.length) {
+		ret = _rq_map_opa(or, &or->in);
 		if (ret) {
 			OSD_DEBUG("blk_rq_append_bio in failed\n");
 			return ret;
diff --git a/include/scsi/osd_initiator.h b/include/scsi/osd_initiator.h
index 7a317aa..fa32074 100644
--- a/include/scsi/osd_initiator.h
+++ b/include/scsi/osd_initiator.h
@@ -78,6 +78,21 @@ static inline void osd_dev_set_ver(struct osd_dev *od, enum osd_std_version v)
 #endif
 }
 
+struct osd_pages_array {
+	struct page **pages;
+	unsigned nr_pages;
+	unsigned long offset;
+	unsigned long length;
+};
+
+static inline void opa_map_kern(struct osd_pages_array *opa, void *buff,
+				unsigned long length)
+{
+	WARN_ON(opa->pages || opa->offset);
+	opa->offset = (unsigned long)buff;
+	opa->length = length;
+}
+
 struct osd_request;
 typedef void (osd_req_done_fn)(struct osd_request *or, void *private);
 
@@ -96,7 +111,7 @@ struct osd_request {
 	} set_attr, enc_get_attr, get_attr;
 
 	struct _osd_io_info {
-		struct bio *bio;
+		struct osd_pages_array opa;
 		u64 total_bytes;
 		struct request *req;
 		struct _osd_req_data_segment *last_seg;
@@ -364,12 +379,16 @@ void osd_req_remove_object(struct osd_request *or, struct osd_obj_id *);
 
 void __deprecated osd_req_write_old(struct osd_request *or,
 	const struct osd_obj_id *, struct bio *data_out, u64 offset);
+
+void osd_req_write(struct osd_request *or,
+	const struct osd_obj_id *, const struct osd_pages_array *opa,
+	u64 offset);
 int osd_req_write_kern(struct osd_request *or,
 	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len);
 void osd_req_append(struct osd_request *or,
-	const struct osd_obj_id *, struct bio *data_out);/* NI */
-void osd_req_create_write(struct osd_request *or,
-	const struct osd_obj_id *, struct bio *data_out, u64 offset);/* NI */
+	const struct osd_obj_id *, struct osd_pages_array *opa_out);/* NI */
+void osd_req_create_write(struct osd_request *or, /* NI */
+	const struct osd_obj_id *, struct osd_pages_array *opa_out, u64 offset);
 void osd_req_clear(struct osd_request *or,
 	const struct osd_obj_id *, u64 offset, u64 len);/* NI */
 void osd_req_punch(struct osd_request *or,
@@ -381,6 +400,10 @@ void osd_req_flush_object(struct osd_request *or,
 
 void __deprecated osd_req_read_old(struct osd_request *or,
 	const struct osd_obj_id *, struct bio *data_in, u64 offset);
+
+void osd_req_read(struct osd_request *or,
+	const struct osd_obj_id *, const struct osd_pages_array *opa_in,
+	u64 offset);
 int osd_req_read_kern(struct osd_request *or,
 	const struct osd_obj_id *obj, u64 offset, void *buff, u64 len);
 
-- 
1.6.2.1

[RFC C 4/5] exofs: No bio for you

[Boaz Harrosh]

bio based API to osd_initiator is depreciated. Move to
a page array API when writing/reading pages.

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 fs/exofs/common.h |    4 --
 fs/exofs/inode.c  |  108 +++++++++++++++++++++++++++--------------------------
 2 files changed, 55 insertions(+), 57 deletions(-)

diff --git a/fs/exofs/common.h b/fs/exofs/common.h
index f02cb4c..8a56338 100644
--- a/fs/exofs/common.h
+++ b/fs/exofs/common.h
@@ -44,10 +44,6 @@
 #include <scsi/osd_initiator.h>
 #include <scsi/osd_sec.h>
 
-/* FIXME: Temporarly until exofs tree syncs with scsi-misc's osd_initiator */
-#define osd_req_write osd_req_write_old
-#define osd_req_read  osd_req_read_old
-
 /****************************************************************************
  * Object ID related defines
  * NOTE: inode# = object ID - EXOFS_OBJ_OFF
diff --git a/fs/exofs/inode.c b/fs/exofs/inode.c
index 0f52e76..31cfd6d 100644
--- a/fs/exofs/inode.c
+++ b/fs/exofs/inode.c
@@ -49,9 +49,8 @@ struct page_collect {
 	struct inode *inode;
 	unsigned expected_pages;
 
-	struct bio *bio;
-	unsigned nr_pages;
-	unsigned long length;
+	struct osd_pages_array opa;
+	unsigned pg_alloc_count;
 	long pg_first;
 };
 
@@ -66,9 +65,8 @@ void _pcol_init(struct page_collect *pcol, unsigned expected_pages,
 	pcol->inode = inode;
 	pcol->expected_pages = expected_pages;
 
-	pcol->bio = NULL;
-	pcol->nr_pages = 0;
-	pcol->length = 0;
+	memset(&pcol->opa, 0, sizeof(pcol->opa));
+	pcol->pg_alloc_count = 0;
 	pcol->pg_first = -1;
 
 	EXOFS_DBGMSG("_pcol_init ino=0x%lx expected_pages=%u\n", inode->i_ino,
@@ -77,11 +75,10 @@ void _pcol_init(struct page_collect *pcol, unsigned expected_pages,
 
 void _pcol_reset(struct page_collect *pcol)
 {
-	pcol->expected_pages -= min(pcol->nr_pages, pcol->expected_pages);
+	pcol->expected_pages -= min(pcol->opa.nr_pages, pcol->expected_pages);
 
-	pcol->bio = NULL;
-	pcol->nr_pages = 0;
-	pcol->length = 0;
+	memset(&pcol->opa, 0, sizeof(pcol->opa));
+	pcol->pg_alloc_count = 0;
 	pcol->pg_first = -1;
 	EXOFS_DBGMSG("_pcol_reset ino=0x%lx expected_pages=%u\n",
 		     pcol->inode->i_ino, pcol->expected_pages);
@@ -98,9 +95,12 @@ int pcol_try_alloc(struct page_collect *pcol)
 	int pages = min_t(unsigned, pcol->expected_pages, BIO_MAX_PAGES);
 
 	for (; pages; pages >>= 1) {
-		pcol->bio = bio_alloc(GFP_KERNEL, pages);
-		if (likely(pcol->bio))
+		pcol->opa.pages = kcalloc(pages, sizeof(struct page *),
+					  GFP_KERNEL);
+		if (likely(pcol->opa.pages)) {
+			pcol->pg_alloc_count = pages;
 			return 0;
+		}
 	}
 
 	EXOFS_ERR("Failed to kcalloc expected_pages=%d\n",
@@ -110,21 +110,25 @@ int pcol_try_alloc(struct page_collect *pcol)
 
 void pcol_free(struct page_collect *pcol)
 {
-	bio_put(pcol->bio);
-	pcol->bio = NULL;
+	kfree(pcol->opa.pages);
+	pcol->opa.pages = NULL;
+	pcol->pg_alloc_count = 0;
 }
 
-int pcol_add_page(struct page_collect *pcol, struct page *page, unsigned len)
+int pcol_add_page(struct page_collect *pcol, struct page *page, int len)
 {
-	int added_len = bio_add_pc_page(pcol->req_q, pcol->bio, page, len, 0);
-	if (unlikely(len != added_len))
+	if (pcol->opa.nr_pages >= pcol->pg_alloc_count)
 		return -ENOMEM;
 
-	++pcol->nr_pages;
-	pcol->length += len;
+	pcol->opa.pages[pcol->opa.nr_pages++] = page;
+	pcol->opa.length += len;
 	return 0;
 }
 
+#define __pcol_for_each_segment(page, pcol, i, s) \
+	for (page = pcol->opa.pages[s], i = s; i < pcol->opa.nr_pages; \
+		page = pcol->opa.pages[++(i)])
+
 static int update_read_page(struct page *page, int ret)
 {
 	if (ret == 0) {
@@ -164,7 +168,7 @@ static int _readpage(struct page *page, bool is_sync);
 static int __readpages_done(struct osd_request *or, struct page_collect *pcol,
 			    bool do_unlock)
 {
-	struct bio_vec *bvec;
+	struct page *page;
 	int i;
 	u64 resid;
 	u64 good_bytes;
@@ -174,19 +178,18 @@ static int __readpages_done(struct osd_request *or, struct page_collect *pcol,
 	osd_end_request(or);
 
 	if (!ret)
-		good_bytes = pcol->length;
+		good_bytes = pcol->opa.length;
 	else if (ret && !resid)
 		good_bytes = 0;
 	else
-		good_bytes = pcol->length - resid;
+		good_bytes = pcol->opa.length - resid;
 
 	EXOFS_DBGMSG("readpages_done(%ld) good_bytes=%llx"
 		     " length=%zx nr_pages=%u\n",
-		     pcol->inode->i_ino, _LLU(good_bytes), pcol->length,
-		     pcol->nr_pages);
+		     pcol->inode->i_ino, _LLU(good_bytes), pcol->opa.length,
+		     pcol->opa.nr_pages);
 
-	__bio_for_each_segment(bvec, pcol->bio, i, 0) {
-		struct page *page = bvec->bv_page;
+	__pcol_for_each_segment(page, pcol, i, 0) {
 		struct inode *inode = page->mapping->host;
 
 		if (inode != pcol->inode)
@@ -210,7 +213,8 @@ static int __readpages_done(struct osd_request *or, struct page_collect *pcol,
 			_readpage(page, false);
 		}
 
-		length += bvec->bv_len;
+		length += min_t(u64,
+				pcol->opa.length - length, PAGE_CACHE_SIZE);
 	}
 
 	pcol_free(pcol);
@@ -229,12 +233,10 @@ static void readpages_done(struct osd_request *or, void *p)
 
 void _unlock_pcol_pages(struct page_collect *pcol, int ret, int rw)
 {
-	struct bio_vec *bvec;
+	struct page *page;
 	int i;
 
-	__bio_for_each_segment(bvec, pcol->bio, i, 0) {
-		struct page *page = bvec->bv_page;
-
+	__pcol_for_each_segment(page, pcol, i, 0) {
 		if (rw == READ)
 			update_read_page(page, ret);
 		else
@@ -255,11 +257,11 @@ int read_exec(struct page_collect *pcol, bool is_sync)
 	loff_t i_start = pcol->pg_first << PAGE_CACHE_SHIFT;
 	int ret;
 
-	if (!pcol->bio)
+	if (!pcol->opa.pages)
 		return 0;
 
 	/* see comment in _readpage() about sync reads */
-	WARN_ON(is_sync && (pcol->nr_pages != 1));
+	WARN_ON(is_sync && (pcol->opa.nr_pages != 1));
 
 	or = osd_start_request(pcol->sbi->s_dev, GFP_KERNEL);
 	if (unlikely(!or)) {
@@ -267,7 +269,7 @@ int read_exec(struct page_collect *pcol, bool is_sync)
 		goto err;
 	}
 
-	osd_req_read(or, &obj, pcol->bio, i_start);
+	osd_req_read(or, &obj, &pcol->opa, i_start);
 
 	if (is_sync) {
 		exofs_sync_op(or, pcol->sbi->s_timeout, oi->i_cred);
@@ -288,7 +290,7 @@ int read_exec(struct page_collect *pcol, bool is_sync)
 	atomic_inc(&pcol->sbi->s_curr_pending);
 
 	EXOFS_DBGMSG("read_exec obj=%llx start=%llx length=%zx\n",
-		  obj.id, _LLU(i_start), pcol->length);
+		  obj.id, _LLU(i_start), pcol->opa.length);
 
 	/* pages ownership was passed to pcol_copy */
 	_pcol_reset(pcol);
@@ -346,7 +348,7 @@ try_again:
 
 	if (unlikely(pcol->pg_first == -1)) {
 		pcol->pg_first = page->index;
-	} else if (unlikely((pcol->pg_first + pcol->nr_pages) !=
+	} else if (unlikely((pcol->pg_first + pcol->opa.nr_pages) !=
 		   page->index)) {
 		/* Discontinuity detected, split the request */
 		ret = read_exec(pcol, false);
@@ -355,7 +357,7 @@ try_again:
 		goto try_again;
 	}
 
-	if (!pcol->bio) {
+	if (!pcol->opa.pages) {
 		ret = pcol_try_alloc(pcol);
 		if (unlikely(ret))
 			goto fail;
@@ -371,7 +373,7 @@ try_again:
 	if (ret) {
 		EXOFS_DBGMSG("Failed pcol_add_page pages[i]=%p "
 			  "len=%zx nr_pages=%u length=%zx\n",
-			  page, len, pcol->nr_pages, pcol->length);
+			  page, len, pcol->opa.nr_pages, pcol->opa.length);
 
 		/* split the request, and start again with current page */
 		ret = read_exec(pcol, false);
@@ -439,31 +441,30 @@ static int exofs_writepage(struct page *page, struct writeback_control *wbc2);
 static void writepages_done(struct osd_request *or, void *p)
 {
 	struct page_collect *pcol = p;
-	struct bio_vec *bvec;
+	struct page *page;
 	int i;
 	u64 resid;
 	u64  good_bytes;
 	u64  length = 0;
-
+	
 	int ret = exofs_check_ok_resid(or, NULL, &resid);
 
 	osd_end_request(or);
 	atomic_dec(&pcol->sbi->s_curr_pending);
 
 	if (likely(!ret))
-		good_bytes = pcol->length;
+		good_bytes = pcol->opa.length;
 	else if (ret && !resid)
 		good_bytes = 0;
 	else
-		good_bytes = pcol->length - resid;
+		good_bytes = pcol->opa.length - resid;
 
 	EXOFS_DBGMSG("writepages_done(%lx) good_bytes=%llx"
 		     " length=%zx nr_pages=%u\n",
-		     pcol->inode->i_ino, _LLU(good_bytes), pcol->length,
-		     pcol->nr_pages);
+		     pcol->inode->i_ino, _LLU(good_bytes), pcol->opa.length,
+		     pcol->opa.nr_pages);
 
-	__bio_for_each_segment(bvec, pcol->bio, i, 0) {
-		struct page *page = bvec->bv_page;
+	__pcol_for_each_segment(page, pcol, i, 0) {
 		struct inode *inode = page->mapping->host;
 
 		if (inode != pcol->inode)
@@ -485,7 +486,8 @@ static void writepages_done(struct osd_request *or, void *p)
 			exofs_writepage(page, NULL);
 		}
 
-		length += bvec->bv_len;
+		length += min_t(u64,
+				pcol->opa.length - length, PAGE_CACHE_SIZE);
 	}
 
 	pcol_free(pcol);
@@ -503,7 +505,7 @@ int write_exec(struct page_collect *pcol)
 	loff_t i_start = pcol->pg_first << PAGE_CACHE_SHIFT;
 	int ret;
 
-	if (!pcol->bio)
+	if (!pcol->opa.pages)
 		return 0;
 
 	or = osd_start_request(pcol->sbi->s_dev, GFP_KERNEL);
@@ -522,7 +524,7 @@ int write_exec(struct page_collect *pcol)
 
 	*pcol_copy = *pcol;
 
-	osd_req_write(or, &obj, pcol_copy->bio, i_start);
+	osd_req_write(or, &obj, &pcol_copy->opa, i_start);
 	ret = exofs_async_op(or, writepages_done, pcol_copy, oi->i_cred);
 	if (unlikely(ret)) {
 		EXOFS_ERR("write_exec: exofs_async_op() Faild\n");
@@ -532,7 +534,7 @@ int write_exec(struct page_collect *pcol)
 	atomic_inc(&pcol->sbi->s_curr_pending);
 	EXOFS_DBGMSG("write_exec(%lx, %lx) start=%llx length=%zx\n",
 		  pcol->inode->i_ino, pcol->pg_first, _LLU(i_start),
-		  pcol->length);
+		  pcol->opa.length);
 	/* pages ownership was passed to pcol_copy */
 	_pcol_reset(pcol);
 	return 0;
@@ -586,7 +588,7 @@ try_again:
 
 	if (unlikely(pcol->pg_first == -1)) {
 		pcol->pg_first = page->index;
-	} else if (unlikely((pcol->pg_first + pcol->nr_pages) !=
+	} else if (unlikely((pcol->pg_first + pcol->opa.nr_pages) !=
 		   page->index)) {
 		/* Discontinuity detected, split the request */
 		ret = write_exec(pcol);
@@ -595,7 +597,7 @@ try_again:
 		goto try_again;
 	}
 
-	if (!pcol->bio) {
+	if (!pcol->opa.pages) {
 		ret = pcol_try_alloc(pcol);
 		if (unlikely(ret))
 			goto fail;
@@ -608,7 +610,7 @@ try_again:
 	if (unlikely(ret)) {
 		EXOFS_DBGMSG("Failed pcol_add_page "
 			     "nr_pages=%u total_length=%zx\n",
-			     pcol->nr_pages, pcol->length);
+			     pcol->opa.nr_pages, pcol->opa.length);
 
 		/* split the request, next loop will start again */
 		ret = write_exec(pcol);
-- 
1.6.2.1

[RFC C 5/5] libosd: Remove deprecated bio based API

[Boaz Harrosh]

Feel good be happy

Signed-off-by: Boaz Harrosh <bharrosh@panasas.com>
---
 drivers/scsi/osd/osd_initiator.c |   14 --------------
 include/scsi/osd_initiator.h     |    6 ------
 2 files changed, 0 insertions(+), 20 deletions(-)

diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index d18402c..14ab901 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -679,13 +679,6 @@ EXPORT_SYMBOL(osd_req_remove_object);
 	struct osd_obj_id *first, struct osd_obj_id_list *list, unsigned nelem);
 */
 
-void osd_req_write_old(struct osd_request *or,
-	const struct osd_obj_id *obj, struct bio *bio, u64 offset)
-{
-	BUG_ON(1);
-}
-EXPORT_SYMBOL(osd_req_write_old);
-
 void osd_req_write(struct osd_request *or,
 	const struct osd_obj_id *obj, const struct osd_pages_array *opa,
 	u64 offset)
@@ -736,13 +729,6 @@ void osd_req_flush_object(struct osd_request *or,
 }
 EXPORT_SYMBOL(osd_req_flush_object);
 
-void osd_req_read_old(struct osd_request *or,
-	const struct osd_obj_id *obj, struct bio *bio, u64 offset)
-{
-	BUG_ON(1);
-}
-EXPORT_SYMBOL(osd_req_read_old);
-
 void osd_req_read(struct osd_request *or,
 	const struct osd_obj_id *obj, const struct osd_pages_array *opa,
 	u64 offset)
diff --git a/include/scsi/osd_initiator.h b/include/scsi/osd_initiator.h
index fa32074..db96847 100644
--- a/include/scsi/osd_initiator.h
+++ b/include/scsi/osd_initiator.h
@@ -377,9 +377,6 @@ void osd_req_set_member_attrs(struct osd_request *or, ...);/* V2-only NI */
 void osd_req_create_object(struct osd_request *or, struct osd_obj_id *);
 void osd_req_remove_object(struct osd_request *or, struct osd_obj_id *);
 
-void __deprecated osd_req_write_old(struct osd_request *or,
-	const struct osd_obj_id *, struct bio *data_out, u64 offset);
-
 void osd_req_write(struct osd_request *or,
 	const struct osd_obj_id *, const struct osd_pages_array *opa,
 	u64 offset);
@@ -398,9 +395,6 @@ void osd_req_flush_object(struct osd_request *or,
 	const struct osd_obj_id *, enum osd_options_flush_scope_values,
 	/*V2*/ u64 offset, /*V2*/ u64 len);
 
-void __deprecated osd_req_read_old(struct osd_request *or,
-	const struct osd_obj_id *, struct bio *data_in, u64 offset);
-
 void osd_req_read(struct osd_request *or,
 	const struct osd_obj_id *, const struct osd_pages_array *opa_in,
 	u64 offset);
-- 
1.6.2.1