* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios [not found] <1393457997-17618-1-git-send-email-kmo@daterainc.com> @ 2014-02-26 23:39 ` Kent Overstreet 2014-02-27 17:22 ` Matthew Wilcox 2014-03-02 20:31 ` Muthu Kumar 0 siblings, 2 replies; 8+ messages in thread From: Kent Overstreet @ 2014-02-26 23:39 UTC (permalink / raw) The way the block layer is currently written, it goes to great lengths to avoid having to split bios; upper layer code (such as bio_add_page()) checks what the underlying device can handle and tries to always create bios that don't need to be split. But this approach becomes unwieldy and eventually breaks down with stacked devices and devices with dynamic limits, and it adds a lot of complexity. If the block layer could split bios as needed, we could eliminate a lot of complexity elsewhere - particularly in stacked drivers. Code that creates bios can then create whatever size bios are convenient, and more importantly stacked drivers don't have to deal with both their own bio size limitations and the limitations of the (potentially multiple) devices underneath them. In the future this will let us delete merge_bvec_fn and a bunch of other code. We do this by adding calls to blk_queue_split() to the various make_request functions that need it - a few can already handle arbitrary size bios. Note that we add the call _after_ any call to blk_queue_bounce(); this means that blk_queue_split() and blk_recalc_rq_segments() don't need to be concerned with bouncing affecting segment merging. Some make_request_fns were simple enough to audit and verify they don't need blk_queue_split() calls. The skipped ones are: * nfhd_make_request (arch/m68k/emu/nfblock.c) * axon_ram_make_request (arch/powerpc/sysdev/axonram.c) * simdisk_make_request (arch/xtensa/platforms/iss/simdisk.c) * brd_make_request (ramdisk - drivers/block/brd.c) * loop_make_request * null_queue_bio * bcache's make_request fns Some others are almost certainly safe to remove now, but will be left for future patches. Signed-off-by: Kent Overstreet <kmo at daterainc.com> Cc: Jens Axboe <axboe at kernel.dk> Cc: Neil Brown <neilb at suse.de> Cc: Alasdair Kergon <agk at redhat.com> Cc: dm-devel at redhat.com Cc: Lars Ellenberg <drbd-dev at lists.linbit.com> Cc: drbd-user at lists.linbit.com Cc: Asai Thambi S P <asamymuthupa at micron.com> Cc: Sam Bradshaw <sbradshaw at micron.com> Cc: Matthew Wilcox <willy at linux.intel.com> Cc: linux-nvme at lists.infradead.org Cc: Jiri Kosina <jkosina at suse.cz> Cc: Geoff Levand <geoff at infradead.org> Cc: Jim Paris <jim at jtan.com> Cc: Joshua Morris <josh.h.morris at us.ibm.com> Cc: Philip Kelleher <pjk1939 at linux.vnet.ibm.com> Cc: Minchan Kim <minchan at kernel.org> Cc: Nitin Gupta <ngupta at vflare.org> Cc: Martin Schwidefsky <schwidefsky at de.ibm.com> Cc: Heiko Carstens <heiko.carstens at de.ibm.com> Cc: Peng Tao <bergwolf at gmail.com> --- block/blk-core.c | 19 ++-- block/blk-merge.c | 150 ++++++++++++++++++++++++++-- block/blk-mq.c | 2 + drivers/block/drbd/drbd_req.c | 2 + drivers/block/mtip32xx/mtip32xx.c | 6 +- drivers/block/nvme-core.c | 2 + drivers/block/pktcdvd.c | 6 +- drivers/block/ps3vram.c | 2 + drivers/block/rsxx/dev.c | 2 + drivers/block/umem.c | 2 + drivers/block/zram/zram_drv.c | 2 + drivers/md/dm.c | 2 + drivers/md/md.c | 2 + drivers/s390/block/dcssblk.c | 2 + drivers/s390/block/xpram.c | 2 + drivers/staging/lustre/lustre/llite/lloop.c | 2 + include/linux/blkdev.h | 3 + 17 files changed, 185 insertions(+), 23 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 853f927492..d3b0782ec3 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -581,6 +581,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) if (q->id < 0) goto fail_c; + q->bio_split = bioset_create(4, 0); + if (!q->bio_split) + goto fail_id; + q->backing_dev_info.ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE; q->backing_dev_info.state = 0; @@ -590,7 +594,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) err = bdi_init(&q->backing_dev_info); if (err) - goto fail_id; + goto fail_split; setup_timer(&q->backing_dev_info.laptop_mode_wb_timer, laptop_mode_timer_fn, (unsigned long) q); @@ -635,6 +639,8 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) fail_bdi: bdi_destroy(&q->backing_dev_info); +fail_split: + bioset_free(q->bio_split); fail_id: ida_simple_remove(&blk_queue_ida, q->id); fail_c: @@ -1501,6 +1507,8 @@ void blk_queue_bio(struct request_queue *q, struct bio *bio) struct request *req; unsigned int request_count = 0; + blk_queue_split(q, &bio, q->bio_split); + /* * low level driver can indicate that it wants pages above a * certain limit bounced to low memory (ie for highmem, or even @@ -1723,15 +1731,6 @@ generic_make_request_checks(struct bio *bio) goto end_io; } - if (likely(bio_is_rw(bio) && - nr_sectors > queue_max_hw_sectors(q))) { - printk(KERN_ERR "bio too big device %s (%u > %u)\n", - bdevname(bio->bi_bdev, b), - bio_sectors(bio), - queue_max_hw_sectors(q)); - goto end_io; - } - part = bio->bi_bdev->bd_part; if (should_fail_request(part, bio->bi_iter.bi_size) || should_fail_request(&part_to_disk(part)->part0, diff --git a/block/blk-merge.c b/block/blk-merge.c index 6c583f9c5b..0afbe3f1c2 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -9,11 +9,149 @@ #include "blk.h" +static struct bio *blk_bio_discard_split(struct request_queue *q, + struct bio *bio, + struct bio_set *bs) +{ + unsigned int max_discard_sectors, granularity; + int alignment; + sector_t tmp; + unsigned split_sectors; + + /* Zero-sector (unknown) and one-sector granularities are the same. */ + granularity = max(q->limits.discard_granularity >> 9, 1U); + + max_discard_sectors = min(q->limits.max_discard_sectors, UINT_MAX >> 9); + max_discard_sectors -= max_discard_sectors % granularity; + + if (unlikely(!max_discard_sectors)) { + /* XXX: warn */ + return NULL; + } + + if (bio_sectors(bio) <= max_discard_sectors) + return NULL; + + split_sectors = max_discard_sectors; + + /* + * If the next starting sector would be misaligned, stop the discard at + * the previous aligned sector. + */ + alignment = (q->limits.discard_alignment >> 9) % granularity; + + tmp = bio->bi_iter.bi_sector + split_sectors - alignment; + tmp = sector_div(tmp, granularity); + + if (split_sectors > tmp) + split_sectors -= tmp; + + return bio_split(bio, split_sectors, GFP_NOIO, bs); +} + +static struct bio *blk_bio_write_same_split(struct request_queue *q, + struct bio *bio, + struct bio_set *bs) +{ + if (!q->limits.max_write_same_sectors) + return NULL; + + if (bio_sectors(bio) <= q->limits.max_write_same_sectors) + return NULL; + + return bio_split(bio, q->limits.max_write_same_sectors, GFP_NOIO, bs); +} + +static struct bio *blk_bio_segment_split(struct request_queue *q, + struct bio *bio, + struct bio_set *bs) +{ + struct bio *split; + struct bio_vec bv, bvprv; + struct bvec_iter iter; + unsigned seg_size = 0, nsegs = 0; + int prev = 0; + + struct bvec_merge_data bvm = { + .bi_bdev = bio->bi_bdev, + .bi_sector = bio->bi_iter.bi_sector, + .bi_size = 0, + .bi_rw = bio->bi_rw, + }; + + bio_for_each_segment(bv, bio, iter) { + if (q->merge_bvec_fn && + q->merge_bvec_fn(q, &bvm, &bv) < (int) bv.bv_len) + goto split; + + bvm.bi_size += bv.bv_len; + + if (bvm.bi_size >> 9 > queue_max_sectors(q)) + goto split; + + if (prev && blk_queue_cluster(q)) { + if (seg_size + bv.bv_len > queue_max_segment_size(q)) + goto new_segment; + if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bv)) + goto new_segment; + if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bv)) + goto new_segment; + + seg_size += bv.bv_len; + bvprv = bv; + prev = 1; + continue; + } +new_segment: + if (nsegs == queue_max_segments(q)) + goto split; + + nsegs++; + bvprv = bv; + prev = 1; + seg_size = bv.bv_len; + } + + return NULL; +split: + split = bio_clone_bioset(bio, GFP_NOIO, bs); + + split->bi_iter.bi_size -= iter.bi_size; + bio->bi_iter = iter; + + if (bio_integrity(bio)) { + bio_integrity_advance(bio, split->bi_iter.bi_size); + bio_integrity_trim(split, 0, bio_sectors(split)); + } + + return split; +} + +void blk_queue_split(struct request_queue *q, struct bio **bio, + struct bio_set *bs) +{ + struct bio *split; + + if ((*bio)->bi_rw & REQ_DISCARD) + split = blk_bio_discard_split(q, *bio, bs); + else if ((*bio)->bi_rw & REQ_WRITE_SAME) + split = blk_bio_write_same_split(q, *bio, bs); + else + split = blk_bio_segment_split(q, *bio, q->bio_split); + + if (split) { + bio_chain(split, *bio); + generic_make_request(*bio); + *bio = split; + } +} +EXPORT_SYMBOL(blk_queue_split); + static unsigned int __blk_recalc_rq_segments(struct request_queue *q, struct bio *bio) { struct bio_vec bv, bvprv = { NULL }; - int cluster, high, highprv = 1; + int cluster, prev = 0; unsigned int seg_size, nr_phys_segs; struct bio *fbio, *bbio; struct bvec_iter iter; @@ -37,13 +175,7 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q, nr_phys_segs = 0; for_each_bio(bio) { bio_for_each_segment(bv, bio, iter) { - /* - * the trick here is making sure that a high page is - * never considered part of another segment, since that - * might change with the bounce page. - */ - high = page_to_pfn(bv.bv_page) > queue_bounce_pfn(q); - if (!high && !highprv && cluster) { + if (prev && cluster) { if (seg_size + bv.bv_len > queue_max_segment_size(q)) goto new_segment; @@ -63,8 +195,8 @@ new_segment: nr_phys_segs++; bvprv = bv; + prev = 1; seg_size = bv.bv_len; - highprv = high; } bbio = bio; } diff --git a/block/blk-mq.c b/block/blk-mq.c index 6468a715a0..7893e254d8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -915,6 +915,8 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) return; } + blk_queue_split(q, &bio, q->bio_split); + if (use_plug && blk_attempt_plug_merge(q, bio, &request_count)) return; diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c index 104a040f24..941a69c50c 100644 --- a/drivers/block/drbd/drbd_req.c +++ b/drivers/block/drbd/drbd_req.c @@ -1275,6 +1275,8 @@ void drbd_make_request(struct request_queue *q, struct bio *bio) struct drbd_conf *mdev = (struct drbd_conf *) q->queuedata; unsigned long start_time; + blk_queue_split(q, &bio, q->bio_split); + start_time = jiffies; /* diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 516026954b..df733ca685 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -4033,6 +4033,10 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) int nents = 0; int tag = 0, unaligned = 0; + blk_queue_bounce(queue, &bio); + + blk_queue_split(queue, &bio, queue->bio_split); + if (unlikely(dd->dd_flag & MTIP_DDF_STOP_IO)) { if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag))) { @@ -4082,8 +4086,6 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) sg = mtip_hw_get_scatterlist(dd, &tag, unaligned); if (likely(sg != NULL)) { - blk_queue_bounce(queue, &bio); - if (unlikely((bio)->bi_vcnt > MTIP_MAX_SG)) { dev_warn(&dd->pdev->dev, "Maximum number of SGL entries exceeded\n"); diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c index 51824d1f23..e4376b9613 100644 --- a/drivers/block/nvme-core.c +++ b/drivers/block/nvme-core.c @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) struct nvme_queue *nvmeq = get_nvmeq(ns->dev); int result = -EBUSY; + blk_queue_split(q, &bio, q->bio_split); + if (!nvmeq) { put_nvmeq(NULL); bio_endio(bio, -EIO); diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c index a2af73db18..a37acf722b 100644 --- a/drivers/block/pktcdvd.c +++ b/drivers/block/pktcdvd.c @@ -2444,6 +2444,10 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) char b[BDEVNAME_SIZE]; struct bio *split; + blk_queue_bounce(q, &bio); + + blk_queue_split(q, &bio, q->bio_split); + pd = q->queuedata; if (!pd) { pr_err("%s incorrect request queue\n", @@ -2474,8 +2478,6 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) goto end_io; } - blk_queue_bounce(q, &bio); - do { sector_t zone = get_zone(bio->bi_iter.bi_sector, pd); sector_t last_zone = get_zone(bio_end_sector(bio) - 1, pd); diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c index ef45cfb98f..a995972961 100644 --- a/drivers/block/ps3vram.c +++ b/drivers/block/ps3vram.c @@ -603,6 +603,8 @@ static void ps3vram_make_request(struct request_queue *q, struct bio *bio) struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev); int busy; + blk_queue_split(q, &bio, q->bio_split); + dev_dbg(&dev->core, "%s\n", __func__); spin_lock_irq(&priv->lock); diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c index 2839d37e5a..ff074a3cd4 100644 --- a/drivers/block/rsxx/dev.c +++ b/drivers/block/rsxx/dev.c @@ -169,6 +169,8 @@ static void rsxx_make_request(struct request_queue *q, struct bio *bio) struct rsxx_bio_meta *bio_meta; int st = -EINVAL; + blk_queue_split(q, &bio, q->bio_split); + might_sleep(); if (!card) diff --git a/drivers/block/umem.c b/drivers/block/umem.c index 4cf81b5bf0..13d577cfbc 100644 --- a/drivers/block/umem.c +++ b/drivers/block/umem.c @@ -531,6 +531,8 @@ static void mm_make_request(struct request_queue *q, struct bio *bio) (unsigned long long)bio->bi_iter.bi_sector, bio->bi_iter.bi_size); + blk_queue_split(q, &bio, q->bio_split); + spin_lock_irq(&card->lock); *card->biotail = bio; bio->bi_next = NULL; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 011e55d820..ecf9daa01c 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -733,6 +733,8 @@ static void zram_make_request(struct request_queue *queue, struct bio *bio) { struct zram *zram = queue->queuedata; + blk_queue_split(queue, &bio, queue->bio_split); + down_read(&zram->init_lock); if (unlikely(!zram->init_done)) goto error; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 8c53b09b9a..97f70420f2 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1500,6 +1500,8 @@ static void dm_request(struct request_queue *q, struct bio *bio) { struct mapped_device *md = q->queuedata; + blk_queue_split(q, &bio, q->bio_split); + if (dm_request_based(md)) blk_queue_bio(q, bio); else diff --git a/drivers/md/md.c b/drivers/md/md.c index 4ad5cc4e63..1421bc3f7b 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -256,6 +256,8 @@ static void md_make_request(struct request_queue *q, struct bio *bio) int cpu; unsigned int sectors; + blk_queue_split(q, &bio, q->bio_split); + if (mddev == NULL || mddev->pers == NULL || !mddev->ready) { bio_io_error(bio); diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c index ebf41e228e..db33cd3e4c 100644 --- a/drivers/s390/block/dcssblk.c +++ b/drivers/s390/block/dcssblk.c @@ -815,6 +815,8 @@ dcssblk_make_request(struct request_queue *q, struct bio *bio) unsigned long source_addr; unsigned long bytes_done; + blk_queue_split(q, &bio, q->bio_split); + bytes_done = 0; dev_info = bio->bi_bdev->bd_disk->private_data; if (dev_info == NULL) diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c index 6969d39f1e..f03c103f13 100644 --- a/drivers/s390/block/xpram.c +++ b/drivers/s390/block/xpram.c @@ -190,6 +190,8 @@ static void xpram_make_request(struct request_queue *q, struct bio *bio) unsigned long page_addr; unsigned long bytes; + blk_queue_split(q, &bio, q->bio_split); + if ((bio->bi_iter.bi_sector & 7) != 0 || (bio->bi_iter.bi_size & 4095) != 0) /* Request is not page-aligned. */ diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c index 0718905ade..a3f6dc930b 100644 --- a/drivers/staging/lustre/lustre/llite/lloop.c +++ b/drivers/staging/lustre/lustre/llite/lloop.c @@ -344,6 +344,8 @@ static void loop_make_request(struct request_queue *q, struct bio *old_bio) int rw = bio_rw(old_bio); int inactive; + blk_queue_split(q, &old_bio, q->bio_split); + if (!lo) goto err; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 1e1fa3f93d..99e9955c4d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -470,6 +470,7 @@ struct request_queue { wait_queue_head_t mq_freeze_wq; struct percpu_counter mq_usage_counter; struct list_head all_q_node; + struct bio_set *bio_split; }; #define QUEUE_FLAG_QUEUED 1 /* uses generic tag queueing */ @@ -781,6 +782,8 @@ extern void blk_rq_unprep_clone(struct request *rq); extern int blk_insert_cloned_request(struct request_queue *q, struct request *rq); extern void blk_delay_queue(struct request_queue *, unsigned long); +extern void blk_queue_split(struct request_queue *, struct bio **, + struct bio_set *); extern void blk_recount_segments(struct request_queue *, struct bio *); extern int scsi_verify_blk_ioctl(struct block_device *, unsigned int); extern int scsi_cmd_blk_ioctl(struct block_device *, fmode_t, -- 1.9.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-26 23:39 ` [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios Kent Overstreet @ 2014-02-27 17:22 ` Matthew Wilcox 2014-02-27 21:27 ` Kent Overstreet 2014-02-28 23:30 ` Kent Overstreet 2014-03-02 20:31 ` Muthu Kumar 1 sibling, 2 replies; 8+ messages in thread From: Matthew Wilcox @ 2014-02-27 17:22 UTC (permalink / raw) On Wed, Feb 26, 2014@03:39:49PM -0800, Kent Overstreet wrote: > We do this by adding calls to blk_queue_split() to the various > make_request functions that need it - a few can already handle arbitrary > size bios. Note that we add the call _after_ any call to blk_queue_bounce(); > this means that blk_queue_split() and blk_recalc_rq_segments() don't need to > be concerned with bouncing affecting segment merging. > diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c > index 51824d1f23..e4376b9613 100644 > --- a/drivers/block/nvme-core.c > +++ b/drivers/block/nvme-core.c > @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) > struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > int result = -EBUSY; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (!nvmeq) { > put_nvmeq(NULL); > bio_endio(bio, -EIO); I'd suggest that we do: - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); + struct nvme_queue *nvmeq; int result = -EBUSY; + blk_queue_split(q, &bio, q->bio_split); + + nvmeq = get_nvmeq(ns->dev); if (!nvmeq) { so that we're running the blk_queue_split() code outside the get_cpu() call. Now, the NVMe driver has its own rules about when BIOs have to be split. Right now, that's way down inside the nvme_map_bio() call when we're walking the bio to compose the scatterlist. Should we instead have an nvme_bio_split() routine that is called instead of blk_queue_split(), and we can simplify nvme_map_bio() since it'll know that it's working with bios that don't have to be split. In fact, I think it would have little NVMe-specific in it at that point, so we could name __blk_bios_map_sg() better, export it to drivers and call it from nvme_map_bio(), which I think would make everybody happier. > diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c > index a2af73db18..a37acf722b 100644 > --- a/drivers/block/pktcdvd.c > +++ b/drivers/block/pktcdvd.c > @@ -2444,6 +2444,10 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) > char b[BDEVNAME_SIZE]; > struct bio *split; > > + blk_queue_bounce(q, &bio); > + > + blk_queue_split(q, &bio, q->bio_split); > + > pd = q->queuedata; > if (!pd) { > pr_err("%s incorrect request queue\n", > @@ -2474,8 +2478,6 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) > goto end_io; > } > > - blk_queue_bounce(q, &bio); > - > do { > sector_t zone = get_zone(bio->bi_iter.bi_sector, pd); > sector_t last_zone = get_zone(bio_end_sector(bio) - 1, pd); > diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c > index ef45cfb98f..a995972961 100644 > --- a/drivers/block/ps3vram.c > +++ b/drivers/block/ps3vram.c > @@ -603,6 +603,8 @@ static void ps3vram_make_request(struct request_queue *q, struct bio *bio) > struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev); > int busy; > > + blk_queue_split(q, &bio, q->bio_split); > + > dev_dbg(&dev->core, "%s\n", __func__); > > spin_lock_irq(&priv->lock); > diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c > index 2839d37e5a..ff074a3cd4 100644 > --- a/drivers/block/rsxx/dev.c > +++ b/drivers/block/rsxx/dev.c > @@ -169,6 +169,8 @@ static void rsxx_make_request(struct request_queue *q, struct bio *bio) > struct rsxx_bio_meta *bio_meta; > int st = -EINVAL; > > + blk_queue_split(q, &bio, q->bio_split); > + > might_sleep(); > > if (!card) > diff --git a/drivers/block/umem.c b/drivers/block/umem.c > index 4cf81b5bf0..13d577cfbc 100644 > --- a/drivers/block/umem.c > +++ b/drivers/block/umem.c > @@ -531,6 +531,8 @@ static void mm_make_request(struct request_queue *q, struct bio *bio) > (unsigned long long)bio->bi_iter.bi_sector, > bio->bi_iter.bi_size); > > + blk_queue_split(q, &bio, q->bio_split); > + > spin_lock_irq(&card->lock); > *card->biotail = bio; > bio->bi_next = NULL; > diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c > index 011e55d820..ecf9daa01c 100644 > --- a/drivers/block/zram/zram_drv.c > +++ b/drivers/block/zram/zram_drv.c > @@ -733,6 +733,8 @@ static void zram_make_request(struct request_queue *queue, struct bio *bio) > { > struct zram *zram = queue->queuedata; > > + blk_queue_split(queue, &bio, queue->bio_split); > + > down_read(&zram->init_lock); > if (unlikely(!zram->init_done)) > goto error; > diff --git a/drivers/md/dm.c b/drivers/md/dm.c > index 8c53b09b9a..97f70420f2 100644 > --- a/drivers/md/dm.c > +++ b/drivers/md/dm.c > @@ -1500,6 +1500,8 @@ static void dm_request(struct request_queue *q, struct bio *bio) > { > struct mapped_device *md = q->queuedata; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (dm_request_based(md)) > blk_queue_bio(q, bio); > else > diff --git a/drivers/md/md.c b/drivers/md/md.c > index 4ad5cc4e63..1421bc3f7b 100644 > --- a/drivers/md/md.c > +++ b/drivers/md/md.c > @@ -256,6 +256,8 @@ static void md_make_request(struct request_queue *q, struct bio *bio) > int cpu; > unsigned int sectors; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (mddev == NULL || mddev->pers == NULL > || !mddev->ready) { > bio_io_error(bio); > diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c > index ebf41e228e..db33cd3e4c 100644 > --- a/drivers/s390/block/dcssblk.c > +++ b/drivers/s390/block/dcssblk.c > @@ -815,6 +815,8 @@ dcssblk_make_request(struct request_queue *q, struct bio *bio) > unsigned long source_addr; > unsigned long bytes_done; > > + blk_queue_split(q, &bio, q->bio_split); > + > bytes_done = 0; > dev_info = bio->bi_bdev->bd_disk->private_data; > if (dev_info == NULL) > diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c > index 6969d39f1e..f03c103f13 100644 > --- a/drivers/s390/block/xpram.c > +++ b/drivers/s390/block/xpram.c > @@ -190,6 +190,8 @@ static void xpram_make_request(struct request_queue *q, struct bio *bio) > unsigned long page_addr; > unsigned long bytes; > > + blk_queue_split(q, &bio, q->bio_split); > + > if ((bio->bi_iter.bi_sector & 7) != 0 || > (bio->bi_iter.bi_size & 4095) != 0) > /* Request is not page-aligned. */ > diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c > index 0718905ade..a3f6dc930b 100644 > --- a/drivers/staging/lustre/lustre/llite/lloop.c > +++ b/drivers/staging/lustre/lustre/llite/lloop.c > @@ -344,6 +344,8 @@ static void loop_make_request(struct request_queue *q, struct bio *old_bio) > int rw = bio_rw(old_bio); > int inactive; > > + blk_queue_split(q, &old_bio, q->bio_split); > + > if (!lo) > goto err; > > diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h > index 1e1fa3f93d..99e9955c4d 100644 > --- a/include/linux/blkdev.h > +++ b/include/linux/blkdev.h > @@ -470,6 +470,7 @@ struct request_queue { > wait_queue_head_t mq_freeze_wq; > struct percpu_counter mq_usage_counter; > struct list_head all_q_node; > + struct bio_set *bio_split; > }; > > #define QUEUE_FLAG_QUEUED 1 /* uses generic tag queueing */ > @@ -781,6 +782,8 @@ extern void blk_rq_unprep_clone(struct request *rq); > extern int blk_insert_cloned_request(struct request_queue *q, > struct request *rq); > extern void blk_delay_queue(struct request_queue *, unsigned long); > +extern void blk_queue_split(struct request_queue *, struct bio **, > + struct bio_set *); > extern void blk_recount_segments(struct request_queue *, struct bio *); > extern int scsi_verify_blk_ioctl(struct block_device *, unsigned int); > extern int scsi_cmd_blk_ioctl(struct block_device *, fmode_t, > -- > 1.9.0 ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-27 17:22 ` Matthew Wilcox @ 2014-02-27 21:27 ` Kent Overstreet 2014-02-28 23:30 ` Kent Overstreet 1 sibling, 0 replies; 8+ messages in thread From: Kent Overstreet @ 2014-02-27 21:27 UTC (permalink / raw) On Thu, Feb 27, 2014@12:22:54PM -0500, Matthew Wilcox wrote: > On Wed, Feb 26, 2014@03:39:49PM -0800, Kent Overstreet wrote: > > We do this by adding calls to blk_queue_split() to the various > > make_request functions that need it - a few can already handle arbitrary > > size bios. Note that we add the call _after_ any call to blk_queue_bounce(); > > this means that blk_queue_split() and blk_recalc_rq_segments() don't need to > > be concerned with bouncing affecting segment merging. > > > diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c > > index 51824d1f23..e4376b9613 100644 > > --- a/drivers/block/nvme-core.c > > +++ b/drivers/block/nvme-core.c > > @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) > > struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > > int result = -EBUSY; > > > > + blk_queue_split(q, &bio, q->bio_split); > > + > > if (!nvmeq) { > > put_nvmeq(NULL); > > bio_endio(bio, -EIO); > > I'd suggest that we do: > > - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > + struct nvme_queue *nvmeq; > int result = -EBUSY; > > + blk_queue_split(q, &bio, q->bio_split); > + > + nvmeq = get_nvmeq(ns->dev); > if (!nvmeq) { > > so that we're running the blk_queue_split() code outside the get_cpu() > call. Whoops, that's definitely a bug. > Now, the NVMe driver has its own rules about when BIOs have to be split. > Right now, that's way down inside the nvme_map_bio() call when we're > walking the bio to compose the scatterlist. Should we instead have an > nvme_bio_split() routine that is called instead of blk_queue_split(), > and we can simplify nvme_map_bio() since it'll know that it's working > with bios that don't have to be split. > > In fact, I think it would have little NVMe-specific in it at that point, > so we could name __blk_bios_map_sg() better, export it to drivers and > call it from nvme_map_bio(), which I think would make everybody happier. Yes, definitely - and by doing it there we shoudn't even have to split the bios, we can just process them incrementally. I can write a patch for it later if you want to test it. ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-27 17:22 ` Matthew Wilcox 2014-02-27 21:27 ` Kent Overstreet @ 2014-02-28 23:30 ` Kent Overstreet 2014-03-01 17:52 ` Keith Busch 2014-03-13 23:33 ` Keith Busch 1 sibling, 2 replies; 8+ messages in thread From: Kent Overstreet @ 2014-02-28 23:30 UTC (permalink / raw) On Thu, Feb 27, 2014@12:22:54PM -0500, Matthew Wilcox wrote: > On Wed, Feb 26, 2014@03:39:49PM -0800, Kent Overstreet wrote: > > We do this by adding calls to blk_queue_split() to the various > > make_request functions that need it - a few can already handle arbitrary > > size bios. Note that we add the call _after_ any call to blk_queue_bounce(); > > this means that blk_queue_split() and blk_recalc_rq_segments() don't need to > > be concerned with bouncing affecting segment merging. > > > diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c > > index 51824d1f23..e4376b9613 100644 > > --- a/drivers/block/nvme-core.c > > +++ b/drivers/block/nvme-core.c > > @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) > > struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > > int result = -EBUSY; > > > > + blk_queue_split(q, &bio, q->bio_split); > > + > > if (!nvmeq) { > > put_nvmeq(NULL); > > bio_endio(bio, -EIO); > > I'd suggest that we do: > > - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > + struct nvme_queue *nvmeq; > int result = -EBUSY; > > + blk_queue_split(q, &bio, q->bio_split); > + > + nvmeq = get_nvmeq(ns->dev); > if (!nvmeq) { > > so that we're running the blk_queue_split() code outside the get_cpu() > call. > > Now, the NVMe driver has its own rules about when BIOs have to be split. > Right now, that's way down inside the nvme_map_bio() call when we're > walking the bio to compose the scatterlist. Should we instead have an > nvme_bio_split() routine that is called instead of blk_queue_split(), > and we can simplify nvme_map_bio() since it'll know that it's working > with bios that don't have to be split. > > In fact, I think it would have little NVMe-specific in it at that point, > so we could name __blk_bios_map_sg() better, export it to drivers and > call it from nvme_map_bio(), which I think would make everybody happier. Actually, reading nvme_map_bio() (it's different since last I looked at it) it looks like nvme should already be able to handle arbitrary size bios? I do intend to rework the blk_bio_map_sg() (or add a new one?) to incrementally map as much of a bio as will fit in the provided scatterlist, but it looks like nvme has some odd restrictions where it's using BIOVEC_PHYS_MERGABLE()/BIOVEC_NOT_VIRT_MERGABLE() so I dunno if it's worth bothering to try and have it use generic code. However we don't need an explicit split here: if the sg fills up (i.e. the places nvme_split_and_submit() is called), we can just mark the bio as partially completed (set bio->bi_iter = iter, i.e. use the iterator you passed to bio_for_each_segment), then increment bi_remaining (which just counts completions, i.e. bio_endio() calls before the bio is really completed) and resubmit the original bio. No need to allocate a split bio, or loop over the bio again in bio_split(). ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-28 23:30 ` Kent Overstreet @ 2014-03-01 17:52 ` Keith Busch 2014-03-13 23:33 ` Keith Busch 1 sibling, 0 replies; 8+ messages in thread From: Keith Busch @ 2014-03-01 17:52 UTC (permalink / raw) On Fri, 28 Feb 2014, Kent Overstreet wrote: > On Thu, Feb 27, 2014@12:22:54PM -0500, Matthew Wilcox wrote: >> On Wed, Feb 26, 2014@03:39:49PM -0800, Kent Overstreet wrote: >>> We do this by adding calls to blk_queue_split() to the various >>> make_request functions that need it - a few can already handle arbitrary >>> size bios. Note that we add the call _after_ any call to blk_queue_bounce(); >>> this means that blk_queue_split() and blk_recalc_rq_segments() don't need to >>> be concerned with bouncing affecting segment merging. >> >>> diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c >>> index 51824d1f23..e4376b9613 100644 >>> --- a/drivers/block/nvme-core.c >>> +++ b/drivers/block/nvme-core.c >>> @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) >>> struct nvme_queue *nvmeq = get_nvmeq(ns->dev); >>> int result = -EBUSY; >>> >>> + blk_queue_split(q, &bio, q->bio_split); >>> + >>> if (!nvmeq) { >>> put_nvmeq(NULL); >>> bio_endio(bio, -EIO); >> >> I'd suggest that we do: >> >> - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); >> + struct nvme_queue *nvmeq; >> int result = -EBUSY; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> + nvmeq = get_nvmeq(ns->dev); >> if (!nvmeq) { >> >> so that we're running the blk_queue_split() code outside the get_cpu() >> call. >> >> Now, the NVMe driver has its own rules about when BIOs have to be split. >> Right now, that's way down inside the nvme_map_bio() call when we're >> walking the bio to compose the scatterlist. Should we instead have an >> nvme_bio_split() routine that is called instead of blk_queue_split(), >> and we can simplify nvme_map_bio() since it'll know that it's working >> with bios that don't have to be split. >> >> In fact, I think it would have little NVMe-specific in it at that point, >> so we could name __blk_bios_map_sg() better, export it to drivers and >> call it from nvme_map_bio(), which I think would make everybody happier. > > Actually, reading nvme_map_bio() (it's different since last I looked at > it) it looks like nvme should already be able to handle arbitrary size > bios? > > I do intend to rework the blk_bio_map_sg() (or add a new one?) to > incrementally map as much of a bio as will fit in the provided > scatterlist, but it looks like nvme has some odd restrictions where it's > using BIOVEC_PHYS_MERGABLE()/BIOVEC_NOT_VIRT_MERGABLE() so I dunno if > it's worth bothering to try and have it use generic code. Is nvme the only driver that has these kinds of restrictions on segment address offsets? If so, I guess there's no reason to make it generic. > However we don't need an explicit split here: if the sg fills up (i.e. > the places nvme_split_and_submit() is called), we can just mark the bio > as partially completed (set bio->bi_iter = iter, i.e. use the iterator > you passed to bio_for_each_segment), then increment bi_remaining (which > just counts completions, i.e. bio_endio() calls before the bio is really > completed) and resubmit the original bio. No need to allocate a split > bio, or loop over the bio again in bio_split(). We used to manipulate the original bio to track partial completions, but I changed that for reasons that haven't quite yet materialized. If we move the bio's bi_iter, it will make it difficult to retry the original request on intermittent failures, and it will break the integrity verify if the device format supports protection information. It's also more performant to submit all parts at once rather than wait for the previous part to complete before sending the next. ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-28 23:30 ` Kent Overstreet 2014-03-01 17:52 ` Keith Busch @ 2014-03-13 23:33 ` Keith Busch 1 sibling, 0 replies; 8+ messages in thread From: Keith Busch @ 2014-03-13 23:33 UTC (permalink / raw) On Fri, 28 Feb 2014, Kent Overstreet wrote: > On Thu, Feb 27, 2014@12:22:54PM -0500, Matthew Wilcox wrote: >> On Wed, Feb 26, 2014@03:39:49PM -0800, Kent Overstreet wrote: >>> We do this by adding calls to blk_queue_split() to the various >>> make_request functions that need it - a few can already handle arbitrary >>> size bios. Note that we add the call _after_ any call to blk_queue_bounce(); >>> this means that blk_queue_split() and blk_recalc_rq_segments() don't need to >>> be concerned with bouncing affecting segment merging. >> >>> diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c >>> index 51824d1f23..e4376b9613 100644 >>> --- a/drivers/block/nvme-core.c >>> +++ b/drivers/block/nvme-core.c >>> @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) >>> struct nvme_queue *nvmeq = get_nvmeq(ns->dev); >>> int result = -EBUSY; >>> >>> + blk_queue_split(q, &bio, q->bio_split); >>> + >>> if (!nvmeq) { >>> put_nvmeq(NULL); >>> bio_endio(bio, -EIO); >> >> I'd suggest that we do: >> >> - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); >> + struct nvme_queue *nvmeq; >> int result = -EBUSY; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> + nvmeq = get_nvmeq(ns->dev); >> if (!nvmeq) { >> >> so that we're running the blk_queue_split() code outside the get_cpu() >> call. >> >> Now, the NVMe driver has its own rules about when BIOs have to be split. >> Right now, that's way down inside the nvme_map_bio() call when we're >> walking the bio to compose the scatterlist. Should we instead have an >> nvme_bio_split() routine that is called instead of blk_queue_split(), >> and we can simplify nvme_map_bio() since it'll know that it's working >> with bios that don't have to be split. >> >> In fact, I think it would have little NVMe-specific in it at that point, >> so we could name __blk_bios_map_sg() better, export it to drivers and >> call it from nvme_map_bio(), which I think would make everybody happier. > > Actually, reading nvme_map_bio() (it's different since last I looked at > it) it looks like nvme should already be able to handle arbitrary size > bios? > > I do intend to rework the blk_bio_map_sg() (or add a new one?) to > incrementally map as much of a bio as will fit in the provided > scatterlist, but it looks like nvme has some odd restrictions where it's > using BIOVEC_PHYS_MERGABLE()/BIOVEC_NOT_VIRT_MERGABLE() so I dunno if > it's worth bothering to try and have it use generic code. > > However we don't need an explicit split here: if the sg fills up (i.e. > the places nvme_split_and_submit() is called), we can just mark the bio > as partially completed (set bio->bi_iter = iter, i.e. use the iterator > you passed to bio_for_each_segment), then increment bi_remaining (which > just counts completions, i.e. bio_endio() calls before the bio is really > completed) and resubmit the original bio. No need to allocate a split > bio, or loop over the bio again in bio_split(). Hi Kent, I wanted to see about getting rid of nvme_map_bio and do all the splits generically. I built this on top of your original patch and added additional split criteria for NVMe, which includes the "segment" address offset and the vendor specific "stripe_size" (I think "chunk_size" may have been a more appropriate description, but I used someone else's term for this). This is what I came up with; I hoped the additions would be useful, but maybe the additional split criteria applies only to nvme. diff --git a/block/blk-merge.c b/block/blk-merge.c index 0afbe3f..2848181 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -66,11 +66,9 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, struct bio *bio, struct bio_set *bs) { - struct bio *split; struct bio_vec bv, bvprv; struct bvec_iter iter; unsigned seg_size = 0, nsegs = 0; - int prev = 0; struct bvec_merge_data bvm = { .bi_bdev = bio->bi_bdev, @@ -79,17 +77,26 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, .bi_rw = bio->bi_rw, }; + if (bio->bi_iter.bi_size >> 9 > queue_max_sectors(q)) { + bvm.bi_size = queue_max_sectors(q) << 9; + goto split; + } + if (queue_stripe_sectors(q)) { + unsigned split_len = + (queue_stripe_sectors(q) - + (bio->bi_iter.bi_sector & + (queue_stripe_sectors(q) - 1))) << 9; + if (split_len < bio->bi_iter.bi_size) { + bvm.bi_size = split_len; + goto split; + } + } + bio_for_each_segment(bv, bio, iter) { if (q->merge_bvec_fn && q->merge_bvec_fn(q, &bvm, &bv) < (int) bv.bv_len) goto split; - - bvm.bi_size += bv.bv_len; - - if (bvm.bi_size >> 9 > queue_max_sectors(q)) - goto split; - - if (prev && blk_queue_cluster(q)) { + if (nsegs && blk_queue_cluster(q)) { if (seg_size + bv.bv_len > queue_max_segment_size(q)) goto new_segment; if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bv)) @@ -97,34 +104,26 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bv)) goto new_segment; + bvm.bi_size += bv.bv_len; seg_size += bv.bv_len; bvprv = bv; - prev = 1; continue; } new_segment: + if (nsegs && BIOVEC_SEG_OFFSET(q, &bvprv, &bv)) + goto split; if (nsegs == queue_max_segments(q)) goto split; + bvm.bi_size += bv.bv_len; nsegs++; bvprv = bv; - prev = 1; seg_size = bv.bv_len; } return NULL; split: - split = bio_clone_bioset(bio, GFP_NOIO, bs); - - split->bi_iter.bi_size -= iter.bi_size; - bio->bi_iter = iter; - - if (bio_integrity(bio)) { - bio_integrity_advance(bio, split->bi_iter.bi_size); - bio_integrity_trim(split, 0, bio_sectors(split)); - } - - return split; + return bio_split(bio, bvm.bi_size >> 9, GFP_ATOMIC, bs); } void blk_queue_split(struct request_queue *q, struct bio **bio, @@ -303,7 +302,7 @@ new_segment: *bvprv = *bvec; } -static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio, +int blk_bios_map_sg(struct request_queue *q, struct bio *bio, struct scatterlist *sglist, struct scatterlist **sg) { @@ -344,6 +343,7 @@ single_segment: return nsegs; } +EXPORT_SYMBOL(blk_bios_map_sg); /* * map a request to scatterlist, return number of sg entries setup. Caller @@ -356,7 +356,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq, int nsegs = 0; if (rq->bio) - nsegs = __blk_bios_map_sg(q, rq->bio, sglist, &sg); + nsegs = blk_bios_map_sg(q, rq->bio, sglist, &sg); if (unlikely(rq->cmd_flags & REQ_COPY_USER) && (blk_rq_bytes(rq) & q->dma_pad_mask)) { @@ -407,7 +407,7 @@ int blk_bio_map_sg(struct request_queue *q, struct bio *bio, struct bio *next = bio->bi_next; bio->bi_next = NULL; - nsegs = __blk_bios_map_sg(q, bio, sglist, &sg); + nsegs = blk_bios_map_sg(q, bio, sglist, &sg); bio->bi_next = next; if (sg) sg_mark_end(sg); diff --git a/block/blk-settings.c b/block/blk-settings.c index 5d21239..6590703 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -111,8 +111,10 @@ void blk_set_default_limits(struct queue_limits *lim) lim->max_segments = BLK_MAX_SEGMENTS; lim->max_integrity_segments = 0; lim->seg_boundary_mask = BLK_SEG_BOUNDARY_MASK; + lim->seg_offset_mask = 0; lim->max_segment_size = BLK_MAX_SEGMENT_SIZE; lim->max_sectors = lim->max_hw_sectors = BLK_SAFE_MAX_SECTORS; + lim->stripe_sectors = 0; lim->max_write_same_sectors = 0; lim->max_discard_sectors = 0; lim->discard_granularity = 0; @@ -146,6 +148,7 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_hw_sectors = UINT_MAX; lim->max_segment_size = UINT_MAX; lim->max_sectors = UINT_MAX; + lim->stripe_sectors = 0; lim->max_write_same_sectors = UINT_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -262,6 +265,18 @@ void blk_limits_max_hw_sectors(struct queue_limits *limits, unsigned int max_hw_ } EXPORT_SYMBOL(blk_limits_max_hw_sectors); +void blk_queue_stripe_sectors(struct request_queue *q, unsigned int hw_stripe_sectors) +{ + if (hw_stripe_sectors && ((hw_stripe_sectors << 9) < PAGE_CACHE_SIZE || + !is_power_of_2(hw_stripe_sectors))) { + hw_stripe_sectors = 1 << (PAGE_CACHE_SHIFT - 9); + printk(KERN_INFO "%s: set to minimum %d\n", + __func__, hw_stripe_sectors); + } + q->limits.stripe_sectors = hw_stripe_sectors; +} +EXPORT_SYMBOL(blk_queue_stripe_sectors); + /** * blk_queue_max_hw_sectors - set max sectors for a request for this queue * @q: the request queue for the device @@ -527,10 +542,13 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_hw_sectors = min_not_zero(t->max_hw_sectors, b->max_hw_sectors); t->max_write_same_sectors = min(t->max_write_same_sectors, b->max_write_same_sectors); + t->stripe_sectors = min_not_zero(t->stripe_sectors, b->stripe_sectors); t->bounce_pfn = min_not_zero(t->bounce_pfn, b->bounce_pfn); t->seg_boundary_mask = min_not_zero(t->seg_boundary_mask, b->seg_boundary_mask); + t->seg_offset_mask = min_not_zero(t->seg_offset_mask, + b->seg_offset_mask); t->max_segments = min_not_zero(t->max_segments, b->max_segments); t->max_integrity_segments = min_not_zero(t->max_integrity_segments, @@ -758,7 +776,7 @@ EXPORT_SYMBOL_GPL(blk_queue_dma_drain); **/ void blk_queue_segment_boundary(struct request_queue *q, unsigned long mask) { - if (mask < PAGE_CACHE_SIZE - 1) { + if (mask < (PAGE_CACHE_SIZE - 1)) { mask = PAGE_CACHE_SIZE - 1; printk(KERN_INFO "%s: set to minimum %lx\n", __func__, mask); @@ -769,6 +787,23 @@ void blk_queue_segment_boundary(struct request_queue *q, unsigned long mask) EXPORT_SYMBOL(blk_queue_segment_boundary); /** + * blk_queue_segment_offset - set boundary rules for segment start address + * @q: the request queue for the device + * @mask: the memory offset mask at which the request must be split + **/ +void blk_queue_segment_offset(struct request_queue *q, unsigned long mask) +{ + if (mask && mask < (PAGE_CACHE_SIZE - 1)) { + printk(KERN_INFO "%s: mask:%lx set to minimum %lx\n", + __func__, mask, PAGE_CACHE_SIZE - 1); + mask = PAGE_CACHE_SIZE - 1; + } + + q->limits.seg_offset_mask = mask; +} +EXPORT_SYMBOL(blk_queue_segment_offset); + +/** * blk_queue_dma_alignment - set dma length and memory alignment * @q: the request queue for the device * @mask: alignment mask diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c index e4376b9..7980697 100644 --- a/drivers/block/nvme-core.c +++ b/drivers/block/nvme-core.c @@ -471,71 +471,6 @@ int nvme_setup_prps(struct nvme_dev *dev, struct nvme_common_command *cmd, return total_len; } -static int nvme_split_and_submit(struct bio *bio, struct nvme_queue *nvmeq, - int len) -{ - struct bio *split = bio_split(bio, len >> 9, GFP_ATOMIC, NULL); - if (!split) - return -ENOMEM; - - bio_chain(split, bio); - - if (bio_list_empty(&nvmeq->sq_cong)) - add_wait_queue(&nvmeq->sq_full, &nvmeq->sq_cong_wait); - bio_list_add(&nvmeq->sq_cong, split); - bio_list_add(&nvmeq->sq_cong, bio); - - return 0; -} - -/* NVMe scatterlists require no holes in the virtual address */ -#define BIOVEC_NOT_VIRT_MERGEABLE(vec1, vec2) ((vec2)->bv_offset || \ - (((vec1)->bv_offset + (vec1)->bv_len) % PAGE_SIZE)) - -static int nvme_map_bio(struct nvme_queue *nvmeq, struct nvme_iod *iod, - struct bio *bio, enum dma_data_direction dma_dir, int psegs) -{ - struct bio_vec bvec, bvprv; - struct bvec_iter iter; - struct scatterlist *sg = NULL; - int length = 0, nsegs = 0, split_len = bio->bi_iter.bi_size; - int first = 1; - - if (nvmeq->dev->stripe_size) - split_len = nvmeq->dev->stripe_size - - ((bio->bi_iter.bi_sector << 9) & - (nvmeq->dev->stripe_size - 1)); - - sg_init_table(iod->sg, psegs); - bio_for_each_segment(bvec, bio, iter) { - if (!first && BIOVEC_PHYS_MERGEABLE(&bvprv, &bvec)) { - sg->length += bvec.bv_len; - } else { - if (!first && BIOVEC_NOT_VIRT_MERGEABLE(&bvprv, &bvec)) - return nvme_split_and_submit(bio, nvmeq, - length); - - sg = sg ? sg + 1 : iod->sg; - sg_set_page(sg, bvec.bv_page, - bvec.bv_len, bvec.bv_offset); - nsegs++; - } - - if (split_len - length < bvec.bv_len) - return nvme_split_and_submit(bio, nvmeq, split_len); - length += bvec.bv_len; - bvprv = bvec; - first = 0; - } - iod->nents = nsegs; - sg_mark_end(sg); - if (dma_map_sg(nvmeq->q_dmadev, iod->sg, iod->nents, dma_dir) == 0) - return -ENOMEM; - - BUG_ON(length != bio->bi_iter.bi_size); - return length; -} - /* * We reuse the small pool to allocate the 16-byte range here as it is not * worth having a special pool for these or additional cases to handle freeing @@ -607,6 +542,7 @@ int nvme_submit_flush_data(struct nvme_queue *nvmeq, struct nvme_ns *ns) static int nvme_submit_bio_queue(struct nvme_queue *nvmeq, struct nvme_ns *ns, struct bio *bio) { + struct scatterlist *sg = NULL; struct nvme_command *cmnd; struct nvme_iod *iod; enum dma_data_direction dma_dir; @@ -662,14 +598,17 @@ static int nvme_submit_bio_queue(struct nvme_queue *nvmeq, struct nvme_ns *ns, dma_dir = DMA_FROM_DEVICE; } - result = nvme_map_bio(nvmeq, iod, bio, dma_dir, psegs); - if (result <= 0) + sg_init_table(iod->sg, psegs); + iod->nents = blk_bios_map_sg(ns->queue, bio, iod->sg, &sg); + sg_mark_end(sg); + if (dma_map_sg(nvmeq->q_dmadev, iod->sg, iod->nents, dma_dir) == 0) { + result = -ENOMEM; goto free_cmdid; - length = result; + } cmnd->rw.command_id = cmdid; cmnd->rw.nsid = cpu_to_le32(ns->ns_id); - length = nvme_setup_prps(nvmeq->dev, &cmnd->common, iod, length, + length = nvme_setup_prps(nvmeq->dev, &cmnd->common, iod, bio->bi_iter.bi_size, GFP_ATOMIC); cmnd->rw.slba = cpu_to_le64(nvme_block_nr(ns, bio->bi_iter.bi_sector)); cmnd->rw.length = cpu_to_le16((length >> ns->lba_shift) - 1); @@ -734,11 +673,12 @@ static int nvme_process_cq(struct nvme_queue *nvmeq) static void nvme_make_request(struct request_queue *q, struct bio *bio) { struct nvme_ns *ns = q->queuedata; - struct nvme_queue *nvmeq = get_nvmeq(ns->dev); + struct nvme_queue *nvmeq; int result = -EBUSY; blk_queue_split(q, &bio, q->bio_split); + nvmeq = get_nvmeq(ns->dev); if (!nvmeq) { put_nvmeq(NULL); bio_endio(bio, -EIO); @@ -812,6 +752,9 @@ int nvme_submit_sync_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd, int cmdid; struct sync_cmd_info cmdinfo; + if (!nvmeq || nvmeq->q_suspended) + return -EBUSY; + cmdinfo.task = current; cmdinfo.status = -EINTR; @@ -1767,6 +1710,9 @@ static struct nvme_ns *nvme_alloc_ns(struct nvme_dev *dev, unsigned nsid, blk_queue_logical_block_size(ns->queue, 1 << ns->lba_shift); if (dev->max_hw_sectors) blk_queue_max_hw_sectors(ns->queue, dev->max_hw_sectors); + if (dev->stripe_size) + blk_queue_stripe_sectors(ns->queue, dev->stripe_size >> 9); + blk_queue_segment_offset(ns->queue, PAGE_SIZE - 1); disk->major = nvme_major; disk->first_minor = 0; diff --git a/include/linux/bio.h b/include/linux/bio.h index 5a4d39b..7dbbd03 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -186,6 +186,11 @@ static inline void *bio_data(struct bio *bio) #define BIOVEC_SEG_BOUNDARY(q, b1, b2) \ __BIO_SEG_BOUNDARY(bvec_to_phys((b1)), bvec_to_phys((b2)) + (b2)->bv_len, queue_segment_boundary((q))) +#define __BIO_SEG_OFFSET(addr1, addr2, mask) \ + (((addr1) & (mask)) || (((addr2) & (mask)))) +#define BIOVEC_SEG_OFFSET(q, b1, b2) \ + __BIO_SEG_OFFSET(bvec_to_phys((b1)), bvec_to_phys((b2)) + (b2)->bv_len, queue_segment_offset((q))) + #define bio_io_error(bio) bio_endio((bio), -EIO) /* diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 95e789d..08cad22 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -267,9 +267,11 @@ struct blk_queue_tag { struct queue_limits { unsigned long bounce_pfn; unsigned long seg_boundary_mask; + unsigned long seg_offset_mask; unsigned int max_hw_sectors; unsigned int max_sectors; + unsigned int stripe_sectors; unsigned int max_segment_size; unsigned int physical_block_size; unsigned int alignment_offset; @@ -987,6 +989,8 @@ extern int blk_queue_dma_drain(struct request_queue *q, void *buf, unsigned int size); extern void blk_queue_lld_busy(struct request_queue *q, lld_busy_fn *fn); extern void blk_queue_segment_boundary(struct request_queue *, unsigned long); +extern void blk_queue_segment_offset(struct request_queue *, unsigned long); +extern void blk_queue_stripe_sectors(struct request_queue *, unsigned int); extern void blk_queue_prep_rq(struct request_queue *, prep_rq_fn *pfn); extern void blk_queue_unprep_rq(struct request_queue *, unprep_rq_fn *ufn); extern void blk_queue_merge_bvec(struct request_queue *, merge_bvec_fn *); @@ -999,6 +1003,8 @@ extern void blk_queue_flush(struct request_queue *q, unsigned int flush); extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable); extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev); +extern int blk_bios_map_sg(struct request_queue *q, struct bio *bio, struct scatterlist *sglist, + struct scatterlist **sg); extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *); extern int blk_bio_map_sg(struct request_queue *q, struct bio *bio, struct scatterlist *sglist); @@ -1159,11 +1165,21 @@ static inline unsigned long queue_segment_boundary(struct request_queue *q) return q->limits.seg_boundary_mask; } +static inline unsigned long queue_segment_offset(struct request_queue *q) +{ + return q->limits.seg_offset_mask; +} + static inline unsigned int queue_max_sectors(struct request_queue *q) { return q->limits.max_sectors; } +static inline unsigned int queue_stripe_sectors(struct request_queue *q) +{ + return q->limits.stripe_sectors; +} + static inline unsigned int queue_max_hw_sectors(struct request_queue *q) { return q->limits.max_hw_sectors; ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-02-26 23:39 ` [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios Kent Overstreet 2014-02-27 17:22 ` Matthew Wilcox @ 2014-03-02 20:31 ` Muthu Kumar 2014-03-02 20:50 ` Muthu Kumar 1 sibling, 1 reply; 8+ messages in thread From: Muthu Kumar @ 2014-03-02 20:31 UTC (permalink / raw) Kent, The blk_queue_split(), splits a bio into at most two bios right? So, if the original bio spans larger space than two bios can cover (restriction by the lower driver in the stack), this might not work? Am I reading it incorrectly? Thanks! Regards, Muthu On Wed, Feb 26, 2014@3:39 PM, Kent Overstreet <kmo@daterainc.com> wrote: > The way the block layer is currently written, it goes to great lengths > to avoid having to split bios; upper layer code (such as bio_add_page()) > checks what the underlying device can handle and tries to always create > bios that don't need to be split. > > But this approach becomes unwieldy and eventually breaks down with > stacked devices and devices with dynamic limits, and it adds a lot of > complexity. If the block layer could split bios as needed, we could > eliminate a lot of complexity elsewhere - particularly in stacked > drivers. Code that creates bios can then create whatever size bios are > convenient, and more importantly stacked drivers don't have to deal with > both their own bio size limitations and the limitations of the > (potentially multiple) devices underneath them. In the future this will > let us delete merge_bvec_fn and a bunch of other code. > > We do this by adding calls to blk_queue_split() to the various > make_request functions that need it - a few can already handle arbitrary > size bios. Note that we add the call _after_ any call to blk_queue_bounce(); > this means that blk_queue_split() and blk_recalc_rq_segments() don't need to be > concerned with bouncing affecting segment merging. > > Some make_request_fns were simple enough to audit and verify they don't > need blk_queue_split() calls. The skipped ones are: > > * nfhd_make_request (arch/m68k/emu/nfblock.c) > * axon_ram_make_request (arch/powerpc/sysdev/axonram.c) > * simdisk_make_request (arch/xtensa/platforms/iss/simdisk.c) > * brd_make_request (ramdisk - drivers/block/brd.c) > * loop_make_request > * null_queue_bio > * bcache's make_request fns > > Some others are almost certainly safe to remove now, but will be left for future > patches. > > Signed-off-by: Kent Overstreet <kmo at daterainc.com> > Cc: Jens Axboe <axboe at kernel.dk> > Cc: Neil Brown <neilb at suse.de> > Cc: Alasdair Kergon <agk at redhat.com> > Cc: dm-devel at redhat.com > Cc: Lars Ellenberg <drbd-dev at lists.linbit.com> > Cc: drbd-user at lists.linbit.com > Cc: Asai Thambi S P <asamymuthupa at micron.com> > Cc: Sam Bradshaw <sbradshaw at micron.com> > Cc: Matthew Wilcox <willy at linux.intel.com> > Cc: linux-nvme at lists.infradead.org > Cc: Jiri Kosina <jkosina at suse.cz> > Cc: Geoff Levand <geoff at infradead.org> > Cc: Jim Paris <jim at jtan.com> > Cc: Joshua Morris <josh.h.morris at us.ibm.com> > Cc: Philip Kelleher <pjk1939 at linux.vnet.ibm.com> > Cc: Minchan Kim <minchan at kernel.org> > Cc: Nitin Gupta <ngupta at vflare.org> > Cc: Martin Schwidefsky <schwidefsky at de.ibm.com> > Cc: Heiko Carstens <heiko.carstens at de.ibm.com> > Cc: Peng Tao <bergwolf at gmail.com> > --- > block/blk-core.c | 19 ++-- > block/blk-merge.c | 150 ++++++++++++++++++++++++++-- > block/blk-mq.c | 2 + > drivers/block/drbd/drbd_req.c | 2 + > drivers/block/mtip32xx/mtip32xx.c | 6 +- > drivers/block/nvme-core.c | 2 + > drivers/block/pktcdvd.c | 6 +- > drivers/block/ps3vram.c | 2 + > drivers/block/rsxx/dev.c | 2 + > drivers/block/umem.c | 2 + > drivers/block/zram/zram_drv.c | 2 + > drivers/md/dm.c | 2 + > drivers/md/md.c | 2 + > drivers/s390/block/dcssblk.c | 2 + > drivers/s390/block/xpram.c | 2 + > drivers/staging/lustre/lustre/llite/lloop.c | 2 + > include/linux/blkdev.h | 3 + > 17 files changed, 185 insertions(+), 23 deletions(-) > > diff --git a/block/blk-core.c b/block/blk-core.c > index 853f927492..d3b0782ec3 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -581,6 +581,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) > if (q->id < 0) > goto fail_c; > > + q->bio_split = bioset_create(4, 0); > + if (!q->bio_split) > + goto fail_id; > + > q->backing_dev_info.ra_pages = > (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE; > q->backing_dev_info.state = 0; > @@ -590,7 +594,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) > > err = bdi_init(&q->backing_dev_info); > if (err) > - goto fail_id; > + goto fail_split; > > setup_timer(&q->backing_dev_info.laptop_mode_wb_timer, > laptop_mode_timer_fn, (unsigned long) q); > @@ -635,6 +639,8 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) > > fail_bdi: > bdi_destroy(&q->backing_dev_info); > +fail_split: > + bioset_free(q->bio_split); > fail_id: > ida_simple_remove(&blk_queue_ida, q->id); > fail_c: > @@ -1501,6 +1507,8 @@ void blk_queue_bio(struct request_queue *q, struct bio *bio) > struct request *req; > unsigned int request_count = 0; > > + blk_queue_split(q, &bio, q->bio_split); > + > /* > * low level driver can indicate that it wants pages above a > * certain limit bounced to low memory (ie for highmem, or even > @@ -1723,15 +1731,6 @@ generic_make_request_checks(struct bio *bio) > goto end_io; > } > > - if (likely(bio_is_rw(bio) && > - nr_sectors > queue_max_hw_sectors(q))) { > - printk(KERN_ERR "bio too big device %s (%u > %u)\n", > - bdevname(bio->bi_bdev, b), > - bio_sectors(bio), > - queue_max_hw_sectors(q)); > - goto end_io; > - } > - > part = bio->bi_bdev->bd_part; > if (should_fail_request(part, bio->bi_iter.bi_size) || > should_fail_request(&part_to_disk(part)->part0, > diff --git a/block/blk-merge.c b/block/blk-merge.c > index 6c583f9c5b..0afbe3f1c2 100644 > --- a/block/blk-merge.c > +++ b/block/blk-merge.c > @@ -9,11 +9,149 @@ > > #include "blk.h" > > +static struct bio *blk_bio_discard_split(struct request_queue *q, > + struct bio *bio, > + struct bio_set *bs) > +{ > + unsigned int max_discard_sectors, granularity; > + int alignment; > + sector_t tmp; > + unsigned split_sectors; > + > + /* Zero-sector (unknown) and one-sector granularities are the same. */ > + granularity = max(q->limits.discard_granularity >> 9, 1U); > + > + max_discard_sectors = min(q->limits.max_discard_sectors, UINT_MAX >> 9); > + max_discard_sectors -= max_discard_sectors % granularity; > + > + if (unlikely(!max_discard_sectors)) { > + /* XXX: warn */ > + return NULL; > + } > + > + if (bio_sectors(bio) <= max_discard_sectors) > + return NULL; > + > + split_sectors = max_discard_sectors; > + > + /* > + * If the next starting sector would be misaligned, stop the discard at > + * the previous aligned sector. > + */ > + alignment = (q->limits.discard_alignment >> 9) % granularity; > + > + tmp = bio->bi_iter.bi_sector + split_sectors - alignment; > + tmp = sector_div(tmp, granularity); > + > + if (split_sectors > tmp) > + split_sectors -= tmp; > + > + return bio_split(bio, split_sectors, GFP_NOIO, bs); > +} > + > +static struct bio *blk_bio_write_same_split(struct request_queue *q, > + struct bio *bio, > + struct bio_set *bs) > +{ > + if (!q->limits.max_write_same_sectors) > + return NULL; > + > + if (bio_sectors(bio) <= q->limits.max_write_same_sectors) > + return NULL; > + > + return bio_split(bio, q->limits.max_write_same_sectors, GFP_NOIO, bs); > +} > + > +static struct bio *blk_bio_segment_split(struct request_queue *q, > + struct bio *bio, > + struct bio_set *bs) > +{ > + struct bio *split; > + struct bio_vec bv, bvprv; > + struct bvec_iter iter; > + unsigned seg_size = 0, nsegs = 0; > + int prev = 0; > + > + struct bvec_merge_data bvm = { > + .bi_bdev = bio->bi_bdev, > + .bi_sector = bio->bi_iter.bi_sector, > + .bi_size = 0, > + .bi_rw = bio->bi_rw, > + }; > + > + bio_for_each_segment(bv, bio, iter) { > + if (q->merge_bvec_fn && > + q->merge_bvec_fn(q, &bvm, &bv) < (int) bv.bv_len) > + goto split; > + > + bvm.bi_size += bv.bv_len; > + > + if (bvm.bi_size >> 9 > queue_max_sectors(q)) > + goto split; > + > + if (prev && blk_queue_cluster(q)) { > + if (seg_size + bv.bv_len > queue_max_segment_size(q)) > + goto new_segment; > + if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bv)) > + goto new_segment; > + if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bv)) > + goto new_segment; > + > + seg_size += bv.bv_len; > + bvprv = bv; > + prev = 1; > + continue; > + } > +new_segment: > + if (nsegs == queue_max_segments(q)) > + goto split; > + > + nsegs++; > + bvprv = bv; > + prev = 1; > + seg_size = bv.bv_len; > + } > + > + return NULL; > +split: > + split = bio_clone_bioset(bio, GFP_NOIO, bs); > + > + split->bi_iter.bi_size -= iter.bi_size; > + bio->bi_iter = iter; > + > + if (bio_integrity(bio)) { > + bio_integrity_advance(bio, split->bi_iter.bi_size); > + bio_integrity_trim(split, 0, bio_sectors(split)); > + } > + > + return split; > +} > + > +void blk_queue_split(struct request_queue *q, struct bio **bio, > + struct bio_set *bs) > +{ > + struct bio *split; > + > + if ((*bio)->bi_rw & REQ_DISCARD) > + split = blk_bio_discard_split(q, *bio, bs); > + else if ((*bio)->bi_rw & REQ_WRITE_SAME) > + split = blk_bio_write_same_split(q, *bio, bs); > + else > + split = blk_bio_segment_split(q, *bio, q->bio_split); > + > + if (split) { > + bio_chain(split, *bio); > + generic_make_request(*bio); > + *bio = split; > + } > +} > +EXPORT_SYMBOL(blk_queue_split); > + > static unsigned int __blk_recalc_rq_segments(struct request_queue *q, > struct bio *bio) > { > struct bio_vec bv, bvprv = { NULL }; > - int cluster, high, highprv = 1; > + int cluster, prev = 0; > unsigned int seg_size, nr_phys_segs; > struct bio *fbio, *bbio; > struct bvec_iter iter; > @@ -37,13 +175,7 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q, > nr_phys_segs = 0; > for_each_bio(bio) { > bio_for_each_segment(bv, bio, iter) { > - /* > - * the trick here is making sure that a high page is > - * never considered part of another segment, since that > - * might change with the bounce page. > - */ > - high = page_to_pfn(bv.bv_page) > queue_bounce_pfn(q); > - if (!high && !highprv && cluster) { > + if (prev && cluster) { > if (seg_size + bv.bv_len > > queue_max_segment_size(q)) > goto new_segment; > @@ -63,8 +195,8 @@ new_segment: > > nr_phys_segs++; > bvprv = bv; > + prev = 1; > seg_size = bv.bv_len; > - highprv = high; > } > bbio = bio; > } > diff --git a/block/blk-mq.c b/block/blk-mq.c > index 6468a715a0..7893e254d8 100644 > --- a/block/blk-mq.c > +++ b/block/blk-mq.c > @@ -915,6 +915,8 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) > return; > } > > + blk_queue_split(q, &bio, q->bio_split); > + > if (use_plug && blk_attempt_plug_merge(q, bio, &request_count)) > return; > > diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c > index 104a040f24..941a69c50c 100644 > --- a/drivers/block/drbd/drbd_req.c > +++ b/drivers/block/drbd/drbd_req.c > @@ -1275,6 +1275,8 @@ void drbd_make_request(struct request_queue *q, struct bio *bio) > struct drbd_conf *mdev = (struct drbd_conf *) q->queuedata; > unsigned long start_time; > > + blk_queue_split(q, &bio, q->bio_split); > + > start_time = jiffies; > > /* > diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c > index 516026954b..df733ca685 100644 > --- a/drivers/block/mtip32xx/mtip32xx.c > +++ b/drivers/block/mtip32xx/mtip32xx.c > @@ -4033,6 +4033,10 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) > int nents = 0; > int tag = 0, unaligned = 0; > > + blk_queue_bounce(queue, &bio); > + > + blk_queue_split(queue, &bio, queue->bio_split); > + > if (unlikely(dd->dd_flag & MTIP_DDF_STOP_IO)) { > if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, > &dd->dd_flag))) { > @@ -4082,8 +4086,6 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) > > sg = mtip_hw_get_scatterlist(dd, &tag, unaligned); > if (likely(sg != NULL)) { > - blk_queue_bounce(queue, &bio); > - > if (unlikely((bio)->bi_vcnt > MTIP_MAX_SG)) { > dev_warn(&dd->pdev->dev, > "Maximum number of SGL entries exceeded\n"); > diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c > index 51824d1f23..e4376b9613 100644 > --- a/drivers/block/nvme-core.c > +++ b/drivers/block/nvme-core.c > @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) > struct nvme_queue *nvmeq = get_nvmeq(ns->dev); > int result = -EBUSY; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (!nvmeq) { > put_nvmeq(NULL); > bio_endio(bio, -EIO); > diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c > index a2af73db18..a37acf722b 100644 > --- a/drivers/block/pktcdvd.c > +++ b/drivers/block/pktcdvd.c > @@ -2444,6 +2444,10 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) > char b[BDEVNAME_SIZE]; > struct bio *split; > > + blk_queue_bounce(q, &bio); > + > + blk_queue_split(q, &bio, q->bio_split); > + > pd = q->queuedata; > if (!pd) { > pr_err("%s incorrect request queue\n", > @@ -2474,8 +2478,6 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) > goto end_io; > } > > - blk_queue_bounce(q, &bio); > - > do { > sector_t zone = get_zone(bio->bi_iter.bi_sector, pd); > sector_t last_zone = get_zone(bio_end_sector(bio) - 1, pd); > diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c > index ef45cfb98f..a995972961 100644 > --- a/drivers/block/ps3vram.c > +++ b/drivers/block/ps3vram.c > @@ -603,6 +603,8 @@ static void ps3vram_make_request(struct request_queue *q, struct bio *bio) > struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev); > int busy; > > + blk_queue_split(q, &bio, q->bio_split); > + > dev_dbg(&dev->core, "%s\n", __func__); > > spin_lock_irq(&priv->lock); > diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c > index 2839d37e5a..ff074a3cd4 100644 > --- a/drivers/block/rsxx/dev.c > +++ b/drivers/block/rsxx/dev.c > @@ -169,6 +169,8 @@ static void rsxx_make_request(struct request_queue *q, struct bio *bio) > struct rsxx_bio_meta *bio_meta; > int st = -EINVAL; > > + blk_queue_split(q, &bio, q->bio_split); > + > might_sleep(); > > if (!card) > diff --git a/drivers/block/umem.c b/drivers/block/umem.c > index 4cf81b5bf0..13d577cfbc 100644 > --- a/drivers/block/umem.c > +++ b/drivers/block/umem.c > @@ -531,6 +531,8 @@ static void mm_make_request(struct request_queue *q, struct bio *bio) > (unsigned long long)bio->bi_iter.bi_sector, > bio->bi_iter.bi_size); > > + blk_queue_split(q, &bio, q->bio_split); > + > spin_lock_irq(&card->lock); > *card->biotail = bio; > bio->bi_next = NULL; > diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c > index 011e55d820..ecf9daa01c 100644 > --- a/drivers/block/zram/zram_drv.c > +++ b/drivers/block/zram/zram_drv.c > @@ -733,6 +733,8 @@ static void zram_make_request(struct request_queue *queue, struct bio *bio) > { > struct zram *zram = queue->queuedata; > > + blk_queue_split(queue, &bio, queue->bio_split); > + > down_read(&zram->init_lock); > if (unlikely(!zram->init_done)) > goto error; > diff --git a/drivers/md/dm.c b/drivers/md/dm.c > index 8c53b09b9a..97f70420f2 100644 > --- a/drivers/md/dm.c > +++ b/drivers/md/dm.c > @@ -1500,6 +1500,8 @@ static void dm_request(struct request_queue *q, struct bio *bio) > { > struct mapped_device *md = q->queuedata; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (dm_request_based(md)) > blk_queue_bio(q, bio); > else > diff --git a/drivers/md/md.c b/drivers/md/md.c > index 4ad5cc4e63..1421bc3f7b 100644 > --- a/drivers/md/md.c > +++ b/drivers/md/md.c > @@ -256,6 +256,8 @@ static void md_make_request(struct request_queue *q, struct bio *bio) > int cpu; > unsigned int sectors; > > + blk_queue_split(q, &bio, q->bio_split); > + > if (mddev == NULL || mddev->pers == NULL > || !mddev->ready) { > bio_io_error(bio); > diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c > index ebf41e228e..db33cd3e4c 100644 > --- a/drivers/s390/block/dcssblk.c > +++ b/drivers/s390/block/dcssblk.c > @@ -815,6 +815,8 @@ dcssblk_make_request(struct request_queue *q, struct bio *bio) > unsigned long source_addr; > unsigned long bytes_done; > > + blk_queue_split(q, &bio, q->bio_split); > + > bytes_done = 0; > dev_info = bio->bi_bdev->bd_disk->private_data; > if (dev_info == NULL) > diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c > index 6969d39f1e..f03c103f13 100644 > --- a/drivers/s390/block/xpram.c > +++ b/drivers/s390/block/xpram.c > @@ -190,6 +190,8 @@ static void xpram_make_request(struct request_queue *q, struct bio *bio) > unsigned long page_addr; > unsigned long bytes; > > + blk_queue_split(q, &bio, q->bio_split); > + > if ((bio->bi_iter.bi_sector & 7) != 0 || > (bio->bi_iter.bi_size & 4095) != 0) > /* Request is not page-aligned. */ > diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c > index 0718905ade..a3f6dc930b 100644 > --- a/drivers/staging/lustre/lustre/llite/lloop.c > +++ b/drivers/staging/lustre/lustre/llite/lloop.c > @@ -344,6 +344,8 @@ static void loop_make_request(struct request_queue *q, struct bio *old_bio) > int rw = bio_rw(old_bio); > int inactive; > > + blk_queue_split(q, &old_bio, q->bio_split); > + > if (!lo) > goto err; > > diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h > index 1e1fa3f93d..99e9955c4d 100644 > --- a/include/linux/blkdev.h > +++ b/include/linux/blkdev.h > @@ -470,6 +470,7 @@ struct request_queue { > wait_queue_head_t mq_freeze_wq; > struct percpu_counter mq_usage_counter; > struct list_head all_q_node; > + struct bio_set *bio_split; > }; > > #define QUEUE_FLAG_QUEUED 1 /* uses generic tag queueing */ > @@ -781,6 +782,8 @@ extern void blk_rq_unprep_clone(struct request *rq); > extern int blk_insert_cloned_request(struct request_queue *q, > struct request *rq); > extern void blk_delay_queue(struct request_queue *, unsigned long); > +extern void blk_queue_split(struct request_queue *, struct bio **, > + struct bio_set *); > extern void blk_recount_segments(struct request_queue *, struct bio *); > extern int scsi_verify_blk_ioctl(struct block_device *, unsigned int); > extern int scsi_cmd_blk_ioctl(struct block_device *, fmode_t, > -- > 1.9.0 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios 2014-03-02 20:31 ` Muthu Kumar @ 2014-03-02 20:50 ` Muthu Kumar 0 siblings, 0 replies; 8+ messages in thread From: Muthu Kumar @ 2014-03-02 20:50 UTC (permalink / raw) Never mind... The following code covers it: + if (split) { + bio_chain(split, *bio); + generic_make_request(*bio); + *bio = split; + } My other question is, can we avoid calling the queue_split from individual drivers make_request()? Can we move the functionality into generic_make_request()? Thanks. Regards, Muthu On Sun, Mar 2, 2014@12:31 PM, Muthu Kumar <muthu.lkml@gmail.com> wrote: > Kent, > The blk_queue_split(), splits a bio into at most two bios right? So, > if the original bio spans larger space than two bios can cover > (restriction by the lower driver in the stack), this might not work? > Am I reading it incorrectly? > > Thanks! > > Regards, > Muthu > > > > On Wed, Feb 26, 2014@3:39 PM, Kent Overstreet <kmo@daterainc.com> wrote: >> The way the block layer is currently written, it goes to great lengths >> to avoid having to split bios; upper layer code (such as bio_add_page()) >> checks what the underlying device can handle and tries to always create >> bios that don't need to be split. >> >> But this approach becomes unwieldy and eventually breaks down with >> stacked devices and devices with dynamic limits, and it adds a lot of >> complexity. If the block layer could split bios as needed, we could >> eliminate a lot of complexity elsewhere - particularly in stacked >> drivers. Code that creates bios can then create whatever size bios are >> convenient, and more importantly stacked drivers don't have to deal with >> both their own bio size limitations and the limitations of the >> (potentially multiple) devices underneath them. In the future this will >> let us delete merge_bvec_fn and a bunch of other code. >> >> We do this by adding calls to blk_queue_split() to the various >> make_request functions that need it - a few can already handle arbitrary >> size bios. Note that we add the call _after_ any call to blk_queue_bounce(); >> this means that blk_queue_split() and blk_recalc_rq_segments() don't need to be >> concerned with bouncing affecting segment merging. >> >> Some make_request_fns were simple enough to audit and verify they don't >> need blk_queue_split() calls. The skipped ones are: >> >> * nfhd_make_request (arch/m68k/emu/nfblock.c) >> * axon_ram_make_request (arch/powerpc/sysdev/axonram.c) >> * simdisk_make_request (arch/xtensa/platforms/iss/simdisk.c) >> * brd_make_request (ramdisk - drivers/block/brd.c) >> * loop_make_request >> * null_queue_bio >> * bcache's make_request fns >> >> Some others are almost certainly safe to remove now, but will be left for future >> patches. >> >> Signed-off-by: Kent Overstreet <kmo at daterainc.com> >> Cc: Jens Axboe <axboe at kernel.dk> >> Cc: Neil Brown <neilb at suse.de> >> Cc: Alasdair Kergon <agk at redhat.com> >> Cc: dm-devel at redhat.com >> Cc: Lars Ellenberg <drbd-dev at lists.linbit.com> >> Cc: drbd-user at lists.linbit.com >> Cc: Asai Thambi S P <asamymuthupa at micron.com> >> Cc: Sam Bradshaw <sbradshaw at micron.com> >> Cc: Matthew Wilcox <willy at linux.intel.com> >> Cc: linux-nvme at lists.infradead.org >> Cc: Jiri Kosina <jkosina at suse.cz> >> Cc: Geoff Levand <geoff at infradead.org> >> Cc: Jim Paris <jim at jtan.com> >> Cc: Joshua Morris <josh.h.morris at us.ibm.com> >> Cc: Philip Kelleher <pjk1939 at linux.vnet.ibm.com> >> Cc: Minchan Kim <minchan at kernel.org> >> Cc: Nitin Gupta <ngupta at vflare.org> >> Cc: Martin Schwidefsky <schwidefsky at de.ibm.com> >> Cc: Heiko Carstens <heiko.carstens at de.ibm.com> >> Cc: Peng Tao <bergwolf at gmail.com> >> --- >> block/blk-core.c | 19 ++-- >> block/blk-merge.c | 150 ++++++++++++++++++++++++++-- >> block/blk-mq.c | 2 + >> drivers/block/drbd/drbd_req.c | 2 + >> drivers/block/mtip32xx/mtip32xx.c | 6 +- >> drivers/block/nvme-core.c | 2 + >> drivers/block/pktcdvd.c | 6 +- >> drivers/block/ps3vram.c | 2 + >> drivers/block/rsxx/dev.c | 2 + >> drivers/block/umem.c | 2 + >> drivers/block/zram/zram_drv.c | 2 + >> drivers/md/dm.c | 2 + >> drivers/md/md.c | 2 + >> drivers/s390/block/dcssblk.c | 2 + >> drivers/s390/block/xpram.c | 2 + >> drivers/staging/lustre/lustre/llite/lloop.c | 2 + >> include/linux/blkdev.h | 3 + >> 17 files changed, 185 insertions(+), 23 deletions(-) >> >> diff --git a/block/blk-core.c b/block/blk-core.c >> index 853f927492..d3b0782ec3 100644 >> --- a/block/blk-core.c >> +++ b/block/blk-core.c >> @@ -581,6 +581,10 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) >> if (q->id < 0) >> goto fail_c; >> >> + q->bio_split = bioset_create(4, 0); >> + if (!q->bio_split) >> + goto fail_id; >> + >> q->backing_dev_info.ra_pages = >> (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE; >> q->backing_dev_info.state = 0; >> @@ -590,7 +594,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) >> >> err = bdi_init(&q->backing_dev_info); >> if (err) >> - goto fail_id; >> + goto fail_split; >> >> setup_timer(&q->backing_dev_info.laptop_mode_wb_timer, >> laptop_mode_timer_fn, (unsigned long) q); >> @@ -635,6 +639,8 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) >> >> fail_bdi: >> bdi_destroy(&q->backing_dev_info); >> +fail_split: >> + bioset_free(q->bio_split); >> fail_id: >> ida_simple_remove(&blk_queue_ida, q->id); >> fail_c: >> @@ -1501,6 +1507,8 @@ void blk_queue_bio(struct request_queue *q, struct bio *bio) >> struct request *req; >> unsigned int request_count = 0; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> /* >> * low level driver can indicate that it wants pages above a >> * certain limit bounced to low memory (ie for highmem, or even >> @@ -1723,15 +1731,6 @@ generic_make_request_checks(struct bio *bio) >> goto end_io; >> } >> >> - if (likely(bio_is_rw(bio) && >> - nr_sectors > queue_max_hw_sectors(q))) { >> - printk(KERN_ERR "bio too big device %s (%u > %u)\n", >> - bdevname(bio->bi_bdev, b), >> - bio_sectors(bio), >> - queue_max_hw_sectors(q)); >> - goto end_io; >> - } >> - >> part = bio->bi_bdev->bd_part; >> if (should_fail_request(part, bio->bi_iter.bi_size) || >> should_fail_request(&part_to_disk(part)->part0, >> diff --git a/block/blk-merge.c b/block/blk-merge.c >> index 6c583f9c5b..0afbe3f1c2 100644 >> --- a/block/blk-merge.c >> +++ b/block/blk-merge.c >> @@ -9,11 +9,149 @@ >> >> #include "blk.h" >> >> +static struct bio *blk_bio_discard_split(struct request_queue *q, >> + struct bio *bio, >> + struct bio_set *bs) >> +{ >> + unsigned int max_discard_sectors, granularity; >> + int alignment; >> + sector_t tmp; >> + unsigned split_sectors; >> + >> + /* Zero-sector (unknown) and one-sector granularities are the same. */ >> + granularity = max(q->limits.discard_granularity >> 9, 1U); >> + >> + max_discard_sectors = min(q->limits.max_discard_sectors, UINT_MAX >> 9); >> + max_discard_sectors -= max_discard_sectors % granularity; >> + >> + if (unlikely(!max_discard_sectors)) { >> + /* XXX: warn */ >> + return NULL; >> + } >> + >> + if (bio_sectors(bio) <= max_discard_sectors) >> + return NULL; >> + >> + split_sectors = max_discard_sectors; >> + >> + /* >> + * If the next starting sector would be misaligned, stop the discard at >> + * the previous aligned sector. >> + */ >> + alignment = (q->limits.discard_alignment >> 9) % granularity; >> + >> + tmp = bio->bi_iter.bi_sector + split_sectors - alignment; >> + tmp = sector_div(tmp, granularity); >> + >> + if (split_sectors > tmp) >> + split_sectors -= tmp; >> + >> + return bio_split(bio, split_sectors, GFP_NOIO, bs); >> +} >> + >> +static struct bio *blk_bio_write_same_split(struct request_queue *q, >> + struct bio *bio, >> + struct bio_set *bs) >> +{ >> + if (!q->limits.max_write_same_sectors) >> + return NULL; >> + >> + if (bio_sectors(bio) <= q->limits.max_write_same_sectors) >> + return NULL; >> + >> + return bio_split(bio, q->limits.max_write_same_sectors, GFP_NOIO, bs); >> +} >> + >> +static struct bio *blk_bio_segment_split(struct request_queue *q, >> + struct bio *bio, >> + struct bio_set *bs) >> +{ >> + struct bio *split; >> + struct bio_vec bv, bvprv; >> + struct bvec_iter iter; >> + unsigned seg_size = 0, nsegs = 0; >> + int prev = 0; >> + >> + struct bvec_merge_data bvm = { >> + .bi_bdev = bio->bi_bdev, >> + .bi_sector = bio->bi_iter.bi_sector, >> + .bi_size = 0, >> + .bi_rw = bio->bi_rw, >> + }; >> + >> + bio_for_each_segment(bv, bio, iter) { >> + if (q->merge_bvec_fn && >> + q->merge_bvec_fn(q, &bvm, &bv) < (int) bv.bv_len) >> + goto split; >> + >> + bvm.bi_size += bv.bv_len; >> + >> + if (bvm.bi_size >> 9 > queue_max_sectors(q)) >> + goto split; >> + >> + if (prev && blk_queue_cluster(q)) { >> + if (seg_size + bv.bv_len > queue_max_segment_size(q)) >> + goto new_segment; >> + if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bv)) >> + goto new_segment; >> + if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bv)) >> + goto new_segment; >> + >> + seg_size += bv.bv_len; >> + bvprv = bv; >> + prev = 1; >> + continue; >> + } >> +new_segment: >> + if (nsegs == queue_max_segments(q)) >> + goto split; >> + >> + nsegs++; >> + bvprv = bv; >> + prev = 1; >> + seg_size = bv.bv_len; >> + } >> + >> + return NULL; >> +split: >> + split = bio_clone_bioset(bio, GFP_NOIO, bs); >> + >> + split->bi_iter.bi_size -= iter.bi_size; >> + bio->bi_iter = iter; >> + >> + if (bio_integrity(bio)) { >> + bio_integrity_advance(bio, split->bi_iter.bi_size); >> + bio_integrity_trim(split, 0, bio_sectors(split)); >> + } >> + >> + return split; >> +} >> + >> +void blk_queue_split(struct request_queue *q, struct bio **bio, >> + struct bio_set *bs) >> +{ >> + struct bio *split; >> + >> + if ((*bio)->bi_rw & REQ_DISCARD) >> + split = blk_bio_discard_split(q, *bio, bs); >> + else if ((*bio)->bi_rw & REQ_WRITE_SAME) >> + split = blk_bio_write_same_split(q, *bio, bs); >> + else >> + split = blk_bio_segment_split(q, *bio, q->bio_split); >> + >> + if (split) { >> + bio_chain(split, *bio); >> + generic_make_request(*bio); >> + *bio = split; >> + } >> +} >> +EXPORT_SYMBOL(blk_queue_split); >> + >> static unsigned int __blk_recalc_rq_segments(struct request_queue *q, >> struct bio *bio) >> { >> struct bio_vec bv, bvprv = { NULL }; >> - int cluster, high, highprv = 1; >> + int cluster, prev = 0; >> unsigned int seg_size, nr_phys_segs; >> struct bio *fbio, *bbio; >> struct bvec_iter iter; >> @@ -37,13 +175,7 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q, >> nr_phys_segs = 0; >> for_each_bio(bio) { >> bio_for_each_segment(bv, bio, iter) { >> - /* >> - * the trick here is making sure that a high page is >> - * never considered part of another segment, since that >> - * might change with the bounce page. >> - */ >> - high = page_to_pfn(bv.bv_page) > queue_bounce_pfn(q); >> - if (!high && !highprv && cluster) { >> + if (prev && cluster) { >> if (seg_size + bv.bv_len >> > queue_max_segment_size(q)) >> goto new_segment; >> @@ -63,8 +195,8 @@ new_segment: >> >> nr_phys_segs++; >> bvprv = bv; >> + prev = 1; >> seg_size = bv.bv_len; >> - highprv = high; >> } >> bbio = bio; >> } >> diff --git a/block/blk-mq.c b/block/blk-mq.c >> index 6468a715a0..7893e254d8 100644 >> --- a/block/blk-mq.c >> +++ b/block/blk-mq.c >> @@ -915,6 +915,8 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) >> return; >> } >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> if (use_plug && blk_attempt_plug_merge(q, bio, &request_count)) >> return; >> >> diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c >> index 104a040f24..941a69c50c 100644 >> --- a/drivers/block/drbd/drbd_req.c >> +++ b/drivers/block/drbd/drbd_req.c >> @@ -1275,6 +1275,8 @@ void drbd_make_request(struct request_queue *q, struct bio *bio) >> struct drbd_conf *mdev = (struct drbd_conf *) q->queuedata; >> unsigned long start_time; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> start_time = jiffies; >> >> /* >> diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c >> index 516026954b..df733ca685 100644 >> --- a/drivers/block/mtip32xx/mtip32xx.c >> +++ b/drivers/block/mtip32xx/mtip32xx.c >> @@ -4033,6 +4033,10 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) >> int nents = 0; >> int tag = 0, unaligned = 0; >> >> + blk_queue_bounce(queue, &bio); >> + >> + blk_queue_split(queue, &bio, queue->bio_split); >> + >> if (unlikely(dd->dd_flag & MTIP_DDF_STOP_IO)) { >> if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, >> &dd->dd_flag))) { >> @@ -4082,8 +4086,6 @@ static void mtip_make_request(struct request_queue *queue, struct bio *bio) >> >> sg = mtip_hw_get_scatterlist(dd, &tag, unaligned); >> if (likely(sg != NULL)) { >> - blk_queue_bounce(queue, &bio); >> - >> if (unlikely((bio)->bi_vcnt > MTIP_MAX_SG)) { >> dev_warn(&dd->pdev->dev, >> "Maximum number of SGL entries exceeded\n"); >> diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c >> index 51824d1f23..e4376b9613 100644 >> --- a/drivers/block/nvme-core.c >> +++ b/drivers/block/nvme-core.c >> @@ -737,6 +737,8 @@ static void nvme_make_request(struct request_queue *q, struct bio *bio) >> struct nvme_queue *nvmeq = get_nvmeq(ns->dev); >> int result = -EBUSY; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> if (!nvmeq) { >> put_nvmeq(NULL); >> bio_endio(bio, -EIO); >> diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c >> index a2af73db18..a37acf722b 100644 >> --- a/drivers/block/pktcdvd.c >> +++ b/drivers/block/pktcdvd.c >> @@ -2444,6 +2444,10 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) >> char b[BDEVNAME_SIZE]; >> struct bio *split; >> >> + blk_queue_bounce(q, &bio); >> + >> + blk_queue_split(q, &bio, q->bio_split); >> + >> pd = q->queuedata; >> if (!pd) { >> pr_err("%s incorrect request queue\n", >> @@ -2474,8 +2478,6 @@ static void pkt_make_request(struct request_queue *q, struct bio *bio) >> goto end_io; >> } >> >> - blk_queue_bounce(q, &bio); >> - >> do { >> sector_t zone = get_zone(bio->bi_iter.bi_sector, pd); >> sector_t last_zone = get_zone(bio_end_sector(bio) - 1, pd); >> diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c >> index ef45cfb98f..a995972961 100644 >> --- a/drivers/block/ps3vram.c >> +++ b/drivers/block/ps3vram.c >> @@ -603,6 +603,8 @@ static void ps3vram_make_request(struct request_queue *q, struct bio *bio) >> struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev); >> int busy; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> dev_dbg(&dev->core, "%s\n", __func__); >> >> spin_lock_irq(&priv->lock); >> diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c >> index 2839d37e5a..ff074a3cd4 100644 >> --- a/drivers/block/rsxx/dev.c >> +++ b/drivers/block/rsxx/dev.c >> @@ -169,6 +169,8 @@ static void rsxx_make_request(struct request_queue *q, struct bio *bio) >> struct rsxx_bio_meta *bio_meta; >> int st = -EINVAL; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> might_sleep(); >> >> if (!card) >> diff --git a/drivers/block/umem.c b/drivers/block/umem.c >> index 4cf81b5bf0..13d577cfbc 100644 >> --- a/drivers/block/umem.c >> +++ b/drivers/block/umem.c >> @@ -531,6 +531,8 @@ static void mm_make_request(struct request_queue *q, struct bio *bio) >> (unsigned long long)bio->bi_iter.bi_sector, >> bio->bi_iter.bi_size); >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> spin_lock_irq(&card->lock); >> *card->biotail = bio; >> bio->bi_next = NULL; >> diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c >> index 011e55d820..ecf9daa01c 100644 >> --- a/drivers/block/zram/zram_drv.c >> +++ b/drivers/block/zram/zram_drv.c >> @@ -733,6 +733,8 @@ static void zram_make_request(struct request_queue *queue, struct bio *bio) >> { >> struct zram *zram = queue->queuedata; >> >> + blk_queue_split(queue, &bio, queue->bio_split); >> + >> down_read(&zram->init_lock); >> if (unlikely(!zram->init_done)) >> goto error; >> diff --git a/drivers/md/dm.c b/drivers/md/dm.c >> index 8c53b09b9a..97f70420f2 100644 >> --- a/drivers/md/dm.c >> +++ b/drivers/md/dm.c >> @@ -1500,6 +1500,8 @@ static void dm_request(struct request_queue *q, struct bio *bio) >> { >> struct mapped_device *md = q->queuedata; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> if (dm_request_based(md)) >> blk_queue_bio(q, bio); >> else >> diff --git a/drivers/md/md.c b/drivers/md/md.c >> index 4ad5cc4e63..1421bc3f7b 100644 >> --- a/drivers/md/md.c >> +++ b/drivers/md/md.c >> @@ -256,6 +256,8 @@ static void md_make_request(struct request_queue *q, struct bio *bio) >> int cpu; >> unsigned int sectors; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> if (mddev == NULL || mddev->pers == NULL >> || !mddev->ready) { >> bio_io_error(bio); >> diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c >> index ebf41e228e..db33cd3e4c 100644 >> --- a/drivers/s390/block/dcssblk.c >> +++ b/drivers/s390/block/dcssblk.c >> @@ -815,6 +815,8 @@ dcssblk_make_request(struct request_queue *q, struct bio *bio) >> unsigned long source_addr; >> unsigned long bytes_done; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> bytes_done = 0; >> dev_info = bio->bi_bdev->bd_disk->private_data; >> if (dev_info == NULL) >> diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c >> index 6969d39f1e..f03c103f13 100644 >> --- a/drivers/s390/block/xpram.c >> +++ b/drivers/s390/block/xpram.c >> @@ -190,6 +190,8 @@ static void xpram_make_request(struct request_queue *q, struct bio *bio) >> unsigned long page_addr; >> unsigned long bytes; >> >> + blk_queue_split(q, &bio, q->bio_split); >> + >> if ((bio->bi_iter.bi_sector & 7) != 0 || >> (bio->bi_iter.bi_size & 4095) != 0) >> /* Request is not page-aligned. */ >> diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c >> index 0718905ade..a3f6dc930b 100644 >> --- a/drivers/staging/lustre/lustre/llite/lloop.c >> +++ b/drivers/staging/lustre/lustre/llite/lloop.c >> @@ -344,6 +344,8 @@ static void loop_make_request(struct request_queue *q, struct bio *old_bio) >> int rw = bio_rw(old_bio); >> int inactive; >> >> + blk_queue_split(q, &old_bio, q->bio_split); >> + >> if (!lo) >> goto err; >> >> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h >> index 1e1fa3f93d..99e9955c4d 100644 >> --- a/include/linux/blkdev.h >> +++ b/include/linux/blkdev.h >> @@ -470,6 +470,7 @@ struct request_queue { >> wait_queue_head_t mq_freeze_wq; >> struct percpu_counter mq_usage_counter; >> struct list_head all_q_node; >> + struct bio_set *bio_split; >> }; >> >> #define QUEUE_FLAG_QUEUED 1 /* uses generic tag queueing */ >> @@ -781,6 +782,8 @@ extern void blk_rq_unprep_clone(struct request *rq); >> extern int blk_insert_cloned_request(struct request_queue *q, >> struct request *rq); >> extern void blk_delay_queue(struct request_queue *, unsigned long); >> +extern void blk_queue_split(struct request_queue *, struct bio **, >> + struct bio_set *); >> extern void blk_recount_segments(struct request_queue *, struct bio *); >> extern int scsi_verify_blk_ioctl(struct block_device *, unsigned int); >> extern int scsi_cmd_blk_ioctl(struct block_device *, fmode_t, >> -- >> 1.9.0 >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo at vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-03-13 23:33 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1393457997-17618-1-git-send-email-kmo@daterainc.com>
2014-02-26 23:39 ` [PATCH 1/9] block: Make generic_make_request handle arbitrary sized bios Kent Overstreet
2014-02-27 17:22 ` Matthew Wilcox
2014-02-27 21:27 ` Kent Overstreet
2014-02-28 23:30 ` Kent Overstreet
2014-03-01 17:52 ` Keith Busch
2014-03-13 23:33 ` Keith Busch
2014-03-02 20:31 ` Muthu Kumar
2014-03-02 20:50 ` Muthu Kumar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox