From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=95.215.58.38 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794170883; b=MjKsKZNPMTrOYAPxu4csE3jj04sApgN/jQKl3I/Qrz0TN4iixB0OQi9pZcL1+jN+t+Iy PrBQQA/WdtXlZoKGTe/EDlFP38mIHV2fX66h7pA014EUEmpiX68W/lN62TD1Z7mNACT/j yj/WCnTyfQicvKVPqcyFDP1i8DmaI4aGCaFmNDxEM+mJOO+1tWZeflsmYRvFO5GYkbqZv VlUfm8qC/UFjfCx2AGG3l4AbqKrmqro0s7Kqv70h1ZGfmtuwZ5CXxtV2owYB3mwTAf/v3 gu033ZlkVg9R2+kIQJoogpSJep+Dkf534z8NzIG0cz8v4e/3E0ynVEkbimTMxIVZthQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794170883; h=DMARC-Filter:X-Envelope-To:DKIM-Signature:X-Envelope-To:Received: X-Mizu-Trace-ID:X-Migadu-Flow:From:To:Cc:Subject:Date:Message-ID: X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=ccCEPTXfeTW7gEQlR9QKMUHkSDrza9/l9sNjHCD8T/s=; b=i7u6rlD5DLrEVSG6bXHi9bQLAmn2UwTqu/IholtIpDZJU75tL9MNLF/PfIiwfoiq0VU4 tW7/PbrvQs8WQUyWXhPaXg2FhjsESOUfr1mVKT9I/WKfsV5buzk2kjnPTXvhRTf8/oPDW anYNFf07Fe+nGfGtmp+Udcaqtdeh52jd+d0IECJWX8u0F/Tpz9g7JIBDa/bEbcIbO1Ptn UBSrCHykw+0AJoAw3cE7GjPNxKCrv5rl9i+Sk7jrCMA90myxdxU9ZxO8YGCCdBnNdT9OV 575WHv12TbBBtkXLZsr9yv6CUA8I1YVN+Q+cvIjC7/gx274RpfOmpsW1TilJuuggENw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Qx58Uhvx; arc=none smtp.remote-ip=95.215.58.38 Received: from mta1.migadu.com (out-38.mta1.migadu.com [95.215.58.38]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 170EB606A331 for ; Sun, 8 Nov 2026 20:48:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 170EB606A331 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Qx58Uhvx X-Envelope-To: linux-scsi@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=91ZzruK9YdRqDB+8gQEmqkG0seV4+faEgDGR+QsaOUk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1794170881; v=1; x=1794775681; b=Qx58UhvxH/pPt7eIKrC9J6Ed3scRopS4F4xfBqxd3PlJ2X54k8a3XxfhWezlTZYB7dPIOu+Y WK/EwRV4sR4MeHAXqdj2KuIfL+MZ5qM3KKrE0nfzNtpDjS1Nl7rE0Yu4MeNdMZPXeibbPWwJqLM AfTbfjgWShkg9vTz/wvRrMtM= X-Envelope-To: linux-scsi@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id c93b221cbde1e43a; Sun, 08 Nov 2026 20:48:00 +0000 X-Mizu-Trace-ID: c93b221cbde1e43a X-Migadu-Flow: FLOW_OUT From: John Garry To: james.bottomley@hansenpartnership.com, mkp@kernel.org Cc: hare@suse.com, linux-nvme@lists.infradead.org, linux-scsi@vger.kernel.org, snitzer@kernel.org, jmeneghi@redhat.com, bmarzins@redhat.com, dm-devel@lists.linux.dev, hch@lst.de, michael.christie@oracle.com, John Garry Subject: [PATCH v9 05/29] libmultipath: Add bio handling Date: Sun, 8 Nov 2026 20:47:09 +0000 Message-ID: <20261108204733.557995-6-john.garry@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261108204733.557995-1-john.garry@linux.dev> References: <20261108204733.557995-1-john.garry@linux.dev> X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Add support to submit a bio per-path. In addition, for failover, add support to requeue a failed bio. NVMe has almost like-for-like equivalents here: - nvme_available_path() -> mpath_available_path() - nvme_requeue_work() -> mpath_requeue_work() - nvme_ns_head_submit_bio() -> mpath_bdev_submit_bio() For failover, a driver may want to re-submit a bio, so add support to clone a bio prior to submission. A bio which is submitted to a per-path device has flag REQ_MPATH set, same as what is done for NVMe with REQ_NVME_MPATH. Signed-off-by: John Garry --- include/linux/multipath.h | 22 ++++++++ lib/multipath.c | 108 +++++++++++++++++++++++++++++++++++++- 2 files changed, 128 insertions(+), 2 deletions(-) diff --git a/include/linux/multipath.h b/include/linux/multipath.h index 9bbab0aa8961..374f0531f7e0 100644 --- a/include/linux/multipath.h +++ b/include/linux/multipath.h @@ -3,6 +3,7 @@ #define _LIBMULTIPATH_H #include +#include #include extern const struct block_device_operations mpath_ops; @@ -29,8 +30,10 @@ struct mpath_device { }; struct mpath_head_template { + bool (*available_path)(struct mpath_device *); bool (*is_disabled)(struct mpath_device *); bool (*is_optimized)(struct mpath_device *); + struct bio *(*clone_bio)(struct bio *); const struct attribute_group **device_groups; }; @@ -44,6 +47,14 @@ struct mpath_head { refcount_t refcount; enum mpath_iopolicy_e *iopolicy; + + struct bio_list requeue_list; /* list for requeing bio */ + spinlock_t requeue_lock; + struct work_struct requeue_work; /* work struct for requeue */ + + atomic_long_t requeue_no_usable_path_cnt; + atomic_long_t fail_no_avail_path_cnt; + unsigned long flags; struct gendisk *disk; struct work_struct partition_scan_work; @@ -53,6 +64,13 @@ struct mpath_head { struct mpath_device __rcu *current_path[MAX_NUMNODES]; }; +#define REQ_MPATH REQ_DRV + +static inline bool is_mpath_request(struct request *req) +{ + return req->cmd_flags & REQ_MPATH; +} + static inline struct mpath_head *mpath_bd_device_to_head(struct device *dev) { return dev_get_drvdata(dev); @@ -96,4 +114,8 @@ static inline bool mpath_qd_iopolicy(enum mpath_iopolicy_e *iopolicy) return READ_ONCE(*iopolicy) == MPATH_IOPOLICY_QD; } +static inline void mpath_schedule_requeue_work(struct mpath_head *mpath_head) +{ + kblockd_schedule_work(&mpath_head->requeue_work); +} #endif // _LIBMULTIPATH_H diff --git a/lib/multipath.c b/lib/multipath.c index b8ea8478037e..5dc7994f8095 100644 --- a/lib/multipath.c +++ b/lib/multipath.c @@ -7,6 +7,7 @@ #include #include #include +#include static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head); @@ -45,7 +46,6 @@ int mpath_get_iopolicy(char *buf, int iopolicy) } EXPORT_SYMBOL_GPL(mpath_get_iopolicy); - void mpath_synchronize(struct mpath_head *mpath_head) { synchronize_srcu(&mpath_head->srcu); @@ -232,7 +232,6 @@ static struct mpath_device *mpath_numa_path(struct mpath_head *mpath_head) return mpath_device; } -__maybe_unused static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head) { enum mpath_iopolicy_e iopolicy = mpath_read_iopolicy(mpath_head); @@ -247,6 +246,81 @@ static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head) } } +static bool mpath_available_path(struct mpath_head *mpath_head) +{ + struct mpath_device *mpath_device; + + if (!test_bit(MPATH_HEAD_DISK_LIVE, &mpath_head->flags)) + return false; + + list_for_each_entry_srcu(mpath_device, &mpath_head->dev_list, siblings, + srcu_read_lock_held(&mpath_head->srcu)) { + if (mpath_head->mpdt->available_path(mpath_device)) + return true; + } + + return false; +} + +static void mpath_bdev_submit_bio(struct bio *bio) +{ + struct mpath_head *mpath_head = bio->bi_bdev->bd_disk->private_data; + struct device *dev = mpath_head->parent; + struct mpath_device *mpath_device; + int srcu_idx; + + /* + * The mpath_device might be going away and the bio might be moved to a + * different queue in failover, so we need to use the bio_split + * pool from the original queue to allocate the bvecs from. + */ + bio = bio_split_to_limits(bio); + if (!bio) + return; + + srcu_idx = srcu_read_lock(&mpath_head->srcu); + mpath_device = mpath_find_path(mpath_head); + + if (likely(mpath_device)) { + if (mpath_head->mpdt->clone_bio) { + struct bio *orig = bio; + + bio = mpath_head->mpdt->clone_bio(bio); + if (!bio) { + bio_io_error(orig); + goto out; + } + } + + bio_set_dev(bio, mpath_device->disk->part0); + /* + * Use BIO_REMAPPED to skip bio_check_eod() when this bio + * enters submit_bio_noacct() for the per-path device. The EOD + * check already passed on the multipath head. + */ + bio_set_flag(bio, BIO_REMAPPED); + bio->bi_opf |= REQ_MPATH; + trace_block_bio_remap(bio, disk_devt(mpath_head->disk), + bio->bi_iter.bi_sector); + submit_bio_noacct(bio); + } else if (mpath_available_path(mpath_head)) { + dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n"); + + spin_lock_irq(&mpath_head->requeue_lock); + bio_list_add(&mpath_head->requeue_list, bio); + spin_unlock_irq(&mpath_head->requeue_lock); + atomic_long_inc(&mpath_head->requeue_no_usable_path_cnt); + } else { + dev_warn_ratelimited(dev, "no available path - failing I/O\n"); + + bio_io_error(bio); + atomic_long_inc(&mpath_head->fail_no_avail_path_cnt); + } + +out: + srcu_read_unlock(&mpath_head->srcu, srcu_idx); +} + int mpath_get_head(struct mpath_head *mpath_head) { if (!refcount_inc_not_zero(&mpath_head->refcount)) @@ -292,6 +366,7 @@ const struct block_device_operations mpath_ops = { .owner = THIS_MODULE, .open = mpath_bdev_open, .release = mpath_bdev_release, + .submit_bio = mpath_bdev_submit_bio, }; EXPORT_SYMBOL_GPL(mpath_ops); @@ -309,12 +384,35 @@ static void multipath_partition_scan_work(struct work_struct *work) mutex_unlock(&mpath_head->disk->open_mutex); } +static void mpath_requeue_work(struct work_struct *work) +{ + struct mpath_head *mpath_head = + container_of(work, struct mpath_head, requeue_work); + struct bio *bio, *next; + + spin_lock_irq(&mpath_head->requeue_lock); + next = bio_list_get(&mpath_head->requeue_list); + spin_unlock_irq(&mpath_head->requeue_lock); + + while ((bio = next) != NULL) { + next = bio->bi_next; + bio->bi_next = NULL; + submit_bio_noacct(bio); + } +} + void mpath_remove_disk(struct mpath_head *mpath_head) { if (test_and_clear_bit(MPATH_HEAD_DISK_LIVE, &mpath_head->flags)) { struct gendisk *disk = mpath_head->disk; + /* + * requeue I/O after MPATH_HEAD_DISK_LIVE has been cleared + * to allow multipath to fail all I/O. + */ mpath_synchronize(mpath_head); + mpath_schedule_requeue_work(mpath_head); + del_gendisk(disk); } } @@ -326,6 +424,8 @@ void mpath_put_disk(struct mpath_head *mpath_head) return; /* make sure all pending bios are cleaned up */ + kblockd_schedule_work(&mpath_head->requeue_work); + flush_work(&mpath_head->requeue_work); flush_work(&mpath_head->partition_scan_work); put_disk(mpath_head->disk); mpath_head->disk = NULL; @@ -385,6 +485,7 @@ void mpath_device_set_live(struct mpath_device *mpath_device) mutex_unlock(&mpath_head->lock); mpath_synchronize(mpath_head); + mpath_schedule_requeue_work(mpath_head); } EXPORT_SYMBOL_GPL(mpath_device_set_live); @@ -397,6 +498,9 @@ int mpath_head_init(struct mpath_head *mpath_head) INIT_WORK(&mpath_head->partition_scan_work, multipath_partition_scan_work); + INIT_WORK(&mpath_head->requeue_work, mpath_requeue_work); + spin_lock_init(&mpath_head->requeue_lock); + bio_list_init(&mpath_head->requeue_list); return init_srcu_struct(&mpath_head->srcu); } -- 2.43.0