From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3970B606D81C for ; Sun, 8 Nov 2026 20:48:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=UOubftQ+; dkim=pass (1024-bit key, unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Qx58Uhvx DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Unsubscribe-Post: List-Owner; bh=ccCEPTXfeTW7gEQlR9QKMUHkSDrza9/l9sNjHCD8T/s=; b=UOubftQ+yU9bSo und95UfOsBzwc6Ij96hnUH8z8lbO7m1j4kF10hJo4AMQpFdcekvHK12NOhAnKaR9lRXSS/34iwAP8 Ob3mLn4q2jIPhxqtLYpJ8tcGBGDWQWYA7S0ksQpfOegAYZlsEsmX+9dtsaklK+RkUKXYr5QRn8LCp tWCMnL4eWkiykYdLTgy5gNc9XiezRgdi8PNPM/ynDdJtVrykZsMGyVgYSbKXMWSYq1MmwAc2SH2BR PVy63qy9G5IDw88fCOArsbaC9n4/mpEzmEc5Wq1i4fP1jF42gvlvKOBUnkI25FvMi7p/7Stdr1E4V pHg7APYMsdpX7wUrNVOw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.100.1 #2 (Red Hat Linux)) id 1xQ9o5-0000000GboV-3eSU; Sun, 08 Nov 2026 20:48:05 +0000 Received: from out-174.mta0.migadu.com ([2001:41d0:1004:224b::ae] helo=mta0.migadu.com) by bombadil.infradead.org with esmtps (Exim 4.100.1 #2 (Red Hat Linux)) id 1xQ9o3-0000000GbjE-0iU2 for linux-nvme@lists.infradead.org; Sun, 08 Nov 2026 20:48:04 +0000 X-Envelope-To: linux-nvme@lists.infradead.org DKIM-Signature: a=rsa-sha256; bh=91ZzruK9YdRqDB+8gQEmqkG0seV4+faEgDGR+QsaOUk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1794170881; v=1; x=1794775681; b=Qx58UhvxH/pPt7eIKrC9J6Ed3scRopS4F4xfBqxd3PlJ2X54k8a3XxfhWezlTZYB7dPIOu+Y WK/EwRV4sR4MeHAXqdj2KuIfL+MZ5qM3KKrE0nfzNtpDjS1Nl7rE0Yu4MeNdMZPXeibbPWwJqLM AfTbfjgWShkg9vTz/wvRrMtM= X-Envelope-To: linux-nvme@lists.infradead.org Received: by mta10.migadu.com with ESMTPS id c93b221cbde1e43a; Sun, 08 Nov 2026 20:48:00 +0000 X-Mizu-Trace-ID: c93b221cbde1e43a X-Migadu-Flow: FLOW_OUT From: John Garry To: james.bottomley@hansenpartnership.com, mkp@kernel.org Cc: hare@suse.com, linux-nvme@lists.infradead.org, linux-scsi@vger.kernel.org, snitzer@kernel.org, jmeneghi@redhat.com, bmarzins@redhat.com, dm-devel@lists.linux.dev, hch@lst.de, michael.christie@oracle.com, John Garry Subject: [PATCH v9 05/29] libmultipath: Add bio handling Date: Sun, 8 Nov 2026 20:47:09 +0000 Message-ID: <20261108204733.557995-6-john.garry@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261108204733.557995-1-john.garry@linux.dev> References: <20261108204733.557995-1-john.garry@linux.dev> X-Archived-By: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20261108_124803_381818_DD27EC61 X-CRM114-Status: GOOD ( 22.49 ) X-Spam-Score: -2.1 (--) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 Add support to submit a bio per-path. In addition, for failover, add support to requeue a failed bio. NVMe has almost like-for-like equivalents here: - nvme_available_path() -> mpath_available_path() - nvme_requeue_work() -> mpath_requeue_work() - nvme_ns_head_submit_bio() -> mpath_bdev_submit_bio() For failover, a driver may want to re-submit a bio, so add support to clone a bio prior to submission. A bio which is submitted to a per-path device has flag REQ_MPATH set, same as what is done for NVMe with REQ_NVME_MPATH. Signed-off-by: John Garry --- include/linux/multipath.h | 22 ++++++++ lib/multipath.c | 108 +++++++++++++++++++++++++++++++++++++- 2 files changed, 128 insertions(+), 2 deletions(-) diff --git a/include/linux/multipath.h b/include/linux/multipath.h index 9bbab0aa8961..374f0531f7e0 100644 --- a/include/linux/multipath.h +++ b/include/linux/multipath.h @@ -3,6 +3,7 @@ #define _LIBMULTIPATH_H #include +#include #include extern const struct block_device_operations mpath_ops; @@ -29,8 +30,10 @@ struct mpath_device { }; struct mpath_head_template { + bool (*available_path)(struct mpath_device *); bool (*is_disabled)(struct mpath_device *); bool (*is_optimized)(struct mpath_device *); + struct bio *(*clone_bio)(struct bio *); const struct attribute_group **device_groups; }; @@ -44,6 +47,14 @@ struct mpath_head { refcount_t refcount; enum mpath_iopolicy_e *iopolicy; + + struct bio_list requeue_list; /* list for requeing bio */ + spinlock_t requeue_lock; + struct work_struct requeue_work; /* work struct for requeue */ + + atomic_long_t requeue_no_usable_path_cnt; + atomic_long_t fail_no_avail_path_cnt; + unsigned long flags; struct gendisk *disk; struct work_struct partition_scan_work; @@ -53,6 +64,13 @@ struct mpath_head { struct mpath_device __rcu *current_path[MAX_NUMNODES]; }; +#define REQ_MPATH REQ_DRV + +static inline bool is_mpath_request(struct request *req) +{ + return req->cmd_flags & REQ_MPATH; +} + static inline struct mpath_head *mpath_bd_device_to_head(struct device *dev) { return dev_get_drvdata(dev); @@ -96,4 +114,8 @@ static inline bool mpath_qd_iopolicy(enum mpath_iopolicy_e *iopolicy) return READ_ONCE(*iopolicy) == MPATH_IOPOLICY_QD; } +static inline void mpath_schedule_requeue_work(struct mpath_head *mpath_head) +{ + kblockd_schedule_work(&mpath_head->requeue_work); +} #endif // _LIBMULTIPATH_H diff --git a/lib/multipath.c b/lib/multipath.c index b8ea8478037e..5dc7994f8095 100644 --- a/lib/multipath.c +++ b/lib/multipath.c @@ -7,6 +7,7 @@ #include #include #include +#include static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head); @@ -45,7 +46,6 @@ int mpath_get_iopolicy(char *buf, int iopolicy) } EXPORT_SYMBOL_GPL(mpath_get_iopolicy); - void mpath_synchronize(struct mpath_head *mpath_head) { synchronize_srcu(&mpath_head->srcu); @@ -232,7 +232,6 @@ static struct mpath_device *mpath_numa_path(struct mpath_head *mpath_head) return mpath_device; } -__maybe_unused static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head) { enum mpath_iopolicy_e iopolicy = mpath_read_iopolicy(mpath_head); @@ -247,6 +246,81 @@ static struct mpath_device *mpath_find_path(struct mpath_head *mpath_head) } } +static bool mpath_available_path(struct mpath_head *mpath_head) +{ + struct mpath_device *mpath_device; + + if (!test_bit(MPATH_HEAD_DISK_LIVE, &mpath_head->flags)) + return false; + + list_for_each_entry_srcu(mpath_device, &mpath_head->dev_list, siblings, + srcu_read_lock_held(&mpath_head->srcu)) { + if (mpath_head->mpdt->available_path(mpath_device)) + return true; + } + + return false; +} + +static void mpath_bdev_submit_bio(struct bio *bio) +{ + struct mpath_head *mpath_head = bio->bi_bdev->bd_disk->private_data; + struct device *dev = mpath_head->parent; + struct mpath_device *mpath_device; + int srcu_idx; + + /* + * The mpath_device might be going away and the bio might be moved to a + * different queue in failover, so we need to use the bio_split + * pool from the original queue to allocate the bvecs from. + */ + bio = bio_split_to_limits(bio); + if (!bio) + return; + + srcu_idx = srcu_read_lock(&mpath_head->srcu); + mpath_device = mpath_find_path(mpath_head); + + if (likely(mpath_device)) { + if (mpath_head->mpdt->clone_bio) { + struct bio *orig = bio; + + bio = mpath_head->mpdt->clone_bio(bio); + if (!bio) { + bio_io_error(orig); + goto out; + } + } + + bio_set_dev(bio, mpath_device->disk->part0); + /* + * Use BIO_REMAPPED to skip bio_check_eod() when this bio + * enters submit_bio_noacct() for the per-path device. The EOD + * check already passed on the multipath head. + */ + bio_set_flag(bio, BIO_REMAPPED); + bio->bi_opf |= REQ_MPATH; + trace_block_bio_remap(bio, disk_devt(mpath_head->disk), + bio->bi_iter.bi_sector); + submit_bio_noacct(bio); + } else if (mpath_available_path(mpath_head)) { + dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n"); + + spin_lock_irq(&mpath_head->requeue_lock); + bio_list_add(&mpath_head->requeue_list, bio); + spin_unlock_irq(&mpath_head->requeue_lock); + atomic_long_inc(&mpath_head->requeue_no_usable_path_cnt); + } else { + dev_warn_ratelimited(dev, "no available path - failing I/O\n"); + + bio_io_error(bio); + atomic_long_inc(&mpath_head->fail_no_avail_path_cnt); + } + +out: + srcu_read_unlock(&mpath_head->srcu, srcu_idx); +} + int mpath_get_head(struct mpath_head *mpath_head) { if (!refcount_inc_not_zero(&mpath_head->refcount)) @@ -292,6 +366,7 @@ const struct block_device_operations mpath_ops = { .owner = THIS_MODULE, .open = mpath_bdev_open, .release = mpath_bdev_release, + .submit_bio = mpath_bdev_submit_bio, }; EXPORT_SYMBOL_GPL(mpath_ops); @@ -309,12 +384,35 @@ static void multipath_partition_scan_work(struct work_struct *work) mutex_unlock(&mpath_head->disk->open_mutex); } +static void mpath_requeue_work(struct work_struct *work) +{ + struct mpath_head *mpath_head = + container_of(work, struct mpath_head, requeue_work); + struct bio *bio, *next; + + spin_lock_irq(&mpath_head->requeue_lock); + next = bio_list_get(&mpath_head->requeue_list); + spin_unlock_irq(&mpath_head->requeue_lock); + + while ((bio = next) != NULL) { + next = bio->bi_next; + bio->bi_next = NULL; + submit_bio_noacct(bio); + } +} + void mpath_remove_disk(struct mpath_head *mpath_head) { if (test_and_clear_bit(MPATH_HEAD_DISK_LIVE, &mpath_head->flags)) { struct gendisk *disk = mpath_head->disk; + /* + * requeue I/O after MPATH_HEAD_DISK_LIVE has been cleared + * to allow multipath to fail all I/O. + */ mpath_synchronize(mpath_head); + mpath_schedule_requeue_work(mpath_head); + del_gendisk(disk); } } @@ -326,6 +424,8 @@ void mpath_put_disk(struct mpath_head *mpath_head) return; /* make sure all pending bios are cleaned up */ + kblockd_schedule_work(&mpath_head->requeue_work); + flush_work(&mpath_head->requeue_work); flush_work(&mpath_head->partition_scan_work); put_disk(mpath_head->disk); mpath_head->disk = NULL; @@ -385,6 +485,7 @@ void mpath_device_set_live(struct mpath_device *mpath_device) mutex_unlock(&mpath_head->lock); mpath_synchronize(mpath_head); + mpath_schedule_requeue_work(mpath_head); } EXPORT_SYMBOL_GPL(mpath_device_set_live); @@ -397,6 +498,9 @@ int mpath_head_init(struct mpath_head *mpath_head) INIT_WORK(&mpath_head->partition_scan_work, multipath_partition_scan_work); + INIT_WORK(&mpath_head->requeue_work, mpath_requeue_work); + spin_lock_init(&mpath_head->requeue_lock); + bio_list_init(&mpath_head->requeue_list); return init_srcu_struct(&mpath_head->srcu); } -- 2.43.0