* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-22 20:14 [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info() Martin Wilck
@ 2024-08-23 6:45 ` Hannes Reinecke
2024-08-23 7:00 ` Daniel Wagner
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Hannes Reinecke @ 2024-08-23 6:45 UTC (permalink / raw)
To: Martin Wilck, Keith Busch, Jens Axboe, Christoph Hellwig,
Sagi Grimberg
Cc: Daniel Wagner, Stuart Hayes, linux-nvme
On 8/22/24 22:14, Martin Wilck wrote:
> For multipath devices, nvme_update_ns_info() needs to freeze both
> the queue of the path and the queue of the multipath device. For
> both operations, it waits for one RCU grace period to pass, ~25ms
> on my test system. By calling blk_freeze_queue_start() for the
> multipath queue early, we avoid waiting twice; tests using ftrace
> have shown that the second blk_mq_freeze_queue_wait() call finishes
> in just a few microseconds. The path queue is unfrozen before
> calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> possibly outstanding IO in the multipath queue can be flushed.
>
> I tested this using the "controller rescan under I/O load" test
> I submitted recently [1].
>
> [1] https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
> drivers/nvme/host/core.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-22 20:14 [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info() Martin Wilck
2024-08-23 6:45 ` Hannes Reinecke
@ 2024-08-23 7:00 ` Daniel Wagner
2024-08-23 13:38 ` Niklas Cassel
2024-08-25 8:28 ` Sagi Grimberg
3 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2024-08-23 7:00 UTC (permalink / raw)
To: Martin Wilck
Cc: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Hannes Reinecke, Stuart Hayes, linux-nvme
On Thu, Aug 22, 2024 at 10:14:13PM GMT, Martin Wilck wrote:
> For multipath devices, nvme_update_ns_info() needs to freeze both
> the queue of the path and the queue of the multipath device. For
> both operations, it waits for one RCU grace period to pass, ~25ms
> on my test system. By calling blk_freeze_queue_start() for the
> multipath queue early, we avoid waiting twice; tests using ftrace
> have shown that the second blk_mq_freeze_queue_wait() call finishes
> in just a few microseconds. The path queue is unfrozen before
> calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> possibly outstanding IO in the multipath queue can be flushed.
>
> I tested this using the "controller rescan under I/O load" test
> I submitted recently [1].
>
> [1] https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-22 20:14 [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info() Martin Wilck
2024-08-23 6:45 ` Hannes Reinecke
2024-08-23 7:00 ` Daniel Wagner
@ 2024-08-23 13:38 ` Niklas Cassel
2024-08-23 15:26 ` Martin Wilck
2024-08-25 8:28 ` Sagi Grimberg
3 siblings, 1 reply; 7+ messages in thread
From: Niklas Cassel @ 2024-08-23 13:38 UTC (permalink / raw)
To: Martin Wilck
Cc: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Hannes Reinecke, Daniel Wagner, Stuart Hayes, linux-nvme
On Thu, Aug 22, 2024 at 10:14:13PM +0200, Martin Wilck wrote:
> For multipath devices, nvme_update_ns_info() needs to freeze both
> the queue of the path and the queue of the multipath device. For
> both operations, it waits for one RCU grace period to pass, ~25ms
> on my test system. By calling blk_freeze_queue_start() for the
> multipath queue early, we avoid waiting twice; tests using ftrace
> have shown that the second blk_mq_freeze_queue_wait() call finishes
> in just a few microseconds. The path queue is unfrozen before
> calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> possibly outstanding IO in the multipath queue can be flushed.
>
> I tested this using the "controller rescan under I/O load" test
> I submitted recently [1].
>
> [1] https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
> drivers/nvme/host/core.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 33fa01c599ad..e2454398c660 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -2217,6 +2217,9 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
> bool unsupported = false;
> int ret;
>
> + if (nvme_ns_head_multipath(ns->head))
> + blk_freeze_queue_start(ns->head->disk->queue);
> +
From someone reading this code, it looks quite similar to
nvme_mpath_start_freeze().
Perhaps create a new helper, with proper kdoc, and possibly also add kdoc to
nvme_mpath_start_freeze(), so that a user can easily tell (from the kdoc)
when to use which function.
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-23 13:38 ` Niklas Cassel
@ 2024-08-23 15:26 ` Martin Wilck
2024-08-23 15:51 ` Niklas Cassel
0 siblings, 1 reply; 7+ messages in thread
From: Martin Wilck @ 2024-08-23 15:26 UTC (permalink / raw)
To: Niklas Cassel
Cc: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Hannes Reinecke, Daniel Wagner, Stuart Hayes, linux-nvme
On Fri, 2024-08-23 at 15:38 +0200, Niklas Cassel wrote:
> On Thu, Aug 22, 2024 at 10:14:13PM +0200, Martin Wilck wrote:
> > For multipath devices, nvme_update_ns_info() needs to freeze both
> > the queue of the path and the queue of the multipath device. For
> > both operations, it waits for one RCU grace period to pass, ~25ms
> > on my test system. By calling blk_freeze_queue_start() for the
> > multipath queue early, we avoid waiting twice; tests using ftrace
> > have shown that the second blk_mq_freeze_queue_wait() call finishes
> > in just a few microseconds. The path queue is unfrozen before
> > calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> > possibly outstanding IO in the multipath queue can be flushed.
> >
> > I tested this using the "controller rescan under I/O load" test
> > I submitted recently [1].
> >
> > [1]
> > https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
> >
> > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > ---
> > drivers/nvme/host/core.c | 8 ++++++--
> > 1 file changed, 6 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 33fa01c599ad..e2454398c660 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -2217,6 +2217,9 @@ static int nvme_update_ns_info(struct nvme_ns
> > *ns, struct nvme_ns_info *info)
> > bool unsupported = false;
> > int ret;
> >
> > + if (nvme_ns_head_multipath(ns->head))
> > + blk_freeze_queue_start(ns->head->disk->queue);
> > +
>
> From someone reading this code, it looks quite similar to
> nvme_mpath_start_freeze().
That function takes a struct nvme_subsystem as argument, and walks over
all namespaces in that subsystem, whereas here we're just acting on a
single namespace.
> Perhaps create a new helper, with proper kdoc, and possibly also add
> kdoc to
> nvme_mpath_start_freeze(), so that a user can easily tell (from the
> kdoc)
> when to use which function.
What's the benefit of introducing such a trivial helper, used only in a
single place of the code?
Thanks for the suggestion, but I'd like to see other maintainers'
opinions about this.
Thanks,
Martin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-23 15:26 ` Martin Wilck
@ 2024-08-23 15:51 ` Niklas Cassel
0 siblings, 0 replies; 7+ messages in thread
From: Niklas Cassel @ 2024-08-23 15:51 UTC (permalink / raw)
To: Martin Wilck
Cc: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Hannes Reinecke, Daniel Wagner, Stuart Hayes, linux-nvme
On Fri, Aug 23, 2024 at 05:26:56PM +0200, Martin Wilck wrote:
> On Fri, 2024-08-23 at 15:38 +0200, Niklas Cassel wrote:
> > On Thu, Aug 22, 2024 at 10:14:13PM +0200, Martin Wilck wrote:
> > > For multipath devices, nvme_update_ns_info() needs to freeze both
> > > the queue of the path and the queue of the multipath device. For
> > > both operations, it waits for one RCU grace period to pass, ~25ms
> > > on my test system. By calling blk_freeze_queue_start() for the
> > > multipath queue early, we avoid waiting twice; tests using ftrace
> > > have shown that the second blk_mq_freeze_queue_wait() call finishes
> > > in just a few microseconds. The path queue is unfrozen before
> > > calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> > > possibly outstanding IO in the multipath queue can be flushed.
> > >
> > > I tested this using the "controller rescan under I/O load" test
> > > I submitted recently [1].
> > >
> > > [1]
> > > https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
> > >
> > > Signed-off-by: Martin Wilck <mwilck@suse.com>
> > > ---
> > > drivers/nvme/host/core.c | 8 ++++++--
> > > 1 file changed, 6 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > > index 33fa01c599ad..e2454398c660 100644
> > > --- a/drivers/nvme/host/core.c
> > > +++ b/drivers/nvme/host/core.c
> > > @@ -2217,6 +2217,9 @@ static int nvme_update_ns_info(struct nvme_ns
> > > *ns, struct nvme_ns_info *info)
> > > bool unsupported = false;
> > > int ret;
> > >
> > > + if (nvme_ns_head_multipath(ns->head))
> > > + blk_freeze_queue_start(ns->head->disk->queue);
> > > +
> >
> > From someone reading this code, it looks quite similar to
> > nvme_mpath_start_freeze().
>
> That function takes a struct nvme_subsystem as argument, and walks over
> all namespaces in that subsystem, whereas here we're just acting on a
> single namespace.
>
> > Perhaps create a new helper, with proper kdoc, and possibly also add
> > kdoc to
> > nvme_mpath_start_freeze(), so that a user can easily tell (from the
> > kdoc)
> > when to use which function.
>
> What's the benefit of introducing such a trivial helper, used only in a
> single place of the code?
To make nvme_update_ns_info() easier to read, and to keep the existing code
style of calling nvme_mpath_*() functions unconditionally.
Sure, with my suggestion you would need a mpath_freeze() and mpath_unfreeze()
helper.
But if you add a helper that simply does:
if (nvme_ns_head_multipath(ns->head))
blk_freeze_queue_start(ns->head->disk->queue);
nvme_update_ns_info() could call the helper unconditionally,
just like how e.g. nvme_passthru_start() calls nvme_mpath_start_freeze()
unconditionally for both multipath and non-multipath NS heads:
https://github.com/torvalds/linux/blob/v6.11-rc4/drivers/nvme/host/core.c#L1205
(And how core.c calls many other nvme_mpath_*() functions unconditionally.)
> Thanks for the suggestion, but I'd like to see other maintainers'
> opinions about this.
Of course, a suggestion is just a suggestion :)
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info()
2024-08-22 20:14 [PATCH] nvme: core: freeze multipath queue early in nvme_update_ns_info() Martin Wilck
` (2 preceding siblings ...)
2024-08-23 13:38 ` Niklas Cassel
@ 2024-08-25 8:28 ` Sagi Grimberg
3 siblings, 0 replies; 7+ messages in thread
From: Sagi Grimberg @ 2024-08-25 8:28 UTC (permalink / raw)
To: Martin Wilck, Keith Busch, Jens Axboe, Christoph Hellwig
Cc: Hannes Reinecke, Daniel Wagner, Stuart Hayes, linux-nvme
The patch title should explain what it is fixing, i.e. the reason of
its existence.
Perhaps something like:
nvme: shorten multipath ns update duration
On 22/08/2024 23:14, Martin Wilck wrote:
> For multipath devices, nvme_update_ns_info() needs to freeze both
> the queue of the path and the queue of the multipath device. For
> both operations, it waits for one RCU grace period to pass, ~25ms
> on my test system. By calling blk_freeze_queue_start() for the
> multipath queue early, we avoid waiting twice; tests using ftrace
> have shown that the second blk_mq_freeze_queue_wait() call finishes
> in just a few microseconds. The path queue is unfrozen before
> calling blk_mq_freeze_queue_wait() on the multipath queue, so that
> possibly outstanding IO in the multipath queue can be flushed.
>
> I tested this using the "controller rescan under I/O load" test
> I submitted recently [1].
>
> [1] https://lore.kernel.org/linux-nvme/20240822193814.106111-3-mwilck@suse.com/T/#u
>
> Signed-off-by: Martin Wilck <mwilck@suse.com>
> ---
> drivers/nvme/host/core.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 33fa01c599ad..e2454398c660 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -2217,6 +2217,9 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
> bool unsupported = false;
> int ret;
>
> + if (nvme_ns_head_multipath(ns->head))
> + blk_freeze_queue_start(ns->head->disk->queue);
Lets add a small comment here.
> +
> switch (info->ids.csi) {
> case NVME_CSI_ZNS:
> if (!IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
> @@ -2254,7 +2257,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
> struct queue_limits *ns_lim = &ns->disk->queue->limits;
> struct queue_limits lim;
>
> - blk_mq_freeze_queue(ns->head->disk->queue);
> + blk_mq_freeze_queue_wait(ns->head->disk->queue);
> /*
> * queue_limits mixes values that are the hardware limitations
> * for bio splitting with what is the device configuration.
> @@ -2288,7 +2291,8 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
> nvme_mpath_revalidate_paths(ns);
>
> blk_mq_unfreeze_queue(ns->head->disk->queue);
> - }
> + } else if (nvme_ns_head_multipath(ns->head))
> + blk_mq_unfreeze_queue(ns->head->disk->queue);
Right now if ret!=0 you are unfreezing without waiting, you need to wait
for the freeze to
complete before unfreezing.
You should restructure the code to make it so that the freeze_start,
freeze_wait, unfreeze are
paired regardless of the code flows.
^ permalink raw reply [flat|nested] 7+ messages in thread