* [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 15:02 ` Caleb Sander Mateos
2025-04-07 13:15 ` [PATCH 02/13] selftests: ublk: fix ublk_find_tgt() Ming Lei
` (11 subsequent siblings)
12 siblings, 1 reply; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block
Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei, Keith Busch
When one request buffer is leased to io_uring via
io_buffer_register_bvec(), io_uring guarantees that the buffer will
be returned. However ublk aborts request in case that io_uring context
is exiting, then ublk_io_release() may observe freed request, and
kernel panic is triggered.
Fix the issue by delaying to abort zc request until io_uring returns
the buffer back.
Cc: Keith Busch <kbusch@kernel.org>
Fixes: 1f6540e2aabb ("ublk: zc register/unregister bvec")
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
drivers/block/ublk_drv.c | 31 ++++++++++++++++++++++++++++---
1 file changed, 28 insertions(+), 3 deletions(-)
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 2fd05c1bd30b..76caec28e5ac 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -140,6 +140,17 @@ struct ublk_uring_cmd_pdu {
*/
#define UBLK_IO_FLAG_NEED_GET_DATA 0x08
+
+/*
+ * Set when this request buffer is leased to ublk server, and cleared when
+ * the buffer is returned back.
+ *
+ * If this flag is set, this request can't be aborted until buffer is
+ * returned back from io_uring since io_uring is guaranteed to release the
+ * buffer.
+ */
+#define UBLK_IO_FLAG_BUF_LEASED 0x10
+
/* atomic RW with ubq->cancel_lock */
#define UBLK_IO_FLAG_CANCELED 0x80000000
@@ -1550,7 +1561,8 @@ static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq)
rq = blk_mq_tag_to_rq(ub->tag_set.tags[ubq->q_id], i);
if (rq && blk_mq_request_started(rq)) {
io->flags |= UBLK_IO_FLAG_ABORTED;
- __ublk_fail_req(ubq, io, rq);
+ if (!(io->flags & UBLK_IO_FLAG_BUF_LEASED))
+ __ublk_fail_req(ubq, io, rq);
}
}
}
@@ -1874,8 +1886,18 @@ static void ublk_io_release(void *priv)
{
struct request *rq = priv;
struct ublk_queue *ubq = rq->mq_hctx->driver_data;
+ struct ublk_io *io = &ubq->ios[rq->tag];
- ublk_put_req_ref(ubq, rq);
+ io->flags &= ~UBLK_IO_FLAG_BUF_LEASED;
+ /*
+ * request has been aborted, and the queue context is exiting,
+ * and ublk server can't be relied for completing this IO cmd,
+ * so force to complete it
+ */
+ if (unlikely(io->flags & UBLK_IO_FLAG_ABORTED))
+ __ublk_complete_rq(rq);
+ else
+ ublk_put_req_ref(ubq, rq);
}
static int ublk_register_io_buf(struct io_uring_cmd *cmd,
@@ -1958,7 +1980,10 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,
ret = -EINVAL;
switch (_IOC_NR(cmd_op)) {
case UBLK_IO_REGISTER_IO_BUF:
- return ublk_register_io_buf(cmd, ubq, tag, ub_cmd->addr, issue_flags);
+ ret = ublk_register_io_buf(cmd, ubq, tag, ub_cmd->addr, issue_flags);
+ if (!ret)
+ io->flags |= UBLK_IO_FLAG_BUF_LEASED;
+ return ret;
case UBLK_IO_UNREGISTER_IO_BUF:
return ublk_unregister_io_buf(cmd, ub_cmd->addr, issue_flags);
case UBLK_IO_FETCH_REQ:
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer
2025-04-07 13:15 ` [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer Ming Lei
@ 2025-04-07 15:02 ` Caleb Sander Mateos
2025-04-08 2:18 ` Ming Lei
0 siblings, 1 reply; 19+ messages in thread
From: Caleb Sander Mateos @ 2025-04-07 15:02 UTC (permalink / raw)
To: Ming Lei; +Cc: Jens Axboe, linux-block, Uday Shankar, Keith Busch
On Mon, Apr 7, 2025 at 6:15 AM Ming Lei <ming.lei@redhat.com> wrote:
>
> When one request buffer is leased to io_uring via
> io_buffer_register_bvec(), io_uring guarantees that the buffer will
> be returned. However ublk aborts request in case that io_uring context
> is exiting, then ublk_io_release() may observe freed request, and
> kernel panic is triggered.
Not sure I follow how the request can be freed while its buffer is
still registered with io_uring. It looks like __ublk_fail_req()
decrements the ublk request's reference count (ublk_put_req_ref()) and
the reference count shouldn't hit 0 if the io_uring registered buffer
is still holding a reference. Is the problem the if
(ublk_nosrv_should_reissue_outstanding()) case, which calls
blk_mq_requeue_request() without checking the reference count?
Maybe a better place to put the requeue logic would be in
__ublk_complete_rq(). Then both the abort and io_uring buffer
unregister paths can just call ublk_put_req_ref(). And there would be
no need for UBLK_IO_FLAG_BUF_LEASED.
>
> Fix the issue by delaying to abort zc request until io_uring returns
> the buffer back.
>
> Cc: Keith Busch <kbusch@kernel.org>
> Fixes: 1f6540e2aabb ("ublk: zc register/unregister bvec")
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> drivers/block/ublk_drv.c | 31 ++++++++++++++++++++++++++++---
> 1 file changed, 28 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
> index 2fd05c1bd30b..76caec28e5ac 100644
> --- a/drivers/block/ublk_drv.c
> +++ b/drivers/block/ublk_drv.c
> @@ -140,6 +140,17 @@ struct ublk_uring_cmd_pdu {
> */
> #define UBLK_IO_FLAG_NEED_GET_DATA 0x08
>
> +
> +/*
> + * Set when this request buffer is leased to ublk server, and cleared when
> + * the buffer is returned back.
> + *
> + * If this flag is set, this request can't be aborted until buffer is
> + * returned back from io_uring since io_uring is guaranteed to release the
> + * buffer.
> + */
> +#define UBLK_IO_FLAG_BUF_LEASED 0x10
> +
> /* atomic RW with ubq->cancel_lock */
> #define UBLK_IO_FLAG_CANCELED 0x80000000
>
> @@ -1550,7 +1561,8 @@ static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq)
> rq = blk_mq_tag_to_rq(ub->tag_set.tags[ubq->q_id], i);
> if (rq && blk_mq_request_started(rq)) {
> io->flags |= UBLK_IO_FLAG_ABORTED;
> - __ublk_fail_req(ubq, io, rq);
> + if (!(io->flags & UBLK_IO_FLAG_BUF_LEASED))
> + __ublk_fail_req(ubq, io, rq);
> }
> }
> }
> @@ -1874,8 +1886,18 @@ static void ublk_io_release(void *priv)
> {
> struct request *rq = priv;
> struct ublk_queue *ubq = rq->mq_hctx->driver_data;
> + struct ublk_io *io = &ubq->ios[rq->tag];
>
> - ublk_put_req_ref(ubq, rq);
> + io->flags &= ~UBLK_IO_FLAG_BUF_LEASED;
> + /*
> + * request has been aborted, and the queue context is exiting,
> + * and ublk server can't be relied for completing this IO cmd,
> + * so force to complete it
> + */
> + if (unlikely(io->flags & UBLK_IO_FLAG_ABORTED))
> + __ublk_complete_rq(rq);
This unconditionally ends the ublk request without checking the
reference count. Wouldn't that cause use-after-free/double-free issues
if the ublk request's buffer was registered in multiple io_uring
buffer slots?
Best,
Caleb
> + else
> + ublk_put_req_ref(ubq, rq);
> }
>
> static int ublk_register_io_buf(struct io_uring_cmd *cmd,
> @@ -1958,7 +1980,10 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,
> ret = -EINVAL;
> switch (_IOC_NR(cmd_op)) {
> case UBLK_IO_REGISTER_IO_BUF:
> - return ublk_register_io_buf(cmd, ubq, tag, ub_cmd->addr, issue_flags);
> + ret = ublk_register_io_buf(cmd, ubq, tag, ub_cmd->addr, issue_flags);
> + if (!ret)
> + io->flags |= UBLK_IO_FLAG_BUF_LEASED;
> + return ret;
> case UBLK_IO_UNREGISTER_IO_BUF:
> return ublk_unregister_io_buf(cmd, ub_cmd->addr, issue_flags);
> case UBLK_IO_FETCH_REQ:
> --
> 2.47.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer
2025-04-07 15:02 ` Caleb Sander Mateos
@ 2025-04-08 2:18 ` Ming Lei
0 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-08 2:18 UTC (permalink / raw)
To: Caleb Sander Mateos
Cc: Jens Axboe, linux-block, Uday Shankar, Keith Busch, io-uring
On Mon, Apr 07, 2025 at 08:02:24AM -0700, Caleb Sander Mateos wrote:
> On Mon, Apr 7, 2025 at 6:15 AM Ming Lei <ming.lei@redhat.com> wrote:
> >
> > When one request buffer is leased to io_uring via
> > io_buffer_register_bvec(), io_uring guarantees that the buffer will
> > be returned. However ublk aborts request in case that io_uring context
> > is exiting, then ublk_io_release() may observe freed request, and
> > kernel panic is triggered.
>
> Not sure I follow how the request can be freed while its buffer is
> still registered with io_uring. It looks like __ublk_fail_req()
> decrements the ublk request's reference count (ublk_put_req_ref()) and
> the reference count shouldn't hit 0 if the io_uring registered buffer
> is still holding a reference. Is the problem the if
> (ublk_nosrv_should_reissue_outstanding()) case, which calls
> blk_mq_requeue_request() without checking the reference count?
Yeah, that is the problem, the request can be failed immediately after
requeue & re-dispatch, then trigger the panic, and I verified that the
following patch does fix it:
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 2fd05c1bd30b..41bed67508f2 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -1140,6 +1140,25 @@ static void ublk_complete_rq(struct kref *ref)
__ublk_complete_rq(req);
}
+static void ublk_do_fail_rq(struct request *req)
+{
+ struct ublk_queue *ubq = req->mq_hctx->driver_data;
+
+ if (ublk_nosrv_should_reissue_outstanding(ubq->dev))
+ blk_mq_requeue_request(req, false);
+ else
+ __ublk_complete_rq(req);
+}
+
+static void ublk_fail_rq_fn(struct kref *ref)
+{
+ struct ublk_rq_data *data = container_of(ref, struct ublk_rq_data,
+ ref);
+ struct request *req = blk_mq_rq_from_pdu(data);
+
+ ublk_do_fail_rq(req);
+}
+
/*
* Since ublk_rq_task_work_cb always fails requests immediately during
* exiting, __ublk_fail_req() is only called from abort context during
@@ -1153,10 +1172,13 @@ static void __ublk_fail_req(struct ublk_queue *ubq, struct ublk_io *io,
{
WARN_ON_ONCE(io->flags & UBLK_IO_FLAG_ACTIVE);
- if (ublk_nosrv_should_reissue_outstanding(ubq->dev))
- blk_mq_requeue_request(req, false);
- else
- ublk_put_req_ref(ubq, req);
+ if (ublk_need_req_ref(ubq)) {
+ struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+
+ kref_put(&data->ref, ublk_fail_rq_fn);
+ } else {
+ ublk_do_fail_rq(req);
+ }
}
static void ubq_complete_io_cmd(struct ublk_io *io, int res,
Thanks,
Ming
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH 02/13] selftests: ublk: fix ublk_find_tgt()
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
2025-04-07 13:15 ` [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-08 6:05 ` Johannes Thumshirn
2025-04-07 13:15 ` [PATCH 03/13] selftests: ublk: add io_uring uapi header Ming Lei
` (10 subsequent siblings)
12 siblings, 1 reply; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Bounds check for iterator variable `i` is missed, so add it and fix
ublk_find_tgt().
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index 91c282bc7674..5c03c776426f 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -14,13 +14,12 @@ static const struct ublk_tgt_ops *tgt_ops_list[] = {
static const struct ublk_tgt_ops *ublk_find_tgt(const char *name)
{
- const struct ublk_tgt_ops *ops;
int i;
if (name == NULL)
return NULL;
- for (i = 0; sizeof(tgt_ops_list) / sizeof(ops); i++)
+ for (i = 0; i < sizeof(tgt_ops_list) / sizeof(tgt_ops_list[0]); i++)
if (strcmp(tgt_ops_list[i]->name, name) == 0)
return tgt_ops_list[i];
return NULL;
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 02/13] selftests: ublk: fix ublk_find_tgt()
2025-04-07 13:15 ` [PATCH 02/13] selftests: ublk: fix ublk_find_tgt() Ming Lei
@ 2025-04-08 6:05 ` Johannes Thumshirn
2025-04-12 1:25 ` Ming Lei
0 siblings, 1 reply; 19+ messages in thread
From: Johannes Thumshirn @ 2025-04-08 6:05 UTC (permalink / raw)
To: Ming Lei, Jens Axboe, linux-block@vger.kernel.org
Cc: Caleb Sander Mateos, Uday Shankar
On 07.04.25 15:18, Ming Lei wrote:
> Bounds check for iterator variable `i` is missed, so add it and fix
> ublk_find_tgt().
>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> tools/testing/selftests/ublk/kublk.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
> index 91c282bc7674..5c03c776426f 100644
> --- a/tools/testing/selftests/ublk/kublk.c
> +++ b/tools/testing/selftests/ublk/kublk.c
> @@ -14,13 +14,12 @@ static const struct ublk_tgt_ops *tgt_ops_list[] = {
>
> static const struct ublk_tgt_ops *ublk_find_tgt(const char *name)
> {
> - const struct ublk_tgt_ops *ops;
> int i;
>
> if (name == NULL)
> return NULL;
>
> - for (i = 0; sizeof(tgt_ops_list) / sizeof(ops); i++)
> + for (i = 0; i < sizeof(tgt_ops_list) / sizeof(tgt_ops_list[0]); i++)
Why not
for (i=0; i < ARRAY_SIZE(tgt_ops_list); i++)
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 02/13] selftests: ublk: fix ublk_find_tgt()
2025-04-08 6:05 ` Johannes Thumshirn
@ 2025-04-12 1:25 ` Ming Lei
0 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-12 1:25 UTC (permalink / raw)
To: Johannes Thumshirn
Cc: Jens Axboe, linux-block@vger.kernel.org, Caleb Sander Mateos,
Uday Shankar
On Tue, Apr 8, 2025 at 2:05 PM Johannes Thumshirn
<Johannes.Thumshirn@wdc.com> wrote:
>
> On 07.04.25 15:18, Ming Lei wrote:
> > Bounds check for iterator variable `i` is missed, so add it and fix
> > ublk_find_tgt().
> >
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
> > ---
> > tools/testing/selftests/ublk/kublk.c | 3 +--
> > 1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
> > index 91c282bc7674..5c03c776426f 100644
> > --- a/tools/testing/selftests/ublk/kublk.c
> > +++ b/tools/testing/selftests/ublk/kublk.c
> > @@ -14,13 +14,12 @@ static const struct ublk_tgt_ops *tgt_ops_list[] = {
> >
> > static const struct ublk_tgt_ops *ublk_find_tgt(const char *name)
> > {
> > - const struct ublk_tgt_ops *ops;
> > int i;
> >
> > if (name == NULL)
> > return NULL;
> >
> > - for (i = 0; sizeof(tgt_ops_list) / sizeof(ops); i++)
> > + for (i = 0; i < sizeof(tgt_ops_list) / sizeof(tgt_ops_list[0]); i++)
>
> Why not
>
> for (i=0; i < ARRAY_SIZE(tgt_ops_list); i++)
It is because ARRAY_SIZE isn't defined as one generic helper for
test code.
But it is more readable, so I will add it in ublk header.
Thanks,
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 03/13] selftests: ublk: add io_uring uapi header
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
2025-04-07 13:15 ` [PATCH 01/13] ublk: delay aborting zc request until io_uring returns the buffer Ming Lei
2025-04-07 13:15 ` [PATCH 02/13] selftests: ublk: fix ublk_find_tgt() Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-08 6:08 ` Johannes Thumshirn
2025-04-07 13:15 ` [PATCH 04/13] selftests: ublk: cleanup backfile automatically Ming Lei
` (9 subsequent siblings)
12 siblings, 1 reply; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Add io_uring UAPI header so that ublk can work with latest uapi
definition.
Fix the following build failure:
stripe.c: In function ‘stripe_to_uring_op’:
stripe.c:120:29: error: ‘IORING_OP_READV_FIXED’ undeclared (first use in this function); did you mean ‘IORING_OP_READ_FIXED’?
120 | return zc ? IORING_OP_READV_FIXED : IORING_OP_READV;
| ^~~~~~~~~~~~~~~~~~~~~
| IORING_OP_READ_FIXED
Fixes: 57ed58c13256 ("selftests: ublk: enable zero copy for stripe target")
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h
index 760ff8ffb810..d87d6c376283 100644
--- a/tools/testing/selftests/ublk/kublk.h
+++ b/tools/testing/selftests/ublk/kublk.h
@@ -20,6 +20,7 @@
#include <sys/wait.h>
#include <sys/eventfd.h>
#include <sys/uio.h>
+#include <linux/io_uring.h>
#include <liburing.h>
#include <linux/ublk_cmd.h>
#include "ublk_dep.h"
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 04/13] selftests: ublk: cleanup backfile automatically
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (2 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 03/13] selftests: ublk: add io_uring uapi header Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 05/13] selftests: ublk: make sure _add_ublk_dev can return in sub-shell Ming Lei
` (8 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Use global array of $UBLK_BACKFILES for storing all backfile name, then
clean them automatically.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/test_common.sh | 36 ++++++++++++-------
tools/testing/selftests/ublk/test_loop_01.sh | 8 ++---
tools/testing/selftests/ublk/test_loop_02.sh | 8 ++---
tools/testing/selftests/ublk/test_loop_03.sh | 8 ++---
tools/testing/selftests/ublk/test_loop_04.sh | 9 +++--
tools/testing/selftests/ublk/test_loop_05.sh | 8 ++---
.../testing/selftests/ublk/test_stress_01.sh | 16 ++++-----
.../testing/selftests/ublk/test_stress_02.sh | 16 ++++-----
.../testing/selftests/ublk/test_stripe_01.sh | 12 +++----
.../testing/selftests/ublk/test_stripe_02.sh | 13 +++----
.../testing/selftests/ublk/test_stripe_03.sh | 12 +++----
.../testing/selftests/ublk/test_stripe_04.sh | 13 +++----
12 files changed, 70 insertions(+), 89 deletions(-)
diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index a88b35943227..c7d04da7235a 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -30,18 +30,26 @@ _run_fio_verify_io() {
}
_create_backfile() {
- local my_size=$1
- local my_file
+ local index=$1
+ local new_size=$2
+ local old_file
+ local new_file
- my_file=$(mktemp ublk_file_"${my_size}"_XXXXX)
- truncate -s "${my_size}" "${my_file}"
- echo "$my_file"
+ old_file="${UBLK_BACKFILES[$index]}"
+ [ -f "$old_file" ] && rm -f "$old_file"
+
+ new_file=$(mktemp ublk_file_"${new_size}"_XXXXX)
+ truncate -s "${new_size}" "${new_file}"
+ UBLK_BACKFILES["$index"]="$new_file"
}
-_remove_backfile() {
- local file=$1
+_remove_files() {
+ local file
- [ -f "$file" ] && rm -f "$file"
+ for file in "${UBLK_BACKFILES[@]}"; do
+ [ -f "$file" ] && rm -f "$file"
+ done
+ [ -f "$UBLK_TMP" ] && rm -f "$UBLK_TMP"
}
_create_tmp_dir() {
@@ -129,7 +137,10 @@ _show_result()
echo "$1 : [FAIL]"
fi
fi
- [ "$2" -ne 0 ] && exit "$2"
+ if [ "$2" -ne 0 ]; then
+ _remove_files
+ exit "$2"
+ fi
return 0
}
@@ -138,16 +149,16 @@ _check_add_dev()
{
local tid=$1
local code=$2
- shift 2
+
if [ "${code}" -ne 0 ]; then
- _remove_test_files "$@"
_show_result "${tid}" "${code}"
fi
}
_cleanup_test() {
"${UBLK_PROG}" del -a
- rm -f "$UBLK_TMP"
+
+ _remove_files
}
_have_feature()
@@ -247,6 +258,7 @@ UBLK_TMP=$(mktemp ublk_test_XXXXX)
UBLK_PROG=$(_ublk_test_top_dir)/kublk
UBLK_TEST_QUIET=1
UBLK_TEST_SHOW_RESULT=1
+UBLK_BACKFILES=()
export UBLK_PROG
export UBLK_TEST_QUIET
export UBLK_TEST_SHOW_RESULT
diff --git a/tools/testing/selftests/ublk/test_loop_01.sh b/tools/testing/selftests/ublk/test_loop_01.sh
index 1ef8b6044777..833fa0dbc700 100755
--- a/tools/testing/selftests/ublk/test_loop_01.sh
+++ b/tools/testing/selftests/ublk/test_loop_01.sh
@@ -12,10 +12,10 @@ fi
_prep_test "loop" "write and verify test"
-backfile_0=$(_create_backfile 256M)
+_create_backfile 0 256M
-dev_id=$(_add_ublk_dev -t loop "$backfile_0")
-_check_add_dev $TID $? "${backfile_0}"
+dev_id=$(_add_ublk_dev -t loop "${UBLK_BACKFILES[0]}")
+_check_add_dev $TID $?
# run fio over the ublk disk
_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -23,6 +23,4 @@ ERR_CODE=$?
_cleanup_test "loop"
-_remove_backfile "$backfile_0"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_loop_02.sh b/tools/testing/selftests/ublk/test_loop_02.sh
index 03863d825e07..874568b3646b 100755
--- a/tools/testing/selftests/ublk/test_loop_02.sh
+++ b/tools/testing/selftests/ublk/test_loop_02.sh
@@ -8,15 +8,13 @@ ERR_CODE=0
_prep_test "loop" "mkfs & mount & umount"
-backfile_0=$(_create_backfile 256M)
-dev_id=$(_add_ublk_dev -t loop "$backfile_0")
-_check_add_dev $TID $? "$backfile_0"
+_create_backfile 0 256M
+dev_id=$(_add_ublk_dev -t loop "${UBLK_BACKFILES[0]}")
+_check_add_dev $TID $?
_mkfs_mount_test /dev/ublkb"${dev_id}"
ERR_CODE=$?
_cleanup_test "loop"
-_remove_backfile "$backfile_0"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_loop_03.sh b/tools/testing/selftests/ublk/test_loop_03.sh
index e9ca744de8b1..c30f797c6429 100755
--- a/tools/testing/selftests/ublk/test_loop_03.sh
+++ b/tools/testing/selftests/ublk/test_loop_03.sh
@@ -12,9 +12,9 @@ fi
_prep_test "loop" "write and verify over zero copy"
-backfile_0=$(_create_backfile 256M)
-dev_id=$(_add_ublk_dev -t loop -z "$backfile_0")
-_check_add_dev $TID $? "$backfile_0"
+_create_backfile 0 256M
+dev_id=$(_add_ublk_dev -t loop -z "${UBLK_BACKFILES[0]}")
+_check_add_dev $TID $?
# run fio over the ublk disk
_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -22,6 +22,4 @@ ERR_CODE=$?
_cleanup_test "loop"
-_remove_backfile "$backfile_0"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_loop_04.sh b/tools/testing/selftests/ublk/test_loop_04.sh
index 1435422c38ec..b01d75b3214d 100755
--- a/tools/testing/selftests/ublk/test_loop_04.sh
+++ b/tools/testing/selftests/ublk/test_loop_04.sh
@@ -8,15 +8,14 @@ ERR_CODE=0
_prep_test "loop" "mkfs & mount & umount with zero copy"
-backfile_0=$(_create_backfile 256M)
-dev_id=$(_add_ublk_dev -t loop -z "$backfile_0")
-_check_add_dev $TID $? "$backfile_0"
+_create_backfile 0 256M
+
+dev_id=$(_add_ublk_dev -t loop -z "${UBLK_BACKFILES[0]}")
+_check_add_dev $TID $?
_mkfs_mount_test /dev/ublkb"${dev_id}"
ERR_CODE=$?
_cleanup_test "loop"
-_remove_backfile "$backfile_0"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_loop_05.sh b/tools/testing/selftests/ublk/test_loop_05.sh
index 2e6e2e6978fc..de2141533074 100755
--- a/tools/testing/selftests/ublk/test_loop_05.sh
+++ b/tools/testing/selftests/ublk/test_loop_05.sh
@@ -12,10 +12,10 @@ fi
_prep_test "loop" "write and verify test"
-backfile_0=$(_create_backfile 256M)
+_create_backfile 0 256M
-dev_id=$(_add_ublk_dev -q 2 -t loop "$backfile_0")
-_check_add_dev $TID $? "${backfile_0}"
+dev_id=$(_add_ublk_dev -q 2 -t loop "${UBLK_BACKFILES[0]}")
+_check_add_dev $TID $?
# run fio over the ublk disk
_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M
@@ -23,6 +23,4 @@ ERR_CODE=$?
_cleanup_test "loop"
-_remove_backfile "$backfile_0"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh
index a8be24532b24..4c37a2cf13a3 100755
--- a/tools/testing/selftests/ublk/test_stress_01.sh
+++ b/tools/testing/selftests/ublk/test_stress_01.sh
@@ -10,17 +10,13 @@ ublk_io_and_remove()
{
local size=$1
shift 1
- local backfile=""
- if echo "$@" | grep -q "loop"; then
- backfile=${*: -1}
- fi
+
DEV_ID=$(_add_ublk_dev "$@")
- _check_add_dev $TID $? "${backfile}"
+ _check_add_dev $TID $?
[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
if ! __run_io_and_remove "${DEV_ID}" "${size}" "no"; then
echo "/dev/ublkc${DEV_ID} isn't removed"
- _remove_backfile "${backfile}"
exit 255
fi
}
@@ -33,15 +29,15 @@ if [ ${ERR_CODE} -ne 0 ]; then
_show_result $TID $ERR_CODE
fi
-BACK_FILE=$(_create_backfile 256M)
-ublk_io_and_remove 256M -t loop -q 4 "${BACK_FILE}"
+_create_backfile 0 256M
+
+ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}"
ERR_CODE=$?
if [ ${ERR_CODE} -ne 0 ]; then
_show_result $TID $ERR_CODE
fi
-ublk_io_and_remove 256M -t loop -q 4 -z "${BACK_FILE}"
+ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}"
ERR_CODE=$?
_cleanup_test "stress"
-_remove_backfile "${BACK_FILE}"
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh
index 2159e4cc8140..4b6ad441d500 100755
--- a/tools/testing/selftests/ublk/test_stress_02.sh
+++ b/tools/testing/selftests/ublk/test_stress_02.sh
@@ -10,17 +10,13 @@ ublk_io_and_kill_daemon()
{
local size=$1
shift 1
- local backfile=""
- if echo "$@" | grep -q "loop"; then
- backfile=${*: -1}
- fi
+
DEV_ID=$(_add_ublk_dev "$@")
- _check_add_dev $TID $? "${backfile}"
+ _check_add_dev $TID $?
[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)"
if ! __run_io_and_remove "${DEV_ID}" "${size}" "yes"; then
echo "/dev/ublkc${DEV_ID} isn't removed res ${res}"
- _remove_backfile "${backfile}"
exit 255
fi
}
@@ -33,15 +29,15 @@ if [ ${ERR_CODE} -ne 0 ]; then
_show_result $TID $ERR_CODE
fi
-BACK_FILE=$(_create_backfile 256M)
-ublk_io_and_kill_daemon 256M -t loop -q 4 "${BACK_FILE}"
+_create_backfile 0 256M
+
+ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}"
ERR_CODE=$?
if [ ${ERR_CODE} -ne 0 ]; then
_show_result $TID $ERR_CODE
fi
-ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${BACK_FILE}"
+ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}"
ERR_CODE=$?
_cleanup_test "stress"
-_remove_backfile "${BACK_FILE}"
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stripe_01.sh b/tools/testing/selftests/ublk/test_stripe_01.sh
index 7e387ef656ea..4e4f0fdf3c9b 100755
--- a/tools/testing/selftests/ublk/test_stripe_01.sh
+++ b/tools/testing/selftests/ublk/test_stripe_01.sh
@@ -12,19 +12,15 @@ fi
_prep_test "stripe" "write and verify test"
-backfile_0=$(_create_backfile 256M)
-backfile_1=$(_create_backfile 256M)
+_create_backfile 0 256M
+_create_backfile 1 256M
-dev_id=$(_add_ublk_dev -t stripe "$backfile_0" "$backfile_1")
-_check_add_dev $TID $? "${backfile_0}"
+dev_id=$(_add_ublk_dev -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}")
+_check_add_dev $TID $?
# run fio over the ublk disk
_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M
ERR_CODE=$?
_cleanup_test "stripe"
-
-_remove_backfile "$backfile_0"
-_remove_backfile "$backfile_1"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stripe_02.sh b/tools/testing/selftests/ublk/test_stripe_02.sh
index e8a45fa82dde..5820ab2efba4 100755
--- a/tools/testing/selftests/ublk/test_stripe_02.sh
+++ b/tools/testing/selftests/ublk/test_stripe_02.sh
@@ -8,17 +8,14 @@ ERR_CODE=0
_prep_test "stripe" "mkfs & mount & umount"
-backfile_0=$(_create_backfile 256M)
-backfile_1=$(_create_backfile 256M)
-dev_id=$(_add_ublk_dev -t stripe "$backfile_0" "$backfile_1")
-_check_add_dev $TID $? "$backfile_0" "$backfile_1"
+_create_backfile 0 256M
+_create_backfile 1 256M
+
+dev_id=$(_add_ublk_dev -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}")
+_check_add_dev $TID $?
_mkfs_mount_test /dev/ublkb"${dev_id}"
ERR_CODE=$?
_cleanup_test "stripe"
-
-_remove_backfile "$backfile_0"
-_remove_backfile "$backfile_1"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stripe_03.sh b/tools/testing/selftests/ublk/test_stripe_03.sh
index c1b34af36145..20b977e27814 100755
--- a/tools/testing/selftests/ublk/test_stripe_03.sh
+++ b/tools/testing/selftests/ublk/test_stripe_03.sh
@@ -12,19 +12,15 @@ fi
_prep_test "stripe" "write and verify test"
-backfile_0=$(_create_backfile 256M)
-backfile_1=$(_create_backfile 256M)
+_create_backfile 0 256M
+_create_backfile 1 256M
-dev_id=$(_add_ublk_dev -q 2 -t stripe "$backfile_0" "$backfile_1")
-_check_add_dev $TID $? "${backfile_0}"
+dev_id=$(_add_ublk_dev -q 2 -t stripe "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}")
+_check_add_dev $TID $?
# run fio over the ublk disk
_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M
ERR_CODE=$?
_cleanup_test "stripe"
-
-_remove_backfile "$backfile_0"
-_remove_backfile "$backfile_1"
-
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stripe_04.sh b/tools/testing/selftests/ublk/test_stripe_04.sh
index 1f2b642381d1..1b51ed2f1d84 100755
--- a/tools/testing/selftests/ublk/test_stripe_04.sh
+++ b/tools/testing/selftests/ublk/test_stripe_04.sh
@@ -8,17 +8,14 @@ ERR_CODE=0
_prep_test "stripe" "mkfs & mount & umount on zero copy"
-backfile_0=$(_create_backfile 256M)
-backfile_1=$(_create_backfile 256M)
-dev_id=$(_add_ublk_dev -t stripe -z -q 2 "$backfile_0" "$backfile_1")
-_check_add_dev $TID $? "$backfile_0" "$backfile_1"
+_create_backfile 0 256M
+_create_backfile 1 256M
+
+dev_id=$(_add_ublk_dev -t stripe -z -q 2 "${UBLK_BACKFILES[0]}" "${UBLK_BACKFILES[1]}")
+_check_add_dev $TID $?
_mkfs_mount_test /dev/ublkb"${dev_id}"
ERR_CODE=$?
_cleanup_test "stripe"
-
-_remove_backfile "$backfile_0"
-_remove_backfile "$backfile_1"
-
_show_result $TID $ERR_CODE
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 05/13] selftests: ublk: make sure _add_ublk_dev can return in sub-shell
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (3 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 04/13] selftests: ublk: cleanup backfile automatically Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 06/13] selftests: ublk: run stress tests in parallel Ming Lei
` (7 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Detach ublk daemon from the starting process completely by double-fork and
clearing its process group, so that _add_ublk_dev can return from
sub-shell.
Prepare for running ublk test in parallel.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 29 +++++++++++++++----
tools/testing/selftests/ublk/test_common.sh | 15 +++++-----
.../testing/selftests/ublk/test_stress_01.sh | 8 ++---
.../testing/selftests/ublk/test_stress_02.sh | 8 ++---
4 files changed, 39 insertions(+), 21 deletions(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index 5c03c776426f..f01d8618739b 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -654,6 +654,8 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)
if (write(evtfd, &id, sizeof(id)) != sizeof(id))
return -EINVAL;
+ close(evtfd);
+
return 0;
}
@@ -889,22 +891,39 @@ static int cmd_dev_add(struct dev_ctx *ctx)
exit(-1);
}
- setsid();
res = fork();
if (res == 0) {
+ int res2;
+
+ setsid();
+ res2 = fork();
+
+ if (res2 == 0) {
+ /* prepare for detaching */
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+ close(STDERR_FILENO);
run:
- res = __cmd_dev_add(ctx);
- return res;
+ res = __cmd_dev_add(ctx);
+ return res;
+ } else {
+ /* detached from the parent */
+ exit(EXIT_SUCCESS);
+ }
} else if (res > 0) {
uint64_t id;
+ int exit_code = EXIT_FAILURE;
res = read(ctx->_evtfd, &id, sizeof(id));
close(ctx->_evtfd);
if (res == sizeof(id) && id != ERROR_EVTFD_DEVID) {
ctx->dev_id = id - 1;
- return __cmd_dev_list(ctx);
+ if (__cmd_dev_list(ctx) >= 0)
+ exit_code = EXIT_SUCCESS;
}
- exit(EXIT_FAILURE);
+ /* wait for child */
+ wait(NULL);
+ exit(exit_code);
} else {
return res;
}
diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index c7d04da7235a..c43bd1d5c9c0 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -170,7 +170,6 @@ _have_feature()
}
_add_ublk_dev() {
- local kublk_temp;
local dev_id;
if [ ! -c /dev/ublk-control ]; then
@@ -182,17 +181,17 @@ _add_ublk_dev() {
fi
fi
- kublk_temp=$(mktemp /tmp/kublk-XXXXXX)
- if ! "${UBLK_PROG}" add "$@" > "${kublk_temp}" 2>&1; then
+ if ! dev_id=$("${UBLK_PROG}" add "$@" | grep "dev id" | awk -F '[ :]' '{print $3}'); then
echo "fail to add ublk dev $*"
- rm -f "${kublk_temp}"
return 255
fi
-
- dev_id=$(grep "dev id" "${kublk_temp}" | awk -F '[ :]' '{print $3}')
udevadm settle
- rm -f "${kublk_temp}"
- echo "${dev_id}"
+
+ if [[ "$dev_id" =~ ^[0-9]+$ ]]; then
+ echo "${dev_id}"
+ else
+ return 255
+ fi
}
# kill the ublk daemon and return ublk device state
diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh
index 4c37a2cf13a3..61fdbdfe70bc 100755
--- a/tools/testing/selftests/ublk/test_stress_01.sh
+++ b/tools/testing/selftests/ublk/test_stress_01.sh
@@ -4,19 +4,19 @@
. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
TID="stress_01"
ERR_CODE=0
-DEV_ID=-1
ublk_io_and_remove()
{
local size=$1
+ local dev_id
shift 1
- DEV_ID=$(_add_ublk_dev "$@")
+ dev_id=$(_add_ublk_dev "$@")
_check_add_dev $TID $?
[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
- if ! __run_io_and_remove "${DEV_ID}" "${size}" "no"; then
- echo "/dev/ublkc${DEV_ID} isn't removed"
+ if ! __run_io_and_remove "$dev_id" "${size}" "no"; then
+ echo "/dev/ublkc$dev_id isn't removed"
exit 255
fi
}
diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh
index 4b6ad441d500..7643e58637c8 100755
--- a/tools/testing/selftests/ublk/test_stress_02.sh
+++ b/tools/testing/selftests/ublk/test_stress_02.sh
@@ -4,19 +4,19 @@
. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
TID="stress_02"
ERR_CODE=0
-DEV_ID=-1
ublk_io_and_kill_daemon()
{
local size=$1
+ local dev_id
shift 1
- DEV_ID=$(_add_ublk_dev "$@")
+ dev_id=$(_add_ublk_dev "$@")
_check_add_dev $TID $?
[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)"
- if ! __run_io_and_remove "${DEV_ID}" "${size}" "yes"; then
- echo "/dev/ublkc${DEV_ID} isn't removed res ${res}"
+ if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then
+ echo "/dev/ublkc$dev_id isn't removed res ${res}"
exit 255
fi
}
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 06/13] selftests: ublk: run stress tests in parallel
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (4 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 05/13] selftests: ublk: make sure _add_ublk_dev can return in sub-shell Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 07/13] selftests: ublk: add two stress tests for zero copy feature Ming Lei
` (6 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Run stress tests in parallel, meantime add shell local function to
simplify the two stress tests.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/test_common.sh | 34 +++++++++++++++-
.../testing/selftests/ublk/test_stress_01.sh | 39 +++++++------------
.../testing/selftests/ublk/test_stress_02.sh | 39 +++++++------------
3 files changed, 63 insertions(+), 49 deletions(-)
diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index c43bd1d5c9c0..87fd0c824b77 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -230,7 +230,7 @@ __run_io_and_remove()
local kill_server=$3
fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \
- --rw=readwrite --iodepth=64 --size="${size}" --numjobs=4 \
+ --rw=readwrite --iodepth=256 --size="${size}" --numjobs=4 \
--runtime=20 --time_based > /dev/null 2>&1 &
sleep 2
if [ "${kill_server}" = "yes" ]; then
@@ -248,6 +248,38 @@ __run_io_and_remove()
wait
}
+run_io_and_remove()
+{
+ local size=$1
+ local dev_id
+ shift 1
+
+ dev_id=$(_add_ublk_dev "$@")
+ _check_add_dev "$TID" $?
+
+ [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
+ if ! __run_io_and_remove "$dev_id" "${size}" "no"; then
+ echo "/dev/ublkc$dev_id isn't removed"
+ exit 255
+ fi
+}
+
+run_io_and_kill_daemon()
+{
+ local size=$1
+ local dev_id
+ shift 1
+
+ dev_id=$(_add_ublk_dev "$@")
+ _check_add_dev "$TID" $?
+
+ [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)"
+ if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then
+ echo "/dev/ublkc$dev_id isn't removed res ${res}"
+ exit 255
+ fi
+}
+
_ublk_test_top_dir()
{
cd "$(dirname "$0")" && pwd
diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh
index 61fdbdfe70bc..7d3150f057d4 100755
--- a/tools/testing/selftests/ublk/test_stress_01.sh
+++ b/tools/testing/selftests/ublk/test_stress_01.sh
@@ -7,37 +7,28 @@ ERR_CODE=0
ublk_io_and_remove()
{
- local size=$1
- local dev_id
- shift 1
-
- dev_id=$(_add_ublk_dev "$@")
- _check_add_dev $TID $?
-
- [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
- if ! __run_io_and_remove "$dev_id" "${size}" "no"; then
- echo "/dev/ublkc$dev_id isn't removed"
- exit 255
+ run_io_and_remove "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
fi
}
-_prep_test "stress" "run IO and remove device"
-
-ublk_io_and_remove 8G -t null -q 4
-ERR_CODE=$?
-if [ ${ERR_CODE} -ne 0 ]; then
- _show_result $TID $ERR_CODE
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
fi
+_prep_test "stress" "run IO and remove device"
+
_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
-ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}"
-ERR_CODE=$?
-if [ ${ERR_CODE} -ne 0 ]; then
- _show_result $TID $ERR_CODE
-fi
+ublk_io_and_remove 8G -t null -q 4 &
+ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" &
+ublk_io_and_remove 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
-ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}"
-ERR_CODE=$?
_cleanup_test "stress"
_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh
index 7643e58637c8..1a9065125ae1 100755
--- a/tools/testing/selftests/ublk/test_stress_02.sh
+++ b/tools/testing/selftests/ublk/test_stress_02.sh
@@ -5,39 +5,30 @@
TID="stress_02"
ERR_CODE=0
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
ublk_io_and_kill_daemon()
{
- local size=$1
- local dev_id
- shift 1
-
- dev_id=$(_add_ublk_dev "$@")
- _check_add_dev $TID $?
-
- [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)"
- if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then
- echo "/dev/ublkc$dev_id isn't removed res ${res}"
- exit 255
+ run_io_and_kill_daemon "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
fi
}
_prep_test "stress" "run IO and kill ublk server"
-ublk_io_and_kill_daemon 8G -t null -q 4
-ERR_CODE=$?
-if [ ${ERR_CODE} -ne 0 ]; then
- _show_result $TID $ERR_CODE
-fi
-
_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
-ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}"
-ERR_CODE=$?
-if [ ${ERR_CODE} -ne 0 ]; then
- _show_result $TID $ERR_CODE
-fi
+ublk_io_and_kill_daemon 8G -t null -q 4 &
+ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" &
+ublk_io_and_kill_daemon 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
-ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}"
-ERR_CODE=$?
_cleanup_test "stress"
_show_result $TID $ERR_CODE
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 07/13] selftests: ublk: add two stress tests for zero copy feature
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (5 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 06/13] selftests: ublk: run stress tests in parallel Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 08/13] selftests: ublk: setup ring with IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN Ming Lei
` (5 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Add stress_03 & stress_04 for covering zero copy feature.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/Makefile | 2 +
.../testing/selftests/ublk/test_stress_03.sh | 38 +++++++++++++++++++
.../testing/selftests/ublk/test_stress_04.sh | 37 ++++++++++++++++++
3 files changed, 77 insertions(+)
create mode 100755 tools/testing/selftests/ublk/test_stress_03.sh
create mode 100755 tools/testing/selftests/ublk/test_stress_04.sh
diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile
index c7781efea0f3..7311e8f6bee7 100644
--- a/tools/testing/selftests/ublk/Makefile
+++ b/tools/testing/selftests/ublk/Makefile
@@ -21,6 +21,8 @@ TEST_PROGS += test_stripe_04.sh
TEST_PROGS += test_stress_01.sh
TEST_PROGS += test_stress_02.sh
+TEST_PROGS += test_stress_03.sh
+TEST_PROGS += test_stress_04.sh
TEST_GEN_PROGS_EXTENDED = kublk
diff --git a/tools/testing/selftests/ublk/test_stress_03.sh b/tools/testing/selftests/ublk/test_stress_03.sh
new file mode 100755
index 000000000000..e0854f71d35b
--- /dev/null
+++ b/tools/testing/selftests/ublk/test_stress_03.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
+TID="stress_03"
+ERR_CODE=0
+
+ublk_io_and_remove()
+{
+ run_io_and_remove "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
+ fi
+}
+
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+if ! _have_feature "ZERO_COPY"; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+_prep_test "stress" "run IO and remove device(zero copy)"
+
+_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
+
+ublk_io_and_remove 8G -t null -q 4 -z &
+ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" &
+ublk_io_and_remove 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+_cleanup_test "stress"
+_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_04.sh b/tools/testing/selftests/ublk/test_stress_04.sh
new file mode 100755
index 000000000000..1798a98387e8
--- /dev/null
+++ b/tools/testing/selftests/ublk/test_stress_04.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
+TID="stress_04"
+ERR_CODE=0
+
+ublk_io_and_kill_daemon()
+{
+ run_io_and_kill_daemon "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
+ fi
+}
+
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
+fi
+if ! _have_feature "ZERO_COPY"; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+_prep_test "stress" "run IO and kill ublk server(zero copy)"
+
+_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
+
+ublk_io_and_kill_daemon 8G -t null -q 4 -z &
+ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" &
+ublk_io_and_kill_daemon 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+_cleanup_test "stress"
+_show_result $TID $ERR_CODE
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 08/13] selftests: ublk: setup ring with IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (6 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 07/13] selftests: ublk: add two stress tests for zero copy feature Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 09/13] selftests: ublk: set queue pthread's cpu affinity Ming Lei
` (4 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
It is observed that this way is more efficient for fast nvme backing
file.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index f01d8618739b..a78eaa123859 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -346,7 +346,9 @@ static int ublk_queue_init(struct ublk_queue *q)
}
ret = ublk_setup_ring(&q->ring, ring_depth, cq_depth,
- IORING_SETUP_COOP_TASKRUN);
+ IORING_SETUP_COOP_TASKRUN |
+ IORING_SETUP_SINGLE_ISSUER |
+ IORING_SETUP_DEFER_TASKRUN);
if (ret < 0) {
ublk_err("ublk dev %d queue %d setup io_uring failed %d\n",
q->dev->dev_info.dev_id, q->q_id, ret);
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 09/13] selftests: ublk: set queue pthread's cpu affinity
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (7 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 08/13] selftests: ublk: setup ring with IORING_SETUP_SINGLE_ISSUER/IORING_SETUP_DEFER_TASKRUN Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 10/13] selftests: ublk: increase max nr_queues and queue depth Ming Lei
` (3 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
In NUMA machine, ublk IO performance is very sensitive with queue
pthread's affinity setting.
Retrieve queue's affinity and select the 1st cpu as queue thread's sched
affinity, and it is observed that single cpu task affinity can get
stable & good performance if client application is put on proper cpu.
Dump this info when adding one ublk device. Use shmem to communicate
queue's tid between parent and daemon.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 156 +++++++++++++++++++++++++--
tools/testing/selftests/ublk/kublk.h | 11 +-
2 files changed, 159 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index a78eaa123859..464abacdbbcb 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -206,10 +206,73 @@ static const char *ublk_dev_state_desc(struct ublk_dev *dev)
};
}
+static void ublk_print_cpu_set(const cpu_set_t *set, char *buf, unsigned len)
+{
+ unsigned done = 0;
+ int i;
+
+ for (i = 0; i < CPU_SETSIZE; i++) {
+ if (CPU_ISSET(i, set))
+ done += snprintf(&buf[done], len - done, "%d ", i);
+ }
+}
+
+static void ublk_adjust_affinity(cpu_set_t *set)
+{
+ int j, updated = 0;
+
+ /*
+ * Just keep the 1st CPU now.
+ *
+ * In future, auto affinity selection can be tried.
+ */
+ for (j = 0; j < CPU_SETSIZE; j++) {
+ if (CPU_ISSET(j, set)) {
+ if (!updated) {
+ updated = 1;
+ continue;
+ }
+ CPU_CLR(j, set);
+ }
+ }
+}
+
+/* Caller must free the allocated buffer */
+static int ublk_ctrl_get_affinity(struct ublk_dev *ctrl_dev, cpu_set_t **ptr_buf)
+{
+ struct ublk_ctrl_cmd_data data = {
+ .cmd_op = UBLK_U_CMD_GET_QUEUE_AFFINITY,
+ .flags = CTRL_CMD_HAS_DATA | CTRL_CMD_HAS_BUF,
+ };
+ cpu_set_t *buf;
+ int i, ret;
+
+ buf = malloc(sizeof(cpu_set_t) * ctrl_dev->dev_info.nr_hw_queues);
+ if (!buf)
+ return -ENOMEM;
+
+ for (i = 0; i < ctrl_dev->dev_info.nr_hw_queues; i++) {
+ data.data[0] = i;
+ data.len = sizeof(cpu_set_t);
+ data.addr = (__u64)&buf[i];
+
+ ret = __ublk_ctrl_cmd(ctrl_dev, &data);
+ if (ret < 0) {
+ free(buf);
+ return ret;
+ }
+ ublk_adjust_affinity(&buf[i]);
+ }
+
+ *ptr_buf = buf;
+ return 0;
+}
+
static void ublk_ctrl_dump(struct ublk_dev *dev)
{
struct ublksrv_ctrl_dev_info *info = &dev->dev_info;
struct ublk_params p;
+ cpu_set_t *affinity;
int ret;
ret = ublk_ctrl_get_params(dev, &p);
@@ -218,12 +281,31 @@ static void ublk_ctrl_dump(struct ublk_dev *dev)
return;
}
+ ret = ublk_ctrl_get_affinity(dev, &affinity);
+ if (ret < 0) {
+ ublk_err("failed to get affinity %m\n");
+ return;
+ }
+
ublk_log("dev id %d: nr_hw_queues %d queue_depth %d block size %d dev_capacity %lld\n",
info->dev_id, info->nr_hw_queues, info->queue_depth,
1 << p.basic.logical_bs_shift, p.basic.dev_sectors);
ublk_log("\tmax rq size %d daemon pid %d flags 0x%llx state %s\n",
info->max_io_buf_bytes, info->ublksrv_pid, info->flags,
ublk_dev_state_desc(dev));
+
+ if (affinity) {
+ char buf[512];
+ int i;
+
+ for (i = 0; i < info->nr_hw_queues; i++) {
+ ublk_print_cpu_set(&affinity[i], buf, sizeof(buf));
+ printf("\tqueue %u: tid %d affinity(%s)\n",
+ i, dev->q[i].tid, buf);
+ }
+ free(affinity);
+ }
+
fflush(stdout);
}
@@ -603,9 +685,24 @@ static int ublk_process_io(struct ublk_queue *q)
return reapped;
}
+static void ublk_queue_set_sched_affinity(const struct ublk_queue *q,
+ cpu_set_t *cpuset)
+{
+ if (sched_setaffinity(0, sizeof(*cpuset), cpuset) < 0)
+ ublk_err("ublk dev %u queue %u set affinity failed",
+ q->dev->dev_info.dev_id, q->q_id);
+}
+
+struct ublk_queue_info {
+ struct ublk_queue *q;
+ sem_t *queue_sem;
+ cpu_set_t *affinity;
+};
+
static void *ublk_io_handler_fn(void *data)
{
- struct ublk_queue *q = data;
+ struct ublk_queue_info *info = data;
+ struct ublk_queue *q = info->q;
int dev_id = q->dev->dev_info.dev_id;
int ret;
@@ -615,6 +712,10 @@ static void *ublk_io_handler_fn(void *data)
dev_id, q->q_id);
return NULL;
}
+ /* IO perf is sensitive with queue pthread affinity on NUMA machine*/
+ ublk_queue_set_sched_affinity(q, info->affinity);
+ sem_post(info->queue_sem);
+
ublk_dbg(UBLK_DBG_QUEUE, "tid %d: ublk dev %d queue %d started\n",
q->tid, dev_id, q->q_id);
@@ -640,7 +741,7 @@ static void ublk_set_parameters(struct ublk_dev *dev)
dev->dev_info.dev_id, ret);
}
-static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)
+static int ublk_send_dev_event(const struct dev_ctx *ctx, struct ublk_dev *dev, int dev_id)
{
uint64_t id;
int evtfd = ctx->_evtfd;
@@ -653,10 +754,14 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)
else
id = ERROR_EVTFD_DEVID;
+ if (dev && ctx->shadow_dev)
+ memcpy(&ctx->shadow_dev->q, &dev->q, sizeof(dev->q));
+
if (write(evtfd, &id, sizeof(id)) != sizeof(id))
return -EINVAL;
close(evtfd);
+ shmdt(ctx->shadow_dev);
return 0;
}
@@ -664,24 +769,46 @@ static int ublk_send_dev_event(const struct dev_ctx *ctx, int dev_id)
static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev)
{
- int ret, i;
- void *thread_ret;
const struct ublksrv_ctrl_dev_info *dinfo = &dev->dev_info;
+ struct ublk_queue_info *qinfo;
+ cpu_set_t *affinity_buf;
+ void *thread_ret;
+ sem_t queue_sem;
+ int ret, i;
ublk_dbg(UBLK_DBG_DEV, "%s enter\n", __func__);
+ qinfo = (struct ublk_queue_info *)calloc(sizeof(struct ublk_queue_info),
+ dinfo->nr_hw_queues);
+ if (!qinfo)
+ return -ENOMEM;
+
+ sem_init(&queue_sem, 0, 0);
ret = ublk_dev_prep(ctx, dev);
if (ret)
return ret;
+ ret = ublk_ctrl_get_affinity(dev, &affinity_buf);
+ if (ret)
+ return ret;
+
for (i = 0; i < dinfo->nr_hw_queues; i++) {
dev->q[i].dev = dev;
dev->q[i].q_id = i;
+
+ qinfo[i].q = &dev->q[i];
+ qinfo[i].queue_sem = &queue_sem;
+ qinfo[i].affinity = &affinity_buf[i];
pthread_create(&dev->q[i].thread, NULL,
ublk_io_handler_fn,
- &dev->q[i]);
+ &qinfo[i]);
}
+ for (i = 0; i < dinfo->nr_hw_queues; i++)
+ sem_wait(&queue_sem);
+ free(qinfo);
+ free(affinity_buf);
+
/* everything is fine now, start us */
ublk_set_parameters(dev);
ret = ublk_ctrl_start_dev(dev, getpid());
@@ -694,7 +821,7 @@ static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev)
if (ctx->fg)
ublk_ctrl_dump(dev);
else
- ublk_send_dev_event(ctx, dev->dev_info.dev_id);
+ ublk_send_dev_event(ctx, dev, dev->dev_info.dev_id);
/* wait until we are terminated */
for (i = 0; i < dinfo->nr_hw_queues; i++)
@@ -873,7 +1000,7 @@ static int __cmd_dev_add(const struct dev_ctx *ctx)
fail:
if (ret < 0)
- ublk_send_dev_event(ctx, -1);
+ ublk_send_dev_event(ctx, dev, -1);
ublk_ctrl_deinit(dev);
return ret;
}
@@ -887,6 +1014,16 @@ static int cmd_dev_add(struct dev_ctx *ctx)
if (ctx->fg)
goto run;
+ ctx->_shmid = shmget(IPC_PRIVATE, sizeof(struct ublk_dev), IPC_CREAT | 0666);
+ if (ctx->_shmid < 0) {
+ ublk_err("%s: failed to shmget %s\n", __func__, strerror(errno));
+ exit(-1);
+ }
+ ctx->shadow_dev = (struct ublk_dev *)shmat(ctx->_shmid, NULL, 0);
+ if (ctx->shadow_dev == (struct ublk_dev *)-1) {
+ ublk_err("%s: failed to shmat %s\n", __func__, strerror(errno));
+ exit(-1);
+ }
ctx->_evtfd = eventfd(0, 0);
if (ctx->_evtfd < 0) {
ublk_err("%s: failed to create eventfd %s\n", __func__, strerror(errno));
@@ -923,6 +1060,8 @@ static int cmd_dev_add(struct dev_ctx *ctx)
if (__cmd_dev_list(ctx) >= 0)
exit_code = EXIT_SUCCESS;
}
+ shmdt(ctx->shadow_dev);
+ shmctl(ctx->_shmid, IPC_RMID, NULL);
/* wait for child */
wait(NULL);
exit(exit_code);
@@ -989,6 +1128,9 @@ static int __cmd_dev_list(struct dev_ctx *ctx)
ublk_err("%s: can't get dev info from %d: %d\n",
__func__, ctx->dev_id, ret);
} else {
+ if (ctx->shadow_dev)
+ memcpy(&dev->q, ctx->shadow_dev->q, sizeof(dev->q));
+
ublk_ctrl_dump(dev);
}
diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h
index d87d6c376283..526b55bceb27 100644
--- a/tools/testing/selftests/ublk/kublk.h
+++ b/tools/testing/selftests/ublk/kublk.h
@@ -20,10 +20,15 @@
#include <sys/wait.h>
#include <sys/eventfd.h>
#include <sys/uio.h>
+#include <sys/ipc.h>
+#include <sys/shm.h>
#include <linux/io_uring.h>
#include <liburing.h>
-#include <linux/ublk_cmd.h>
+#include <semaphore.h>
+
+/* allow ublk_dep.h to override ublk_cmd.h */
#include "ublk_dep.h"
+#include <linux/ublk_cmd.h>
#define __maybe_unused __attribute__((unused))
#define MAX_BACK_FILES 4
@@ -72,6 +77,10 @@ struct dev_ctx {
unsigned int chunk_size;
int _evtfd;
+ int _shmid;
+
+ /* built from shmem, only for ublk_dump_dev() */
+ struct ublk_dev *shadow_dev;
};
struct ublk_ctrl_cmd_data {
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 10/13] selftests: ublk: increase max nr_queues and queue depth
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (8 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 09/13] selftests: ublk: set queue pthread's cpu affinity Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 11/13] selftests: ublk: support target specific command line Ming Lei
` (2 subsequent siblings)
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Increase max nr_queues to 32, and queue depth to 1024.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 2 +-
tools/testing/selftests/ublk/kublk.h | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index 464abacdbbcb..11cc8a1df2b8 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -1204,7 +1204,7 @@ static int cmd_dev_get_features(void)
static int cmd_dev_help(char *exe)
{
printf("%s add -t [null|loop] [-q nr_queues] [-d depth] [-n dev_id] [backfile1] [backfile2] ...\n", exe);
- printf("\t default: nr_queues=2(max 4), depth=128(max 128), dev_id=-1(auto allocation)\n");
+ printf("\t default: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n");
printf("%s del [-n dev_id] -a \n", exe);
printf("\t -a delete all devices -n delete specified device\n");
printf("%s list [-n dev_id] -a \n", exe);
diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h
index 526b55bceb27..5b6b473e1c9a 100644
--- a/tools/testing/selftests/ublk/kublk.h
+++ b/tools/testing/selftests/ublk/kublk.h
@@ -48,8 +48,8 @@
#define UBLKSRV_IO_IDLE_SECS 20
#define UBLK_IO_MAX_BYTES (1 << 20)
-#define UBLK_MAX_QUEUES 4
-#define UBLK_QUEUE_DEPTH 128
+#define UBLK_MAX_QUEUES 32
+#define UBLK_QUEUE_DEPTH 1024
#define UBLK_DBG_DEV (1U << 0)
#define UBLK_DBG_QUEUE (1U << 1)
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 11/13] selftests: ublk: support target specific command line
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (9 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 10/13] selftests: ublk: increase max nr_queues and queue depth Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 12/13] selftests: ublk: support user recovery Ming Lei
2025-04-07 13:15 ` [PATCH 13/13] selftests: ublk: add test_stress_05.sh Ming Lei
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Support target specific command line for making related command line code
handling more readable & clean.
Also helps for adding new features.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/kublk.c | 59 +++++++++++++++++++++++----
tools/testing/selftests/ublk/kublk.h | 20 +++++++--
tools/testing/selftests/ublk/stripe.c | 28 ++++++++++++-
3 files changed, 95 insertions(+), 12 deletions(-)
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index 11cc8a1df2b8..2e1963bee21c 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -5,6 +5,8 @@
#include "kublk.h"
+#define MAX_NR_TGT_ARG 64
+
unsigned int ublk_dbg_mask = UBLK_LOG;
static const struct ublk_tgt_ops *tgt_ops_list[] = {
&null_tgt_ops,
@@ -1203,12 +1205,25 @@ static int cmd_dev_get_features(void)
static int cmd_dev_help(char *exe)
{
- printf("%s add -t [null|loop] [-q nr_queues] [-d depth] [-n dev_id] [backfile1] [backfile2] ...\n", exe);
- printf("\t default: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n");
+ int i;
+
+ printf("%s add -t [null|loop|stripe] [-q nr_queues] [-d depth] [-n dev_id]\n", exe);
+ printf("\t[--foreground] [--quiet] [-z] [--debug_mask mask]\n");
+ printf("\t[target options] [backfile1] [backfile2] ...\n");
+ printf("\tdefault: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n");
+
+ for (i = 0; i < sizeof(tgt_ops_list) / sizeof(tgt_ops_list[0]); i++) {
+ const struct ublk_tgt_ops *ops = tgt_ops_list[i];
+
+ if (ops->usage)
+ ops->usage(ops);
+ }
+ printf("\n");
+
printf("%s del [-n dev_id] -a \n", exe);
- printf("\t -a delete all devices -n delete specified device\n");
+ printf("\t -a delete all devices -n delete specified device\n\n");
printf("%s list [-n dev_id] -a \n", exe);
- printf("\t -a list all devices, -n list specified device, default -a \n");
+ printf("\t -a list all devices, -n list specified device, default -a \n\n");
printf("%s features\n", exe);
return 0;
}
@@ -1225,9 +1240,9 @@ int main(int argc, char *argv[])
{ "quiet", 0, NULL, 0 },
{ "zero_copy", 0, NULL, 'z' },
{ "foreground", 0, NULL, 0 },
- { "chunk_size", 1, NULL, 0 },
{ 0, 0, 0, 0 }
};
+ const struct ublk_tgt_ops *ops = NULL;
int option_idx, opt;
const char *cmd = argv[1];
struct dev_ctx ctx = {
@@ -1235,13 +1250,15 @@ int main(int argc, char *argv[])
.nr_hw_queues = 2,
.dev_id = -1,
.tgt_type = "unknown",
- .chunk_size = 65536, /* def chunk size is 64K */
};
int ret = -EINVAL, i;
+ int tgt_argc = 1;
+ char *tgt_argv[MAX_NR_TGT_ARG] = { NULL };
if (argc == 1)
return ret;
+ opterr = 0;
optind = 2;
while ((opt = getopt_long(argc, argv, "t:n:d:q:az",
longopts, &option_idx)) != -1) {
@@ -1272,8 +1289,26 @@ int main(int argc, char *argv[])
ublk_dbg_mask = 0;
if (!strcmp(longopts[option_idx].name, "foreground"))
ctx.fg = 1;
- if (!strcmp(longopts[option_idx].name, "chunk_size"))
- ctx.chunk_size = strtol(optarg, NULL, 10);
+ break;
+ case '?':
+ /*
+ * target requires every option must have argument
+ */
+ if (argv[optind][0] == '-' || argv[optind - 1][0] != '-') {
+ fprintf(stderr, "every target option requires argument: %s %s\n",
+ argv[optind - 1], argv[optind]);
+ exit(EXIT_FAILURE);
+ }
+
+ if (tgt_argc < (MAX_NR_TGT_ARG - 1) / 2) {
+ tgt_argv[tgt_argc++] = argv[optind - 1];
+ tgt_argv[tgt_argc++] = argv[optind];
+ } else {
+ fprintf(stderr, "too many target options\n");
+ exit(EXIT_FAILURE);
+ }
+ optind += 1;
+ break;
}
}
@@ -1282,6 +1317,14 @@ int main(int argc, char *argv[])
ctx.files[ctx.nr_files++] = argv[i++];
}
+ ops = ublk_find_tgt(ctx.tgt_type);
+ if (ops && ops->parse_cmd_line) {
+ optind = 0;
+
+ tgt_argv[0] = ctx.tgt_type;
+ ops->parse_cmd_line(&ctx, tgt_argc, tgt_argv);
+ }
+
if (!strcmp(cmd, "add"))
ret = cmd_dev_add(&ctx);
else if (!strcmp(cmd, "del"))
diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h
index 5b6b473e1c9a..96a5eff436b3 100644
--- a/tools/testing/selftests/ublk/kublk.h
+++ b/tools/testing/selftests/ublk/kublk.h
@@ -61,6 +61,11 @@
struct ublk_dev;
struct ublk_queue;
+struct stripe_ctx {
+ /* stripe */
+ unsigned int chunk_size;
+};
+
struct dev_ctx {
char tgt_type[16];
unsigned long flags;
@@ -73,14 +78,15 @@ struct dev_ctx {
unsigned int all:1;
unsigned int fg:1;
- /* stripe */
- unsigned int chunk_size;
-
int _evtfd;
int _shmid;
/* built from shmem, only for ublk_dump_dev() */
struct ublk_dev *shadow_dev;
+
+ union {
+ struct stripe_ctx stripe;
+ };
};
struct ublk_ctrl_cmd_data {
@@ -117,6 +123,14 @@ struct ublk_tgt_ops {
int (*queue_io)(struct ublk_queue *, int tag);
void (*tgt_io_done)(struct ublk_queue *,
int tag, const struct io_uring_cqe *);
+
+ /*
+ * Target specific command line handling
+ *
+ * each option requires argument for target command line
+ */
+ void (*parse_cmd_line)(struct dev_ctx *ctx, int argc, char *argv[]);
+ void (*usage)(const struct ublk_tgt_ops *ops);
};
struct ublk_tgt {
diff --git a/tools/testing/selftests/ublk/stripe.c b/tools/testing/selftests/ublk/stripe.c
index 179731c3dd6f..5dbd6392d83d 100644
--- a/tools/testing/selftests/ublk/stripe.c
+++ b/tools/testing/selftests/ublk/stripe.c
@@ -281,7 +281,7 @@ static int ublk_stripe_tgt_init(const struct dev_ctx *ctx, struct ublk_dev *dev)
.max_sectors = dev->dev_info.max_io_buf_bytes >> 9,
},
};
- unsigned chunk_size = ctx->chunk_size;
+ unsigned chunk_size = ctx->stripe.chunk_size;
struct stripe_conf *conf;
unsigned chunk_shift;
loff_t bytes = 0;
@@ -344,10 +344,36 @@ static void ublk_stripe_tgt_deinit(struct ublk_dev *dev)
backing_file_tgt_deinit(dev);
}
+static void ublk_stripe_cmd_line(struct dev_ctx *ctx, int argc, char *argv[])
+{
+ static const struct option longopts[] = {
+ { "chunk_size", 1, NULL, 0 },
+ { 0, 0, 0, 0 }
+ };
+ int option_idx, opt;
+
+ ctx->stripe.chunk_size = 65536;
+ while ((opt = getopt_long(argc, argv, "",
+ longopts, &option_idx)) != -1) {
+ switch (opt) {
+ case 0:
+ if (!strcmp(longopts[option_idx].name, "chunk_size"))
+ ctx->stripe.chunk_size = strtol(optarg, NULL, 10);
+ }
+ }
+}
+
+static void ublk_stripe_usage(const struct ublk_tgt_ops *ops)
+{
+ printf("\tstripe: [--chunk_size chunk_size (default 65536)]\n");
+}
+
const struct ublk_tgt_ops stripe_tgt_ops = {
.name = "stripe",
.init_tgt = ublk_stripe_tgt_init,
.deinit_tgt = ublk_stripe_tgt_deinit,
.queue_io = ublk_stripe_queue_io,
.tgt_io_done = ublk_stripe_io_done,
+ .parse_cmd_line = ublk_stripe_cmd_line,
+ .usage = ublk_stripe_usage,
};
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 12/13] selftests: ublk: support user recovery
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (10 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 11/13] selftests: ublk: support target specific command line Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
2025-04-07 13:15 ` [PATCH 13/13] selftests: ublk: add test_stress_05.sh Ming Lei
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Add user recovery feature.
Meantime add user recovery test: generic_04 and generic_05(zero copy)
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/Makefile | 2 +
tools/testing/selftests/ublk/kublk.c | 96 +++++++++++++++++--
tools/testing/selftests/ublk/kublk.h | 1 +
tools/testing/selftests/ublk/test_common.sh | 57 ++++++++++-
.../testing/selftests/ublk/test_generic_04.sh | 40 ++++++++
.../testing/selftests/ublk/test_generic_05.sh | 44 +++++++++
6 files changed, 230 insertions(+), 10 deletions(-)
create mode 100755 tools/testing/selftests/ublk/test_generic_04.sh
create mode 100755 tools/testing/selftests/ublk/test_generic_05.sh
diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile
index 7311e8f6bee7..d93373384e93 100644
--- a/tools/testing/selftests/ublk/Makefile
+++ b/tools/testing/selftests/ublk/Makefile
@@ -6,6 +6,8 @@ LDLIBS += -lpthread -lm -luring
TEST_PROGS := test_generic_01.sh
TEST_PROGS += test_generic_02.sh
TEST_PROGS += test_generic_03.sh
+TEST_PROGS += test_generic_04.sh
+TEST_PROGS += test_generic_05.sh
TEST_PROGS += test_null_01.sh
TEST_PROGS += test_null_02.sh
diff --git a/tools/testing/selftests/ublk/kublk.c b/tools/testing/selftests/ublk/kublk.c
index 2e1963bee21c..677a0ba79f09 100644
--- a/tools/testing/selftests/ublk/kublk.c
+++ b/tools/testing/selftests/ublk/kublk.c
@@ -119,6 +119,27 @@ static int ublk_ctrl_start_dev(struct ublk_dev *dev,
return __ublk_ctrl_cmd(dev, &data);
}
+static int ublk_ctrl_start_user_recovery(struct ublk_dev *dev)
+{
+ struct ublk_ctrl_cmd_data data = {
+ .cmd_op = UBLK_U_CMD_START_USER_RECOVERY,
+ };
+
+ return __ublk_ctrl_cmd(dev, &data);
+}
+
+static int ublk_ctrl_end_user_recovery(struct ublk_dev *dev, int daemon_pid)
+{
+ struct ublk_ctrl_cmd_data data = {
+ .cmd_op = UBLK_U_CMD_END_USER_RECOVERY,
+ .flags = CTRL_CMD_HAS_DATA,
+ };
+
+ dev->dev_info.ublksrv_pid = data.data[0] = daemon_pid;
+
+ return __ublk_ctrl_cmd(dev, &data);
+}
+
static int ublk_ctrl_add_dev(struct ublk_dev *dev)
{
struct ublk_ctrl_cmd_data data = {
@@ -812,8 +833,12 @@ static int ublk_start_daemon(const struct dev_ctx *ctx, struct ublk_dev *dev)
free(affinity_buf);
/* everything is fine now, start us */
- ublk_set_parameters(dev);
- ret = ublk_ctrl_start_dev(dev, getpid());
+ if (ctx->recovery)
+ ret = ublk_ctrl_end_user_recovery(dev, getpid());
+ else {
+ ublk_set_parameters(dev);
+ ret = ublk_ctrl_start_dev(dev, getpid());
+ }
if (ret < 0) {
ublk_err("%s: ublk_ctrl_start_dev failed: %d\n", __func__, ret);
goto fail;
@@ -988,7 +1013,10 @@ static int __cmd_dev_add(const struct dev_ctx *ctx)
}
}
- ret = ublk_ctrl_add_dev(dev);
+ if (ctx->recovery)
+ ret = ublk_ctrl_start_user_recovery(dev);
+ else
+ ret = ublk_ctrl_add_dev(dev);
if (ret < 0) {
ublk_err("%s: can't add dev id %d, type %s ret %d\n",
__func__, dev_id, tgt_type, ret);
@@ -1203,12 +1231,14 @@ static int cmd_dev_get_features(void)
return ret;
}
-static int cmd_dev_help(char *exe)
+static void __cmd_create_help(char *exe, bool recovery)
{
int i;
- printf("%s add -t [null|loop|stripe] [-q nr_queues] [-d depth] [-n dev_id]\n", exe);
- printf("\t[--foreground] [--quiet] [-z] [--debug_mask mask]\n");
+ printf("%s %s -t [null|loop|stripe] [-q nr_queues] [-d depth] [-n dev_id]\n",
+ exe, recovery ? "recover" : "add");
+ printf("\t[--foreground] [--quiet] [-z] [--debug_mask mask] [-r 0|1 ] [-g 0|1]\n");
+ printf("\t[-e 0|1 ] [-i 0|1]\n");
printf("\t[target options] [backfile1] [backfile2] ...\n");
printf("\tdefault: nr_queues=2(max 32), depth=128(max 1024), dev_id=-1(auto allocation)\n");
@@ -1218,7 +1248,25 @@ static int cmd_dev_help(char *exe)
if (ops->usage)
ops->usage(ops);
}
+}
+
+static void cmd_add_help(char *exe)
+{
+ __cmd_create_help(exe, false);
+ printf("\n");
+}
+
+static void cmd_recover_help(char *exe)
+{
+ __cmd_create_help(exe, true);
+ printf("\tPlease provide exact command line for creating this device with real dev_id\n");
printf("\n");
+}
+
+static int cmd_dev_help(char *exe)
+{
+ cmd_add_help(exe);
+ cmd_recover_help(exe);
printf("%s del [-n dev_id] -a \n", exe);
printf("\t -a delete all devices -n delete specified device\n\n");
@@ -1240,6 +1288,10 @@ int main(int argc, char *argv[])
{ "quiet", 0, NULL, 0 },
{ "zero_copy", 0, NULL, 'z' },
{ "foreground", 0, NULL, 0 },
+ { "recovery", 1, NULL, 'r' },
+ { "recovery_fail_io", 1, NULL, 'e'},
+ { "recovery_reissue", 1, NULL, 'i'},
+ { "get_data", 1, NULL, 'g'},
{ 0, 0, 0, 0 }
};
const struct ublk_tgt_ops *ops = NULL;
@@ -1254,13 +1306,14 @@ int main(int argc, char *argv[])
int ret = -EINVAL, i;
int tgt_argc = 1;
char *tgt_argv[MAX_NR_TGT_ARG] = { NULL };
+ int value;
if (argc == 1)
return ret;
opterr = 0;
optind = 2;
- while ((opt = getopt_long(argc, argv, "t:n:d:q:az",
+ while ((opt = getopt_long(argc, argv, "t:n:d:q:r:e:i:az",
longopts, &option_idx)) != -1) {
switch (opt) {
case 'a':
@@ -1282,6 +1335,25 @@ int main(int argc, char *argv[])
case 'z':
ctx.flags |= UBLK_F_SUPPORT_ZERO_COPY | UBLK_F_USER_COPY;
break;
+ case 'r':
+ value = strtol(optarg, NULL, 10);
+ if (value)
+ ctx.flags |= UBLK_F_USER_RECOVERY;
+ break;
+ case 'e':
+ value = strtol(optarg, NULL, 10);
+ if (value)
+ ctx.flags |= UBLK_F_USER_RECOVERY | UBLK_F_USER_RECOVERY_FAIL_IO;
+ break;
+ case 'i':
+ value = strtol(optarg, NULL, 10);
+ if (value)
+ ctx.flags |= UBLK_F_USER_RECOVERY | UBLK_F_USER_RECOVERY_REISSUE;
+ break;
+ case 'g':
+ value = strtol(optarg, NULL, 10);
+ if (value)
+ ctx.flags |= UBLK_F_NEED_GET_DATA;
case 0:
if (!strcmp(longopts[option_idx].name, "debug_mask"))
ublk_dbg_mask = strtol(optarg, NULL, 16);
@@ -1327,7 +1399,15 @@ int main(int argc, char *argv[])
if (!strcmp(cmd, "add"))
ret = cmd_dev_add(&ctx);
- else if (!strcmp(cmd, "del"))
+ else if (!strcmp(cmd, "recover")) {
+ if (ctx.dev_id < 0) {
+ fprintf(stderr, "device id isn't provided for recovering\n");
+ ret = -EINVAL;
+ } else {
+ ctx.recovery = 1;
+ ret = cmd_dev_add(&ctx);
+ }
+ } else if (!strcmp(cmd, "del"))
ret = cmd_dev_del(&ctx);
else if (!strcmp(cmd, "list")) {
ctx.all = 1;
diff --git a/tools/testing/selftests/ublk/kublk.h b/tools/testing/selftests/ublk/kublk.h
index 96a5eff436b3..f5bf38d1da36 100644
--- a/tools/testing/selftests/ublk/kublk.h
+++ b/tools/testing/selftests/ublk/kublk.h
@@ -77,6 +77,7 @@ struct dev_ctx {
unsigned int logging:1;
unsigned int all:1;
unsigned int fg:1;
+ unsigned int recovery:1;
int _evtfd;
int _shmid;
diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index 87fd0c824b77..e822b2a2729a 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -169,8 +169,11 @@ _have_feature()
return 1
}
-_add_ublk_dev() {
+_create_ublk_dev() {
local dev_id;
+ local cmd=$1
+
+ shift 1
if [ ! -c /dev/ublk-control ]; then
return ${UBLK_SKIP_CODE}
@@ -181,7 +184,7 @@ _add_ublk_dev() {
fi
fi
- if ! dev_id=$("${UBLK_PROG}" add "$@" | grep "dev id" | awk -F '[ :]' '{print $3}'); then
+ if ! dev_id=$("${UBLK_PROG}" "$cmd" "$@" | grep "dev id" | awk -F '[ :]' '{print $3}'); then
echo "fail to add ublk dev $*"
return 255
fi
@@ -194,6 +197,23 @@ _add_ublk_dev() {
fi
}
+_add_ublk_dev() {
+ _create_ublk_dev "add" "$@"
+}
+
+_recover_ublk_dev() {
+ local dev_id
+ local state
+
+ dev_id=$(_create_ublk_dev "recover" "$@")
+ for ((j=0;j<20;j++)); do
+ state=$(_get_ublk_dev_state "${dev_id}")
+ [ "$state" == "LIVE" ] && break
+ sleep 1
+ done
+ echo "$state"
+}
+
# kill the ublk daemon and return ublk device state
__ublk_kill_daemon()
{
@@ -280,6 +300,39 @@ run_io_and_kill_daemon()
fi
}
+run_io_and_recover()
+{
+ local state
+ local dev_id
+
+ dev_id=$(_add_ublk_dev "$@")
+ _check_add_dev "$TID" $?
+
+ fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \
+ --rw=readwrite --iodepth=256 --size="${size}" --numjobs=4 \
+ --runtime=20 --time_based > /dev/null 2>&1 &
+ sleep 4
+
+ state=$(__ublk_kill_daemon "${dev_id}" "QUIESCED")
+ if [ "$state" != "QUIESCED" ]; then
+ echo "device isn't quiesced($state) after killing daemon"
+ return 255
+ fi
+
+ state=$(_recover_ublk_dev -n "$dev_id" "$@")
+ if [ "$state" != "LIVE" ]; then
+ echo "faile to recover to LIVE($state)"
+ return 255
+ fi
+
+ if ! __remove_ublk_dev_return "${dev_id}"; then
+ echo "delete dev ${dev_id} failed"
+ return 255
+ fi
+ wait
+}
+
+
_ublk_test_top_dir()
{
cd "$(dirname "$0")" && pwd
diff --git a/tools/testing/selftests/ublk/test_generic_04.sh b/tools/testing/selftests/ublk/test_generic_04.sh
new file mode 100755
index 000000000000..8a3bc080c577
--- /dev/null
+++ b/tools/testing/selftests/ublk/test_generic_04.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
+
+TID="generic_04"
+ERR_CODE=0
+
+ublk_run_recover_test()
+{
+ run_io_and_recover "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
+ fi
+}
+
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+_prep_test "recover" "basic recover function verification"
+
+_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
+
+ublk_run_recover_test -t null -q 2 -r 1 &
+ublk_run_recover_test -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" &
+ublk_run_recover_test -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+ublk_run_recover_test -t null -q 2 -r 1 -i 1 &
+ublk_run_recover_test -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" &
+ublk_run_recover_test -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+_cleanup_test "recover"
+_show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_generic_05.sh b/tools/testing/selftests/ublk/test_generic_05.sh
new file mode 100755
index 000000000000..714630b4b329
--- /dev/null
+++ b/tools/testing/selftests/ublk/test_generic_05.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
+
+TID="generic_04"
+ERR_CODE=0
+
+ublk_run_recover_test()
+{
+ run_io_and_recover "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
+ fi
+}
+
+if ! _have_program fio; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+if ! _have_feature "ZERO_COPY"; then
+ exit "$UBLK_SKIP_CODE"
+fi
+
+_prep_test "recover" "basic recover function verification (zero copy)"
+
+_create_backfile 0 256M
+_create_backfile 1 128M
+_create_backfile 2 128M
+
+ublk_run_recover_test -t null -q 2 -r 1 -z &
+ublk_run_recover_test -t loop -q 2 -r 1 -z "${UBLK_BACKFILES[0]}" &
+ublk_run_recover_test -t stripe -q 2 -r 1 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+ublk_run_recover_test -t null -q 2 -r 1 -z -i 1 &
+ublk_run_recover_test -t loop -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[0]}" &
+ublk_run_recover_test -t stripe -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+wait
+
+_cleanup_test "recover"
+_show_result $TID $ERR_CODE
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 13/13] selftests: ublk: add test_stress_05.sh
2025-04-07 13:15 [PATCH 00/13] ublk: one driver bug fix and selftest change Ming Lei
` (11 preceding siblings ...)
2025-04-07 13:15 ` [PATCH 12/13] selftests: ublk: support user recovery Ming Lei
@ 2025-04-07 13:15 ` Ming Lei
12 siblings, 0 replies; 19+ messages in thread
From: Ming Lei @ 2025-04-07 13:15 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Caleb Sander Mateos, Uday Shankar, Ming Lei
Add test_stress_05.sh for covering removing device with recovery
enabled.
io-hang has been observed with the following patch:
https://lore.kernel.org/linux-block/20250403-ublk_timeout-v3-1-aa09f76c7451@purestorage.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
tools/testing/selftests/ublk/Makefile | 1 +
.../testing/selftests/ublk/test_stress_05.sh | 64 +++++++++++++++++++
2 files changed, 65 insertions(+)
create mode 100755 tools/testing/selftests/ublk/test_stress_05.sh
diff --git a/tools/testing/selftests/ublk/Makefile b/tools/testing/selftests/ublk/Makefile
index d93373384e93..dddc64036aa1 100644
--- a/tools/testing/selftests/ublk/Makefile
+++ b/tools/testing/selftests/ublk/Makefile
@@ -25,6 +25,7 @@ TEST_PROGS += test_stress_01.sh
TEST_PROGS += test_stress_02.sh
TEST_PROGS += test_stress_03.sh
TEST_PROGS += test_stress_04.sh
+TEST_PROGS += test_stress_05.sh
TEST_GEN_PROGS_EXTENDED = kublk
diff --git a/tools/testing/selftests/ublk/test_stress_05.sh b/tools/testing/selftests/ublk/test_stress_05.sh
new file mode 100755
index 000000000000..a7071b10224d
--- /dev/null
+++ b/tools/testing/selftests/ublk/test_stress_05.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
+TID="stress_05"
+ERR_CODE=0
+
+run_io_and_remove()
+{
+ local size=$1
+ local dev_id
+ local dev_pid
+ shift 1
+
+ dev_id=$(_add_ublk_dev "$@")
+ _check_add_dev $TID $?
+
+ [ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
+
+ fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \
+ --rw=readwrite --iodepth=128 --size="${size}" --numjobs=4 \
+ --runtime=40 --time_based > /dev/null 2>&1 &
+ sleep 4
+
+ dev_pid=$(_get_ublk_daemon_pid "$dev_id")
+ kill -9 "$dev_pid"
+
+ if ! __remove_ublk_dev_return "${dev_id}"; then
+ echo "delete dev ${dev_id} failed"
+ return 255
+ fi
+}
+
+ublk_io_and_remove()
+{
+ run_io_and_remove "$@"
+ ERR_CODE=$?
+ if [ ${ERR_CODE} -ne 0 ]; then
+ echo "$TID failure: $*"
+ _show_result $TID $ERR_CODE
+ fi
+}
+
+_prep_test "stress" "run IO and remove device with recovery enabled"
+
+_create_backfile 0 256M
+_create_backfile 1 256M
+
+for reissue in $(seq 0 1); do
+ ublk_io_and_remove 8G -t null -q 4 -g 1 -r 1 -i "$reissue" &
+ ublk_io_and_remove 256M -t loop -q 4 -g 1 -r 1 -i "$reissue" "${UBLK_BACKFILES[0]}" &
+ wait
+done
+
+if _have_feature "ZERO_COPY"; then
+ for reissue in $(seq 0 1); do
+ ublk_io_and_remove 8G -t null -q 4 -g 1 -z -r 1 -i "$reissue" &
+ ublk_io_and_remove 256M -t loop -q 4 -g 1 -z -r 1 -i "$reissue" "${UBLK_BACKFILES[1]}" &
+ wait
+ done
+fi
+
+_cleanup_test "stress"
+_show_result $TID $ERR_CODE
--
2.47.0
^ permalink raw reply related [flat|nested] 19+ messages in thread