From: Jens Axboe <axboe@kernel.dk>
To: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 05/11] block: avoid ordered task state change for polled IO
Date: Tue, 13 Nov 2018 08:42:27 -0700 [thread overview]
Message-ID: <20181113154233.15256-6-axboe@kernel.dk> (raw)
In-Reply-To: <20181113154233.15256-1-axboe@kernel.dk>
Ensure that writes to the dio/bio waiter field are ordered
correctly. With the smp_rmb() before the READ_ONCE() check,
we should be able to use a more relaxed ordering for the
task state setting.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
fs/block_dev.c | 11 +++++++++--
fs/iomap.c | 6 +++++-
2 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index c039abfb2052..2f920c03996e 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -181,6 +181,7 @@ static void blkdev_bio_end_io_simple(struct bio *bio)
struct task_struct *waiter = bio->bi_private;
WRITE_ONCE(bio->bi_private, NULL);
+ smp_wmb();
wake_up_process(waiter);
}
@@ -237,9 +238,12 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
qc = submit_bio(&bio);
for (;;) {
- set_current_state(TASK_UNINTERRUPTIBLE);
+ __set_current_state(TASK_UNINTERRUPTIBLE);
+
+ smp_rmb();
if (!READ_ONCE(bio.bi_private))
break;
+
if (!(iocb->ki_flags & IOCB_HIPRI) ||
!blk_poll(bdev_get_queue(bdev), qc))
io_schedule();
@@ -305,6 +309,7 @@ static void blkdev_bio_end_io(struct bio *bio)
struct task_struct *waiter = dio->waiter;
WRITE_ONCE(dio->waiter, NULL);
+ smp_wmb();
wake_up_process(waiter);
}
}
@@ -403,7 +408,9 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
return -EIOCBQUEUED;
for (;;) {
- set_current_state(TASK_UNINTERRUPTIBLE);
+ __set_current_state(TASK_UNINTERRUPTIBLE);
+
+ smp_rmb();
if (!READ_ONCE(dio->waiter))
break;
diff --git a/fs/iomap.c b/fs/iomap.c
index f61d13dfdf09..7898927e758e 100644
--- a/fs/iomap.c
+++ b/fs/iomap.c
@@ -1524,7 +1524,9 @@ static void iomap_dio_bio_end_io(struct bio *bio)
if (atomic_dec_and_test(&dio->ref)) {
if (dio->wait_for_completion) {
struct task_struct *waiter = dio->submit.waiter;
+
WRITE_ONCE(dio->submit.waiter, NULL);
+ smp_wmb();
wake_up_process(waiter);
} else if (dio->flags & IOMAP_DIO_WRITE) {
struct inode *inode = file_inode(dio->iocb->ki_filp);
@@ -1888,7 +1890,9 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
return -EIOCBQUEUED;
for (;;) {
- set_current_state(TASK_UNINTERRUPTIBLE);
+ __set_current_state(TASK_UNINTERRUPTIBLE);
+
+ smp_rmb();
if (!READ_ONCE(dio->submit.waiter))
break;
--
2.17.1
next prev parent reply other threads:[~2018-11-13 15:42 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-13 15:42 [PATCHSET v2 0/11] Various block optimizations Jens Axboe
2018-11-13 15:42 ` [PATCH 01/11] nvme: don't disable local ints for polled queue Jens Axboe
2018-11-13 15:42 ` [PATCH 02/11] block: add queue_is_mq() helper Jens Axboe
2018-11-14 15:23 ` Christoph Hellwig
2018-11-13 15:42 ` [PATCH 03/11] blk-mq: embed blk_mq_ops directly in the request queue Jens Axboe
2018-11-13 15:42 ` [PATCH 04/11] blk-rq-qos: inline check for q->rq_qos functions Jens Axboe
2018-11-13 15:42 ` Jens Axboe [this message]
2018-11-14 2:29 ` [PATCH 05/11] block: avoid ordered task state change for polled IO jianchao.wang
2018-11-14 2:35 ` Jens Axboe
2018-11-13 15:42 ` [PATCH 06/11] block: add polled wakeup task helper Jens Axboe
2018-11-13 15:52 ` Keith Busch
2018-11-13 15:58 ` Jens Axboe
2018-11-13 16:59 ` Jens Axboe
2018-11-13 15:42 ` [PATCH 07/11] block: have ->poll_fn() return number of entries polled Jens Axboe
2018-11-13 15:42 ` [PATCH 08/11] blk-mq: when polling for IO, look for any completion Jens Axboe
2018-11-13 15:42 ` [PATCH 09/11] block: make blk_poll() take a parameter on whether to spin or not Jens Axboe
2018-11-13 15:42 ` [PATCH 10/11] block: for async O_DIRECT, mark us as polling if asked to Jens Axboe
2018-11-13 15:42 ` [PATCH 11/11] block: don't plug for aio/O_DIRECT HIPRI IO Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181113154233.15256-6-axboe@kernel.dk \
--to=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).