[PATCH] nvmet: Use direct IO for writes

[PATCH] nvmet: Use direct IO for writes

[Sagi Grimberg]

We're designed to work with high-end devices where
direct IO makes perfect sense. We noticed that we
context switch by scheduling kblockd instead of going
directly to the device without REQ_SYNC for writes.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/io-cmd.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/io-cmd.c b/drivers/nvme/target/io-cmd.c
index 2cd069b691ae..4132b6b98182 100644
--- a/drivers/nvme/target/io-cmd.c
+++ b/drivers/nvme/target/io-cmd.c
@@ -58,6 +58,7 @@ static void nvmet_execute_rw(struct nvmet_req *req)
 
 	if (req->cmd->rw.opcode == nvme_cmd_write) {
 		op = REQ_OP_WRITE;
+		op_flags = WRITE_ODIRECT;
 		if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))
 			op_flags |= REQ_FUA;
 	} else {
-- 
1.9.1

[PATCH] nvmet: Use direct IO for writes

[Christoph Hellwig]

On Wed, Sep 21, 2016@11:10:50AM -0700, Sagi Grimberg wrote:
> We're designed to work with high-end devices where
> direct IO makes perfect sense. We noticed that we
> context switch by scheduling kblockd instead of going
> directly to the device without REQ_SYNC for writes.

This looks reasonable.  But I still wonder why we bother to inject delay
for any fast blk-mq device (background: Sagi told me he is observing
issues without this on a NVMe PCIe card backend)

[PATCH] nvmet: Use direct IO for writes

[Jens Axboe]

On 09/22/2016 07:57 AM, Christoph Hellwig wrote:
> On Wed, Sep 21, 2016@11:10:50AM -0700, Sagi Grimberg wrote:
>> We're designed to work with high-end devices where
>> direct IO makes perfect sense. We noticed that we
>> context switch by scheduling kblockd instead of going
>> directly to the device without REQ_SYNC for writes.
>
> This looks reasonable.  But I still wonder why we bother to inject delay
> for any fast blk-mq device (background: Sagi told me he is observing
> issues without this on a NVMe PCIe card backend)

Batching/merging. It's not like we're doing a calculated explicit delay,
it's just async running the queue.

-- 
Jens Axboe

[PATCH] nvmet: Use direct IO for writes

[Sagi Grimberg]

>> This looks reasonable.  But I still wonder why we bother to inject delay
>> for any fast blk-mq device (background: Sagi told me he is observing
>> issues without this on a NVMe PCIe card backend)
>
> Batching/merging. It's not like we're doing a calculated explicit delay,
> it's just async running the queue.

So are we OK with bypassing that in nvmet?

[PATCH] nvmet: Use direct IO for writes

[Jens Axboe]

On 09/23/2016 03:00 PM, Sagi Grimberg wrote:
>
>>> This looks reasonable.  But I still wonder why we bother to inject delay
>>> for any fast blk-mq device (background: Sagi told me he is observing
>>> issues without this on a NVMe PCIe card backend)
>>
>> Batching/merging. It's not like we're doing a calculated explicit delay,
>> it's just async running the queue.
>
> So are we OK with bypassing that in nvmet?

I'm OK with the patch as posted.

-- 
Jens Axboe

[PATCH] nvmet: Use direct IO for writes

[Sagi Grimberg]

>> So are we OK with bypassing that in nvmet?
>
> I'm OK with the patch as posted.

Thanks, I'll queue it up for 4.9

[PATCH] nvmet: Use direct IO for writes

[Jens Axboe]

On 09/23/2016 03:12 PM, Sagi Grimberg wrote:
>
>>> So are we OK with bypassing that in nvmet?
>>
>> I'm OK with the patch as posted.
>
> Thanks, I'll queue it up for 4.9

You can add my Reviewed-by, fwiw.

-- 
Jens Axboe

[PATCH] nvmet: Use direct IO for writes

[Christoph Hellwig]

On Fri, Sep 23, 2016@02:12:38PM -0700, Sagi Grimberg wrote:
>
>>> So are we OK with bypassing that in nvmet?
>>
>> I'm OK with the patch as posted.
>
> Thanks, I'll queue it up for 4.9

Btw, it might be a good idea to send our for-4.9 branch to Jens
so it gets some linux-next exposure.