From: Keith Busch <kbusch@kernel.org>
To: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>,
marcan@marcan.st, sven@svenpeter.dev, axboe@kernel.dk,
james.smart@broadcom.com, alyssa@rosenzweig.io,
asahi@lists.linux.dev, linux-nvme@lists.infradead.org,
kch@nvidia.com
Subject: Re: [PATCH] nvme: don't set a virt_boundary unless needed
Date: Thu, 21 Dec 2023 10:03:40 -0700 [thread overview]
Message-ID: <ZYRv7Fo6oCHgETyk@kbusch-mbp> (raw)
In-Reply-To: <20231221121746.GA17956@lst.de>
On Thu, Dec 21, 2023 at 01:17:46PM +0100, Christoph Hellwig wrote:
> On Thu, Dec 21, 2023 at 11:30:38AM +0200, Sagi Grimberg wrote:
> >
> >> NVMe PRPs are a pain and force the expensive virt_boundary checking on
> >> block layer, prevent secure passthrough and require scatter/gather I/O
> >> to be split into multiple commands which is problematic for the upcoming
> >> atomic write support.
> >
> > But is the threshold still correct? meaning for I/Os small enough the
> > device will have lower performance? I'm not advocating that we keep it,
> > but we should at least mention the tradeoff in the change log.
>
> Chaitanya benchmarked it on the first generation of devices that
> supported SGLs. On the only SGL-enabled device I have there is no
> performance penality for using SGLs on small transfer, but I'd love
> to see numbers from other setups.
It's the larger transfers where it gets worse. To exaggerate the
difference, consider send a 2MB write with virtually aligned but
discontiguous user buffer: 512 folios.
PRP fits in 1 prp_page_pool block.
SGL needs 3 prp_page_pool blocks, tripling the command's memory usage.
next prev parent reply other threads:[~2023-12-21 17:03 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-21 8:48 [PATCH] nvme: don't set a virt_boundary unless needed Christoph Hellwig
2023-12-21 9:30 ` Sagi Grimberg
2023-12-21 12:17 ` Christoph Hellwig
2023-12-21 12:32 ` Sagi Grimberg
2023-12-21 12:40 ` Christoph Hellwig
2023-12-25 9:13 ` Sagi Grimberg
2023-12-21 17:03 ` Keith Busch [this message]
2023-12-25 9:20 ` Sagi Grimberg
2023-12-22 1:16 ` Max Gurtovoy
2023-12-25 10:08 ` Sagi Grimberg
2023-12-25 10:36 ` Max Gurtovoy
2023-12-25 10:44 ` Sagi Grimberg
2023-12-25 12:31 ` Max Gurtovoy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZYRv7Fo6oCHgETyk@kbusch-mbp \
--to=kbusch@kernel.org \
--cc=alyssa@rosenzweig.io \
--cc=asahi@lists.linux.dev \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=james.smart@broadcom.com \
--cc=kch@nvidia.com \
--cc=linux-nvme@lists.infradead.org \
--cc=marcan@marcan.st \
--cc=sagi@grimberg.me \
--cc=sven@svenpeter.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox