From: Ming Lei <tom.leiming@gmail.com>
To: Qiliang Yuan <qiliang.yuan@linux.dev>
Cc: "Martin K. Petersen" <mkp@kernel.org>,
Shuah Khan <shuah@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
target-devel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-doc@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
io-uring@vger.kernel.org,
Mike Christie <michael.christie@oracle.com>,
Bodo Stroesser <bostroesser@gmail.com>
Subject: Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore
Date: Wed, 18 Nov 2026 20:16:03 -0600 [thread overview]
Message-ID: <av5b4-b48QNzzyxz@fedora-laptop> (raw)
In-Reply-To: <20261119010233.2917434-1-qiliang.yuan@linux.dev>
On Wed, Nov 18, 2026 at 08:02:33PM -0500, Qiliang Yuan wrote:
> On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote:
> > Can you share your setup? It is supposed that you run fio from VM,
> > otherwise you can just compare ublk with scsi backed with tcmu.
> >
> > But if you run fio from VM, I believe you can setup virtio-scsi backed by
> > ublk block device directly, which should cover your requirement, or maybe
> > I miss something?
>
> Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16
> sessions over 127.0.0.1), fio runs in that guest on the initiator
> disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the
> vhost-scsi disk, with the target and the backstore server in the outer
> guest. In the ublk runs, kublk serves the file as /dev/ublkbN and
> target_core_iblock exports that through the same fabric.
>
> You are not missing anything. This backstore has block semantics: the
> kernel emulates the SCSI commands, PR and ALUA included, exactly as for
> iblock, so ublk behind target_core_iblock offers the same function on
> every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of
> the ublk device covers it as well. What the series adds over that is
> only performance, and outside tcm_loop the numbers I sent do not show a
> consistent gain.
But in your last reply, not see the setup(virtio-scsi over ublk) and
perf number/compare with your scsi:target optimization.
>
> [...]
>
> > My question is actually why there is regression wrt. 4k randread:
> >
> > workload TCM/USER (MB/s) uring (MB/s) change
> > 4k randread, 1 job 426 (409-463) 369 (366-384) -13%
> >
> > Maybe you can focus on single job perf only, which is easy to compare and
> > investigate.
>
> It comes from the scheduler moving fio between CPUs. The pwrite() I
> pointed to in my last mail is not it: doing that copy inside
> io_uring_enter() instead makes no difference.
>
> tcm_loop queues the LIO submission and completion work on the CPU that
> submitted the command, so with one job, fio's CPU always has bound
> kworkers and ksoftirqd runnable next to fio. The load balancer cannot
> move those and moves fio instead. This backstore gives it more chances:
> the completion work on fio's CPU handles 14 commands per run here
> against 24 with TCM/USER, so per 1000 I/Os the completion kworker and
> ksoftirqd run about 57 times against 35, fio is preempted as often, and
> it is moved two to three times as often. With balancing on, fio's own
> CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us
> with TCM/USER, spread over the whole submission path rather than in one
> function.
That is one disadvantage for running IO from kernel space, and it is also one
reason why ioutgt ( https://github.com/io-target/ioutgt ) performs much better
than nvmet, especially for nvmet/tcp, not mention ioutgt doesn't implement
io_uring rxzc yet, and there are also other network optimizations available,
all can be applied flexibly.
>
> Turning load balancing off for fio's CPUs and nothing else reverses the
> result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run,
> five alternating runs, median (min-max). fio runs in a cpuset of CPUs
> 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either
> a normal member (balanced) or an isolated partition (not balanced):
>
> fio CPUs 4-31, backstore bandwidth (MB/s) change
> balanced, TCM/USER 439 (398-462)
> balanced, uring 379 (371-388) -13.7%
> not balanced, TCM/USER 662 (544-710)
> not balanced, uring 760 (575-798) +14.8%
>
> Booting the guest with idle=poll removes the gap as well: with fio free
> to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s
> here.
>
> That points at what each switch costs in a guest. An idle vCPU executes
> HLT, which exits to the host and puts the vCPU thread to sleep; waking
> it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host
> wakeup and a VM entry, and the host may run the thread on another
> physical CPU, so the guest code then runs with cold caches. With
> idle=poll the vCPUs spin instead, stay on their physical CPUs and take
> the IPI without an exit. This backstore makes more of those switches
> per I/O, so it pays more of that cost, and fio's slowdown spread over
> the whole submission path looks like cold caches.
>
> What is measured is that idle=poll removes the gap and that the gap
> follows the switches and migrations. Which part of the cost is the exit
> and wakeup latency and which part is the vCPU thread landing on another
> physical CPU is inferred: I cannot trace the host side here. Waking an
> idle CPU is much cheaper on bare metal, so I expect the gap to be
> smaller there, but I have not measured it. idle=poll is only a
> diagnostic, it burns a host CPU per idle vCPU.
>
> [...]
>
> > When userspace target becomes more efficient, I'd suggest you to think
> > about why these applications can't be improved or refactored for the
> > potential (bigger)perf requirement.
>
> Agreed. Since this backstore only takes block commands, an application
> that can move to it can just as well move to ublk, and one that needs
> SCSI semantics in userspace is not served by it either.
Why? SCSI semantics can be served in userspace, just like existing
virtio-scsi user backend.
Such as, uring-scsi may be supported in guest IO emulation application(qemu),
or one uring-virtqueue bridge, the later may be done in both userspace or
vhost_scsi, I guess.
Thanks,
Ming
prev parent reply other threads:[~2026-11-19 2:16 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 0:17 [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 01/18] scsi: target: uring: Introduce " Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 02/18] scsi: target: uring: Allocate per-queue command descriptors Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 03/18] scsi: target: uring: Create a character device per backstore device Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 04/18] scsi: target: uring: Map the command descriptors into the server Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 05/18] scsi: target: uring: Queue commands until a server fetches them Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 06/18] scsi: target: uring: Hand commands out through TCMUR_IO_FETCH Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 07/18] scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 08/18] scsi: target: uring: Copy command data with pread() and pwrite() Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 09/18] scsi: target: uring: Fail in-flight commands when the server exits Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 10/18] scsi: target: uring: Time out commands the server does not complete Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 11/18] scsi: target: uring: Drop aborted commands that are still queued Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 12/18] scsi: target: uring: Support SYNCHRONIZE CACHE Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 13/18] scsi: target: uring: Implement UNMAP Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 14/18] scsi: target: uring: Handle WRITE SAME Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 15/18] scsi: target: uring: Register command data as io_uring fixed buffers Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 16/18] selftests: target_core_uring: Add a minimal server Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 17/18] selftests: target_core_uring: Test a device exported through tcm_loop Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 18/18] Documentation: target: Describe the uring backstore Qiliang Yuan
2026-11-16 2:32 ` [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Ming Lei
2026-11-17 2:46 ` Qiliang Yuan
2026-11-18 1:40 ` Ming Lei
2026-11-19 1:02 ` Qiliang Yuan
2026-11-19 2:16 ` Ming Lei [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av5b4-b48QNzzyxz@fedora-laptop \
--to=tom.leiming@gmail.com \
--cc=axboe@kernel.dk \
--cc=bostroesser@gmail.com \
--cc=corbet@lwn.net \
--cc=io-uring@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=michael.christie@oracle.com \
--cc=mkp@kernel.org \
--cc=qiliang.yuan@linux.dev \
--cc=rdunlap@infradead.org \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=target-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox