Linux Documentation
 help / color / mirror / Atom feed
From: Ming Lei <tom.leiming@gmail.com>
To: Qiliang Yuan <qiliang.yuan@linux.dev>
Cc: "Martin K. Petersen" <mkp@kernel.org>,
	Shuah Khan <shuah@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>,
	Randy Dunlap <rdunlap@infradead.org>,
	linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
	target-devel@vger.kernel.org, linux-kselftest@vger.kernel.org,
	linux-doc@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
	io-uring@vger.kernel.org,
	Mike Christie <michael.christie@oracle.com>,
	Bodo Stroesser <bostroesser@gmail.com>
Subject: Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore
Date: Wed, 18 Nov 2026 20:16:03 -0600	[thread overview]
Message-ID: <av5b4-b48QNzzyxz@fedora-laptop> (raw)
In-Reply-To: <20261119010233.2917434-1-qiliang.yuan@linux.dev>

On Wed, Nov 18, 2026 at 08:02:33PM -0500, Qiliang Yuan wrote:
> On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote:
> > Can you share your setup? It is supposed that you run fio from VM,
> > otherwise you can just compare ublk with scsi backed with tcmu.
> >
> > But if you run fio from VM, I believe you can setup virtio-scsi backed by
> > ublk block device directly, which should cover your requirement, or maybe
> > I miss something?
> 
> Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16
> sessions over 127.0.0.1), fio runs in that guest on the initiator
> disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the
> vhost-scsi disk, with the target and the backstore server in the outer
> guest. In the ublk runs, kublk serves the file as /dev/ublkbN and
> target_core_iblock exports that through the same fabric.
> 
> You are not missing anything. This backstore has block semantics: the
> kernel emulates the SCSI commands, PR and ALUA included, exactly as for
> iblock, so ublk behind target_core_iblock offers the same function on
> every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of
> the ublk device covers it as well. What the series adds over that is
> only performance, and outside tcm_loop the numbers I sent do not show a
> consistent gain.

But in your last reply, not see the setup(virtio-scsi over ublk) and
perf number/compare with your scsi:target optimization.

> 
> [...]
> 
> > My question is actually why there is regression wrt. 4k randread:
> >
> >  workload               TCM/USER (MB/s)   uring (MB/s)          change
> >   4k randread, 1 job      426 (409-463)      369 (366-384)         -13%
> >
> > Maybe you can focus on single job perf only, which is easy to compare and
> > investigate.
> 
> It comes from the scheduler moving fio between CPUs. The pwrite() I
> pointed to in my last mail is not it: doing that copy inside
> io_uring_enter() instead makes no difference.
> 
> tcm_loop queues the LIO submission and completion work on the CPU that
> submitted the command, so with one job, fio's CPU always has bound
> kworkers and ksoftirqd runnable next to fio. The load balancer cannot
> move those and moves fio instead. This backstore gives it more chances:
> the completion work on fio's CPU handles 14 commands per run here
> against 24 with TCM/USER, so per 1000 I/Os the completion kworker and
> ksoftirqd run about 57 times against 35, fio is preempted as often, and
> it is moved two to three times as often. With balancing on, fio's own
> CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us
> with TCM/USER, spread over the whole submission path rather than in one
> function.

That is one disadvantage for running IO from kernel space, and it is also one
reason why ioutgt ( https://github.com/io-target/ioutgt ) performs much better
than nvmet, especially for nvmet/tcp, not mention ioutgt doesn't implement
io_uring rxzc yet, and there are also other network optimizations available,
all can be applied flexibly.

> 
> Turning load balancing off for fio's CPUs and nothing else reverses the
> result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run,
> five alternating runs, median (min-max). fio runs in a cpuset of CPUs
> 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either
> a normal member (balanced) or an isolated partition (not balanced):
> 
>   fio CPUs 4-31, backstore      bandwidth (MB/s)   change
>   balanced, TCM/USER               439 (398-462)
>   balanced, uring                  379 (371-388)   -13.7%
>   not balanced, TCM/USER           662 (544-710)
>   not balanced, uring              760 (575-798)   +14.8%
> 
> Booting the guest with idle=poll removes the gap as well: with fio free
> to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s
> here.
> 
> That points at what each switch costs in a guest. An idle vCPU executes
> HLT, which exits to the host and puts the vCPU thread to sleep; waking
> it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host
> wakeup and a VM entry, and the host may run the thread on another
> physical CPU, so the guest code then runs with cold caches. With
> idle=poll the vCPUs spin instead, stay on their physical CPUs and take
> the IPI without an exit. This backstore makes more of those switches
> per I/O, so it pays more of that cost, and fio's slowdown spread over
> the whole submission path looks like cold caches.
> 
> What is measured is that idle=poll removes the gap and that the gap
> follows the switches and migrations. Which part of the cost is the exit
> and wakeup latency and which part is the vCPU thread landing on another
> physical CPU is inferred: I cannot trace the host side here. Waking an
> idle CPU is much cheaper on bare metal, so I expect the gap to be
> smaller there, but I have not measured it. idle=poll is only a
> diagnostic, it burns a host CPU per idle vCPU.
> 
> [...]
> 
> > When userspace target becomes more efficient, I'd suggest you to think
> > about why these applications can't be improved or refactored for the
> > potential (bigger)perf requirement.
> 
> Agreed. Since this backstore only takes block commands, an application
> that can move to it can just as well move to ublk, and one that needs
> SCSI semantics in userspace is not served by it either.

Why? SCSI semantics can be served in userspace, just like existing
virtio-scsi user backend.

Such as, uring-scsi may be supported in guest IO emulation application(qemu),
or one uring-virtqueue bridge, the later may be done in both userspace or
vhost_scsi, I guess.

Thanks,
Ming

      reply	other threads:[~2026-11-19  2:16 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  0:17 [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 01/18] scsi: target: uring: Introduce " Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 02/18] scsi: target: uring: Allocate per-queue command descriptors Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 03/18] scsi: target: uring: Create a character device per backstore device Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 04/18] scsi: target: uring: Map the command descriptors into the server Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 05/18] scsi: target: uring: Queue commands until a server fetches them Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 06/18] scsi: target: uring: Hand commands out through TCMUR_IO_FETCH Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 07/18] scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 08/18] scsi: target: uring: Copy command data with pread() and pwrite() Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 09/18] scsi: target: uring: Fail in-flight commands when the server exits Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 10/18] scsi: target: uring: Time out commands the server does not complete Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 11/18] scsi: target: uring: Drop aborted commands that are still queued Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 12/18] scsi: target: uring: Support SYNCHRONIZE CACHE Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 13/18] scsi: target: uring: Implement UNMAP Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 14/18] scsi: target: uring: Handle WRITE SAME Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 15/18] scsi: target: uring: Register command data as io_uring fixed buffers Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 16/18] selftests: target_core_uring: Add a minimal server Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 17/18] selftests: target_core_uring: Test a device exported through tcm_loop Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 18/18] Documentation: target: Describe the uring backstore Qiliang Yuan
2026-11-16  2:32 ` [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Ming Lei
2026-11-17  2:46   ` Qiliang Yuan
2026-11-18  1:40     ` Ming Lei
2026-11-19  1:02       ` Qiliang Yuan
2026-11-19  2:16         ` Ming Lei [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av5b4-b48QNzzyxz@fedora-laptop \
    --to=tom.leiming@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=bostroesser@gmail.com \
    --cc=corbet@lwn.net \
    --cc=io-uring@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=michael.christie@oracle.com \
    --cc=mkp@kernel.org \
    --cc=qiliang.yuan@linux.dev \
    --cc=rdunlap@infradead.org \
    --cc=shuah@kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=target-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox