Re: [PATCH v25 07/20] block/mq-deadline: Enable zoned write pipelining

[Bart Van Assche <bvanassche@acm.org>]

On 10/21/25 2:01 PM, Damien Le Moal wrote:
> On 10/21/25 03:28, Bart Van Assche wrote:
>> On 10/17/25 10:31 PM, Damien Le Moal wrote:
>>> Maybe we need to rethink this, restarting from your main use case and why
>>> performance is not good. I think that said main use case is f2fs. So what
>>> happens with write throughput with it ? Why doesn't merging of small writes in
>>> the zone write plugs improve performance ? Wouldn't small modifications to f2fs
>>> zone write path improve things ?
>>
>> F2FS typically generates large writes if the I/O bandwidth is high (100
>> MiB or more). Write pipelining improves write performance even for large
>> writes but not by a spectacular percentage. Write pipelining only
>> results in a drastic performance improvement if the write size is kept
>> small (e.g. 4 KiB).
> 
> But you are talking about high queue dpeth 4K write pattern, right ? And if yes,
> BIO merging in the zone write plugs should generate much larger commands anyway.
> Have you verified that this is working as expected ?

Write pipelining improves performance even if bio merging is enabled
because with write pipelining enabled the Linux kernel doesn't wait for
a prior write to complete before the next write is sent to the storage
device.

>>> If the answers to all of the above is "no/does not work", what about a different
>>> approach: zone write plugging v2 with a single thread per CPU that does the
>>> pipelining without to force changes to other layers/change the API all over the
>>> block layer ?
>>
>> The block layer changes that I'm proposing are small, easy to maintain
>> and not invasive. Using a mutex when pipelining writes only as I
>> proposed in a previous email is a solution that will yield better
>> performance than delegating work to another thread. Obtaining an
>> uncontended mutex takes less than a microsecond. Delegating work to
>> another thread introduces a delay of 10 to 100 microseconds.
>>
>>> Unless you have a neat way to recreate the problem without Zoned UFS devices ?
>>
>> This patch series adds support in both the scsi_debug and null_blk
>> drivers for write pipelining. If the mq-deadline patches from this
>> series are reverted then the attached shell script sporadically reports
>> a write error on my test setup for the mq-deadline test cases.
> 
> I am not trying to check the correctness of your patches. I was wondering if
> there is an easy way to recreate the performance difference you are seeing with
> zoned UFS device easily. E.g. the 4 K write case you are describing above.

Yes, there is an easy way to recreate the performance difference. The
shell script attached to my previous email tests multiple combinations
of I/O schedulers, queue depths and write pipelining enabled/disabled.
The script that I shared disables I/O merging. Even if make the
following changes in that shell script:

--- a/test-pipelining-zoned-writes
+++ b/test-pipelining-zoned-writes
@@ -147,11 +147,11 @@ for mode in "none 0" "none 1" "mq-deadline 0" 
"mq-deadline 1"; do
         # Disable block layer request merging.
         dev="/dev/block/${basename}"
      fi
-    run_cmd "echo 4096 > /sys/class/block/${basename}/queue/max_sectors_kb"
+    #run_cmd "echo 4096 > 
/sys/class/block/${basename}/queue/max_sectors_kb"
      # 0: disable I/O statistics
      run_cmd "echo 0 > /sys/class/block/${basename}/queue/iostats"
      # 2: do not attempt any merges
-    run_cmd "echo 2 > /sys/class/block/${basename}/queue/nomerges"
+    #run_cmd "echo 2 > /sys/class/block/${basename}/queue/nomerges"
      # 2: complete on the requesting CPU
      run_cmd "echo 2 > /sys/class/block/${basename}/queue/rq_affinity"
      for iopattern in write randwrite; do

then I still see a significant performance improvement for the null_blk
driver (command-line option -n):

==== iosched=none preserves_write_order=0 iopattern=write qd=1
   write: IOPS=6503, BW=25.4MiB/s (26.6MB/s)(762MiB/30003msec); 762 zone 
resets
==== iosched=none preserves_write_order=0 iopattern=randwrite qd=1
   write: IOPS=6469, BW=25.3MiB/s (26.5MB/s)(758MiB/30003msec); 758 zone 
resets
==== iosched=none preserves_write_order=1 iopattern=write qd=1
   write: IOPS=5566, BW=21.7MiB/s (22.8MB/s)(652MiB/30010msec); 652 zone 
resets
==== iosched=none preserves_write_order=1 iopattern=write qd=64
   write: IOPS=15.3k, BW=59.9MiB/s (62.8MB/s)(1797MiB/30001msec); 1796 
zone resets
==== iosched=none preserves_write_order=1 iopattern=randwrite qd=1
   write: IOPS=5575, BW=21.8MiB/s (22.8MB/s)(653MiB/30005msec); 653 zone 
resets
==== iosched=none preserves_write_order=1 iopattern=randwrite qd=64
   write: IOPS=15.5k, BW=60.7MiB/s (63.7MB/s)(1821MiB/30001msec); 1821 
zone resets

As one can see above, if the queue depth equals 64 and with write
pipelining enabled, IOPS are about 2.5x higher compared to
queue depth 1 and write pipelining disabled.

Thanks,

Bart.