* [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup
@ 2022-11-29 23:28 Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES Chaitanya Kulkarni
` (7 more replies)
0 siblings, 8 replies; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="y", Size: 52663 bytes --]
Hi,
Add support for non-trivial I/O operations REQ_OP_WRITE_ZEROES
and REQ_OP_ZONE_RESET in null_blk so we can write the blktests.
This also has few cleanup patches.
Below is the test report for REQ_OP_WRITE_ZEROES with ext2/ext4
and REQ_OP_ZONE_RESET_ALL with module param parameter and
blkzone reset command.
-ck
v3->v4:-
1. Add REQ_OP_ZONE_RESET_ALL patch in this series.
v2->v3:-
1. Fix the commit log in patch 3. (Bart)
2. Fix the commit log in patch 4. (Bart)
v1->v2:-
1. Merge patch 1 and 2 (Damien).
2. Change SECT_INX_IN_PAGE->SECT_OFFSET_IN_PAGE. (Damien)
3. Add reviewed-by tags. (Damien)
4. Add discard/write-zeroes module param to set the respective
max sector length.
Chaitanya Kulkarni (8):
null_blk: allow REQ_OP_WRITE_ZEROES
null_blk: code cleaup
null_blk: initialize cmd->bio in __alloc_cmd()
null_blk: don't use magic numbers in the code
null_blk: remove extra space in switch condition
null_blk: add param to set max disacrd sectors
null_blk: add param to set max write-zeroes sects
null-blk: allow REQ_OP_ZONE_RESET_ALL to configure
drivers/block/null_blk/main.c | 156 +++++++++++++++++++++++-------
drivers/block/null_blk/null_blk.h | 4 +
drivers/block/null_blk/zoned.c | 3 +-
3 files changed, 126 insertions(+), 37 deletions(-)
* zone_reset_all module param test :-
null-blk-big (for-next) # gitlog -9
e6379b3d9131 (HEAD -> for-next) null-blk: allow REQ_OP_ZONE_RESET_ALL to configure
2e183ad4c9fe null_blk: add param to set max write-zeroes sects
7835bf24ff04 null_blk: add param to set max disacrd sectors
d0b50155e1b0 null_blk: remove extra space in switch condition
5ba8509a3ea9 null_blk: don't use magic numbers in the code
e5ad7ffea415 null_blk: initialize cmd->bio in __alloc_cmd()
e8afe2c633e0 null_blk: code cleaup
6739fbf5fbb1 null_blk: allow REQ_OP_WRITE_ZEROES
14c66466ce0d Merge branch 'for-next' of git://git.kernel.dk/linux-block into for-next
null-blk-big (for-next) #
linux-block (for-next) # modprobe null_blk zoned=1 zone_size=128 gb=1 zone_reset_all=0
linux-block (for-next) # blkzone reset /dev/nullb0
linux-block (for-next) # dmesg -c
[ 1512.814822] null_blk: disk nullb0 created
[ 1512.814826] null_blk: module loaded
linux-block (for-next) # dd if=/dev/zero of=/dev/nullb0
dd: writing to '/dev/nullb0': No space left on device
2097153+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.75027 s, 123 MB/s
linux-block (for-next) # blkzone reset /dev/nullb0
linux-block (for-next) # dmesg -c
[ 1538.241929] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241939] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241942] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241945] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241947] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241949] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241951] null_blk: null_zone_mgmt 613 ZONE_RESET
[ 1538.241953] null_blk: null_zone_mgmt 613 ZONE_RESET
linux-block (for-next) # modprobe -r null_blk
linux-block (for-next) # modprobe null_blk zoned=1 zone_size=128 gb=1
linux-block (for-next) # dmesg -c
[ 1558.097021] null_blk: disk nullb0 created
[ 1558.097026] null_blk: module loaded
linux-block (for-next) # dd if=/dev/zero of=/dev/nullb0
dd: writing to '/dev/nullb0': No space left on device
2097153+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.70702 s, 123 MB/s
linux-block (for-next) # blkzone reset /dev/nullb0
linux-block (for-next) # dmesg -c
[ 1575.708141] null_blk: null_zone_mgmt 613 ZONE_RESET_ALL
linux-block (for-next) #
----------------------------------------------------------------------
* blktests :-
blktests (master) # ./check block
block/001 (stress device hotplugging) [passed]
runtime 51.784s ... 51.521s
block/002 (remove a device while running blktrace) [passed]
runtime 1.275s ... 0.819s
block/006 (run null-blk in blocking mode) [passed]
read iops 8698266 ... 276934
runtime 9.911s ... 257.702s
block/007 => nullb1 (test classic and hybrid IO polling) [passed]
auto hybrid poll read iops 618264 ... 621194
auto hybrid poll read lat mean 1.369138 ... 1.360774
auto hybrid poll system cpu 72.340426% ... 71.496437%
classic poll read iops 622669 ... 613920
classic poll read lat mean 1.361180 ... 1.383144
classic poll system cpu 77.142857% ... 73.474178%
fixed hybrid poll read iops 615361 ... 612485
fixed hybrid poll read lat mean 1.380764 ... 1.387007
fixed hybrid poll system cpu 74.352941% ... 73.067916%
no poll read iops 615361 ... 615361
no poll read lat mean 1.381291 ... 1.374778
no poll system cpu 72.470588% ... 78.117647%
runtime 2.511s ... 2.522s
block/009 (check page-cache coherency after BLKDISCARD) [passed]
runtime 0.612s ... 0.590s
block/010 (run I/O on null_blk with shared and non-shared tags) [passed]
Individual tags read iops 7742139 ... 7772627
runtime 21.273s ... 21.055s
Shared tags read iops 7666079 ... 7685394
block/014 (run null-blk with blk-mq and timeout injection configured) [passed]
runtime 49.537s ... 60.790s
block/015 (run null-blk on different schedulers with requeue injection configured) [passed]
runtime 45.369s ... 45.660s
block/016 (send a signal to a process waiting on a frozen queue) [passed]
runtime 7.084s ... 7.074s
block/017 (do I/O and check the inflight counter) [passed]
runtime 1.701s ... 1.708s
block/018 (do I/O and check iostats times) [passed]
runtime 5.114s ... 5.111s
block/020 (run null-blk on different schedulers with only one hardware tag) [passed]
runtime 41.520s ... 41.569s
block/021 (read/write nr_requests on null-blk with different schedulers) [passed]
runtime 5.431s ... 5.380s
block/022 (Test hang caused by freeze/unfreeze sequence) [passed]
runtime 30.162s ... 30.142s
block/023 (do I/O on all null_blk queue modes) [passed]
runtime 0.270s ... 0.254s
block/024 (do I/O faster than a jiffy and check iostats times) [passed]
runtime ... 2.653s
block/025 (do a huge discard with 4k sector size) [passed]
runtime ... 3.935s
block/028 (do I/O on scsi_debug with DIF/DIX enabled) [passed]
runtime 13.590s ... 13.213s
block/029 (trigger blk_mq_update_nr_hw_queues()) [passed]
runtime 30.301s ... 30.297s
block/030 (trigger the blk_mq_realloc_hw_ctxs() error path) [passed]
runtime 2.167s ... 2.231s
block/031 (do IO on null-blk with a host tag set) [passed]
runtime 30.346s ... 30.317s
----------------------------------------------------------------------
* REQ_OP_WRITE_ZEROES tests file systems on null_blk :-
blktests (master) # cdblk
linux-block (for-next) # ./zeroout.sh 1
#################### BLKISZ 512 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
├── zone_reset_all
└── zone_size
0 directories, 32 files
ODD:-
19+1 records in
19+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000397535 s, 25.2 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0000512 a a a a a a a a a a a a a a a a
0000528
0001024 a a a a a a a a a a a a a a a a
0001040
0001536 a a a a a a a a a a a a a a a a
0001552
0002048 a a a a a a a a a a a a a a a a
0002064
0002560 a a a a a a a a a a a a a a a a
0002576
0003072 a a a a a a a a a a a a a a a a
0003088
0003584 a a a a a a a a a a a a a a a a
0003600
0004096 a a a a a a a a a a a a a a a a
0004112
0004608 a a a a a a a a a a a a a a a a
0004624
------------------------------------------------------
0000000 a a a a a a a a a a a a a a a a
0000016
0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000528
0001024 a a a a a a a a a a a a a a a a
0001040
0001536 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0001552
0002048 a a a a a a a a a a a a a a a a
0002064
0002560 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0002576
0003072 a a a a a a a a a a a a a a a a
0003088
0003584 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0003600
0004096 a a a a a a a a a a a a a a a a
0004112
0004608 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0004624
------------------------------------------------------
EVEN:-
19+1 records in
19+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000354233 s, 28.2 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0000512 a a a a a a a a a a a a a a a a
0000528
0001024 a a a a a a a a a a a a a a a a
0001040
0001536 a a a a a a a a a a a a a a a a
0001552
0002048 a a a a a a a a a a a a a a a a
0002064
0002560 a a a a a a a a a a a a a a a a
0002576
0003072 a a a a a a a a a a a a a a a a
0003088
0003584 a a a a a a a a a a a a a a a a
0003600
0004096 a a a a a a a a a a a a a a a a
0004112
0004608 a a a a a a a a a a a a a a a a
0004624
------------------------------------------------------
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000016
0000512 a a a a a a a a a a a a a a a a
0000528
0001024 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0001040
0001536 a a a a a a a a a a a a a a a a
0001552
0002048 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0002064
0002560 a a a a a a a a a a a a a a a a
0002576
0003072 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0003088
0003584 a a a a a a a a a a a a a a a a
0003600
0004096 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0004112
0004608 a a a a a a a a a a a a a a a a
0004624
------------------------------------------------------
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 559bb828-96da-493e-9f1a-e95ca2511cc1
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[ 2318.588649] null_blk: disk nullb0 created
[ 2319.711011] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[ 2319.711647] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1068MiB/s][r=273k IOPS][eta 00m:05s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33178: Tue Nov 29 11:41:06 2022
read: IOPS=273k, BW=1068MiB/s (1120MB/s)(2526MiB/2366msec)
slat (nsec): min=2144, max=86435, avg=2581.56, stdev=533.42
clat (nsec): min=1312, max=254073, avg=55146.61, stdev=2945.44
lat (usec): min=4, max=260, avg=57.77, stdev= 3.03
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 54], 20.00th=[ 54],
| 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 55],
| 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 60], 95.00th=[ 62],
| 99.00th=[ 64], 99.50th=[ 67], 99.90th=[ 73], 99.95th=[ 100],
| 99.99th=[ 113]
write: IOPS=77.4k, BW=303MiB/s (317MB/s)(4000MiB/13222msec); 0 zone resets
slat (usec): min=3, max=425, avg=12.17, stdev= 8.79
clat (nsec): min=811, max=912994, avg=194013.00, stdev=43636.73
lat (usec): min=6, max=1091, avg=206.26, stdev=45.56
clat percentiles (usec):
| 1.00th=[ 128], 5.00th=[ 143], 10.00th=[ 153], 20.00th=[ 163],
| 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 196],
| 70.00th=[ 206], 80.00th=[ 217], 90.00th=[ 229], 95.00th=[ 277],
| 99.00th=[ 363], 99.50th=[ 412], 99.90th=[ 529], 99.95th=[ 594],
| 99.99th=[ 725]
bw ( KiB/s): min=158880, max=368952, per=97.94%, avg=303407.41, stdev=46322.27, samples=27
iops : min=39720, max=92238, avg=75851.85, stdev=11580.56, samples=27
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.69%
lat (usec) : 250=57.76%, 500=3.47%, 750=0.07%, 1000=0.01%
cpu : usr=22.96%, sys=75.13%, ctx=3281, majf=0, minf=15174
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=646667,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1068MiB/s (1120MB/s), 1068MiB/s-1068MiB/s (1120MB/s-1120MB/s), io=2526MiB (2649MB), run=2366-2366msec
WRITE: bw=303MiB/s (317MB/s), 303MiB/s-303MiB/s (317MB/s-317MB/s), io=4000MiB (4194MB), run=13222-13222msec
Disk stats (read/write):
nullb0: ios=591402/1024000, merge=0/0, ticks=679/1716, in_queue=2395, util=99.37%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
last mounted on /mnt/nullb0 on Tue Nov 29 11:40:50 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 16c104f4-70b9-4dbc-bd64-3e2a455fc2c5
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[ 2335.638785] EXT4-fs (nullb0): unmounting filesystem.
[ 2335.667259] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][70.6%][r=1063MiB/s][r=272k IOPS][eta 00m:05s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33191: Tue Nov 29 11:41:19 2022
read: IOPS=272k, BW=1062MiB/s (1113MB/s)(2528MiB/2381msec)
slat (nsec): min=2114, max=57911, avg=2609.46, stdev=523.69
clat (nsec): min=1192, max=113706, avg=55470.72, stdev=2535.05
lat (usec): min=3, max=116, avg=58.12, stdev= 2.60
clat percentiles (nsec):
| 1.00th=[52992], 5.00th=[53504], 10.00th=[53504], 20.00th=[54016],
| 30.00th=[54016], 40.00th=[54528], 50.00th=[54528], 60.00th=[55040],
| 70.00th=[55552], 80.00th=[56064], 90.00th=[59136], 95.00th=[61696],
| 99.00th=[64256], 99.50th=[65280], 99.90th=[68096], 99.95th=[70144],
| 99.99th=[82432]
write: IOPS=98.6k, BW=385MiB/s (404MB/s)(4000MiB/10384msec); 0 zone resets
slat (usec): min=3, max=3670, avg= 9.50, stdev= 8.89
clat (nsec): min=901, max=3807.5k, avg=152411.99, stdev=37693.76
lat (usec): min=6, max=3814, avg=161.97, stdev=39.12
clat percentiles (usec):
| 1.00th=[ 116], 5.00th=[ 125], 10.00th=[ 131], 20.00th=[ 137],
| 30.00th=[ 143], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],
| 70.00th=[ 159], 80.00th=[ 165], 90.00th=[ 174], 95.00th=[ 180],
| 99.00th=[ 215], 99.50th=[ 255], 99.90th=[ 326], 99.95th=[ 486],
| 99.99th=[ 1029]
bw ( KiB/s): min=338664, max=438728, per=98.90%, avg=390095.24, stdev=30398.74, samples=21
iops : min=84666, max=109682, avg=97523.81, stdev=7599.67, samples=21
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.73%
lat (usec) : 250=60.93%, 500=0.32%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=27.52%, sys=72.30%, ctx=25, majf=0, minf=15185
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647104,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1062MiB/s (1113MB/s), 1062MiB/s-1062MiB/s (1113MB/s-1113MB/s), io=2528MiB (2651MB), run=2381-2381msec
WRITE: bw=385MiB/s (404MB/s), 385MiB/s-385MiB/s (404MB/s-404MB/s), io=4000MiB (4194MB), run=10384-10384msec
Disk stats (read/write):
nullb0: ios=614200/1024237, merge=0/5794, ticks=691/1819, in_queue=2509, util=99.28%
#################### BLKISZ 1024 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
├── zone_reset_all
└── zone_size
0 directories, 32 files
ODD:-
9+1 records in
9+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000487677 s, 20.5 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0001024 a a a a a a a a a a a a a a a a
0001040
0002048 a a a a a a a a a a a a a a a a
0002064
0003072 a a a a a a a a a a a a a a a a
0003088
0004096 a a a a a a a a a a a a a a a a
0004112
0005120 a a a a a a a a a a a a a a a a
0005136
0006144 a a a a a a a a a a a a a a a a
0006160
0007168 a a a a a a a a a a a a a a a a
0007184
0008192 a a a a a a a a a a a a a a a a
0008208
0009216 a a a a a a a a a a a a a a a a
0009232
------------------------------------------------------
0000000 a a a a a a a a a a a a a a a a
0000016
0001024 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0001040
0002048 a a a a a a a a a a a a a a a a
0002064
0003072 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0003088
0004096 a a a a a a a a a a a a a a a a
0004112
0005120 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0005136
0006144 a a a a a a a a a a a a a a a a
0006160
0007168 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0007184
0008192 a a a a a a a a a a a a a a a a
0008208
0009216 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0009232
------------------------------------------------------
EVEN:-
9+1 records in
9+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000338302 s, 29.6 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0001024 a a a a a a a a a a a a a a a a
0001040
0002048 a a a a a a a a a a a a a a a a
0002064
0003072 a a a a a a a a a a a a a a a a
0003088
0004096 a a a a a a a a a a a a a a a a
0004112
0005120 a a a a a a a a a a a a a a a a
0005136
0006144 a a a a a a a a a a a a a a a a
0006160
0007168 a a a a a a a a a a a a a a a a
0007184
0008192 a a a a a a a a a a a a a a a a
0008208
0009216 a a a a a a a a a a a a a a a a
0009232
------------------------------------------------------
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000016
0001024 a a a a a a a a a a a a a a a a
0001040
0002048 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0002064
0003072 a a a a a a a a a a a a a a a a
0003088
0004096 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0004112
0005120 a a a a a a a a a a a a a a a a
0005136
0006144 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0006160
0007168 a a a a a a a a a a a a a a a a
0007184
0008192 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0008208
0009216 a a a a a a a a a a a a a a a a
0009232
------------------------------------------------------
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 76101011-10f1-4ec6-9d98-2ddd161bd864
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[ 2349.239397] null_blk: disk nullb0 created
[ 2350.358369] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[ 2350.359025] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][78.9%][r=1104MiB/s][r=283k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33272: Tue Nov 29 11:41:36 2022
read: IOPS=282k, BW=1101MiB/s (1154MB/s)(2529MiB/2298msec)
slat (nsec): min=2074, max=80162, avg=2467.59, stdev=537.50
clat (nsec): min=1322, max=131670, avg=53508.87, stdev=2599.84
lat (usec): min=3, max=134, avg=56.02, stdev= 2.66
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52],
| 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 53],
| 70.00th=[ 53], 80.00th=[ 54], 90.00th=[ 58], 95.00th=[ 60],
| 99.00th=[ 63], 99.50th=[ 64], 99.90th=[ 68], 99.95th=[ 72],
| 99.99th=[ 110]
write: IOPS=79.2k, BW=309MiB/s (325MB/s)(4000MiB/12925msec); 0 zone resets
slat (usec): min=3, max=436, avg=11.88, stdev= 9.13
clat (nsec): min=771, max=817313, avg=189666.35, stdev=42086.13
lat (usec): min=6, max=1163, avg=201.62, stdev=43.78
clat percentiles (usec):
| 1.00th=[ 127], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 161],
| 30.00th=[ 172], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194],
| 70.00th=[ 200], 80.00th=[ 208], 90.00th=[ 219], 95.00th=[ 247],
| 99.00th=[ 355], 99.50th=[ 367], 99.90th=[ 523], 99.95th=[ 594],
| 99.99th=[ 676]
bw ( KiB/s): min=263160, max=371280, per=99.42%, avg=315076.62, stdev=30770.94, samples=26
iops : min=65790, max=92820, avg=78769.23, stdev=7692.66, samples=26
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.74%
lat (usec) : 250=58.22%, 500=2.96%, 750=0.07%, 1000=0.01%
cpu : usr=24.01%, sys=73.91%, ctx=3158, majf=0, minf=15193
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647497,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1101MiB/s (1154MB/s), 1101MiB/s-1101MiB/s (1154MB/s-1154MB/s), io=2529MiB (2652MB), run=2298-2298msec
WRITE: bw=309MiB/s (325MB/s), 309MiB/s-309MiB/s (325MB/s-325MB/s), io=4000MiB (4194MB), run=12925-12925msec
Disk stats (read/write):
nullb0: ios=623394/1024000, merge=0/0, ticks=634/1557, in_queue=2192, util=99.35%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
last mounted on /mnt/nullb0 on Tue Nov 29 11:41:21 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 77e31a3f-74b7-4d5a-a6ac-b301b57e5641
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[ 2365.921130] EXT4-fs (nullb0): unmounting filesystem.
[ 2365.946875] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1097MiB/s][r=281k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33285: Tue Nov 29 11:41:49 2022
read: IOPS=281k, BW=1096MiB/s (1149MB/s)(2527MiB/2306msec)
slat (nsec): min=1994, max=109890, avg=2499.66, stdev=567.50
clat (nsec): min=1223, max=234395, avg=53755.24, stdev=3190.19
lat (usec): min=4, max=240, avg=56.30, stdev= 3.29
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52],
| 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 53],
| 70.00th=[ 54], 80.00th=[ 55], 90.00th=[ 58], 95.00th=[ 61],
| 99.00th=[ 64], 99.50th=[ 66], 99.90th=[ 83], 99.95th=[ 99],
| 99.99th=[ 123]
write: IOPS=99.2k, BW=388MiB/s (406MB/s)(4000MiB/10321msec); 0 zone resets
slat (usec): min=3, max=3517, avg= 9.44, stdev= 7.93
clat (nsec): min=942, max=3691.6k, avg=151485.29, stdev=34257.79
lat (usec): min=12, max=3706, avg=160.98, stdev=35.57
clat percentiles (usec):
| 1.00th=[ 115], 5.00th=[ 124], 10.00th=[ 129], 20.00th=[ 137],
| 30.00th=[ 141], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],
| 70.00th=[ 159], 80.00th=[ 165], 90.00th=[ 174], 95.00th=[ 180],
| 99.00th=[ 210], 99.50th=[ 245], 99.90th=[ 326], 99.95th=[ 437],
| 99.99th=[ 586]
bw ( KiB/s): min=285432, max=445696, per=98.30%, avg=390095.24, stdev=38760.46, samples=21
iops : min=71358, max=111424, avg=97523.81, stdev=9690.11, samples=21
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.70%
lat (usec) : 250=61.03%, 500=0.26%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=27.24%, sys=72.61%, ctx=26, majf=0, minf=15179
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=646864,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1096MiB/s (1149MB/s), 1096MiB/s-1096MiB/s (1149MB/s-1149MB/s), io=2527MiB (2650MB), run=2306-2306msec
WRITE: bw=388MiB/s (406MB/s), 388MiB/s-388MiB/s (406MB/s-406MB/s), io=4000MiB (4194MB), run=10321-10321msec
Disk stats (read/write):
nullb0: ios=582149/1024235, merge=0/5742, ticks=586/1682, in_queue=2269, util=99.25%
#################### BLKISZ 2048 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
├── zone_reset_all
└── zone_size
0 directories, 32 files
ODD:-
4+1 records in
4+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.00036307 s, 27.5 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0002048 a a a a a a a a a a a a a a a a
0002064
0004096 a a a a a a a a a a a a a a a a
0004112
0006144 a a a a a a a a a a a a a a a a
0006160
0008192 a a a a a a a a a a a a a a a a
0008208
0010240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0010256
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0014336 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0014352
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0018432 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0018448
------------------------------------------------------
0000000 a a a a a a a a a a a a a a a a
0000016
0002048 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0002064
0004096 a a a a a a a a a a a a a a a a
0004112
0006144 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0006160
0008192 a a a a a a a a a a a a a a a a
0008208
0010240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0010256
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0014336 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0014352
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0018432 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0018448
------------------------------------------------------
EVEN:-
4+1 records in
4+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000343111 s, 29.2 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0002048 a a a a a a a a a a a a a a a a
0002064
0004096 a a a a a a a a a a a a a a a a
0004112
0006144 a a a a a a a a a a a a a a a a
0006160
0008192 a a a a a a a a a a a a a a a a
0008208
0010240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0010256
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0014336 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0014352
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0018432 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0018448
------------------------------------------------------
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000016
0002048 a a a a a a a a a a a a a a a a
0002064
0004096 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0004112
0006144 a a a a a a a a a a a a a a a a
0006160
0008192 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0008208
0010240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0010256
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0014336 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0014352
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0018432 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0018448
------------------------------------------------------
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 1b57f322-be38-4950-a963-94dd4276b5bb
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[ 2379.405522] null_blk: disk nullb0 created
[ 2380.523797] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[ 2380.524358] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][80.0%][r=1061MiB/s][r=272k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33365: Tue Nov 29 11:42:07 2022
read: IOPS=269k, BW=1050MiB/s (1101MB/s)(2530MiB/2409msec)
slat (nsec): min=1913, max=60406, avg=2571.22, stdev=660.83
clat (nsec): min=1202, max=125759, avg=56086.29, stdev=3692.00
lat (usec): min=4, max=128, avg=58.72, stdev= 3.80
clat percentiles (nsec):
| 1.00th=[49920], 5.00th=[51456], 10.00th=[51968], 20.00th=[53504],
| 30.00th=[54528], 40.00th=[55040], 50.00th=[55552], 60.00th=[56064],
| 70.00th=[56576], 80.00th=[57600], 90.00th=[61184], 95.00th=[64256],
| 99.00th=[67072], 99.50th=[69120], 99.90th=[74240], 99.95th=[78336],
| 99.99th=[89600]
write: IOPS=75.1k, BW=293MiB/s (308MB/s)(4000MiB/13631msec); 0 zone resets
slat (usec): min=3, max=402, avg=12.46, stdev= 9.23
clat (nsec): min=882, max=1006.9k, avg=200071.94, stdev=43702.18
lat (usec): min=6, max=1156, avg=212.62, stdev=45.49
clat percentiles (usec):
| 1.00th=[ 135], 5.00th=[ 149], 10.00th=[ 159], 20.00th=[ 172],
| 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 196], 60.00th=[ 204],
| 70.00th=[ 210], 80.00th=[ 219], 90.00th=[ 231], 95.00th=[ 285],
| 99.00th=[ 363], 99.50th=[ 379], 99.90th=[ 537], 99.95th=[ 611],
| 99.99th=[ 725]
bw ( KiB/s): min=85440, max=354232, per=97.36%, avg=292570.86, stdev=50623.71, samples=28
iops : min=21360, max=88558, avg=73142.86, stdev=12656.04, samples=28
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.47%, 100=38.28%
lat (usec) : 250=57.61%, 500=3.56%, 750=0.08%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=23.26%, sys=74.74%, ctx=3384, majf=0, minf=15199
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647741,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1050MiB/s (1101MB/s), 1050MiB/s-1050MiB/s (1101MB/s-1101MB/s), io=2530MiB (2653MB), run=2409-2409msec
WRITE: bw=293MiB/s (308MB/s), 293MiB/s-293MiB/s (308MB/s-308MB/s), io=4000MiB (4194MB), run=13631-13631msec
Disk stats (read/write):
nullb0: ios=606253/1024000, merge=0/0, ticks=609/1680, in_queue=2290, util=99.40%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
last mounted on /mnt/nullb0 on Tue Nov 29 11:41:51 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 6fe66e2f-3816-42b8-a491-5de76a872b6f
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[ 2396.902577] EXT4-fs (nullb0): unmounting filesystem.
[ 2396.925680] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1125MiB/s][r=288k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33379: Tue Nov 29 11:42:20 2022
read: IOPS=288k, BW=1124MiB/s (1179MB/s)(2529MiB/2250msec)
slat (nsec): min=1973, max=104008, avg=2402.39, stdev=562.74
clat (nsec): min=1183, max=344946, avg=52394.60, stdev=3095.91
lat (usec): min=3, max=347, avg=54.84, stdev= 3.16
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 51],
| 30.00th=[ 51], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 52],
| 70.00th=[ 53], 80.00th=[ 53], 90.00th=[ 57], 95.00th=[ 59],
| 99.00th=[ 62], 99.50th=[ 63], 99.90th=[ 68], 99.95th=[ 72],
| 99.99th=[ 119]
write: IOPS=99.1k, BW=387MiB/s (406MB/s)(4000MiB/10332msec); 0 zone resets
slat (usec): min=3, max=3493, avg= 9.46, stdev= 8.55
clat (nsec): min=922, max=3689.5k, avg=151643.40, stdev=35090.99
lat (usec): min=7, max=3702, avg=161.16, stdev=36.36
clat percentiles (usec):
| 1.00th=[ 119], 5.00th=[ 128], 10.00th=[ 133], 20.00th=[ 139],
| 30.00th=[ 143], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],
| 70.00th=[ 159], 80.00th=[ 163], 90.00th=[ 169], 95.00th=[ 178],
| 99.00th=[ 204], 99.50th=[ 229], 99.90th=[ 293], 99.95th=[ 363],
| 99.99th=[ 494]
bw ( KiB/s): min=283352, max=422080, per=98.40%, avg=390095.24, stdev=30150.48, samples=21
iops : min=70838, max=105520, avg=97523.81, stdev=7537.59, samples=21
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=1.71%, 100=37.02%
lat (usec) : 250=61.07%, 500=0.19%, 750=0.01%
lat (msec) : 4=0.01%
cpu : usr=27.72%, sys=72.09%, ctx=34, majf=0, minf=15194
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647483,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1124MiB/s (1179MB/s), 1124MiB/s-1124MiB/s (1179MB/s-1179MB/s), io=2529MiB (2652MB), run=2250-2250msec
WRITE: bw=387MiB/s (406MB/s), 387MiB/s-387MiB/s (406MB/s-406MB/s), io=4000MiB (4194MB), run=10332-10332msec
Disk stats (read/write):
nullb0: ios=593747/1024233, merge=0/5673, ticks=538/1638, in_queue=2176, util=99.26%
#################### BLKISZ 4096 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
├── zone_reset_all
└── zone_size
0 directories, 32 files
ODD:-
2+1 records in
2+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000348442 s, 28.7 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0004096 a a a a a a a a a a a a a a a a
0004112
0008192 a a a a a a a a a a a a a a a a
0008208
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0020480 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0020496
0024576 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0024592
0028672 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0028688
0032768 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0032784
0036864 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0036880
------------------------------------------------------
0000000 a a a a a a a a a a a a a a a a
0000016
0004096 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0004112
0008192 a a a a a a a a a a a a a a a a
0008208
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0020480 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0020496
0024576 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0024592
0028672 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0028688
0032768 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0032784
0036864 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0036880
------------------------------------------------------
EVEN:-
2+1 records in
2+1 records out
10002 bytes (10 kB, 9.8 KiB) copied, 0.000225138 s, 44.4 MB/s
0000000 a a a a a a a a a a a a a a a a
0000016
0004096 a a a a a a a a a a a a a a a a
0004112
0008192 a a a a a a a a a a a a a a a a
0008208
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0020480 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0020496
0024576 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0024592
0028672 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0028688
0032768 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0032784
0036864 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0036880
------------------------------------------------------
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000016
0004096 a a a a a a a a a a a a a a a a
0004112
0008192 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0008208
0012288 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0012304
0016384 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0016400
0020480 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0020496
0024576 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0024592
0028672 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0028688
0032768 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0032784
0036864 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0036880
------------------------------------------------------
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 19fd9837-934d-47fc-8f7f-bab84f720e6e
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[ 2410.378712] null_blk: disk nullb0 created
[ 2411.491527] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[ 2411.492107] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][78.9%][r=1142MiB/s][r=292k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33461: Tue Nov 29 11:42:37 2022
read: IOPS=291k, BW=1138MiB/s (1193MB/s)(2529MiB/2222msec)
slat (nsec): min=1873, max=133413, avg=2348.75, stdev=527.11
clat (nsec): min=1302, max=268730, avg=51768.86, stdev=3074.23
lat (usec): min=3, max=274, avg=54.16, stdev= 3.17
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 51],
| 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 51],
| 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 56], 95.00th=[ 58],
| 99.00th=[ 62], 99.50th=[ 65], 99.90th=[ 81], 99.95th=[ 84],
| 99.99th=[ 110]
write: IOPS=78.8k, BW=308MiB/s (323MB/s)(4000MiB/12997msec); 0 zone resets
slat (usec): min=3, max=439, avg=11.93, stdev= 9.13
clat (nsec): min=691, max=1021.1k, avg=190729.24, stdev=42245.49
lat (usec): min=7, max=1196, avg=202.74, stdev=43.94
clat percentiles (usec):
| 1.00th=[ 126], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 163],
| 30.00th=[ 172], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 194],
| 70.00th=[ 202], 80.00th=[ 208], 90.00th=[ 219], 95.00th=[ 255],
| 99.00th=[ 355], 99.50th=[ 367], 99.90th=[ 519], 99.95th=[ 586],
| 99.99th=[ 693]
bw ( KiB/s): min=261816, max=365464, per=99.98%, avg=315076.92, stdev=31029.00, samples=26
iops : min=65454, max=91366, avg=78769.23, stdev=7757.25, samples=26
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=4.08%, 100=34.65%
lat (usec) : 250=58.18%, 500=3.02%, 750=0.07%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=23.78%, sys=74.14%, ctx=3225, majf=0, minf=15191
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647342,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1138MiB/s (1193MB/s), 1138MiB/s-1138MiB/s (1193MB/s-1193MB/s), io=2529MiB (2652MB), run=2222-2222msec
WRITE: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=4000MiB (4194MB), run=12997-12997msec
Disk stats (read/write):
nullb0: ios=623488/1024000, merge=0/0, ticks=565/1518, in_queue=2082, util=99.36%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
last mounted on /mnt/nullb0 on Tue Nov 29 11:42:22 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 4b42ba06-00a0-4c88-9a51-1626adbce878
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[ 2427.057182] EXT4-fs (nullb0): unmounting filesystem.
[ 2427.079726] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1133MiB/s][r=290k IOPS][eta 00m:04s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=33475: Tue Nov 29 11:42:50 2022
read: IOPS=289k, BW=1129MiB/s (1184MB/s)(2528MiB/2240msec)
slat (nsec): min=1914, max=93848, avg=2394.61, stdev=533.46
clat (nsec): min=1203, max=144454, avg=52202.30, stdev=2627.71
lat (usec): min=3, max=146, avg=54.64, stdev= 2.69
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 51],
| 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 52],
| 70.00th=[ 52], 80.00th=[ 53], 90.00th=[ 57], 95.00th=[ 59],
| 99.00th=[ 61], 99.50th=[ 63], 99.90th=[ 67], 99.95th=[ 70],
| 99.99th=[ 105]
write: IOPS=100k, BW=391MiB/s (410MB/s)(4000MiB/10229msec); 0 zone resets
slat (usec): min=3, max=3675, avg= 9.36, stdev= 8.20
clat (nsec): min=922, max=7011.3k, avg=150140.47, stdev=36842.61
lat (usec): min=11, max=7022, avg=159.55, stdev=38.40
clat percentiles (usec):
| 1.00th=[ 114], 5.00th=[ 123], 10.00th=[ 128], 20.00th=[ 135],
| 30.00th=[ 141], 40.00th=[ 145], 50.00th=[ 149], 60.00th=[ 153],
| 70.00th=[ 157], 80.00th=[ 163], 90.00th=[ 172], 95.00th=[ 180],
| 99.00th=[ 210], 99.50th=[ 235], 99.90th=[ 293], 99.95th=[ 351],
| 99.99th=[ 498]
bw ( KiB/s): min=200472, max=448840, per=97.42%, avg=390094.86, stdev=52556.93, samples=21
iops : min=50118, max=112210, avg=97523.81, stdev=13139.30, samples=21
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=1.31%, 100=37.43%
lat (usec) : 250=61.08%, 500=0.18%, 750=0.01%
lat (msec) : 4=0.01%, 10=0.01%
cpu : usr=27.58%, sys=72.25%, ctx=28, majf=0, minf=15187
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=647247,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1129MiB/s (1184MB/s), 1129MiB/s-1129MiB/s (1184MB/s-1184MB/s), io=2528MiB (2651MB), run=2240-2240msec
WRITE: bw=391MiB/s (410MB/s), 391MiB/s-391MiB/s (410MB/s-410MB/s), io=4000MiB (4194MB), run=10229-10229msec
Disk stats (read/write):
nullb0: ios=625624/1024236, merge=0/5772, ticks=563/1615, in_queue=2178, util=99.26%
--
2.29.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-30 7:59 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 2/8] null_blk: code cleaup Chaitanya Kulkarni
` (6 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
Add a helper function to enable the REQ_OP_WRITE_ZEROES operations
when null_blk.
Since write-zeroes is a non-trivial I/O operation we need this to
add a blktest so we can test the non-trivial I/O path from the
application to the block layer.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/block/null_blk/main.c | 59 ++++++++++++++++++++++++++++++-
drivers/block/null_blk/null_blk.h | 1 +
2 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 1f154f92f4c2..2d592b4eb815 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -209,6 +209,10 @@ static bool g_discard;
module_param_named(discard, g_discard, bool, 0444);
MODULE_PARM_DESC(discard, "Support discard operations (requires memory-backed null_blk device). Default: false");
+static bool g_write_zeroes;
+module_param_named(write_zeroes, g_write_zeroes, bool, 0444);
+MODULE_PARM_DESC(write_zeroes, "Support write-zeores operations. Default: false");
+
static unsigned long g_cache_size;
module_param_named(cache_size, g_cache_size, ulong, 0444);
MODULE_PARM_DESC(mbps, "Cache size in MiB for memory-backed device. Default: 0 (none)");
@@ -416,6 +420,7 @@ NULLB_DEVICE_ATTR(blocking, bool, NULL);
NULLB_DEVICE_ATTR(use_per_node_hctx, bool, NULL);
NULLB_DEVICE_ATTR(memory_backed, bool, NULL);
NULLB_DEVICE_ATTR(discard, bool, NULL);
+NULLB_DEVICE_ATTR(write_zeroes, bool, NULL);
NULLB_DEVICE_ATTR(mbps, uint, NULL);
NULLB_DEVICE_ATTR(cache_size, ulong, NULL);
NULLB_DEVICE_ATTR(zoned, bool, NULL);
@@ -540,6 +545,7 @@ static struct configfs_attribute *nullb_device_attrs[] = {
&nullb_device_attr_power,
&nullb_device_attr_memory_backed,
&nullb_device_attr_discard,
+ &nullb_device_attr_write_zeroes,
&nullb_device_attr_mbps,
&nullb_device_attr_cache_size,
&nullb_device_attr_badblocks,
@@ -614,7 +620,7 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page)
"poll_queues,power,queue_mode,shared_tag_bitmap,size,"
"submit_queues,use_per_node_hctx,virt_boundary,zoned,"
"zone_capacity,zone_max_active,zone_max_open,"
- "zone_nr_conv,zone_size\n");
+ "zone_nr_conv,zone_size,write_zeroes\n");
}
CONFIGFS_ATTR_RO(memb_group_, features);
@@ -678,6 +684,7 @@ static struct nullb_device *null_alloc_dev(void)
dev->blocking = g_blocking;
dev->memory_backed = g_memory_backed;
dev->discard = g_discard;
+ dev->write_zeroes = g_write_zeroes;
dev->cache_size = g_cache_size;
dev->mbps = g_mbps;
dev->use_per_node_hctx = g_use_per_node_hctx;
@@ -870,6 +877,24 @@ static void null_free_sector(struct nullb *nullb, sector_t sector,
}
}
+static void null_zero_sector(struct nullb_device *d, sector_t sect,
+ sector_t nr_sects, bool cache)
+{
+ struct radix_tree_root *root = cache ? &d->cache : &d->data;
+ struct nullb_page *t_page;
+ unsigned int offset;
+ void *dest;
+
+ t_page = radix_tree_lookup(root, sect >> PAGE_SECTORS_SHIFT);
+ if (!t_page)
+ return;
+
+ offset = (sect & SECTOR_MASK) << SECTOR_SHIFT;
+ dest = kmap_atomic(t_page->page);
+ memset(dest + offset, 0, SECTOR_SIZE * nr_sects);
+ kunmap_atomic(dest);
+}
+
static struct nullb_page *null_radix_tree_insert(struct nullb *nullb, u64 idx,
struct nullb_page *t_page, bool is_cache)
{
@@ -1186,6 +1211,27 @@ blk_status_t null_handle_discard(struct nullb_device *dev,
return BLK_STS_OK;
}
+static blk_status_t null_handle_write_zeroes(struct nullb_device *dev,
+ sector_t sector, sector_t nr_sectors)
+{
+ unsigned int bytes_left = nr_sectors << 9;
+ struct nullb *nullb = dev->nullb;
+ size_t curr_bytes;
+
+ spin_lock_irq(&nullb->lock);
+ while (bytes_left > 0) {
+ curr_bytes = min_t(size_t, bytes_left, nullb->dev->blocksize);
+ nr_sectors = curr_bytes >> SECTOR_SHIFT;
+ null_zero_sector(nullb->dev, sector, nr_sectors, false);
+ if (null_cache_active(nullb))
+ null_zero_sector(nullb->dev, sector, nr_sectors, true);
+ sector += nr_sectors;
+ bytes_left -= curr_bytes;
+ }
+ spin_unlock_irq(&nullb->lock);
+ return BLK_STS_OK;
+}
+
static int null_handle_flush(struct nullb *nullb)
{
int err;
@@ -1352,6 +1398,9 @@ static inline blk_status_t null_handle_memory_backed(struct nullb_cmd *cmd,
if (op == REQ_OP_DISCARD)
return null_handle_discard(dev, sector, nr_sectors);
+ if (op == REQ_OP_WRITE_ZEROES)
+ return null_handle_write_zeroes(dev, sector, nr_sectors);
+
if (dev->queue_mode == NULL_Q_BIO)
err = null_handle_bio(cmd);
else
@@ -1800,6 +1849,13 @@ static void null_config_discard(struct nullb *nullb)
blk_queue_max_discard_sectors(nullb->q, UINT_MAX >> 9);
}
+static void null_config_write_zeroes(struct nullb *nullb)
+{
+ if (!nullb->dev->write_zeroes)
+ return;
+ blk_queue_max_write_zeroes_sectors(nullb->q, UINT_MAX >> 9);
+}
+
static const struct block_device_operations null_bio_ops = {
.owner = THIS_MODULE,
.submit_bio = null_submit_bio,
@@ -2111,6 +2167,7 @@ static int null_add_dev(struct nullb_device *dev)
blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1);
null_config_discard(nullb);
+ null_config_write_zeroes(nullb);
if (config_item_name(&dev->item)) {
/* Use configfs dir name as the device name */
diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h
index 94ff68052b1e..2c0c9c29158f 100644
--- a/drivers/block/null_blk/null_blk.h
+++ b/drivers/block/null_blk/null_blk.h
@@ -111,6 +111,7 @@ struct nullb_device {
bool power; /* power on/off the device */
bool memory_backed; /* if data is stored in memory */
bool discard; /* if support discard */
+ bool write_zeroes; /* if support write_zeroes */
bool zoned; /* if device is zoned */
bool virt_boundary; /* virtual boundary on/off for the device */
bool no_sched; /* no IO scheduler for the device */
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES
2022-11-29 23:28 ` [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES Chaitanya Kulkarni
@ 2022-11-30 7:59 ` Shinichiro Kawasaki
2022-11-30 23:29 ` Chaitanya Kulkarni
0 siblings, 1 reply; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-11-30 7:59 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 29, 2022 / 15:28, Chaitanya Kulkarni wrote:
> Add a helper function to enable the REQ_OP_WRITE_ZEROES operations
> when null_blk.
>
> Since write-zeroes is a non-trivial I/O operation we need this to
> add a blktest so we can test the non-trivial I/O path from the
> application to the block layer.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
This motivation sounds good. I tried this patch. With a quick test it looks
working good for me. Please find minor comments in line.
[...]
> +static void null_zero_sector(struct nullb_device *d, sector_t sect,
> + sector_t nr_sects, bool cache)
> +{
> + struct radix_tree_root *root = cache ? &d->cache : &d->data;
> + struct nullb_page *t_page;
> + unsigned int offset;
> + void *dest;
> +
> + t_page = radix_tree_lookup(root, sect >> PAGE_SECTORS_SHIFT);
> + if (!t_page)
> + return;
> +
> + offset = (sect & SECTOR_MASK) << SECTOR_SHIFT;
> + dest = kmap_atomic(t_page->page);
> + memset(dest + offset, 0, SECTOR_SIZE * nr_sects);
> + kunmap_atomic(dest);
> +}
Did you consider to call null_lookup_page() for __null_lookup_page() from
null_zero_sector()? It may simplify this function a bit.
> +
> static struct nullb_page *null_radix_tree_insert(struct nullb *nullb, u64 idx,
> struct nullb_page *t_page, bool is_cache)
> {
> @@ -1186,6 +1211,27 @@ blk_status_t null_handle_discard(struct nullb_device *dev,
> return BLK_STS_OK;
> }
>
> +static blk_status_t null_handle_write_zeroes(struct nullb_device *dev,
> + sector_t sector, sector_t nr_sectors)
> +{
> + unsigned int bytes_left = nr_sectors << 9;
> + struct nullb *nullb = dev->nullb;
> + size_t curr_bytes;
> +
> + spin_lock_irq(&nullb->lock);
> + while (bytes_left > 0) {
> + curr_bytes = min_t(size_t, bytes_left, nullb->dev->blocksize);
> + nr_sectors = curr_bytes >> SECTOR_SHIFT;
> + null_zero_sector(nullb->dev, sector, nr_sectors, false);
> + if (null_cache_active(nullb))
> + null_zero_sector(nullb->dev, sector, nr_sectors, true);
> + sector += nr_sectors;
> + bytes_left -= curr_bytes;
> + }
> + spin_unlock_irq(&nullb->lock);
> + return BLK_STS_OK;
> +}
> +
> static int null_handle_flush(struct nullb *nullb)
> {
> int err;
> @@ -1352,6 +1398,9 @@ static inline blk_status_t null_handle_memory_backed(struct nullb_cmd *cmd,
> if (op == REQ_OP_DISCARD)
> return null_handle_discard(dev, sector, nr_sectors);
>
> + if (op == REQ_OP_WRITE_ZEROES)
> + return null_handle_write_zeroes(dev, sector, nr_sectors);
> +
> if (dev->queue_mode == NULL_Q_BIO)
> err = null_handle_bio(cmd);
> else
> @@ -1800,6 +1849,13 @@ static void null_config_discard(struct nullb *nullb)
> blk_queue_max_discard_sectors(nullb->q, UINT_MAX >> 9);
> }
>
> +static void null_config_write_zeroes(struct nullb *nullb)
> +{
> + if (!nullb->dev->write_zeroes)
> + return;
> + blk_queue_max_write_zeroes_sectors(nullb->q, UINT_MAX >> 9);
Just comment: this value UINT_MAX >> 9 sounds a bit weird, but probably ok. This
value was introduced by commit 306eb6b4ad4f ("nullb: support discard") to call
blk_queue_max_discard_sectors(). I guess you chose the same value for write
zeroes.
> +}
> +
> static const struct block_device_operations null_bio_ops = {
> .owner = THIS_MODULE,
> .submit_bio = null_submit_bio,
> @@ -2111,6 +2167,7 @@ static int null_add_dev(struct nullb_device *dev)
> blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1);
>
> null_config_discard(nullb);
> + null_config_write_zeroes(nullb);
>
> if (config_item_name(&dev->item)) {
> /* Use configfs dir name as the device name */
> diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h
> index 94ff68052b1e..2c0c9c29158f 100644
> --- a/drivers/block/null_blk/null_blk.h
> +++ b/drivers/block/null_blk/null_blk.h
> @@ -111,6 +111,7 @@ struct nullb_device {
> bool power; /* power on/off the device */
> bool memory_backed; /* if data is stored in memory */
> bool discard; /* if support discard */
> + bool write_zeroes; /* if support write_zeroes */
> bool zoned; /* if device is zoned */
> bool virt_boundary; /* virtual boundary on/off for the device */
> bool no_sched; /* no IO scheduler for the device */
> --
> 2.29.0
>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES
2022-11-30 7:59 ` Shinichiro Kawasaki
@ 2022-11-30 23:29 ` Chaitanya Kulkarni
2022-12-01 1:21 ` Shinichiro Kawasaki
0 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-30 23:29 UTC (permalink / raw)
To: Shinichiro Kawasaki, Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
>> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
>
> This motivation sounds good. I tried this patch. With a quick test it looks
> working good for me. Please find minor comments in line.
>
> [...]
>
>> +static void null_zero_sector(struct nullb_device *d, sector_t sect,
>> + sector_t nr_sects, bool cache)
>> +{
>> + struct radix_tree_root *root = cache ? &d->cache : &d->data;
>> + struct nullb_page *t_page;
>> + unsigned int offset;
>> + void *dest;
>> +
>> + t_page = radix_tree_lookup(root, sect >> PAGE_SECTORS_SHIFT);
>> + if (!t_page)
>> + return;
>> +
>> + offset = (sect & SECTOR_MASK) << SECTOR_SHIFT;
>> + dest = kmap_atomic(t_page->page);
>> + memset(dest + offset, 0, SECTOR_SIZE * nr_sects);
>> + kunmap_atomic(dest);
>> +}
>
> Did you consider to call null_lookup_page() for __null_lookup_page() from
> null_zero_sector()? It may simplify this function a bit.
>
I found this clear and easy than going over call chain when
debugging.
>> +
>> static struct nullb_page *null_radix_tree_insert(struct nullb *nullb, u64 idx,
[...]
>>
>> +static void null_config_write_zeroes(struct nullb *nullb)
>> +{
>> + if (!nullb->dev->write_zeroes)
>> + return;
>> + blk_queue_max_write_zeroes_sectors(nullb->q, UINT_MAX >> 9);
>
> Just comment: this value UINT_MAX >> 9 sounds a bit weird, but probably ok. This
> value was introduced by commit 306eb6b4ad4f ("nullb: support discard") to call
> blk_queue_max_discard_sectors(). I guess you chose the same value for write
> zeroes.
>
Yes indeed, plz have a look end patches to allow user to set this value.
-ck
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES
2022-11-30 23:29 ` Chaitanya Kulkarni
@ 2022-12-01 1:21 ` Shinichiro Kawasaki
0 siblings, 0 replies; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-12-01 1:21 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 30, 2022 / 23:29, Chaitanya Kulkarni wrote:
> >> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> >
> > This motivation sounds good. I tried this patch. With a quick test it looks
> > working good for me. Please find minor comments in line.
> >
> > [...]
> >
> >> +static void null_zero_sector(struct nullb_device *d, sector_t sect,
> >> + sector_t nr_sects, bool cache)
> >> +{
> >> + struct radix_tree_root *root = cache ? &d->cache : &d->data;
> >> + struct nullb_page *t_page;
> >> + unsigned int offset;
> >> + void *dest;
> >> +
> >> + t_page = radix_tree_lookup(root, sect >> PAGE_SECTORS_SHIFT);
> >> + if (!t_page)
> >> + return;
> >> +
> >> + offset = (sect & SECTOR_MASK) << SECTOR_SHIFT;
> >> + dest = kmap_atomic(t_page->page);
> >> + memset(dest + offset, 0, SECTOR_SIZE * nr_sects);
> >> + kunmap_atomic(dest);
> >> +}
> >
> > Did you consider to call null_lookup_page() for __null_lookup_page() from
> > null_zero_sector()? It may simplify this function a bit.
> >
>
> I found this clear and easy than going over call chain when
> debugging.
Okay, then the patch looks good to me :)
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH V4 2/8] null_blk: code cleaup
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-30 8:00 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd() Chaitanya Kulkarni
` (5 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
Introduce and use two new macros for calculating the page index from
given sector and index (offset) of the sector in the page.
The newly added macros makes code easy to read with meaningful name and
explanation comments attached to it.
While at it adjust the code in the null_free_sector() to return early
to get rid of the extra identation.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/block/null_blk/main.c | 37 ++++++++++++++++++++---------------
1 file changed, 21 insertions(+), 16 deletions(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 2d592b4eb815..fa47fab279c3 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -14,6 +14,11 @@
#undef pr_fmt
#define pr_fmt(fmt) "null_blk: " fmt
+/* Gives page index for which this sector belongs to. */
+#define PAGE_IDX_FROM_SECT(sect) (sect >> PAGE_SECTORS_SHIFT)
+/* Gives index (offset) of the sector within page. */
+#define SECT_OFFSET_IN_PAGE(sect) ((sect & SECTOR_MASK) << SECTOR_SHIFT)
+
#define FREE_BATCH 16
#define TICKS_PER_SEC 50ULL
@@ -860,20 +865,20 @@ static void null_free_sector(struct nullb *nullb, sector_t sector,
struct radix_tree_root *root;
root = is_cache ? &nullb->dev->cache : &nullb->dev->data;
- idx = sector >> PAGE_SECTORS_SHIFT;
+ idx = PAGE_IDX_FROM_SECT(sector);
sector_bit = (sector & SECTOR_MASK);
t_page = radix_tree_lookup(root, idx);
- if (t_page) {
- __clear_bit(sector_bit, t_page->bitmap);
-
- if (null_page_empty(t_page)) {
- ret = radix_tree_delete_item(root, idx, t_page);
- WARN_ON(ret != t_page);
- null_free_page(ret);
- if (is_cache)
- nullb->dev->curr_cache -= PAGE_SIZE;
- }
+ if (!t_page)
+ return;
+ __clear_bit(sector_bit, t_page->bitmap);
+
+ if (null_page_empty(t_page)) {
+ ret = radix_tree_delete_item(root, idx, t_page);
+ WARN_ON(ret != t_page);
+ null_free_page(ret);
+ if (is_cache)
+ nullb->dev->curr_cache -= PAGE_SIZE;
}
}
@@ -885,11 +890,11 @@ static void null_zero_sector(struct nullb_device *d, sector_t sect,
unsigned int offset;
void *dest;
- t_page = radix_tree_lookup(root, sect >> PAGE_SECTORS_SHIFT);
+ t_page = radix_tree_lookup(root, PAGE_IDX_FROM_SECT(sect));
if (!t_page)
return;
- offset = (sect & SECTOR_MASK) << SECTOR_SHIFT;
+ offset = SECT_OFFSET_IN_PAGE(sect);
dest = kmap_atomic(t_page->page);
memset(dest + offset, 0, SECTOR_SIZE * nr_sects);
kunmap_atomic(dest);
@@ -949,7 +954,7 @@ static struct nullb_page *__null_lookup_page(struct nullb *nullb,
struct nullb_page *t_page;
struct radix_tree_root *root;
- idx = sector >> PAGE_SECTORS_SHIFT;
+ idx = PAGE_IDX_FROM_SECT(sector);
sector_bit = (sector & SECTOR_MASK);
root = is_cache ? &nullb->dev->cache : &nullb->dev->data;
@@ -1125,7 +1130,7 @@ static int copy_to_nullb(struct nullb *nullb, struct page *source,
if (null_cache_active(nullb) && !is_fua)
null_make_cache_space(nullb, PAGE_SIZE);
- offset = (sector & SECTOR_MASK) << SECTOR_SHIFT;
+ offset = SECT_OFFSET_IN_PAGE(sector);
t_page = null_insert_page(nullb, sector,
!null_cache_active(nullb) || is_fua);
if (!t_page)
@@ -1159,7 +1164,7 @@ static int copy_from_nullb(struct nullb *nullb, struct page *dest,
while (count < n) {
temp = min_t(size_t, nullb->dev->blocksize, n - count);
- offset = (sector & SECTOR_MASK) << SECTOR_SHIFT;
+ offset = SECT_OFFSET_IN_PAGE(sector);
t_page = null_lookup_page(nullb, sector, false,
!null_cache_active(nullb));
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH V4 2/8] null_blk: code cleaup
2022-11-29 23:28 ` [PATCH V4 2/8] null_blk: code cleaup Chaitanya Kulkarni
@ 2022-11-30 8:00 ` Shinichiro Kawasaki
0 siblings, 0 replies; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-11-30 8:00 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 29, 2022 / 15:28, Chaitanya Kulkarni wrote:
> Introduce and use two new macros for calculating the page index from
> given sector and index (offset) of the sector in the page.
> The newly added macros makes code easy to read with meaningful name and
> explanation comments attached to it.
>
> While at it adjust the code in the null_free_sector() to return early
> to get rid of the extra identation.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Looks good to me.
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd()
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 2/8] null_blk: code cleaup Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-30 8:01 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 4/8] null_blk: don't use magic numbers in the code Chaitanya Kulkarni
` (4 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
The function __alloc_cmd() is responsible to allocate tag and
initialize the different members of the null_cmd structure e.g.
cmd->tag, cmd->error, and cmd->nq, Move only member bio that is initialized
from alloc_cmd() into __alloc_cmd().
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
---
drivers/block/null_blk/main.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index fa47fab279c3..84f5ca0cc79c 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -743,7 +743,7 @@ static void free_cmd(struct nullb_cmd *cmd)
static enum hrtimer_restart null_cmd_timer_expired(struct hrtimer *timer);
-static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq)
+static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq, struct bio *bio)
{
struct nullb_cmd *cmd;
unsigned int tag;
@@ -754,6 +754,7 @@ static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq)
cmd->tag = tag;
cmd->error = BLK_STS_OK;
cmd->nq = nq;
+ cmd->bio = bio;
if (nq->dev->irqmode == NULL_IRQ_TIMER) {
hrtimer_init(&cmd->timer, CLOCK_MONOTONIC,
HRTIMER_MODE_REL);
@@ -775,11 +776,9 @@ static struct nullb_cmd *alloc_cmd(struct nullb_queue *nq, struct bio *bio)
* This avoids multiple return statements, multiple calls to
* __alloc_cmd() and a fast path call to prepare_to_wait().
*/
- cmd = __alloc_cmd(nq);
- if (cmd) {
- cmd->bio = bio;
+ cmd = __alloc_cmd(nq, bio);
+ if (cmd)
return cmd;
- }
prepare_to_wait(&nq->wait, &wait, TASK_UNINTERRUPTIBLE);
io_schedule();
finish_wait(&nq->wait, &wait);
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd()
2022-11-29 23:28 ` [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd() Chaitanya Kulkarni
@ 2022-11-30 8:01 ` Shinichiro Kawasaki
0 siblings, 0 replies; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-11-30 8:01 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 29, 2022 / 15:28, Chaitanya Kulkarni wrote:
> The function __alloc_cmd() is responsible to allocate tag and
> initialize the different members of the null_cmd structure e.g.
> cmd->tag, cmd->error, and cmd->nq, Move only member bio that is initialized
> from alloc_cmd() into __alloc_cmd().
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> Reviewed-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
Looks good to me.
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH V4 4/8] null_blk: don't use magic numbers in the code
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
` (2 preceding siblings ...)
2022-11-29 23:28 ` [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd() Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 5/8] null_blk: remove extra space in switch condition Chaitanya Kulkarni
` (3 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
Instead of using the hardcoded value use meaningful macro for tag
available value of -1U in get_tag() and __alloc_cmd().
While at it return early on error to get rid of the extra indentation
in __alloc_cmd().
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
---
drivers/block/null_blk/main.c | 30 +++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 84f5ca0cc79c..db849e6bc475 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -24,6 +24,8 @@
#define TICKS_PER_SEC 50ULL
#define TIMER_INTERVAL (NSEC_PER_SEC / TICKS_PER_SEC)
+#define NULL_REQ_TAG_NOT_AVAILABLE (-1U)
+
#ifdef CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION
static DECLARE_FAULT_ATTR(null_timeout_attr);
static DECLARE_FAULT_ATTR(null_requeue_attr);
@@ -730,7 +732,7 @@ static unsigned int get_tag(struct nullb_queue *nq)
do {
tag = find_first_zero_bit(nq->tag_map, nq->queue_depth);
if (tag >= nq->queue_depth)
- return -1U;
+ return NULL_REQ_TAG_NOT_AVAILABLE;
} while (test_and_set_bit_lock(tag, nq->tag_map));
return tag;
@@ -749,21 +751,19 @@ static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq, struct bio *bio)
unsigned int tag;
tag = get_tag(nq);
- if (tag != -1U) {
- cmd = &nq->cmds[tag];
- cmd->tag = tag;
- cmd->error = BLK_STS_OK;
- cmd->nq = nq;
- cmd->bio = bio;
- if (nq->dev->irqmode == NULL_IRQ_TIMER) {
- hrtimer_init(&cmd->timer, CLOCK_MONOTONIC,
- HRTIMER_MODE_REL);
- cmd->timer.function = null_cmd_timer_expired;
- }
- return cmd;
+ if (tag == NULL_REQ_TAG_NOT_AVAILABLE)
+ return NULL;
+ cmd = &nq->cmds[tag];
+ cmd->tag = tag;
+ cmd->error = BLK_STS_OK;
+ cmd->nq = nq;
+ cmd->bio = bio;
+ if (nq->dev->irqmode == NULL_IRQ_TIMER) {
+ hrtimer_init(&cmd->timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ cmd->timer.function = null_cmd_timer_expired;
}
-
- return NULL;
+ return cmd;
}
static struct nullb_cmd *alloc_cmd(struct nullb_queue *nq, struct bio *bio)
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH V4 5/8] null_blk: remove extra space in switch condition
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
` (3 preceding siblings ...)
2022-11-29 23:28 ` [PATCH V4 4/8] null_blk: don't use magic numbers in the code Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 6/8] null_blk: add param to set max disacrd sectors Chaitanya Kulkarni
` (2 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
The extra space in after switch condition does not follow kernel coding
standards, remove extra space in switch condition end_cmd().
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
---
drivers/block/null_blk/main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index db849e6bc475..96b8aca5abda 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -789,7 +789,7 @@ static void end_cmd(struct nullb_cmd *cmd)
{
int queue_mode = cmd->nq->dev->queue_mode;
- switch (queue_mode) {
+ switch (queue_mode) {
case NULL_Q_MQ:
blk_mq_end_request(cmd->rq, cmd->error);
return;
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH V4 6/8] null_blk: add param to set max disacrd sectors
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
` (4 preceding siblings ...)
2022-11-29 23:28 ` [PATCH V4 5/8] null_blk: remove extra space in switch condition Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-30 8:07 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 8/8] null-blk: allow REQ_OP_ZONE_RESET_ALL to configure Chaitanya Kulkarni
7 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
Instead of hardcoding value for the maximum discard sector to
UINT_MAX >> 9, allow user to set the value with newly added module
parameter max_discard_sectors.
To retain the backward compatibility make set default value to
UINT_MAX >> 9.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/block/null_blk/main.c | 10 +++++++++-
drivers/block/null_blk/null_blk.h | 1 +
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 96b8aca5abda..2f787807cf63 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -216,6 +216,10 @@ static bool g_discard;
module_param_named(discard, g_discard, bool, 0444);
MODULE_PARM_DESC(discard, "Support discard operations (requires memory-backed null_blk device). Default: false");
+static unsigned int g_max_discard_sectors = UINT_MAX >> 9;
+module_param_named(max_discard_sectors, g_max_discard_sectors, uint, 0444);
+MODULE_PARM_DESC(max_discard_sectors, "Maximum size of a REQ_OP_DISCARD command (in 512B sectors).");
+
static bool g_write_zeroes;
module_param_named(write_zeroes, g_write_zeroes, bool, 0444);
MODULE_PARM_DESC(write_zeroes, "Support write-zeores operations. Default: false");
@@ -420,6 +424,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL);
NULLB_DEVICE_ATTR(queue_mode, uint, NULL);
NULLB_DEVICE_ATTR(blocksize, uint, NULL);
NULLB_DEVICE_ATTR(max_sectors, uint, NULL);
+NULLB_DEVICE_ATTR(max_discard_sectors, uint, NULL);
NULLB_DEVICE_ATTR(irqmode, uint, NULL);
NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL);
NULLB_DEVICE_ATTR(index, uint, NULL);
@@ -544,6 +549,7 @@ static struct configfs_attribute *nullb_device_attrs[] = {
&nullb_device_attr_queue_mode,
&nullb_device_attr_blocksize,
&nullb_device_attr_max_sectors,
+ &nullb_device_attr_max_discard_sectors,
&nullb_device_attr_irqmode,
&nullb_device_attr_hw_queue_depth,
&nullb_device_attr_index,
@@ -686,6 +692,7 @@ static struct nullb_device *null_alloc_dev(void)
dev->queue_mode = g_queue_mode;
dev->blocksize = g_bs;
dev->max_sectors = g_max_sectors;
+ dev->max_discard_sectors = g_max_discard_sectors;
dev->irqmode = g_irqmode;
dev->hw_queue_depth = g_hw_queue_depth;
dev->blocking = g_blocking;
@@ -1850,7 +1857,8 @@ static void null_config_discard(struct nullb *nullb)
}
nullb->q->limits.discard_granularity = nullb->dev->blocksize;
- blk_queue_max_discard_sectors(nullb->q, UINT_MAX >> 9);
+ blk_queue_max_discard_sectors(nullb->q,
+ nullb->dev->max_discard_sectors);
}
static void null_config_write_zeroes(struct nullb *nullb)
diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h
index 2c0c9c29158f..09940211326d 100644
--- a/drivers/block/null_blk/null_blk.h
+++ b/drivers/block/null_blk/null_blk.h
@@ -102,6 +102,7 @@ struct nullb_device {
unsigned int queue_mode; /* block interface */
unsigned int blocksize; /* block size */
unsigned int max_sectors; /* Max sectors per command */
+ unsigned int max_discard_sectors; /* Max discard sectors per command */
unsigned int irqmode; /* IRQ completion handler */
unsigned int hw_queue_depth; /* queue depth */
unsigned int index; /* index of the disk, only valid with a disk */
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH V4 6/8] null_blk: add param to set max disacrd sectors
2022-11-29 23:28 ` [PATCH V4 6/8] null_blk: add param to set max disacrd sectors Chaitanya Kulkarni
@ 2022-11-30 8:07 ` Shinichiro Kawasaki
0 siblings, 0 replies; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-11-30 8:07 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 29, 2022 / 15:28, Chaitanya Kulkarni wrote:
> Instead of hardcoding value for the maximum discard sector to
> UINT_MAX >> 9, allow user to set the value with newly added module
> parameter max_discard_sectors.
>
> To retain the backward compatibility make set default value to
> UINT_MAX >> 9.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Looks good to me.
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
` (5 preceding siblings ...)
2022-11-29 23:28 ` [PATCH V4 6/8] null_blk: add param to set max disacrd sectors Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
2022-11-30 8:09 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 8/8] null-blk: allow REQ_OP_ZONE_RESET_ALL to configure Chaitanya Kulkarni
7 siblings, 1 reply; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
Instead of hardcoding value for the maximum write-zeroes sector to
UINT_MAX >> 9, allow user to set the value with newly added module
parameter max_write_zeroes_sectors.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/block/null_blk/main.c | 10 +++++++++-
drivers/block/null_blk/null_blk.h | 1 +
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 2f787807cf63..8b7f42024f14 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -224,6 +224,10 @@ static bool g_write_zeroes;
module_param_named(write_zeroes, g_write_zeroes, bool, 0444);
MODULE_PARM_DESC(write_zeroes, "Support write-zeores operations. Default: false");
+static unsigned int g_max_write_zeroes_sectors = UINT_MAX >> 9;
+module_param_named(max_write_zeroes_sectors, g_max_write_zeroes_sectors, uint, 0444);
+MODULE_PARM_DESC(max_write_zeroes_sectors, "Maximum size of a REQ_OP_WRITE_ZEROES command (in 512B sectors).");
+
static unsigned long g_cache_size;
module_param_named(cache_size, g_cache_size, ulong, 0444);
MODULE_PARM_DESC(mbps, "Cache size in MiB for memory-backed device. Default: 0 (none)");
@@ -425,6 +429,7 @@ NULLB_DEVICE_ATTR(queue_mode, uint, NULL);
NULLB_DEVICE_ATTR(blocksize, uint, NULL);
NULLB_DEVICE_ATTR(max_sectors, uint, NULL);
NULLB_DEVICE_ATTR(max_discard_sectors, uint, NULL);
+NULLB_DEVICE_ATTR(max_write_zeroes_sectors, uint, NULL);
NULLB_DEVICE_ATTR(irqmode, uint, NULL);
NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL);
NULLB_DEVICE_ATTR(index, uint, NULL);
@@ -550,6 +555,7 @@ static struct configfs_attribute *nullb_device_attrs[] = {
&nullb_device_attr_blocksize,
&nullb_device_attr_max_sectors,
&nullb_device_attr_max_discard_sectors,
+ &nullb_device_attr_max_write_zeroes_sectors,
&nullb_device_attr_irqmode,
&nullb_device_attr_hw_queue_depth,
&nullb_device_attr_index,
@@ -693,6 +699,7 @@ static struct nullb_device *null_alloc_dev(void)
dev->blocksize = g_bs;
dev->max_sectors = g_max_sectors;
dev->max_discard_sectors = g_max_discard_sectors;
+ dev->max_write_zeroes_sectors = g_max_write_zeroes_sectors;
dev->irqmode = g_irqmode;
dev->hw_queue_depth = g_hw_queue_depth;
dev->blocking = g_blocking;
@@ -1865,7 +1872,8 @@ static void null_config_write_zeroes(struct nullb *nullb)
{
if (!nullb->dev->write_zeroes)
return;
- blk_queue_max_write_zeroes_sectors(nullb->q, UINT_MAX >> 9);
+ blk_queue_max_write_zeroes_sectors(nullb->q,
+ nullb->dev->max_write_zeroes_sectors);
}
static const struct block_device_operations null_bio_ops = {
diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h
index 09940211326d..e692c2a7369e 100644
--- a/drivers/block/null_blk/null_blk.h
+++ b/drivers/block/null_blk/null_blk.h
@@ -103,6 +103,7 @@ struct nullb_device {
unsigned int blocksize; /* block size */
unsigned int max_sectors; /* Max sectors per command */
unsigned int max_discard_sectors; /* Max discard sectors per command */
+ unsigned int max_write_zeroes_sectors; /* Max write-zeroes sectors per command */
unsigned int irqmode; /* IRQ completion handler */
unsigned int hw_queue_depth; /* queue depth */
unsigned int index; /* index of the disk, only valid with a disk */
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects
2022-11-29 23:28 ` [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects Chaitanya Kulkarni
@ 2022-11-30 8:09 ` Shinichiro Kawasaki
0 siblings, 0 replies; 16+ messages in thread
From: Shinichiro Kawasaki @ 2022-11-30 8:09 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
axboe@kernel.dk, damien.lemoal@opensource.wdc.com,
bvanassche@acm.org, vincent.fu@samsung.com
On Nov 29, 2022 / 15:28, Chaitanya Kulkarni wrote:
> Instead of hardcoding value for the maximum write-zeroes sector to
> UINT_MAX >> 9, allow user to set the value with newly added module
> parameter max_write_zeroes_sectors.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Nit: it would be the better to note in the commit message why the default value
is UINT_MAX >> 9. Other than that, looks good.
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
--
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH V4 8/8] null-blk: allow REQ_OP_ZONE_RESET_ALL to configure
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
` (6 preceding siblings ...)
2022-11-29 23:28 ` [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects Chaitanya Kulkarni
@ 2022-11-29 23:28 ` Chaitanya Kulkarni
7 siblings, 0 replies; 16+ messages in thread
From: Chaitanya Kulkarni @ 2022-11-29 23:28 UTC (permalink / raw)
To: linux-block, linux-kernel
Cc: axboe, damien.lemoal, bvanassche, shinichiro.kawasaki, vincent.fu,
Chaitanya Kulkarni
For a Zoned Block Device zone reset all is emulated if underlaying
device doesn't support REQ_OP_ZONE_RESET_ALL operation. In null_blk
Zoned mode there is no way to test zone reset all emulation present in
the block layer since we enable it by default :-
blkdev_zone_mgmt()
blkdev_zone_reset_all_emulation() <---
blkdev_zone_reset_all()
Add a module parameter zone_reset_all to enable or disable
REQ_OP_ZONE_RESET_ALL, enable it by default to retain the existing
behaviour.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
---
drivers/block/null_blk/main.c | 9 ++++++++-
drivers/block/null_blk/null_blk.h | 1 +
drivers/block/null_blk/zoned.c | 3 ++-
3 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 8b7f42024f14..5dc69f42b46c 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -260,6 +260,10 @@ static unsigned int g_zone_max_active;
module_param_named(zone_max_active, g_zone_max_active, uint, 0444);
MODULE_PARM_DESC(zone_max_active, "Maximum number of active zones when block device is zoned. Default: 0 (no limit)");
+static bool g_zone_reset_all = true;
+module_param_named(zone_reset_all, g_zone_reset_all, bool, 0444);
+MODULE_PARM_DESC(zone_reset_all, "Allow REQ_OP_ZONE_RESET_ALL. Default: true");
+
static struct nullb_device *null_alloc_dev(void);
static void null_free_dev(struct nullb_device *dev);
static void null_del_dev(struct nullb *nullb);
@@ -446,6 +450,7 @@ NULLB_DEVICE_ATTR(zone_capacity, ulong, NULL);
NULLB_DEVICE_ATTR(zone_nr_conv, uint, NULL);
NULLB_DEVICE_ATTR(zone_max_open, uint, NULL);
NULLB_DEVICE_ATTR(zone_max_active, uint, NULL);
+NULLB_DEVICE_ATTR(zone_reset_all, bool, NULL);
NULLB_DEVICE_ATTR(virt_boundary, bool, NULL);
NULLB_DEVICE_ATTR(no_sched, bool, NULL);
NULLB_DEVICE_ATTR(shared_tag_bitmap, bool, NULL);
@@ -574,6 +579,7 @@ static struct configfs_attribute *nullb_device_attrs[] = {
&nullb_device_attr_zone_nr_conv,
&nullb_device_attr_zone_max_open,
&nullb_device_attr_zone_max_active,
+ &nullb_device_attr_zone_reset_all,
&nullb_device_attr_virt_boundary,
&nullb_device_attr_no_sched,
&nullb_device_attr_shared_tag_bitmap,
@@ -639,7 +645,7 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page)
"poll_queues,power,queue_mode,shared_tag_bitmap,size,"
"submit_queues,use_per_node_hctx,virt_boundary,zoned,"
"zone_capacity,zone_max_active,zone_max_open,"
- "zone_nr_conv,zone_size,write_zeroes\n");
+ "zone_nr_conv,zone_size,zone_reset_all,write_zeroes\n");
}
CONFIGFS_ATTR_RO(memb_group_, features);
@@ -715,6 +721,7 @@ static struct nullb_device *null_alloc_dev(void)
dev->zone_nr_conv = g_zone_nr_conv;
dev->zone_max_open = g_zone_max_open;
dev->zone_max_active = g_zone_max_active;
+ dev->zone_reset_all = g_zone_reset_all;
dev->virt_boundary = g_virt_boundary;
dev->no_sched = g_no_sched;
dev->shared_tag_bitmap = g_shared_tag_bitmap;
diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h
index e692c2a7369e..e7efe8de4ebf 100644
--- a/drivers/block/null_blk/null_blk.h
+++ b/drivers/block/null_blk/null_blk.h
@@ -115,6 +115,7 @@ struct nullb_device {
bool discard; /* if support discard */
bool write_zeroes; /* if support write_zeroes */
bool zoned; /* if device is zoned */
+ bool zone_reset_all; /* if support REQ_OP_ZONE_RESET_ALL */
bool virt_boundary; /* virtual boundary on/off for the device */
bool no_sched; /* no IO scheduler for the device */
bool shared_tag_bitmap; /* use hostwide shared tags */
diff --git a/drivers/block/null_blk/zoned.c b/drivers/block/null_blk/zoned.c
index 55a69e48ef8b..7310d1c3f9ec 100644
--- a/drivers/block/null_blk/zoned.c
+++ b/drivers/block/null_blk/zoned.c
@@ -160,7 +160,8 @@ int null_register_zoned_dev(struct nullb *nullb)
struct request_queue *q = nullb->q;
disk_set_zoned(nullb->disk, BLK_ZONED_HM);
- blk_queue_flag_set(QUEUE_FLAG_ZONE_RESETALL, q);
+ if (dev->zone_reset_all)
+ blk_queue_flag_set(QUEUE_FLAG_ZONE_RESETALL, q);
blk_queue_required_elevator_features(q, ELEVATOR_F_ZBD_SEQ_WRITE);
if (queue_is_mq(q)) {
--
2.29.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2022-12-01 1:21 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-11-29 23:28 [PATCH V4 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 1/8] null_blk: allow REQ_OP_WRITE_ZEROES Chaitanya Kulkarni
2022-11-30 7:59 ` Shinichiro Kawasaki
2022-11-30 23:29 ` Chaitanya Kulkarni
2022-12-01 1:21 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 2/8] null_blk: code cleaup Chaitanya Kulkarni
2022-11-30 8:00 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 3/8] null_blk: initialize cmd->bio in __alloc_cmd() Chaitanya Kulkarni
2022-11-30 8:01 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 4/8] null_blk: don't use magic numbers in the code Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 5/8] null_blk: remove extra space in switch condition Chaitanya Kulkarni
2022-11-29 23:28 ` [PATCH V4 6/8] null_blk: add param to set max disacrd sectors Chaitanya Kulkarni
2022-11-30 8:07 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 7/8] null_blk: add param to set max write-zeroes sects Chaitanya Kulkarni
2022-11-30 8:09 ` Shinichiro Kawasaki
2022-11-29 23:28 ` [PATCH V4 8/8] null-blk: allow REQ_OP_ZONE_RESET_ALL to configure Chaitanya Kulkarni
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox