From: "Luís Henriques" <lhenriques@suse.de>
To: Jens Axboe <axboe@kernel.dk>
Cc: fio@vger.kernel.org
Subject: Re: [PATCH] fio: make sure io_u->file isn't NULL before using it
Date: Mon, 11 Oct 2021 16:45:07 +0100 [thread overview]
Message-ID: <YWRcA/2AIb3pMOkG@suse.de> (raw)
In-Reply-To: <e5cc8d8e-e1ba-a97f-b761-c697bdf136b4@kernel.dk>
[-- Attachment #1: Type: text/plain, Size: 2622 bytes --]
On Mon, Oct 11, 2021 at 09:15:47AM -0600, Jens Axboe wrote:
> On 10/11/21 7:44 AM, Luís Henriques wrote:
> > On Mon, Oct 11, 2021 at 06:58:52AM -0600, Jens Axboe wrote:
> >> On 10/11/21 4:27 AM, Luís Henriques wrote:
> >>> While running fstests generic/095 against ext4 on a zram device I started
> >>> seeing fio crashing. Fix it by making sure io_u->file isn't NULL before
> >>> accessing it.
> >>>
> >>> Signed-off-by: Luís Henriques <lhenriques@suse.de>
> >>> ---
> >>> io_u.c | 3 ++-
> >>> 1 file changed, 2 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/io_u.c b/io_u.c
> >>> index 5289b5d1d9c6..b8e715d4118c 100644
> >>> --- a/io_u.c
> >>> +++ b/io_u.c
> >>> @@ -2009,7 +2009,8 @@ static void io_completed(struct thread_data *td, struct io_u **io_u_ptr,
> >>> io_u->xfer_buf += bytes;
> >>> io_u->offset += bytes;
> >>> td->ts.short_io_u[io_u->ddir]++;
> >>> - if (io_u->offset < io_u->file->real_file_size) {
> >>> + if (io_u->file &&
> >>> + (io_u->offset < io_u->file->real_file_size)) {
> >>> requeue_io_u(td, io_u_ptr);
> >>> return;
> >>> }
> >>
> >> This will prevent the crash, but I'm wondering why io_u-> == NULL for this case.
> >> It really should be a valid file.
> >>
> >> Can you let me know exactly how you're reproducing this? Then I'll give it
> >> a whirl too.
> >
> > Sure, here's my recipe for reproducing this bug:
> >
> > # modprobe zram num_devices=2
> > # echo 1G > /sys/block/zram0/disksize
> > # echo 1G > /sys/block/zram1/disksize
> > # mkfs.ext4 /dev/zram0
> > # mkfs.ext4 /dev/zram1
> >
> > # ./check generic/095
>
> I know you fixed the issue in the test by now, but any chance you can try
> this one and see if it correctly reports full residual instead of trying
> to requeue and crash?
Sure! So, I patched fio with this change and executed the generic/095
test twice: one run without the test fix, another with the test fix. I
can confirm that fio didn't crash in any of the runs, but I'm not sure how
to check if it's correctly reporting residual.
I'm attaching the test logs (for the failure run) in case that helps.
Cheers,
--
Luís
>
> diff --git a/io_u.c b/io_u.c
> index 5289b5d1d9c6..586a4befdce0 100644
> --- a/io_u.c
> +++ b/io_u.c
> @@ -2004,7 +2004,7 @@ static void io_completed(struct thread_data *td, struct io_u **io_u_ptr,
> * Make sure we notice short IO from here, and requeue them
> * appropriately!
> */
> - if (io_u->resid) {
> + if (bytes && io_u->resid) {
> io_u->xfer_buflen = io_u->resid;
> io_u->xfer_buf += bytes;
> io_u->offset += bytes;
>
> --
> Jens Axboe
>
[-- Attachment #2: 095.full --]
[-- Type: text/plain, Size: 55020 bytes --]
fio --bs=8k --iodepth=16 --iodepth_batch=8 --randrepeat=1 --size=1m --directory=/tmp/mnt/scratch --numjobs=5 --name=job1 --ioengine=sync --bs=1k --direct=1 --rw=randread --filename=file1:file2 --name=job2 --ioengine=libaio --rw=randwrite --direct=1 --filename=file1:file2 --name=job3 --bs=1k --ioengine=posixaio --rw=randwrite --direct=1 --filename=file1:file2 --name=job4 --ioengine=splice --direct=1 --rw=randwrite --filename=file1:file2 --name=job5 --bs=1k --ioengine=sync --rw=randread --filename=file1:file2 --name=job6 --ioengine=posixaio --rw=randwrite --filename=file1:file2 --name=job7 --ioengine=splice --rw=randwrite --filename=file1:file2 --name=job8 --ioengine=mmap --rw=randwrite --bs=1k --filename=file1:file2 --name=job9 --ioengine=mmap --rw=randwrite --direct=1 --bs=4096 --filename=file1:file2
Discarding device blocks: 0/262144\b\b\b\b\b\b\b\b\b\b\b\b\b \b\b\b\b\b\b\b\b\b\b\b\b\bdone
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: e77d8742-2325-4f0c-a67c-9900cb195663
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: 0/8\b\b\b \b\b\bdone
Writing inode tables: 0/8\b\b\b \b\b\bdone
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: 0/8\b\b\b \b\b\bdone
job1: (g=0): rw=randread, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=sync, iodepth=16
...
job2: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
...
job3: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=posixaio, iodepth=16
...
job4: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=splice, iodepth=16
...
job5: (g=0): rw=randread, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=sync, iodepth=16
...
job6: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=16
...
job7: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=splice, iodepth=16
...
job8: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=mmap, iodepth=16
...
job9: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=16
...
fio-3.28-57-g63b8-dirty
Starting 45 processes
job1: Laying out IO files (2 files / total 1MiB)
fio: pid=2364, got signal=6
fio: pid=2367, got signal=6
fio: pid=2368, got signal=6
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2354: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2355: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=3, majf=3, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2356: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=6, majf=6, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2357: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job1: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2358: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=8, majf=8, minf=24
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2359: Mon Oct 11 16:33:05 2021
write: IOPS=1600, BW=12.5MiB/s (13.1MB/s)(1024KiB/80msec); 0 zone resets
slat (usec): min=25, max=37322, avg=2684.09, stdev=7381.40
clat (usec): min=4, max=12289, avg=3860.08, stdev=3161.06
lat (usec): min=77, max=47504, avg=6544.27, stdev=9652.17
clat percentiles (usec):
| 1.00th=[ 82], 5.00th=[ 652], 10.00th=[ 1221], 20.00th=[ 1598],
| 30.00th=[ 1713], 40.00th=[ 1778], 50.00th=[ 2343], 60.00th=[ 3359],
| 70.00th=[ 3949], 80.00th=[ 6980], 90.00th=[ 8586], 95.00th=[10814],
| 99.00th=[11994], 99.50th=[12256], 99.90th=[12256], 99.95th=[12256],
| 99.99th=[12256]
lat (usec) : 10=0.78%, 100=0.78%, 250=0.78%, 500=0.78%, 750=3.12%
lat (usec) : 1000=2.34%
lat (msec) : 2=38.28%, 4=24.22%, 10=21.88%, 20=7.03%
cpu : usr=0.00%, sys=13.92%, ctx=61, majf=3, minf=19
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2360: Mon Oct 11 16:33:05 2021
write: IOPS=2206, BW=17.2MiB/s (18.1MB/s)(1024KiB/58msec); 0 zone resets
slat (usec): min=19, max=22227, avg=1293.17, stdev=4243.34
clat (nsec): min=1744, max=13937k, avg=2624525.45, stdev=3543041.64
lat (usec): min=55, max=22739, avg=3917.79, stdev=5242.31
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 233], 10.00th=[ 437], 20.00th=[ 545],
| 30.00th=[ 570], 40.00th=[ 594], 50.00th=[ 627], 60.00th=[ 889],
| 70.00th=[ 2671], 80.00th=[ 3556], 90.00th=[ 9503], 95.00th=[11469],
| 99.00th=[12649], 99.50th=[13960], 99.90th=[13960], 99.95th=[13960],
| 99.99th=[13960]
lat (usec) : 2=0.78%, 100=0.78%, 250=3.91%, 500=5.47%, 750=46.88%
lat (usec) : 1000=3.91%
lat (msec) : 2=4.69%, 4=14.06%, 10=12.50%, 20=7.03%
cpu : usr=8.77%, sys=0.00%, ctx=33, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2361: Mon Oct 11 16:33:05 2021
write: IOPS=1662, BW=13.0MiB/s (13.6MB/s)(1024KiB/77msec); 0 zone resets
slat (usec): min=21, max=31476, avg=2285.17, stdev=6451.78
clat (nsec): min=1461, max=8546.3k, avg=3394454.86, stdev=2584376.00
lat (usec): min=30, max=38816, avg=5679.71, stdev=7944.51
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 243], 10.00th=[ 938], 20.00th=[ 1336],
| 30.00th=[ 1762], 40.00th=[ 2114], 50.00th=[ 2278], 60.00th=[ 2573],
| 70.00th=[ 5080], 80.00th=[ 5932], 90.00th=[ 7963], 95.00th=[ 8291],
| 99.00th=[ 8586], 99.50th=[ 8586], 99.90th=[ 8586], 99.95th=[ 8586],
| 99.99th=[ 8586]
lat (usec) : 2=0.78%, 50=0.78%, 100=1.56%, 250=2.34%, 500=1.56%
lat (usec) : 1000=3.91%
lat (msec) : 2=24.22%, 4=34.38%, 10=30.47%
cpu : usr=0.00%, sys=6.58%, ctx=77, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2362: Mon Oct 11 16:33:05 2021
write: IOPS=1684, BW=13.2MiB/s (13.8MB/s)(1024KiB/76msec); 0 zone resets
slat (usec): min=18, max=34986, avg=2547.30, stdev=6865.46
clat (nsec): min=1454, max=12139k, avg=3714736.58, stdev=3282289.17
lat (usec): min=26, max=45556, avg=6262.11, stdev=9137.41
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 190], 10.00th=[ 343], 20.00th=[ 465],
| 30.00th=[ 1942], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2802],
| 70.00th=[ 3916], 80.00th=[ 6783], 90.00th=[ 8455], 95.00th=[11207],
| 99.00th=[11994], 99.50th=[12125], 99.90th=[12125], 99.95th=[12125],
| 99.99th=[12125]
lat (usec) : 2=0.78%, 50=0.78%, 100=0.78%, 250=4.69%, 500=13.28%
lat (msec) : 2=13.28%, 4=36.72%, 10=22.66%, 20=7.03%
cpu : usr=0.00%, sys=5.33%, ctx=90, majf=3, minf=19
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job2: (groupid=0, jobs=1): err= 0: pid=2363: Mon Oct 11 16:33:05 2021
write: IOPS=1662, BW=13.0MiB/s (13.6MB/s)(1024KiB/77msec); 0 zone resets
slat (usec): min=20, max=36554, avg=2615.40, stdev=7062.59
clat (nsec): min=1561, max=13205k, avg=3852028.32, stdev=3205515.13
lat (usec): min=36, max=47273, avg=6467.50, stdev=9399.31
clat percentiles (usec):
| 1.00th=[ 38], 5.00th=[ 343], 10.00th=[ 1020], 20.00th=[ 1500],
| 30.00th=[ 1663], 40.00th=[ 1975], 50.00th=[ 2769], 60.00th=[ 3359],
| 70.00th=[ 4424], 80.00th=[ 6259], 90.00th=[ 8160], 95.00th=[11207],
| 99.00th=[13173], 99.50th=[13173], 99.90th=[13173], 99.95th=[13173],
| 99.99th=[13173]
lat (usec) : 2=0.78%, 50=0.78%, 100=0.78%, 250=1.56%, 500=3.12%
lat (usec) : 750=0.78%, 1000=1.56%
lat (msec) : 2=31.25%, 4=28.91%, 10=23.44%, 20=7.03%
cpu : usr=0.00%, sys=6.58%, ctx=81, majf=1, minf=21
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=6.2%, 16=93.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=98.2%, 8=1.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.0%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2364: Mon Oct 11 16:33:05 2021
lat (msec) : 10=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2365: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=2, majf=1, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.6%, 4=94.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2366: Mon Oct 11 16:33:05 2021
cpu : usr=0.00%, sys=0.00%, ctx=5, majf=3, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.6%, 4=94.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2367: Mon Oct 11 16:33:05 2021
lat (usec) : 1000=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job3: (groupid=0, jobs=1): err=22 (file:io_u.c:1845, func=io_u error, error=Invalid argument): pid=2368: Mon Oct 11 16:33:05 2021
lat (msec) : 4=18.75%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=5.3%, 4=84.2%, 8=5.3%, 16=5.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,16,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2369: Mon Oct 11 16:33:05 2021
write: IOPS=1641, BW=12.8MiB/s (13.4MB/s)(1024KiB/78msec); 0 zone resets
clat (usec): min=32, max=18196, avg=511.89, stdev=1705.10
lat (usec): min=32, max=18196, avg=512.09, stdev=1705.10
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 61],
| 30.00th=[ 73], 40.00th=[ 88], 50.00th=[ 149], 60.00th=[ 176],
| 70.00th=[ 285], 80.00th=[ 494], 90.00th=[ 955], 95.00th=[ 2245],
| 99.00th=[ 3851], 99.50th=[18220], 99.90th=[18220], 99.95th=[18220],
| 99.99th=[18220]
lat (usec) : 50=14.06%, 100=28.91%, 250=25.00%, 500=12.50%, 750=7.81%
lat (usec) : 1000=3.12%
lat (msec) : 2=3.12%, 4=4.69%, 20=0.78%
cpu : usr=0.00%, sys=12.99%, ctx=83, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2370: Mon Oct 11 16:33:05 2021
write: IOPS=2064, BW=16.1MiB/s (16.9MB/s)(1024KiB/62msec); 0 zone resets
clat (usec): min=25, max=16986, avg=470.55, stdev=1703.89
lat (usec): min=25, max=16986, avg=470.70, stdev=1703.90
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 33], 10.00th=[ 34], 20.00th=[ 38],
| 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 55],
| 70.00th=[ 76], 80.00th=[ 363], 90.00th=[ 947], 95.00th=[ 2147],
| 99.00th=[ 5604], 99.50th=[16909], 99.90th=[16909], 99.95th=[16909],
| 99.99th=[16909]
lat (usec) : 50=56.25%, 100=16.41%, 250=6.25%, 500=8.59%, 750=0.78%
lat (usec) : 1000=2.34%
lat (msec) : 2=3.12%, 4=3.91%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=9.84%, ctx=45, majf=2, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2371: Mon Oct 11 16:33:05 2021
write: IOPS=1729, BW=13.5MiB/s (14.2MB/s)(1024KiB/74msec); 0 zone resets
clat (usec): min=30, max=18586, avg=476.04, stdev=1774.05
lat (usec): min=31, max=18586, avg=476.26, stdev=1774.05
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34],
| 30.00th=[ 56], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 163],
| 70.00th=[ 206], 80.00th=[ 343], 90.00th=[ 840], 95.00th=[ 1614],
| 99.00th=[ 5407], 99.50th=[18482], 99.90th=[18482], 99.95th=[18482],
| 99.99th=[18482]
lat (usec) : 50=25.00%, 100=27.34%, 250=20.31%, 500=14.06%, 750=3.12%
lat (usec) : 1000=2.34%
lat (msec) : 2=3.12%, 4=2.34%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=13.70%, ctx=49, majf=2, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2372: Mon Oct 11 16:33:05 2021
write: IOPS=1706, BW=13.3MiB/s (14.0MB/s)(1024KiB/75msec); 0 zone resets
clat (usec): min=32, max=17596, avg=565.07, stdev=1719.54
lat (usec): min=32, max=17596, avg=565.27, stdev=1719.57
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 67],
| 30.00th=[ 78], 40.00th=[ 104], 50.00th=[ 130], 60.00th=[ 180],
| 70.00th=[ 289], 80.00th=[ 379], 90.00th=[ 1303], 95.00th=[ 2966],
| 99.00th=[ 4228], 99.50th=[17695], 99.90th=[17695], 99.95th=[17695],
| 99.99th=[17695]
lat (usec) : 50=15.62%, 100=23.44%, 250=26.56%, 500=16.41%, 750=3.91%
lat (usec) : 1000=2.34%
lat (msec) : 2=5.47%, 4=3.91%, 10=1.56%, 20=0.78%
cpu : usr=6.76%, sys=8.11%, ctx=102, majf=0, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job4: (groupid=0, jobs=1): err= 0: pid=2373: Mon Oct 11 16:33:05 2021
write: IOPS=1620, BW=12.7MiB/s (13.3MB/s)(1024KiB/79msec); 0 zone resets
clat (usec): min=35, max=6962, avg=520.71, stdev=1210.46
lat (usec): min=35, max=6962, avg=520.98, stdev=1210.49
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 65], 20.00th=[ 86],
| 30.00th=[ 100], 40.00th=[ 118], 50.00th=[ 151], 60.00th=[ 172],
| 70.00th=[ 221], 80.00th=[ 326], 90.00th=[ 1156], 95.00th=[ 3163],
| 99.00th=[ 6849], 99.50th=[ 6980], 99.90th=[ 6980], 99.95th=[ 6980],
| 99.99th=[ 6980]
lat (usec) : 50=8.59%, 100=20.31%, 250=45.31%, 500=12.50%, 750=1.56%
lat (usec) : 1000=1.56%
lat (msec) : 2=2.34%, 4=4.69%, 10=3.12%
cpu : usr=0.00%, sys=19.23%, ctx=113, majf=6, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2374: Mon Oct 11 16:33:05 2021
read: IOPS=85.3k, BW=83.3MiB/s (87.4MB/s)(1024KiB/12msec)
clat (nsec): min=582, max=104452, avg=965.48, stdev=3594.28
lat (nsec): min=622, max=104492, avg=1006.11, stdev=3595.08
clat percentiles (nsec):
| 1.00th=[ 588], 5.00th=[ 604], 10.00th=[ 620], 20.00th=[ 652],
| 30.00th=[ 684], 40.00th=[ 700], 50.00th=[ 716], 60.00th=[ 732],
| 70.00th=[ 748], 80.00th=[ 772], 90.00th=[ 812], 95.00th=[ 900],
| 99.00th=[ 4048], 99.50th=[ 5280], 99.90th=[ 46336], 99.95th=[104960],
| 99.99th=[104960]
lat (nsec) : 750=69.43%, 1000=26.46%
lat (usec) : 2=0.68%, 4=2.34%, 10=0.78%, 20=0.10%, 50=0.10%
lat (usec) : 250=0.10%
cpu : usr=9.09%, sys=0.00%, ctx=9, majf=6, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2375: Mon Oct 11 16:33:05 2021
read: IOPS=73.1k, BW=71.4MiB/s (74.9MB/s)(1024KiB/14msec)
clat (nsec): min=562, max=17276, avg=759.74, stdev=815.84
lat (nsec): min=603, max=17596, avg=799.78, stdev=822.30
clat percentiles (nsec):
| 1.00th=[ 588], 5.00th=[ 596], 10.00th=[ 604], 20.00th=[ 620],
| 30.00th=[ 644], 40.00th=[ 660], 50.00th=[ 684], 60.00th=[ 700],
| 70.00th=[ 724], 80.00th=[ 748], 90.00th=[ 788], 95.00th=[ 820],
| 99.00th=[ 2768], 99.50th=[ 4896], 99.90th=[16064], 99.95th=[17280],
| 99.99th=[17280]
lat (nsec) : 750=81.25%, 1000=16.70%
lat (usec) : 2=0.68%, 4=0.68%, 10=0.49%, 20=0.20%
cpu : usr=0.00%, sys=7.69%, ctx=4, majf=3, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2376: Mon Oct 11 16:33:05 2021
read: IOPS=93.1k, BW=90.9MiB/s (95.3MB/s)(1024KiB/11msec)
clat (nsec): min=594, max=14728, avg=1045.56, stdev=1284.62
lat (nsec): min=630, max=14769, avg=1086.76, stdev=1293.48
clat percentiles (nsec):
| 1.00th=[ 604], 5.00th=[ 620], 10.00th=[ 636], 20.00th=[ 660],
| 30.00th=[ 684], 40.00th=[ 700], 50.00th=[ 716], 60.00th=[ 732],
| 70.00th=[ 756], 80.00th=[ 788], 90.00th=[ 1064], 95.00th=[ 3568],
| 99.00th=[ 5664], 99.50th=[ 8256], 99.90th=[14144], 99.95th=[14784],
| 99.99th=[14784]
lat (nsec) : 750=67.48%, 1000=22.27%
lat (usec) : 2=1.95%, 4=5.76%, 10=2.15%, 20=0.39%
cpu : usr=10.00%, sys=0.00%, ctx=6, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2377: Mon Oct 11 16:33:05 2021
read: IOPS=85.3k, BW=83.3MiB/s (87.4MB/s)(1024KiB/12msec)
clat (nsec): min=641, max=16976, avg=878.51, stdev=857.12
lat (nsec): min=681, max=17021, avg=923.44, stdev=864.64
clat percentiles (nsec):
| 1.00th=[ 652], 5.00th=[ 668], 10.00th=[ 684], 20.00th=[ 700],
| 30.00th=[ 724], 40.00th=[ 748], 50.00th=[ 772], 60.00th=[ 788],
| 70.00th=[ 804], 80.00th=[ 828], 90.00th=[ 868], 95.00th=[ 908],
| 99.00th=[ 4320], 99.50th=[ 5024], 99.90th=[14784], 99.95th=[17024],
| 99.99th=[17024]
lat (nsec) : 750=39.75%, 1000=57.13%
lat (usec) : 2=0.49%, 4=1.37%, 10=1.07%, 20=0.20%
cpu : usr=9.09%, sys=0.00%, ctx=9, majf=8, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job5: (groupid=0, jobs=1): err= 0: pid=2378: Mon Oct 11 16:33:05 2021
read: IOPS=68.3k, BW=66.7MiB/s (69.9MB/s)(1024KiB/15msec)
clat (nsec): min=642, max=27395, avg=978.34, stdev=1424.98
lat (nsec): min=679, max=27933, avg=1023.85, stdev=1434.63
clat percentiles (nsec):
| 1.00th=[ 652], 5.00th=[ 668], 10.00th=[ 676], 20.00th=[ 700],
| 30.00th=[ 732], 40.00th=[ 756], 50.00th=[ 772], 60.00th=[ 796],
| 70.00th=[ 812], 80.00th=[ 844], 90.00th=[ 892], 95.00th=[ 1096],
| 99.00th=[ 6048], 99.50th=[ 8640], 99.90th=[19328], 99.95th=[27520],
| 99.99th=[27520]
lat (nsec) : 750=37.30%, 1000=57.23%
lat (usec) : 2=2.15%, 4=1.07%, 10=1.86%, 20=0.29%, 50=0.10%
cpu : usr=7.14%, sys=0.00%, ctx=9, majf=8, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2379: Mon Oct 11 16:33:05 2021
write: IOPS=1855, BW=14.5MiB/s (15.2MB/s)(1024KiB/69msec); 0 zone resets
slat (nsec): min=141, max=75144, avg=1565.61, stdev=7509.80
clat (usec): min=8, max=33482, avg=6816.01, stdev=8391.68
lat (usec): min=9, max=33483, avg=6817.58, stdev=8392.09
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 19], 10.00th=[ 38], 20.00th=[ 147],
| 30.00th=[ 1467], 40.00th=[ 2180], 50.00th=[ 3556], 60.00th=[ 5014],
| 70.00th=[ 7504], 80.00th=[13435], 90.00th=[17695], 95.00th=[28967],
| 99.00th=[33162], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424],
| 99.99th=[33424]
lat (usec) : 10=3.12%, 20=2.34%, 50=5.47%, 100=7.03%, 250=3.91%
lat (usec) : 500=1.56%, 750=1.56%, 1000=0.78%
lat (msec) : 2=12.50%, 4=14.84%, 10=25.00%, 20=12.50%, 50=9.38%
cpu : usr=0.00%, sys=0.00%, ctx=74, majf=2, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=62.5%, 16=37.5%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=96.1%, 8=3.4%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2380: Mon Oct 11 16:33:05 2021
write: IOPS=2560, BW=20.0MiB/s (21.0MB/s)(1024KiB/50msec); 0 zone resets
slat (nsec): min=203, max=148143, avg=2370.61, stdev=13430.86
clat (usec): min=6, max=25433, avg=4766.72, stdev=7242.47
lat (usec): min=6, max=25434, avg=4769.09, stdev=7244.42
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 40],
| 30.00th=[ 235], 40.00th=[ 1045], 50.00th=[ 1385], 60.00th=[ 2966],
| 70.00th=[ 4555], 80.00th=[ 6718], 90.00th=[22414], 95.00th=[22938],
| 99.00th=[23462], 99.50th=[25560], 99.90th=[25560], 99.95th=[25560],
| 99.99th=[25560]
lat (usec) : 10=2.34%, 20=4.69%, 50=16.41%, 100=2.34%, 250=4.69%
lat (usec) : 500=1.56%, 750=0.78%, 1000=4.69%
lat (msec) : 2=15.62%, 4=10.16%, 10=23.44%, 20=1.56%, 50=11.72%
cpu : usr=2.04%, sys=0.00%, ctx=41, majf=1, minf=28
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=43.8%, 16=56.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.5%, 8=3.4%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2381: Mon Oct 11 16:33:05 2021
write: IOPS=1910, BW=14.9MiB/s (15.7MB/s)(1024KiB/67msec); 0 zone resets
slat (nsec): min=145, max=63825, avg=1455.53, stdev=6646.46
clat (usec): min=7, max=26472, avg=6863.34, stdev=7084.59
lat (usec): min=8, max=26473, avg=6864.79, stdev=7085.53
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 23], 10.00th=[ 46], 20.00th=[ 285],
| 30.00th=[ 1287], 40.00th=[ 2278], 50.00th=[ 4686], 60.00th=[ 6783],
| 70.00th=[11469], 80.00th=[13435], 90.00th=[17433], 95.00th=[19792],
| 99.00th=[24511], 99.50th=[26346], 99.90th=[26346], 99.95th=[26346],
| 99.99th=[26346]
lat (usec) : 10=1.56%, 20=3.12%, 50=7.03%, 100=5.47%, 250=1.56%
lat (usec) : 500=3.12%, 750=3.91%, 1000=3.12%
lat (msec) : 2=7.81%, 4=11.72%, 10=19.53%, 20=27.34%, 50=4.69%
cpu : usr=0.00%, sys=0.00%, ctx=135, majf=5, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.4%, 8=1.1%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2382: Mon Oct 11 16:33:05 2021
write: IOPS=4266, BW=33.3MiB/s (35.0MB/s)(1024KiB/30msec); 0 zone resets
slat (nsec): min=133, max=106117, avg=2044.11, stdev=9968.18
clat (usec): min=7, max=11660, avg=3046.86, stdev=3452.44
lat (usec): min=7, max=11662, avg=3048.91, stdev=3452.70
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 22], 10.00th=[ 34], 20.00th=[ 91],
| 30.00th=[ 619], 40.00th=[ 930], 50.00th=[ 2057], 60.00th=[ 2737],
| 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[10814], 95.00th=[11600],
| 99.00th=[11600], 99.50th=[11600], 99.90th=[11600], 99.95th=[11600],
| 99.99th=[11600]
lat (usec) : 10=1.56%, 20=2.34%, 50=10.94%, 100=5.47%, 250=7.03%
lat (usec) : 750=11.72%, 1000=1.56%
lat (msec) : 2=7.81%, 4=26.56%, 10=14.06%, 20=10.94%
cpu : usr=0.00%, sys=0.00%, ctx=37, majf=3, minf=26
IO depths : 1=0.0%, 2=6.2%, 4=6.2%, 8=75.0%, 16=12.5%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.5%, 8=3.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job6: (groupid=0, jobs=1): err= 0: pid=2383: Mon Oct 11 16:33:05 2021
write: IOPS=3200, BW=25.0MiB/s (26.2MB/s)(1024KiB/40msec); 0 zone resets
slat (nsec): min=244, max=66266, avg=2227.13, stdev=7105.77
clat (usec): min=7, max=24308, avg=3503.31, stdev=6587.20
lat (usec): min=11, max=24309, avg=3505.53, stdev=6587.27
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 19], 10.00th=[ 34], 20.00th=[ 139],
| 30.00th=[ 155], 40.00th=[ 273], 50.00th=[ 725], 60.00th=[ 889],
| 70.00th=[ 1385], 80.00th=[ 5473], 90.00th=[18482], 95.00th=[19006],
| 99.00th=[24249], 99.50th=[24249], 99.90th=[24249], 99.95th=[24249],
| 99.99th=[24249]
lat (usec) : 10=0.78%, 20=4.69%, 50=7.03%, 100=5.47%, 250=21.88%
lat (usec) : 500=4.69%, 750=10.94%, 1000=8.59%
lat (msec) : 2=13.28%, 10=9.38%, 20=9.38%, 50=3.91%
cpu : usr=2.56%, sys=0.00%, ctx=26, majf=2, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=68.8%, 16=31.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.1%, 8=5.0%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2384: Mon Oct 11 16:33:05 2021
write: IOPS=3121, BW=24.4MiB/s (25.6MB/s)(1024KiB/41msec); 0 zone resets
clat (usec): min=3, max=14454, avg=233.47, stdev=1499.03
lat (usec): min=3, max=14454, avg=233.55, stdev=1499.06
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 6], 95.00th=[ 122],
| 99.00th=[ 8455], 99.50th=[14484], 99.90th=[14484], 99.95th=[14484],
| 99.99th=[14484]
lat (usec) : 4=22.66%, 10=71.09%, 50=0.78%, 250=0.78%, 750=0.78%
lat (msec) : 2=1.56%, 4=0.78%, 10=0.78%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=13, majf=1, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2385: Mon Oct 11 16:33:05 2021
write: IOPS=3200, BW=25.0MiB/s (26.2MB/s)(1024KiB/40msec); 0 zone resets
clat (usec): min=3, max=11767, avg=214.75, stdev=1229.67
lat (usec): min=3, max=11767, avg=214.82, stdev=1229.69
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 130],
| 99.00th=[ 5014], 99.50th=[11731], 99.90th=[11731], 99.95th=[11731],
| 99.99th=[11731]
lat (usec) : 4=21.88%, 10=69.53%, 20=0.78%, 50=1.56%, 100=0.78%
lat (usec) : 250=0.78%, 750=0.78%
lat (msec) : 2=0.78%, 4=0.78%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=12, majf=2, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2386: Mon Oct 11 16:33:05 2021
write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1024KiB/43msec); 0 zone resets
clat (usec): min=3, max=12613, avg=270.36, stdev=1358.23
lat (usec): min=3, max=12613, avg=270.44, stdev=1358.28
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 1188],
| 99.00th=[ 6063], 99.50th=[12649], 99.90th=[12649], 99.95th=[12649],
| 99.99th=[12649]
lat (usec) : 4=10.94%, 10=79.69%, 20=1.56%, 50=0.78%, 250=0.78%
lat (usec) : 750=0.78%
lat (msec) : 2=0.78%, 4=2.34%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=16, majf=5, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2387: Mon Oct 11 16:33:05 2021
write: IOPS=1600, BW=12.5MiB/s (13.1MB/s)(1024KiB/80msec); 0 zone resets
clat (usec): min=3, max=13691, avg=554.03, stdev=1567.65
lat (usec): min=3, max=13691, avg=554.20, stdev=1567.66
clat percentiles (usec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 10],
| 30.00th=[ 77], 40.00th=[ 99], 50.00th=[ 135], 60.00th=[ 174],
| 70.00th=[ 208], 80.00th=[ 392], 90.00th=[ 1057], 95.00th=[ 2802],
| 99.00th=[ 7111], 99.50th=[13698], 99.90th=[13698], 99.95th=[13698],
| 99.99th=[13698]
lat (usec) : 4=0.78%, 10=20.31%, 20=2.34%, 50=1.56%, 100=17.97%
lat (usec) : 250=30.47%, 500=9.38%, 750=3.91%, 1000=2.34%
lat (msec) : 2=3.91%, 4=3.91%, 10=2.34%, 20=0.78%
cpu : usr=1.27%, sys=2.53%, ctx=125, majf=4, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job7: (groupid=0, jobs=1): err= 0: pid=2388: Mon Oct 11 16:33:05 2021
write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1024KiB/43msec); 0 zone resets
clat (usec): min=3, max=11465, avg=238.97, stdev=1232.98
lat (usec): min=3, max=11465, avg=239.06, stdev=1233.02
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 10], 95.00th=[ 619],
| 99.00th=[ 5342], 99.50th=[11469], 99.90th=[11469], 99.95th=[11469],
| 99.99th=[11469]
lat (usec) : 4=10.94%, 10=81.25%, 20=0.78%, 500=0.78%, 750=1.56%
lat (msec) : 2=0.78%, 4=1.56%, 10=1.56%, 20=0.78%
cpu : usr=0.00%, sys=0.00%, ctx=18, majf=5, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,128,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2389: Mon Oct 11 16:33:05 2021
write: IOPS=29.3k, BW=28.6MiB/s (30.0MB/s)(1024KiB/35msec); 0 zone resets
clat (nsec): min=52, max=4005.3k, avg=21454.89, stdev=228411.55
lat (nsec): min=87, max=4005.4k, avg=21563.28, stdev=228409.29
clat percentiles (nsec):
| 1.00th=[ 59], 5.00th=[ 90], 10.00th=[ 112],
| 20.00th=[ 181], 30.00th=[ 217], 40.00th=[ 282],
| 50.00th=[ 1304], 60.00th=[ 1496], 70.00th=[ 3216],
| 80.00th=[ 3536], 90.00th=[ 5152], 95.00th=[ 11584],
| 99.00th=[ 272384], 99.50th=[1482752], 99.90th=[3620864],
| 99.95th=[4014080], 99.99th=[4014080]
lat (nsec) : 100=7.71%, 250=29.88%, 500=5.47%, 750=0.39%, 1000=0.20%
lat (usec) : 2=21.19%, 4=20.41%, 10=7.81%, 20=4.88%, 50=0.49%
lat (usec) : 100=0.20%, 250=0.29%, 500=0.39%, 750=0.10%
lat (msec) : 2=0.20%, 4=0.29%, 10=0.10%
cpu : usr=8.82%, sys=0.00%, ctx=28, majf=582, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2390: Mon Oct 11 16:33:05 2021
write: IOPS=29.3k, BW=28.6MiB/s (30.0MB/s)(1024KiB/35msec); 0 zone resets
clat (nsec): min=50, max=3672.6k, avg=21308.01, stdev=213632.75
lat (nsec): min=85, max=3672.7k, avg=21416.38, stdev=213630.86
clat percentiles (nsec):
| 1.00th=[ 56], 5.00th=[ 75], 10.00th=[ 104],
| 20.00th=[ 177], 30.00th=[ 211], 40.00th=[ 378],
| 50.00th=[ 1304], 60.00th=[ 2928], 70.00th=[ 3248],
| 80.00th=[ 3632], 90.00th=[ 5472], 95.00th=[ 11840],
| 99.00th=[ 296960], 99.50th=[1515520], 99.90th=[2932736],
| 99.95th=[3686400], 99.99th=[3686400]
lat (nsec) : 100=9.38%, 250=26.95%, 500=5.37%, 750=0.39%
lat (usec) : 2=16.70%, 4=25.68%, 10=8.89%, 20=4.69%, 50=0.88%
lat (usec) : 500=0.20%, 750=0.20%
lat (msec) : 2=0.20%, 4=0.49%
cpu : usr=8.82%, sys=0.00%, ctx=20, majf=599, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2391: Mon Oct 11 16:33:05 2021
write: IOPS=41.0k, BW=40.0MiB/s (41.9MB/s)(1024KiB/25msec); 0 zone resets
clat (nsec): min=49, max=2654.1k, avg=8722.09, stdev=117739.66
lat (nsec): min=84, max=2654.2k, avg=8769.59, stdev=117739.76
clat percentiles (nsec):
| 1.00th=[ 51], 5.00th=[ 58], 10.00th=[ 74],
| 20.00th=[ 96], 30.00th=[ 114], 40.00th=[ 147],
| 50.00th=[ 205], 60.00th=[ 1224], 70.00th=[ 1320],
| 80.00th=[ 1432], 90.00th=[ 1848], 95.00th=[ 2416],
| 99.00th=[ 10048], 99.50th=[ 411648], 99.90th=[1990656],
| 99.95th=[2637824], 99.99th=[2637824]
lat (nsec) : 50=0.20%, 100=22.07%, 250=28.81%, 500=2.25%, 750=0.10%
lat (usec) : 2=38.09%, 4=4.88%, 10=2.54%, 20=0.29%, 50=0.10%
lat (usec) : 500=0.29%, 750=0.10%
lat (msec) : 2=0.20%, 4=0.10%
cpu : usr=4.17%, sys=0.00%, ctx=13, majf=477, minf=21
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2392: Mon Oct 11 16:33:05 2021
write: IOPS=33.0k, BW=32.3MiB/s (33.8MB/s)(1024KiB/31msec); 0 zone resets
clat (nsec): min=53, max=3093.6k, avg=17587.87, stdev=179151.15
lat (nsec): min=91, max=3093.7k, avg=17708.65, stdev=179151.47
clat percentiles (nsec):
| 1.00th=[ 56], 5.00th=[ 59], 10.00th=[ 80],
| 20.00th=[ 181], 30.00th=[ 241], 40.00th=[ 282],
| 50.00th=[ 1240], 60.00th=[ 3216], 70.00th=[ 3504],
| 80.00th=[ 3792], 90.00th=[ 5472], 95.00th=[ 11072],
| 99.00th=[ 189440], 99.50th=[ 937984], 99.90th=[2736128],
| 99.95th=[3096576], 99.99th=[3096576]
lat (nsec) : 100=12.40%, 250=20.12%, 500=15.14%, 750=0.39%
lat (usec) : 2=9.67%, 4=24.22%, 10=10.84%, 20=5.47%, 50=0.68%
lat (usec) : 250=0.20%, 500=0.10%, 750=0.10%, 1000=0.20%
lat (msec) : 2=0.10%, 4=0.39%
cpu : usr=0.00%, sys=13.33%, ctx=18, majf=531, minf=21
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job8: (groupid=0, jobs=1): err= 0: pid=2393: Mon Oct 11 16:33:05 2021
write: IOPS=39.4k, BW=38.5MiB/s (40.3MB/s)(1024KiB/26msec); 0 zone resets
clat (nsec): min=50, max=3405.8k, avg=9250.34, stdev=122881.30
lat (nsec): min=85, max=3405.8k, avg=9331.42, stdev=122880.20
clat percentiles (nsec):
| 1.00th=[ 55], 5.00th=[ 59], 10.00th=[ 70],
| 20.00th=[ 85], 30.00th=[ 99], 40.00th=[ 117],
| 50.00th=[ 215], 60.00th=[ 1272], 70.00th=[ 1416],
| 80.00th=[ 1576], 90.00th=[ 4576], 95.00th=[ 5472],
| 99.00th=[ 34560], 99.50th=[ 399360], 99.90th=[1155072],
| 99.95th=[3391488], 99.99th=[3391488]
lat (nsec) : 100=30.66%, 250=20.41%, 500=4.69%, 750=0.49%
lat (usec) : 2=27.73%, 4=3.81%, 10=9.47%, 20=1.46%, 50=0.49%
lat (usec) : 250=0.10%, 500=0.20%, 750=0.10%, 1000=0.10%
lat (msec) : 2=0.20%, 4=0.10%
cpu : usr=8.00%, sys=0.00%, ctx=18, majf=450, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2394: Mon Oct 11 16:33:05 2021
write: IOPS=7757, BW=30.1MiB/s (31.5MB/s)(1016KiB/33msec); 0 zone resets
clat (usec): min=23, max=3399, avg=49.62, stdev=216.57
lat (usec): min=23, max=3399, avg=49.70, stdev=216.57
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 25], 20.00th=[ 26],
| 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 30], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 35], 90.00th=[ 37], 95.00th=[ 49],
| 99.00th=[ 388], 99.50th=[ 578], 99.90th=[ 3392], 99.95th=[ 3392],
| 99.99th=[ 3392]
lat (usec) : 50=94.92%, 100=1.56%, 250=1.17%, 500=0.78%, 750=0.39%
lat (msec) : 4=0.39%
cpu : usr=0.00%, sys=25.00%, ctx=20, majf=256, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=2, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 0: pid=2395: Mon Oct 11 16:33:05 2021
write: IOPS=14.2k, BW=55.6MiB/s (58.3MB/s)(1024KiB/18msec); 0 zone resets
clat (usec): min=24, max=2564, avg=60.04, stdev=225.97
lat (usec): min=24, max=2564, avg=60.14, stdev=225.97
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
| 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 34],
| 70.00th=[ 35], 80.00th=[ 39], 90.00th=[ 44], 95.00th=[ 50],
| 99.00th=[ 742], 99.50th=[ 2409], 99.90th=[ 2573], 99.95th=[ 2573],
| 99.99th=[ 2573]
lat (usec) : 50=94.92%, 100=2.34%, 250=0.39%, 500=0.78%, 750=0.78%
lat (msec) : 4=0.78%
cpu : usr=0.00%, sys=47.06%, ctx=27, majf=261, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=0, first_error=0/<Success>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2396: Mon Oct 11 16:33:05 2021
write: IOPS=12.2k, BW=47.4MiB/s (49.7MB/s)(1020KiB/21msec); 0 zone resets
clat (usec): min=25, max=4368, avg=70.05, stdev=295.82
lat (usec): min=25, max=4368, avg=70.18, stdev=295.83
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 36], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 74], 95.00th=[ 88],
| 99.00th=[ 857], 99.50th=[ 1680], 99.90th=[ 4359], 99.95th=[ 4359],
| 99.99th=[ 4359]
lat (usec) : 50=82.81%, 100=14.45%, 250=0.78%, 750=0.39%, 1000=0.39%
lat (msec) : 2=0.39%, 10=0.39%
cpu : usr=15.00%, sys=35.00%, ctx=18, majf=264, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=1, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2397: Mon Oct 11 16:33:05 2021
write: IOPS=7529, BW=29.1MiB/s (30.5MB/s)(1012KiB/34msec); 0 zone resets
clat (usec): min=23, max=3350, avg=66.34, stdev=234.18
lat (usec): min=23, max=3350, avg=66.48, stdev=234.18
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 38],
| 70.00th=[ 42], 80.00th=[ 47], 90.00th=[ 53], 95.00th=[ 78],
| 99.00th=[ 1029], 99.50th=[ 1172], 99.90th=[ 3359], 99.95th=[ 3359],
| 99.99th=[ 3359]
lat (usec) : 50=86.72%, 100=7.81%, 250=1.17%, 500=1.56%, 750=0.39%
lat (msec) : 2=0.78%, 4=0.39%
cpu : usr=0.00%, sys=30.30%, ctx=19, majf=256, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=3, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
job9: (groupid=0, jobs=1): err= 5 (file:engines/mmap.c:257, func=msync, error=Input/output error): pid=2398: Mon Oct 11 16:33:05 2021
write: IOPS=5953, BW=23.0MiB/s (24.1MB/s)(1012KiB/43msec); 0 zone resets
clat (usec): min=25, max=4743, avg=85.07, stdev=312.28
lat (usec): min=25, max=4743, avg=85.33, stdev=312.27
clat percentiles (usec):
| 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 35],
| 30.00th=[ 37], 40.00th=[ 44], 50.00th=[ 49], 60.00th=[ 55],
| 70.00th=[ 60], 80.00th=[ 65], 90.00th=[ 80], 95.00th=[ 141],
| 99.00th=[ 881], 99.50th=[ 1090], 99.90th=[ 4752], 99.95th=[ 4752],
| 99.99th=[ 4752]
lat (usec) : 50=50.78%, 100=42.19%, 250=2.73%, 500=1.17%, 750=0.78%
lat (usec) : 1000=0.39%
lat (msec) : 2=0.39%, 10=0.39%
cpu : usr=7.14%, sys=21.43%, ctx=34, majf=257, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
errors : total=3, first_error=5/<Input/output error>
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=333MiB/s (350MB/s), 66.7MiB/s-90.9MiB/s (69.9MB/s-95.3MB/s), io=5120KiB (5243kB), run=11-15msec
WRITE: bw=375MiB/s (393MB/s), 12.5MiB/s-55.6MiB/s (13.1MB/s-58.3MB/s), io=30.0MiB (31.4MB), run=18-80msec
Disk stats (read/write):
zram1: ios=549/3114, merge=0/0, ticks=0/40, in_queue=40, util=29.07%
fio: file hash not empty on exit
[-- Attachment #3: 095.out.bad --]
[-- Type: text/plain, Size: 2494 bytes --]
QA output created by 095
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=45056, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=133120, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=51200, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=342016, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=275456, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=416768, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=233472, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=247808, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=177152, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: read offset=30720, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=92160, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=158720, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=513024, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=33792, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=449536, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=43008, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=114688, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=241664, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=115712, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file2: Invalid argument: write offset=370688, buflen=1024
fio: io_u_queue.h:37: io_u_qpush: Assertion '0' failed.
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=222208, buflen=1024
fio: io_u error on file /tmp/mnt/scratch/file1: Invalid argument: write offset=43008, buflen=1024
fio: pid=2364, got signal=6
fio: pid=2367, got signal=6
fio: pid=2368, got signal=6
fio: file hash not empty on exit
Silence is golden
next prev parent reply other threads:[~2021-10-11 15:45 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-01 17:11 generic/095 failing in ext4 and xfs Luis Henriques
2021-10-01 20:07 ` Theodore Ts'o
2021-10-01 20:46 ` Jens Axboe
2021-10-01 21:59 ` Theodore Ts'o
2021-10-02 10:16 ` Luis Henriques
2021-10-02 14:59 ` Jens Axboe
2021-10-04 10:08 ` Luis Henriques
2021-10-04 10:15 ` Luis Henriques
2021-10-04 12:17 ` Luis Henriques
2021-10-04 16:18 ` Theodore Ts'o
2021-10-06 13:39 ` Luis Henriques
2021-10-11 10:27 ` [PATCH] fio: make sure io_u->file isn't NULL before using it Luís Henriques
2021-10-11 12:58 ` Jens Axboe
2021-10-11 13:44 ` Luís Henriques
2021-10-11 15:15 ` Jens Axboe
2021-10-11 15:45 ` Luís Henriques [this message]
2021-10-10 8:31 ` generic/095 failing in ext4 and xfs Zorro Lang
2021-10-11 9:09 ` Luís Henriques
2021-10-11 9:31 ` Ming Lei
2021-10-11 10:16 ` Luís Henriques
2021-10-11 11:13 ` Ming Lei
2021-10-11 13:41 ` Luís Henriques
2021-10-11 12:44 ` Theodore Ts'o
2021-10-11 13:41 ` Luís Henriques
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YWRcA/2AIb3pMOkG@suse.de \
--to=lhenriques@suse.de \
--cc=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.