* random_generator=lfsr overhead with more disks?
@ 2018-03-13 19:23 Michael Green
2018-03-16 18:45 ` Jeff Furlong
0 siblings, 1 reply; 11+ messages in thread
From: Michael Green @ 2018-03-13 19:23 UTC (permalink / raw)
To: fio@vger.kernel.org
Hello collective wisdom,
First post here, apologize if this has been asked before and would appreciate appropriate pointers or tips how to search the archives more effectively.
I work for E8 Storage and been testing our product with FIO for quite some time now. I’ve encountered this interesting affect that random_generator=lfsr seems to have on the results. Would appreciate your thoughts.
Here are two jobs. Both are 4k 100 RR, attacking 1 volume. The first job is with lfsr, the second without.
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3774MB/0KB/0KB /s] [966K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=25976: Tue Mar 13 20:56:48 2018
read : io=447350MB, bw=3727.1MB/s, iops=954339, runt=120001msec
slat (usec): min=1, max=540, avg= 3.81, stdev= 4.14
clat (usec): min=46, max=4793, avg=330.67, stdev=76.39
lat (usec): min=50, max=4838, avg=334.65, stdev=76.47
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3729MB/0KB/0KB /s] [955K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=18605: Tue Mar 13 20:17:27 2018
read : io=447243MB, bw=3726.2MB/s, iops=954110, runt=120001msec
slat (usec): min=1, max=629, avg= 3.73, stdev= 4.08
clat (usec): min=45, max=4327, avg=330.66, stdev=77.61
lat (usec): min=47, max=4329, avg=334.55, stdev=77.65
Now the same with 16 volumes. Notice how latency went up and IOPS down with lfsr generator:
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120
--filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=0): [r(8)] [12.7% done] [3641MB/0KB/0KB /s] [932K/0/0 iops] [eta 13m:50s] s]
PT7: (groupid=0, jobs=8): err= 0: pid=27644: Tue Mar 13 21:04:24 2018
read : io=443136MB, bw=3692.8MB/s, iops=945349, runt=120001msec
slat (usec): min=1, max=1409, avg= 4.41, stdev= 6.90
clat (usec): min=24, max=5222, avg=332.95, stdev=113.74
lat (usec): min=38, max=5250, avg=337.53, stdev=113.11
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120
--filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=128): [r(8)] [100.0% done] [3199MB/0KB/0KB /s] [819K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=29711: Tue Mar 13 21:16:13 2018
read : io=382920MB, bw=3190.1MB/s, iops=816888, runt=120001msec
slat (usec): min=1, max=1069, avg= 3.77, stdev= 4.33
clat (usec): min=16, max=3593, avg=387.12, stdev=189.09
lat (usec): min=36, max=3613, avg=391.06, stdev=189.05
Between 1 and 16, I’ve tried 4, 8 and 12 volumes. The performance gradually worsens with LFSR as the number of volumes increases, but stays flat without LFSR.
--
Thanks,
Michael Green
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: random_generator=lfsr overhead with more disks?
2018-03-13 19:23 random_generator=lfsr overhead with more disks? Michael Green
@ 2018-03-16 18:45 ` Jeff Furlong
2018-03-16 21:16 ` Michael Green
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Jeff Furlong @ 2018-03-16 18:45 UTC (permalink / raw)
To: Michael Green, fio@vger.kernel.org
A few comments:
-How does the cpu utilization look compared between the lfsr and default tausworthe generators? Is the cpu util actually higher on lfsr? If it is near saturated on 16 devices, then naturally throughput would decrease.
-It seems you are intentionally trying to write each block once. So not using the norandommap parameter.
-Does a runtime of 2 minutes collect enough data points to assess the condition? What happens if you extend the runtime?
Regards,
Jeff
-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Michael Green
Sent: Tuesday, March 13, 2018 12:24 PM
To: fio@vger.kernel.org
Subject: random_generator=lfsr overhead with more disks?
Hello collective wisdom,
First post here, apologize if this has been asked before and would appreciate appropriate pointers or tips how to search the archives more effectively.
I work for E8 Storage and been testing our product with FIO for quite some time now. I’ve encountered this interesting affect that random_generator=lfsr seems to have on the results. Would appreciate your thoughts.
Here are two jobs. Both are 4k 100 RR, attacking 1 volume. The first job is with lfsr, the second without.
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40 ...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3774MB/0KB/0KB /s] [966K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=25976: Tue Mar 13 20:56:48 2018
read : io=447350MB, bw=3727.1MB/s, iops=954339, runt=120001msec
slat (usec): min=1, max=540, avg= 3.81, stdev= 4.14
clat (usec): min=46, max=4793, avg=330.67, stdev=76.39
lat (usec): min=50, max=4838, avg=334.65, stdev=76.47
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40 ...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3729MB/0KB/0KB /s] [955K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=18605: Tue Mar 13 20:17:27 2018
read : io=447243MB, bw=3726.2MB/s, iops=954110, runt=120001msec
slat (usec): min=1, max=629, avg= 3.73, stdev= 4.08
clat (usec): min=45, max=4327, avg=330.66, stdev=77.61
lat (usec): min=47, max=4329, avg=334.55, stdev=77.65
Now the same with 16 volumes. Notice how latency went up and IOPS down with lfsr generator:
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40 ...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=0): [r(8)] [12.7% done] [3641MB/0KB/0KB /s] [932K/0/0 iops] [eta 13m:50s] s]
PT7: (groupid=0, jobs=8): err= 0: pid=27644: Tue Mar 13 21:04:24 2018
read : io=443136MB, bw=3692.8MB/s, iops=945349, runt=120001msec
slat (usec): min=1, max=1409, avg= 4.41, stdev= 6.90
clat (usec): min=24, max=5222, avg=332.95, stdev=113.74
lat (usec): min=38, max=5250, avg=337.53, stdev=113.11
[root@sm28 csv]# /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40 ...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=128): [r(8)] [100.0% done] [3199MB/0KB/0KB /s] [819K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=29711: Tue Mar 13 21:16:13 2018
read : io=382920MB, bw=3190.1MB/s, iops=816888, runt=120001msec
slat (usec): min=1, max=1069, avg= 3.77, stdev= 4.33
clat (usec): min=16, max=3593, avg=387.12, stdev=189.09
lat (usec): min=36, max=3613, avg=391.06, stdev=189.05
Between 1 and 16, I’ve tried 4, 8 and 12 volumes. The performance gradually worsens with LFSR as the number of volumes increases, but stays flat without LFSR.
--
Thanks,
Michael Green--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-16 18:45 ` Jeff Furlong
2018-03-16 21:16 ` Michael Green
@ 2018-03-16 21:16 ` Michael Green
2018-03-21 14:34 ` Jens Axboe
2 siblings, 0 replies; 11+ messages in thread
From: Michael Green @ 2018-03-16 21:16 UTC (permalink / raw)
To: Jeff Furlong, fio@vger.kernel.org
> -How does the cpu utilization look compared between the lfsr and default tausworthe generators? Is the cpu util actually higher on lfsr?
It is actually less when using LFSR with 16 volumes:
Without LFSR (CPU utilization is 5th column):
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
PPID PID PRI PSR %CPU STAT WCHAN COMMAND
2239 24684 19 0 0.7 Sl+ hrtimer_nanosl fio
2239 24684 19 0 0.2 Sl+ futex_wait_que fio
2239 24684 19 3 60.4 Rl+ - fio
2239 24684 19 6 60.5 Sl+ - fio
2239 24684 19 1 61.2 Rl+ - fio
2239 24684 19 23 62.5 Sl+ - fio
2239 24684 19 7 60.6 Rl+ - fio
2239 24684 19 18 61.0 Rl+ - fio
2239 24684 19 16 61.0 Rl+ - fio
2239 24684 19 4 62.0 Rl+ - fio
With LFSR:
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 25589 19 20 2.7 Sl+ hrtimer_nanosl fio
2239 25589 19 25 0.3 Sl+ futex_wait_que fio
2239 25589 19 16 80.1 Rl+ - fio
2239 25589 19 22 78.0 Rl+ - fio
2239 25589 19 5 82.6 Rl+ - fio
2239 25589 19 4 78.0 Rl+ - fio
2239 25589 19 18 82.7 Rl+ - fio
2239 25589 19 17 80.7 Sl+ 14051005663345 fio
2239 25589 19 23 76.6 Sl+ - fio
2239 25589 19 0 77.6 Rl+ - fio
But almost the same with 1 volume:
Without LFSR
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 27559 19 7 4.0 Sl+ hrtimer_nanosl fio
2239 27559 19 2 0.0 Sl+ futex_wait_que fio
2239 27559 19 18 79.4 Rl+ - fio
2239 27559 19 17 79.0 Rl+ - fio
2239 27559 19 6 78.6 Rl+ - fio
2239 27559 19 5 77.6 Rl+ - fio
2239 27559 19 23 82.6 Rl+ - fio
2239 27559 19 3 77.4 Rl+ - fio
2239 27559 19 4 77.1 Sl+ - fio
2239 27559 19 16 75.9 Rl+ - fio
With LFSR:
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 27174 19 20 2.0 Sl+ hrtimer_nanosl fio
2239 27174 19 25 0.0 Sl+ futex_wait_que fio
2239 27174 19 16 80.0 Rl+ - fio
2239 27174 19 6 78.4 Rl+ - fio
2239 27174 19 3 77.5 Rl+ - fio
2239 27174 19 5 76.4 Sl+ - fio
2239 27174 19 19 77.1 Rl+ - fio
2239 27174 19 4 76.4 Rl+ - fio
2239 27174 19 7 77.0 Rl+ - fio
2239 27174 19 1 75.8 Rl+ - fio
> -Does a runtime of 2 minutes collect enough data points to assess the condition? What happens if you extend the runtime?
Yes, it does. Shortening the test to 20 sec or extending to 10 min, doesn’t change a thing.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-16 18:45 ` Jeff Furlong
@ 2018-03-16 21:16 ` Michael Green
[not found] ` <BN3PR0401MB145715070B7B9F0C71A902318FD70@BN3PR0401MB1457.namprd04.prod.outlook.com>
2018-03-16 21:16 ` Michael Green
2018-03-21 14:34 ` Jens Axboe
2 siblings, 1 reply; 11+ messages in thread
From: Michael Green @ 2018-03-16 21:16 UTC (permalink / raw)
To: Jeff Furlong, fio@vger.kernel.org
> -How does the cpu utilization look compared between the lfsr and default tausworthe generators? Is the cpu util actually higher on lfsr?
It is actually less when using LFSR with 16 volumes:
Without LFSR (CPU utilization is 5th column):
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
PPID PID PRI PSR %CPU STAT WCHAN COMMAND
2239 24684 19 0 0.7 Sl+ hrtimer_nanosl fio
2239 24684 19 0 0.2 Sl+ futex_wait_que fio
2239 24684 19 3 60.4 Rl+ - fio
2239 24684 19 6 60.5 Sl+ - fio
2239 24684 19 1 61.2 Rl+ - fio
2239 24684 19 23 62.5 Sl+ - fio
2239 24684 19 7 60.6 Rl+ - fio
2239 24684 19 18 61.0 Rl+ - fio
2239 24684 19 16 61.0 Rl+ - fio
2239 24684 19 4 62.0 Rl+ - fio
With LFSR:
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 25589 19 20 2.7 Sl+ hrtimer_nanosl fio
2239 25589 19 25 0.3 Sl+ futex_wait_que fio
2239 25589 19 16 80.1 Rl+ - fio
2239 25589 19 22 78.0 Rl+ - fio
2239 25589 19 5 82.6 Rl+ - fio
2239 25589 19 4 78.0 Rl+ - fio
2239 25589 19 18 82.7 Rl+ - fio
2239 25589 19 17 80.7 Sl+ 14051005663345 fio
2239 25589 19 23 76.6 Sl+ - fio
2239 25589 19 0 77.6 Rl+ - fio
But almost the same with 1 volume:
Without LFSR
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 27559 19 7 4.0 Sl+ hrtimer_nanosl fio
2239 27559 19 2 0.0 Sl+ futex_wait_que fio
2239 27559 19 18 79.4 Rl+ - fio
2239 27559 19 17 79.0 Rl+ - fio
2239 27559 19 6 78.6 Rl+ - fio
2239 27559 19 5 77.6 Rl+ - fio
2239 27559 19 23 82.6 Rl+ - fio
2239 27559 19 3 77.4 Rl+ - fio
2239 27559 19 4 77.1 Sl+ - fio
2239 27559 19 16 75.9 Rl+ - fio
With LFSR:
[root@sm28 ~]# ps -eLo ppid,pid,pri,psr,pcpu,stat,wchan:14,comm | grep fio
2239 27174 19 20 2.0 Sl+ hrtimer_nanosl fio
2239 27174 19 25 0.0 Sl+ futex_wait_que fio
2239 27174 19 16 80.0 Rl+ - fio
2239 27174 19 6 78.4 Rl+ - fio
2239 27174 19 3 77.5 Rl+ - fio
2239 27174 19 5 76.4 Sl+ - fio
2239 27174 19 19 77.1 Rl+ - fio
2239 27174 19 4 76.4 Rl+ - fio
2239 27174 19 7 77.0 Rl+ - fio
2239 27174 19 1 75.8 Rl+ - fio
> -Does a runtime of 2 minutes collect enough data points to assess the condition? What happens if you extend the runtime?
Yes, it does. Shortening the test to 20 sec or extending to 10 min, doesn’t change a thing.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
[not found] ` <SN6PR04MB4064D865EBEB79DE2219218D8FD70@SN6PR04MB4064.namprd04.prod.outlook.com>
@ 2018-03-16 22:32 ` Michael Green
2018-03-17 5:42 ` Sitsofe Wheeler
0 siblings, 1 reply; 11+ messages in thread
From: Michael Green @ 2018-03-16 22:32 UTC (permalink / raw)
To: Jeff Furlong, fio
Possibly. Who can have a look at the code?
> On Mar 16, 2018, at 6:30 PM, Jeff Furlong <jeff.furlong@wdc.com> wrote:
>
> OK, 61% cpu with lfsr and 82% cpu without lfsr. But the throughput is propositionally higher without lfsr, so perhaps that's why cpu util is higher.
>
> I'm wondering if the lfsr code is single threaded in nature or stuck waiting on a mutex, thereby slowing down the throughput and hence the lower latency.
>
> Regards,
> Jeff
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-16 22:32 ` Michael Green
@ 2018-03-17 5:42 ` Sitsofe Wheeler
2018-03-29 7:24 ` Michael Green
0 siblings, 1 reply; 11+ messages in thread
From: Sitsofe Wheeler @ 2018-03-17 5:42 UTC (permalink / raw)
To: Michael Green; +Cc: Jeff Furlong, fio
On 16 March 2018 at 22:32, Michael Green <mishagreen@gmail.com> wrote:
> Possibly. Who can have a look at the code?
>
>> On Mar 16, 2018, at 6:30 PM, Jeff Furlong <jeff.furlong@wdc.com> wrote:
>>
>> OK, 61% cpu with lfsr and 82% cpu without lfsr. But the throughput is propositionally higher without lfsr, so perhaps that's why cpu util is higher.
>>
>> I'm wondering if the lfsr code is single threaded in nature or stuck waiting on a mutex, thereby slowing down the throughput and hence the lower latency.
Sounds unlikely - there are no locks at all in
https://github.com/axboe/fio/blob/master/lib/lfsr.c and each file gets
its own lsfr generator.
> fio-2.2.11
> Starting 8 threads
Could you repeat the problem on a recent version of fio (see
https://github.com/axboe/fio/releases for what we're up to)? If it
happens there we could do with the output from Linux's perf
for each of the runs to see what might be sucking the time. It would also
help if you strip the line you are using down to the bare minimum that
still shows the problem (e.g. if you can remove numa, lock it to CPUs
make it happen on a pure randread workload etc).
I did two runs using the null ioengine on a four CPU system:
fio --thread=1 --direct=1 --group_reporting=1 --ioengine=null
--name=PT7 --rw=randrw --rwmixread=100 --iodepth=1 --numjobs=4
--bs=4096 --size=450GiB --runtime=10
PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=null, iodepth=40
...
fio-3.5-57-g5b2f-dirty
[...]
Run status group 0 (all jobs):
READ: bw=545MiB/s (572MB/s), 545MiB/s-545MiB/s (572MB/s-572MB/s),
io=5455MiB (5720MB), run=10001-10001msec
fio --thread=1 --direct=1 --group_reporting=1 --ioengine=null
--name=PT7 --rw=randrw --rwmixread=100 --iodepth=1 --numjobs=4
--bs=4096 --size=450GiB --runtime=10 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=null, iodepth=40
...
fio-3.5-57-g5b2f-dirty
[...]
Run status group 0 (all jobs):
READ: bw=637MiB/s (668MB/s), 637MiB/s-637MiB/s (668MB/s-668MB/s),
io=6374MiB (6684MB), run=10001-10001msec
Here the bandwidth was higher with --random_generator=lfsr suggesting
it had lower overhead in my case.
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-16 18:45 ` Jeff Furlong
2018-03-16 21:16 ` Michael Green
2018-03-16 21:16 ` Michael Green
@ 2018-03-21 14:34 ` Jens Axboe
2018-03-29 7:28 ` Michael Green
2 siblings, 1 reply; 11+ messages in thread
From: Jens Axboe @ 2018-03-21 14:34 UTC (permalink / raw)
To: Jeff Furlong, Michael Green, fio@vger.kernel.org
On 3/16/18 12:45 PM, Jeff Furlong wrote:
> A few comments:
>
> -How does the cpu utilization look compared between the lfsr and
> default tausworthe generators? Is the cpu util actually higher on
> lfsr? If it is near saturated on 16 devices, then naturally
> throughput would decrease.
The LFSR generator is constant overhead, and it should not degrade as
more jobs are using it. There's no shared state at all.
If I were to guess, then the identical offsets of the jobs is probably a
likely clue, since that is the only real difference between the two
random generators. Are these 16 volumes all completely separate?
Finally, you are having 8 jobs all accessing the same 16 devices. You'll
likely get more consistent performance if you split things up a bit by
having 4 jobs, each doing IO to just two of the disks. This shouldn't
really be impacted by the random generator, but your current setup is
prone to performance fluctuations.
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-17 5:42 ` Sitsofe Wheeler
@ 2018-03-29 7:24 ` Michael Green
2018-03-29 8:56 ` Sitsofe Wheeler
0 siblings, 1 reply; 11+ messages in thread
From: Michael Green @ 2018-03-29 7:24 UTC (permalink / raw)
To: fio; +Cc: Jeff Furlong, Sitsofe Wheeler
Sorry for the delay here. Please see below inline.
> On Mar 17, 2018, at 1:42 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> Could you repeat the problem on a recent version of fio (see
> https://github.com/axboe/fio/releases for what we're up to)?
Sure. Here are the results with latest fio.
With LFSR
[root@sm28 fio-master]# fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=40
...
fio-3.5-80-gb348
Starting 8 threads
Jobs: 8 (f=128): [r(8)][100.0%][r=3532MiB/s,w=0KiB/s][r=904k,w=0 IOPS][eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=28376: Thu Mar 29 07:35:44 2018
read: IOPS=895k, BW=3496MiB/s (3666MB/s)(410GiB/120002msec)
slat (nsec): min=1619, max=964358, avg=3682.85, stdev=4806.21
clat (usec): min=25, max=13002, avg=353.13, stdev=192.55
lat (usec): min=35, max=13007, avg=356.96, stdev=192.42
clat percentiles (usec):
| 1.00th=[ 91], 5.00th=[ 141], 10.00th=[ 169], 20.00th=[ 206],
| 30.00th=[ 239], 40.00th=[ 269], 50.00th=[ 306], 60.00th=[ 347],
| 70.00th=[ 400], 80.00th=[ 474], 90.00th=[ 603], 95.00th=[ 742],
| 99.00th=[ 1020], 99.50th=[ 1123], 99.90th=[ 1352], 99.95th=[ 1434],
| 99.99th=[ 1647]
bw ( KiB/s): min=373440, max=481952, per=12.50%, avg=447480.00, stdev=12566.36, samples=1918
iops : min=93360, max=120488, avg=111869.99, stdev=3141.58, samples=1918
lat (usec) : 50=0.02%, 100=1.45%, 250=32.29%, 500=49.13%, 750=12.35%
lat (usec) : 1000=3.60%
lat (msec) : 2=1.15%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=17.66%, sys=49.66%, ctx=9942380, majf=0, minf=4898
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=107404316,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=40
Run status group 0 (all jobs):
READ: bw=3496MiB/s (3666MB/s), 3496MiB/s-3496MiB/s (3666MB/s-3666MB/s), io=410GiB (440GB), run=120002-120002msec
Disk stats (read/write):
e8b0: ios=6710489/0, merge=0/0, ticks=1818366/0, in_queue=1822176, util=99.74%
e8b1: ios=6710487/0, merge=0/0, ticks=1993186/0, in_queue=1995781, util=99.75%
e8b2: ios=6710490/0, merge=0/0, ticks=2076054/0, in_queue=2080427, util=99.76%
e8b3: ios=6710491/0, merge=0/0, ticks=2101963/0, in_queue=2107744, util=99.80%
e8b4: ios=6710493/0, merge=0/0, ticks=2167111/0, in_queue=2169552, util=99.79%
e8b5: ios=6710496/0, merge=0/0, ticks=2149837/0, in_queue=2153109, util=99.85%
e8b6: ios=6710497/0, merge=0/0, ticks=1966688/0, in_queue=1970940, util=99.85%
e8b7: ios=6710496/0, merge=0/0, ticks=1984307/0, in_queue=1989317, util=99.87%
e8b8: ios=6710497/0, merge=0/0, ticks=1985081/0, in_queue=1989662, util=99.88%
e8b9: ios=6710498/0, merge=0/0, ticks=1995815/0, in_queue=2000669, util=99.92%
e8b10: ios=6710498/0, merge=0/0, ticks=2005176/0, in_queue=2009368, util=99.94%
e8b11: ios=6710498/0, merge=0/0, ticks=2022758/0, in_queue=2027682, util=99.99%
e8b12: ios=6710499/0, merge=0/0, ticks=1996747/0, in_queue=2001118, util=100.00%
e8b13: ios=6710502/0, merge=0/0, ticks=2034211/0, in_queue=2039490, util=100.00%
e8b14: ios=6710502/0, merge=0/0, ticks=2035394/0, in_queue=2040469, util=100.00%
e8b15: ios=6710505/0, merge=0/0, ticks=2010598/0, in_queue=2017014, util=100.00%
Without LFSR
[root@sm28 fio-master]# fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=40
...
fio-3.5-80-gb348
Starting 8 threads
Jobs: 8 (f=128): [r(8)][100.0%][r=3729MiB/s,w=0KiB/s][r=955k,w=0 IOPS][eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=28564: Thu Mar 29 07:40:09 2018
read: IOPS=943k, BW=3684MiB/s (3863MB/s)(432GiB/120007msec)
slat (nsec): min=1656, max=1601.7k, avg=4342.55, stdev=6792.60
clat (usec): min=31, max=12646, avg=333.84, stdev=102.52
lat (usec): min=38, max=12648, avg=338.34, stdev=101.86
clat percentiles (usec):
| 1.00th=[ 120], 5.00th=[ 167], 10.00th=[ 204], 20.00th=[ 249],
| 30.00th=[ 281], 40.00th=[ 310], 50.00th=[ 334], 60.00th=[ 363],
| 70.00th=[ 392], 80.00th=[ 420], 90.00th=[ 457], 95.00th=[ 482],
| 99.00th=[ 545], 99.50th=[ 594], 99.90th=[ 865], 99.95th=[ 955],
| 99.99th=[ 1254]
bw ( KiB/s): min=390520, max=518192, per=12.50%, avg=471560.73, stdev=13752.55, samples=1914
iops : min=97630, max=129548, avg=117890.14, stdev=3438.13, samples=1914
lat (usec) : 50=0.01%, 100=0.37%, 250=20.22%, 500=76.31%, 750=2.87%
lat (usec) : 1000=0.20%
lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=23.47%, sys=59.78%, ctx=10930907, majf=0, minf=299365
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=113187098,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=40
Run status group 0 (all jobs):
READ: bw=3684MiB/s (3863MB/s), 3684MiB/s-3684MiB/s (3863MB/s-3863MB/s), io=432GiB (464GB), run=120007-120007msec
Disk stats (read/write):
e8b0: ios=7070562/0, merge=0/0, ticks=1862741/0, in_queue=1868144, util=99.78%
e8b1: ios=7070570/0, merge=0/0, ticks=1996123/0, in_queue=1999170, util=99.79%
e8b2: ios=7070580/0, merge=0/0, ticks=2019351/0, in_queue=2024920, util=99.78%
e8b3: ios=7070581/0, merge=0/0, ticks=2018430/0, in_queue=2024167, util=99.80%
e8b4: ios=7070585/0, merge=0/0, ticks=2069985/0, in_queue=2072530, util=99.78%
e8b5: ios=7070586/0, merge=0/0, ticks=2032085/0, in_queue=2035097, util=99.81%
e8b6: ios=7070586/0, merge=0/0, ticks=1838167/0, in_queue=1842676, util=99.79%
e8b7: ios=7070589/0, merge=0/0, ticks=1838162/0, in_queue=1843175, util=99.83%
e8b8: ios=7070587/0, merge=0/0, ticks=1837259/0, in_queue=1842283, util=99.86%
e8b9: ios=7070595/0, merge=0/0, ticks=1836983/0, in_queue=1841398, util=99.89%
e8b10: ios=7070601/0, merge=0/0, ticks=1835967/0, in_queue=1840679, util=99.91%
e8b11: ios=7070605/0, merge=0/0, ticks=1835374/0, in_queue=1840121, util=99.94%
e8b12: ios=7070609/0, merge=0/0, ticks=1834964/0, in_queue=1839546, util=99.98%
e8b13: ios=7070609/0, merge=0/0, ticks=1835191/0, in_queue=1839604, util=100.00%
e8b14: ios=7070608/0, merge=0/0, ticks=1835130/0, in_queue=1840084, util=100.00%
e8b15: ios=7070611/0, merge=0/0, ticks=1835606/0, in_queue=1842956, util=100.00%
Initially, I thought that there is a clear improvement here as the latency gap is smaller now. But then I decided to follow your advice of stripping the command down to the bare minimum.
> It would also
> help if you strip the line you are using down to the bare minimum that
> still shows the problem (e.g. if you can remove numa, lock it to CPUs
> make it happen on a pure randread workload etc).
Now when it comes to removing the flags, I didn’t want to remove the --numa flag because in E8 architecture that may adversely affect latency standard deviation. Reason being that FIO may occasionally get scheduled to run on the same core where where e8 driver is running. So the two flags that I could get rid of were --iomem_align and --size. After experimenting a bit, I eventually set out to run a series of tests with every permutation of the two flags, including a test without them. For every permutation, the test was repeated 3 times and latencies recorded. Nothing else was running on either the storage controller or the host during the testing.
base command was
fio --name=global --thread=1 --direct=1 --group_reporting=1 --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e
8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --random_generator=lfsr
and
fio --name=global --thread=1 --direct=1 --group_reporting=1 --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e
8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio
The only difference is --random_generator=lfsr or not.
The iomem and size flags were iomem_align=4k and size=450GiB
Here are the latency numbers I’ve got:
LFSR
+ iomem + size = 348.07, 347.68, 347.59
nothing = 344.28, 344.84, 345.04
+ size=349.28, 348.37, 348.37
+ iomem = 346.81, 346.05, 344.74
NO LFSR
+ iomem + size = 344.03, 344.29, 345.47
nothing = 347.05, 346.00, 346.03
+ size = 345.43, 343.55, 343.98
+ iomem = 347.87, 347.02, 347.84
It appears that with both flags, LFSR is still behind NO LFSR even so minimally. When both flags are omitted the picture reverses.
Looking at the overall picture, I cannot identify clear winner here. It’s almost like the differences are within measurement error.
> If it
> happens there we could do with the output from Linux's perf
Not sure what “Linux’s perf” exactly means.
Michael
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-21 14:34 ` Jens Axboe
@ 2018-03-29 7:28 ` Michael Green
2018-03-29 8:50 ` Sitsofe Wheeler
0 siblings, 1 reply; 11+ messages in thread
From: Michael Green @ 2018-03-29 7:28 UTC (permalink / raw)
To: Jens Axboe; +Cc: Jeff Furlong, fio@vger.kernel.org
> On Mar 21, 2018, at 7:34 AM, Jens Axboe <axboe@kernel.dk> wrote:
>
> On 3/16/18 12:45 PM, Jeff Furlong wrote:
>> A few comments:
>>
>> -How does the cpu utilization look compared between the lfsr and
>> default tausworthe generators? Is the cpu util actually higher on
>> lfsr? If it is near saturated on 16 devices, then naturally
>> throughput would decrease.
>
> The LFSR generator is constant overhead, and it should not degrade as
> more jobs are using it. There's no shared state at all.
Actually it’s only one job.
>
> If I were to guess, then the identical offsets of the jobs is probably a
> likely clue, since that is the only real difference between the two
> random generators. Are these 16 volumes all completely separate?
>
The 16 volumes are coming from the same E8 storage controller. the volumes share the same RAID6 array and the data traffic is flowing through the same Ethernet infrastructure, 100Gb end to en.
From E8 host driver perspective, there is no overhead for more volumes - 1 or 1000 it’s all the same.
Michael
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-29 7:28 ` Michael Green
@ 2018-03-29 8:50 ` Sitsofe Wheeler
0 siblings, 0 replies; 11+ messages in thread
From: Sitsofe Wheeler @ 2018-03-29 8:50 UTC (permalink / raw)
To: Michael Green; +Cc: Jens Axboe, Jeff Furlong, fio@vger.kernel.org
On 29 March 2018 at 08:28, Michael Green <mishagreen@gmail.com> wrote:
>
>> On Mar 21, 2018, at 7:34 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 3/16/18 12:45 PM, Jeff Furlong wrote:
>>> A few comments:
>>>
>>> -How does the cpu utilization look compared between the lfsr and
>>> default tausworthe generators? Is the cpu util actually higher on
>>> lfsr? If it is near saturated on 16 devices, then naturally
>>> throughput would decrease.
>>
>> The LFSR generator is constant overhead, and it should not degrade as
>> more jobs are using it. There's no shared state at all.
>
> Actually it’s only one job.
In this case we mean "fio jobs" (rather than fio invocations). From
the line in one of your other emails:
fio --name=global --thread=1 --direct=1 --group_reporting=1 --name=PT7
--rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096
--runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15'
--ioengine=libaio --random_generator=lfsr
You have --numjobs=8
(http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-numjobs
) there so you are spawning 8 fio internal jobs which all work on
those same files simultaneously.
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: random_generator=lfsr overhead with more disks?
2018-03-29 7:24 ` Michael Green
@ 2018-03-29 8:56 ` Sitsofe Wheeler
0 siblings, 0 replies; 11+ messages in thread
From: Sitsofe Wheeler @ 2018-03-29 8:56 UTC (permalink / raw)
To: Michael Green; +Cc: fio, Jeff Furlong
On 29 March 2018 at 08:24, Michael Green <mishagreen@gmail.com> wrote:
> Sorry for the delay here. Please see below inline.
>
>> On Mar 17, 2018, at 1:42 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>
>> Could you repeat the problem on a recent version of fio (see
>> https://github.com/axboe/fio/releases for what we're up to)?
>
> Sure. Here are the results with latest fio.
>
> With LFSR
>
> [root@sm28 fio-master]# fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
> PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=40
> ...
> fio-3.5-80-gb348
> Starting 8 threads
> Jobs: 8 (f=128): [r(8)][100.0%][r=3532MiB/s,w=0KiB/s][r=904k,w=0 IOPS][eta 00m:00s]
> PT7: (groupid=0, jobs=8): err= 0: pid=28376: Thu Mar 29 07:35:44 2018
> read: IOPS=895k, BW=3496MiB/s (3666MB/s)(410GiB/120002msec)
> slat (nsec): min=1619, max=964358, avg=3682.85, stdev=4806.21
> clat (usec): min=25, max=13002, avg=353.13, stdev=192.55
> lat (usec): min=35, max=13007, avg=356.96, stdev=192.42
> clat percentiles (usec):
> | 1.00th=[ 91], 5.00th=[ 141], 10.00th=[ 169], 20.00th=[ 206],
> | 30.00th=[ 239], 40.00th=[ 269], 50.00th=[ 306], 60.00th=[ 347],
> | 70.00th=[ 400], 80.00th=[ 474], 90.00th=[ 603], 95.00th=[ 742],
> | 99.00th=[ 1020], 99.50th=[ 1123], 99.90th=[ 1352], 99.95th=[ 1434],
> | 99.99th=[ 1647]
> bw ( KiB/s): min=373440, max=481952, per=12.50%, avg=447480.00, stdev=12566.36, samples=1918
> iops : min=93360, max=120488, avg=111869.99, stdev=3141.58, samples=1918
> lat (usec) : 50=0.02%, 100=1.45%, 250=32.29%, 500=49.13%, 750=12.35%
> lat (usec) : 1000=3.60%
> lat (msec) : 2=1.15%, 4=0.01%, 10=0.01%, 20=0.01%
> cpu : usr=17.66%, sys=49.66%, ctx=9942380, majf=0, minf=4898
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
> issued rwts: total=107404316,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=40
>
> Run status group 0 (all jobs):
> READ: bw=3496MiB/s (3666MB/s), 3496MiB/s-3496MiB/s (3666MB/s-3666MB/s), io=410GiB (440GB), run=120002-120002msec
>
> Disk stats (read/write):
> e8b0: ios=6710489/0, merge=0/0, ticks=1818366/0, in_queue=1822176, util=99.74%
> e8b1: ios=6710487/0, merge=0/0, ticks=1993186/0, in_queue=1995781, util=99.75%
> e8b2: ios=6710490/0, merge=0/0, ticks=2076054/0, in_queue=2080427, util=99.76%
> e8b3: ios=6710491/0, merge=0/0, ticks=2101963/0, in_queue=2107744, util=99.80%
> e8b4: ios=6710493/0, merge=0/0, ticks=2167111/0, in_queue=2169552, util=99.79%
> e8b5: ios=6710496/0, merge=0/0, ticks=2149837/0, in_queue=2153109, util=99.85%
> e8b6: ios=6710497/0, merge=0/0, ticks=1966688/0, in_queue=1970940, util=99.85%
> e8b7: ios=6710496/0, merge=0/0, ticks=1984307/0, in_queue=1989317, util=99.87%
> e8b8: ios=6710497/0, merge=0/0, ticks=1985081/0, in_queue=1989662, util=99.88%
> e8b9: ios=6710498/0, merge=0/0, ticks=1995815/0, in_queue=2000669, util=99.92%
> e8b10: ios=6710498/0, merge=0/0, ticks=2005176/0, in_queue=2009368, util=99.94%
> e8b11: ios=6710498/0, merge=0/0, ticks=2022758/0, in_queue=2027682, util=99.99%
> e8b12: ios=6710499/0, merge=0/0, ticks=1996747/0, in_queue=2001118, util=100.00%
> e8b13: ios=6710502/0, merge=0/0, ticks=2034211/0, in_queue=2039490, util=100.00%
> e8b14: ios=6710502/0, merge=0/0, ticks=2035394/0, in_queue=2040469, util=100.00%
> e8b15: ios=6710505/0, merge=0/0, ticks=2010598/0, in_queue=2017014, util=100.00%
>
>
> Without LFSR
>
> [root@sm28 fio-master]# fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0
> PT7: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=40
> ...
> fio-3.5-80-gb348
> Starting 8 threads
> Jobs: 8 (f=128): [r(8)][100.0%][r=3729MiB/s,w=0KiB/s][r=955k,w=0 IOPS][eta 00m:00s]
> PT7: (groupid=0, jobs=8): err= 0: pid=28564: Thu Mar 29 07:40:09 2018
> read: IOPS=943k, BW=3684MiB/s (3863MB/s)(432GiB/120007msec)
> slat (nsec): min=1656, max=1601.7k, avg=4342.55, stdev=6792.60
> clat (usec): min=31, max=12646, avg=333.84, stdev=102.52
> lat (usec): min=38, max=12648, avg=338.34, stdev=101.86
> clat percentiles (usec):
> | 1.00th=[ 120], 5.00th=[ 167], 10.00th=[ 204], 20.00th=[ 249],
> | 30.00th=[ 281], 40.00th=[ 310], 50.00th=[ 334], 60.00th=[ 363],
> | 70.00th=[ 392], 80.00th=[ 420], 90.00th=[ 457], 95.00th=[ 482],
> | 99.00th=[ 545], 99.50th=[ 594], 99.90th=[ 865], 99.95th=[ 955],
> | 99.99th=[ 1254]
> bw ( KiB/s): min=390520, max=518192, per=12.50%, avg=471560.73, stdev=13752.55, samples=1914
> iops : min=97630, max=129548, avg=117890.14, stdev=3438.13, samples=1914
> lat (usec) : 50=0.01%, 100=0.37%, 250=20.22%, 500=76.31%, 750=2.87%
> lat (usec) : 1000=0.20%
> lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%
> cpu : usr=23.47%, sys=59.78%, ctx=10930907, majf=0, minf=299365
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
> issued rwts: total=113187098,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=40
>
> Run status group 0 (all jobs):
> READ: bw=3684MiB/s (3863MB/s), 3684MiB/s-3684MiB/s (3863MB/s-3863MB/s), io=432GiB (464GB), run=120007-120007msec
>
> Disk stats (read/write):
> e8b0: ios=7070562/0, merge=0/0, ticks=1862741/0, in_queue=1868144, util=99.78%
> e8b1: ios=7070570/0, merge=0/0, ticks=1996123/0, in_queue=1999170, util=99.79%
> e8b2: ios=7070580/0, merge=0/0, ticks=2019351/0, in_queue=2024920, util=99.78%
> e8b3: ios=7070581/0, merge=0/0, ticks=2018430/0, in_queue=2024167, util=99.80%
> e8b4: ios=7070585/0, merge=0/0, ticks=2069985/0, in_queue=2072530, util=99.78%
> e8b5: ios=7070586/0, merge=0/0, ticks=2032085/0, in_queue=2035097, util=99.81%
> e8b6: ios=7070586/0, merge=0/0, ticks=1838167/0, in_queue=1842676, util=99.79%
> e8b7: ios=7070589/0, merge=0/0, ticks=1838162/0, in_queue=1843175, util=99.83%
> e8b8: ios=7070587/0, merge=0/0, ticks=1837259/0, in_queue=1842283, util=99.86%
> e8b9: ios=7070595/0, merge=0/0, ticks=1836983/0, in_queue=1841398, util=99.89%
> e8b10: ios=7070601/0, merge=0/0, ticks=1835967/0, in_queue=1840679, util=99.91%
> e8b11: ios=7070605/0, merge=0/0, ticks=1835374/0, in_queue=1840121, util=99.94%
> e8b12: ios=7070609/0, merge=0/0, ticks=1834964/0, in_queue=1839546, util=99.98%
> e8b13: ios=7070609/0, merge=0/0, ticks=1835191/0, in_queue=1839604, util=100.00%
> e8b14: ios=7070608/0, merge=0/0, ticks=1835130/0, in_queue=1840084, util=100.00%
> e8b15: ios=7070611/0, merge=0/0, ticks=1835606/0, in_queue=1842956, util=100.00%
>
>
> Initially, I thought that there is a clear improvement here as the latency gap is smaller now. But then I decided to follow your advice of stripping the command down to the bare minimum.
>
>> It would also
>> help if you strip the line you are using down to the bare minimum that
>> still shows the problem (e.g. if you can remove numa, lock it to CPUs
>> make it happen on a pure randread workload etc).
>
>
> Now when it comes to removing the flags, I didn’t want to remove the --numa flag because in E8 architecture that may adversely affect latency standard deviation. Reason being that FIO may occasionally get scheduled to run on the same core where where e8 driver is running. So the two flags that I could get rid of were --iomem_align and --size. After experimenting a bit, I eventually set out to run a series of tests with every permutation of the two flags, including a test without them. For every permutation, the test was repeated 3 times and latencies recorded. Nothing else was running on either the storage controller or the host during the testing.
>
> base command was
> fio --name=global --thread=1 --direct=1 --group_reporting=1 --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e
> 8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --random_generator=lfsr
>
> and
>
> fio --name=global --thread=1 --direct=1 --group_reporting=1 --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --runtime=120 --filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e
> 8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio
>
> The only difference is --random_generator=lfsr or not.
> The iomem and size flags were iomem_align=4k and size=450GiB
>
> Here are the latency numbers I’ve got:
>
> LFSR
>
> + iomem + size = 348.07, 347.68, 347.59
> nothing = 344.28, 344.84, 345.04
> + size=349.28, 348.37, 348.37
> + iomem = 346.81, 346.05, 344.74
>
> NO LFSR
>
> + iomem + size = 344.03, 344.29, 345.47
> nothing = 347.05, 346.00, 346.03
> + size = 345.43, 343.55, 343.98
> + iomem = 347.87, 347.02, 347.84
>
> It appears that with both flags, LFSR is still behind NO LFSR even so minimally. When both flags are omitted the picture reverses.
> Looking at the overall picture, I cannot identify clear winner here. It’s almost like the differences are within measurement error.
If you start cutting down the number of disks being used does that
help or make things worse?
>> If it
>> happens there we could do with the output from Linux's perf
>
> Not sure what “Linux’s perf” exactly means.
The tool referred to over in http://www.brendangregg.com/perf.html and
https://perf.wiki.kernel.org/index.php/Main_Page . It's distributed
with the kernel but these days most distros should package it.
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-03-29 8:57 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-03-13 19:23 random_generator=lfsr overhead with more disks? Michael Green
2018-03-16 18:45 ` Jeff Furlong
2018-03-16 21:16 ` Michael Green
[not found] ` <BN3PR0401MB145715070B7B9F0C71A902318FD70@BN3PR0401MB1457.namprd04.prod.outlook.com>
[not found] ` <902A0A2A-8708-4A94-A09E-5D3C9BFB48FF@gmail.com>
[not found] ` <SN6PR04MB4064D865EBEB79DE2219218D8FD70@SN6PR04MB4064.namprd04.prod.outlook.com>
2018-03-16 22:32 ` Michael Green
2018-03-17 5:42 ` Sitsofe Wheeler
2018-03-29 7:24 ` Michael Green
2018-03-29 8:56 ` Sitsofe Wheeler
2018-03-16 21:16 ` Michael Green
2018-03-21 14:34 ` Jens Axboe
2018-03-29 7:28 ` Michael Green
2018-03-29 8:50 ` Sitsofe Wheeler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox