random_generator=lfsr overhead with more disks?

[Michael Green <mishagreen@gmail.com>]

Hello collective wisdom,

First post here, apologize if this has been asked before and would appreciate appropriate pointers or tips how to search the archives more effectively.

I work for E8 Storage and been testing our product with FIO for quite some time now. I’ve encountered this interesting affect that random_generator=lfsr seems to have on the results. Would appreciate your thoughts.

Here are two jobs. Both are 4k 100 RR, attacking 1 volume. The first job is with lfsr, the second without.

[root@sm28 csv]#  /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3774MB/0KB/0KB /s] [966K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=25976: Tue Mar 13 20:56:48 2018
  read : io=447350MB, bw=3727.1MB/s, iops=954339, runt=120001msec
    slat (usec): min=1, max=540, avg= 3.81, stdev= 4.14
    clat (usec): min=46, max=4793, avg=330.67, stdev=76.39
     lat (usec): min=50, max=4838, avg=334.65, stdev=76.47



[root@sm28 csv]#  /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120 --filename='/dev/e8b0' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=8): [r(8)] [100.0% done] [3729MB/0KB/0KB /s] [955K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=18605: Tue Mar 13 20:17:27 2018
  read : io=447243MB, bw=3726.2MB/s, iops=954110, runt=120001msec
    slat (usec): min=1, max=629, avg= 3.73, stdev= 4.08
    clat (usec): min=45, max=4327, avg=330.66, stdev=77.61
     lat (usec): min=47, max=4329, avg=334.55, stdev=77.65


Now the same with 16 volumes. Notice how latency went up and IOPS down with lfsr generator:

[root@sm28 csv]#  /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120
--filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=0): [r(8)] [12.7% done] [3641MB/0KB/0KB /s] [932K/0/0 iops] [eta 13m:50s] s]
PT7: (groupid=0, jobs=8): err= 0: pid=27644: Tue Mar 13 21:04:24 2018
  read : io=443136MB, bw=3692.8MB/s, iops=945349, runt=120001msec
    slat (usec): min=1, max=1409, avg= 4.41, stdev= 6.90
    clat (usec): min=24, max=5222, avg=332.95, stdev=113.74
     lat (usec): min=38, max=5250, avg=337.53, stdev=113.11

[root@sm28 csv]#  /opt/E8/bin/fio --name=global --thread=1 --direct=1 --group_reporting=1 --iomem_align=4k --name=PT7 --rw=randrw --rwmixread=100 --iodepth=40 --numjobs=8 --bs=4096 --size=450GiB --runtime=120
--filename='/dev/e8b0:/dev/e8b1:/dev/e8b2:/dev/e8b3:/dev/e8b4:/dev/e8b5:/dev/e8b6:/dev/e8b7:/dev/e8b8:/dev/e8b9:/dev/e8b10:/dev/e8b11:/dev/e8b12:/dev/e8b13:/dev/e8b14:/dev/e8b15' --ioengine=libaio --numa_cpu_nodes=0 --random_generator=lfsr
PT7: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=40
...
fio-2.2.11
Starting 8 threads
Jobs: 8 (f=128): [r(8)] [100.0% done] [3199MB/0KB/0KB /s] [819K/0/0 iops] [eta 00m:00s]
PT7: (groupid=0, jobs=8): err= 0: pid=29711: Tue Mar 13 21:16:13 2018
  read : io=382920MB, bw=3190.1MB/s, iops=816888, runt=120001msec
    slat (usec): min=1, max=1069, avg= 3.77, stdev= 4.33
    clat (usec): min=16, max=3593, avg=387.12, stdev=189.09
     lat (usec): min=36, max=3613, avg=391.06, stdev=189.05

Between 1 and 16, I’ve tried 4, 8 and 12 volumes. The performance gradually worsens with LFSR as the number of volumes increases, but stays flat without LFSR.
--
Thanks,
Michael Green