RE: fio on vmware esxi

[<eliezer@integritech.solutions>]

Mark, Sitsofe,

So I've been testing FIO on VMware using the different ioengines below and found something interesting.
-psync
-pvsync
-sync
-posixaio

The OS is a VMware ESXi:
VMware ESXi 6.5.0 build-8294253

FIO version compiled via Docker (thanks to Mark and Sitsofe):
fio-3.11-22-g425d

I mounted 3 storage devices (2 are different block storage types) on the same baremetal ESXi server:
-500 IOPS NFS
-1000 IOPS Reliant Tier iSCSI Block Storage
-1000 IOPS Perf Tier iSCSI Block Storage

I run FIO with the following parameters and used the different engines I listed above:
--randrepeat=1 
--ioengine=posixaio 
--direct=1 
--sync=1  
--name=mytest
--filename=mytestfile.fio
 --overwrite=1 
--iodepth=64 
--size=100MB 
--readwrite=randrw 
--rwmixread=50 
--rwmixwrite=50
 --bs=16k

Here are the results:

NFS:::
-ioengine=psync on a 500 IOPS NFS storage:::      read=245 write=256 IOPS*
-ioengine=pvsync on a 500 IOPS NFS Storage::::   read=252 write=263 IOPS*
-ioengine=sync on a 500 IOPS NFS Storage::::        read=258 write=270 IOPS*
-ioengine=posixaio on a 500 IOPS NFS Storage:::: read=172 write=180 IOPS

BLOCK STORAGE:::
1000 IOPS Reliant Tier
-ioengine=psync on a 1000 IOPS Reliant storage:::     read=249 write=260 IOPS
-ioengine=pvsync on a 1000 IOPS Reliant storage:::    read=321 write=335 IOPS
-ioengine=sync on a 1000 IOPS Reliant storage:::        read=236 write=246 IOPS
-ioengine=posixaio on a 1000 IOPS Reliant storage:::  read=535 write=559 IOPS*

1000 IOPS Perf Tier
-ioengine=psync on a 1000 IOPS Perf storage:::     read=177 write=185 IOPS
-ioengine=pvsync on a 1000 IOPS Perf storage:::    read=176 write=184 IOPS
-ioengine=sync on a 1000 IOPS Perf storage:::      read=145 write=152 IOPS
-ioengine=posixaio on a 1000 IOPS Perf storage:::  read=528 write=551 IOPS*

Based on the FIO results I posted above, I've concluded that the suitable FIO engine for Block Storage is "posixaio"; and for NFS it can be psync or pvsync or sync.
If I use the FIO engine "posixaio" on VMware to test block storage, I'm seeing the expected IOPS.  
However, if I use the same FIO engine ("posixaio") to test NFS, I'm NOT seeing the expected IOPS.  I'd have to use either psync, or pvsync, or sync to see the IOPS I'm expecting.
  These tests are on the same baremetal server running the version of VMWare I posted above.

Can someone please shed some light on why FIO results is skewed when "posixaio" engine is used to test an NFS storage?   The same goes with ioengines psync/pvsync/sync skews IOPS  results on a block storage device?

Thanks in advance.

Eliezer