Re: Terrible performance of sequential O_DIRECT 4k writes in SAN environment. ~3 times slower then Solars 10 with the same HBA/Storage.

Christoph Hellwig
2014-01-08T15:26:10Z

On Wed, Jan 08, 2014 at 04:43:07PM +0200, Sergey Meirovich wrote:
> Results are almost the same:
> 14.68Mb/sec 3758.02 Requests/sec
>

On my laptop SSD I get the following results (sometimes up to 200MB/s,
sometimes down to 100MB/s, always in the 40k to 50k IOps range):

time elapsed (sec.): 5
bandwidth (MiB/s): 160.00
IOps: 40960.00

The IOps are more than the hardware is physically capable of, but given
that you didn't specify O_SYNC this seems sensible given that we never
have to flush the disk cache.

Could it be that your array has WCE=0? In Linux we'll never enable the
cache automatically, but Solaris does at least when using ZFS. Try
running:

sdparm --set=WCE /dev/sdX

and try again.