Re: [ofa-general] iSer and Direct IO

[Cameron Harr <charr@fusionio.com>]

Eli Dorfman wrote:
> On Thu, May 15, 2008 at 3:12 AM, Cameron Harr <charr@fusionio.com> wrote:
>   
>> My problem though is that I'm a
>> little suspicious of some of the iSer numbers and whether they are
>> really using Direct IO. For example, you'll see below in some of my FIO
>> results that I'm getting a write B/W of 799.1 MB/s at one point. That's
>> way above what I can get natively on the device (~650 MB/s DIO) and is
>> more along the lines of buffered IO. If the IOP numbers are also using
>> some kind of caching, that could possibly taint them also. Does anyone
>> know if specifying DIO will really bypass all buffers or if something is
>> getting cached in the agents (iscsi, tgtd)?
>>
>>     
>
> Your question should be posted on linux-scsi.
> See the following link that explains about DIO
> http://tldp.org/HOWTO/SCSI-Generic-HOWTO/dio.html
>
> Please check with sgp_dd to avoid any caching.
>   

Well, I posted here because I was looking more at iSer characteristics 
than DIO. Things seemed to behave differently on iSer than what I'd 
expect and what srpt shows. Also, I have trust issues with sg*_dd. On 
the local box, they give me impossible numbers, whereas dd is where I'd 
expect it:
----
[root@test05 ~]# sgp_dd dio=1 if=/dev/zero of=/dev/fioa bs=512 bpt=2048 
count=16777216 time=1
time to transfer data was 5.556115 secs, 1546.03 MB/sec
[root@test05 ~]# sg_dd dio=1 if=/dev/zero of=/dev/fioa bs=512 bpt=2048 
count=16777216 time=1
time to transfer data: 5.565360 secs at 1543.46 MB/sec
[root@test05 ~]# dd oflag=direct if=/dev/zero of=/dev/fioa bs=1M count=8192
8589934592 bytes (8.6 GB) copied, 12.7761 seconds, 672 MB/s
----
Using iSer, with the small transfer chunks, sgp_dd has numbers that are 
in line with what I'd expect for DIO while sg_dd doesn't:
---------
sgp_dd: 200.64 MB/s
sg_dd:  735.42 MB/s
dd:     62.3 MB/s
--------
But for larger transfers (with 1M block transfers), both sgp_dd and 
sg_dd show well above what I think I can be getting:
-------
sgp_dd: 882.43
sg_dd:  819.89
dd:     731 MB/s #Which is still high, and which makes me suspect iSer
-------

The page Eli linked states "Direct IO support is designed in such a way 
that if it is requested and cannot be performed then the command will 
still be performed using indirect IO." So I'm wondering if for some 
reason here DIO can't be used with iSer? (BTW, /proc/scsi/sg/allow_dio is 1)
Cameron