Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald <Martin@lichtvoll.de>]

Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
> >> On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> >> On 15 August 2012 03:36, Martin Steigerwald
> >> > > >> >> <Martin@lichtvoll.de>
> >> > > >> >> 
> >> > > >> >> wrote:
> >> > > >> >> > Hi Greg,
> >> > > >> > 
> >> > > >> > [�]
> >> > > >> > 
> >> > > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe"
> >> > > >> >> >> <axboe@kernel.dk>
> > 
> > wrote:
> >> > > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> >> > [�]
> >> > 
> >> > > >> >> Is it possible to read from more than file in a single
> >> > > >> >> job, in a round-robin fashion? I tried putting more than
> >> > > >> >> one file in a single job, but it only opened one file. If
> >> > > >> >> you mean to just do random reads in a single file - I've
> >> > > >> >> tried that, and the throughput is unrealistically low. I
> >> > > >> >> suspect it's because the read-ahead buffer cannot be
> >> > > >> >> effective for random accesses.  Of course, reading
> >> > > >> >> sequentially from a single file will result in a
> >> > > >> >> throughput that is far too high to simulate the
> >> > > >> >> application.
> >> > > >> > 
> >> > > >> > Have you tried
> >> > > >> > 
> >> > > >> >        nrfiles=int
> >> > > >> >        
> >> > > >> >               Number of files to use for this job. 
> >> > > >> >               Default: 1.
> >> > > >> >        
> >> > > >> >        openfiles=int
> >> > > >> >        
> >> > > >> >               Number of files to keep open at the same
> >> > > >> >               time. Default: nrfiles.
> >> > > >> >        
> >> > > >> >        file_service_type=str
> >> > > > 
> >> > > > [�]
> >> > > > 
> >> > > >> > ? (see fio manpage).
> >> > > >> > 
> >> > > >> > It seems to me that all you need is nrfiles. I�d bet that
> >> > > >> > fio distributes
> >> > > >> > the I/O size given among those files, but AFAIR there is
> >> > > >> > something about
> >> > > >> > that in fio documentation as well.
> >> > > >> > 
> >> > > >> > Use the doc! ;)
> >> > > > 
> >> > > > [�]
> >> > > > 
> >> > > >> Yes, I have tried all that, and it works, except that it
> >> > > >> causes disk queuing, as I stated in my first post. I thought
> >> > > >> you meant to put all the files into a single [job name]
> >> > > >> section of the ini file, to enforce single threaded io.
> >> > > > 
> >> > > > With just one job running at once?
> >> > > > 
> >> > > > Can you post an example job file?
> >> > > > 
> >> > > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
> >> > > > 
> >> > > > I only know the behaviour of fio on Linux where I/O depth of
> >> > > > greater than one is only possible with libaio and direct=1.
> >> > > > The manpage hints at I/O depth is one for all synchronous I/O
> >> > > > engines, so I�d bet that refers to Windows as well.
> >> > > > 
> >> > > > Other than that I have no idea.
> >> > 
> >> > [�]
> >> > 
> >> > > One INI file, but a seperate [job name] section for each file,
> >> > > yes. According to Jens, because each [job name] is a seperate
> >> > > thread, and iodepth acts at the thread level, there will still
> >> > > be queuing at the device level. If there were a way to do what
> >> > > I want I think Jens would have told me, unfortunately.   ;)
> >> > > 
> >> > > direct io does at least allow me to do cache-less reads though -
> >> > > thankyou.
> >> > 
> >> > My suggestion is to use one job with several files.
> >> > 
> >> > martin@merkaba:/tmp> cat severalfiles.job
> >> > [global]
> >> > size=1G
> >> > nrfiles=100
> >> > 
> >> > [read]
> >> > rw=read
> >> > 
> >> > [write]
> >> > stonewall
> >> > rw=write
> >> > 
> >> > (now these are two jobs, but stonewall lets the write job run
> >> > after the read one with cache invalidation if not disabled and if
> >> > supported by OS)
> >> > 
> >> > martin@merkaba:/tmp> fio severalfiles.job
> >> > read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> >> > write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> >> > 2.0.8
> >> > Starting 2 processes
> >> > read: Laying out IO file(s) (100 file(s) / 1023MB)
> >> > write: Laying out IO file(s) (100 file(s) / 1023MB)
> >> > Jobs: 1 (f=100)
> >> > read: (groupid=0, jobs=1): err= 0: pid=23377
> >> > [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs
> >> > �]
> >> > 
> >> > 
> >> > martin@merkaba:/tmp> ls -lh read.1.* | head
> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
> > 
> > [�]
> > 
> >> > [� only first ten displayed �]
> >> > 
> >> > martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
> >> > 100
> >> > 
> >> > 100 files a 11M, due to rounding issues that may nicely add up to
> >> > the one GiB.
> >> > 
> >> > Raw sizes are:
> >> > 
> >> > martin@merkaba:/tmp> ls -l read.1.* | head
> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
> > 
> > [�]
> > 
> >> > Note: When I used filename, fio just created one files regardless
> >> > of the nrfiles setting. I would have expected it to use the
> >> > filename as a prefix. There might be some way to have it do that.
> >> > 
> >> > Ciao,
> >> 
> >> Thanks - that runs, but it's still queuing. As I said before, I
> >> can't use the sync engine - I receive an error. Is there a
> >> synchronous engine available for Windows? Perhaps that's the only
> >> problem. Can you check to see whether your system is queuing at the
> >> file system/device level when you run that test?
> >> 
> >> I had attempted to put the files in a single job earlier - I think
> >> it may have been successfully accessing both files, but I didn't
> >> notice it in the output. I'm a raw beginner.
> > 
> > Did you try with
> > 
> > ioengine=windowsaio
> > 
> > +
> > 
> > iodepth=1 (should be default however I think)
> > 
> > 
> > Otherwise I have no idea. I never used fio on Windows so far.
> > 
> > It might help when you try to explain exactly which problem you want
> > to solve by the fio measurements. Multimedia streaming. Is it to
> > slow? What is
> > 
> > it why you want to do these measurements?
> 
> They are both defaults, and the output shows that both are being used.
> If you could tell me whether your system is generating queuing it
> would help, because if yours queues even when using the sync io
> engine, it means I'm wasting my time and fio simply needs to be
> augmented to support strict single threaded operation over multiple
> files.
> 
> I am wanting to determine whether the application in question is
> extracting a reasonable number of real time streams from any given
> storage system.

Just for the record since you got it working on Windows as well � it works for me:

merkaba:/tmp> cat severalfiles.job 
[global]
size=1G
nrfiles=100

[read]
rw=read

merkaba:/tmp> fio severalfiles.job
read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process

read: (groupid=0, jobs=1): err= 0: pid=4579
  read : io=1023.9MB, bw=2409.7MB/s, iops=616705 , runt=   425msec
    clat (usec): min=0 , max=54 , avg= 1.08, stdev= 0.64
     lat (usec): min=0 , max=54 , avg= 1.13, stdev= 0.66
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    1], 10.00th=[    1], 20.00th=[    1],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    2],
     | 99.00th=[    2], 99.50th=[    2], 99.90th=[   14], 99.95th=[   16],
     | 99.99th=[   23]
    lat (usec) : 2=92.14%, 4=7.74%, 10=0.01%, 20=0.10%, 50=0.01%
    lat (usec) : 100=0.01%
  cpu          : usr=22.41%, sys=76.42%, ctx=421, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=262100/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1023.9MB, aggrb=2409.7MB/s, minb=2409.7MB/s, maxb=2409.7MB/s, mint=425msec, maxt=425msec

(if you wonder about the figures � thats RAM testing � Linux tmpfs ;)

100% at IO depth 1.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7