FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jeffrey McVay (jmcvay)]

[-- Attachment #1.1: Type: text/plain, Size: 883 bytes --]

On windows runing 4kB random Writes to a ramdisk with numjobs=4 and iodepth=64 FIO reports clat and lat maximums times that match the runtime of the test. Changing the numjobs=1 and iodepth=1 yields max latencies that meet expectations.  As a further sanity test I ran similar workloads using IOMeter with max response times that were in line with expectations.

To reproduce:

1.       Download and install the freeware version of Dataram ramdisk from http://memory.dataram.com/products-and-services/software/ramdisk

2.       Run FIO 2.0.9 with PhysD1_Fio_4kRandWr_4x64

3.       Notice the max entries for clat and lat

4.       For comparison run FIO with PhysD1_Fio4kRandWr_1x1

Attached are the FIO jobfiles, latency logs for the two FIO runs, IOMeter ICF files, IOMeter csv files, and screenshots comparing similar runs of FIO and IOMeter.

Cheers,
Jeff




[-- Attachment #1.2: Type: text/html, Size: 7616 bytes --]

[-- Attachment #2: Fio_ramdisk.zip --]
[-- Type: application/x-zip-compressed, Size: 258816 bytes --]

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

On 24/09/2012 06:41, Jeffrey McVay (jmcvay) wrote:
>
> On windows runing 4kB random Writes to a ramdisk with numjobs=4 and 
> iodepth=64 FIO reports clat and lat maximums times that match the 
> runtime of the test. Changing the numjobs=1 and iodepth=1 yields max 
> latencies that meet expectations.  As a further sanity test I ran 
> similar workloads using IOMeter with max response times that were in 
> line with expectations.
>
> To reproduce:
>
> 1.Download and install the freeware version of Dataram ramdisk from 
> http://memory.dataram.com/products-and-services/software/ramdisk
>
> 2.Run FIO 2.0.9 with PhysD1_Fio_4kRandWr_4x64
>
> 3.Notice the max entries for clat and lat
>
> 4.For comparison run FIO with PhysD1_Fio4kRandWr_1x1
>
> Attached are the FIO jobfiles, latency logs for the two FIO runs, 
> IOMeter ICF files, IOMeter csv files, and screenshots comparing 
> similar runs of FIO and IOMeter.
>

Thanks. I've replicated the problem and will work out what's going wrong.

-- 
Bruce Cran

RE: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jeffrey McVay (jmcvay)]

> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Bruce Cran
> Sent: Monday, September 24, 2012 6:28 AM
> To: Jeffrey McVay (jmcvay)
> Cc: fio@vger.kernel.org; Jens Axboe
> Subject: Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme
> latencies
> 
> On 24/09/2012 06:41, Jeffrey McVay (jmcvay) wrote:
> >
> > On windows runing 4kB random Writes to a ramdisk with numjobs=4 and
> > iodepth=64 FIO reports clat and lat maximums times that match the
> > runtime of the test. Changing the numjobs=1 and iodepth=1 yields max
> > latencies that meet expectations.  As a further sanity test I ran
> > similar workloads using IOMeter with max response times that were in
> > line with expectations.
> >
> > To reproduce:
> >
> > 1.Download and install the freeware version of Dataram ramdisk from
> > http://memory.dataram.com/products-and-services/software/ramdisk
> >
> > 2.Run FIO 2.0.9 with PhysD1_Fio_4kRandWr_4x64
> >
> > 3.Notice the max entries for clat and lat
> >
> > 4.For comparison run FIO with PhysD1_Fio4kRandWr_1x1
> >
> > Attached are the FIO jobfiles, latency logs for the two FIO runs,
> > IOMeter ICF files, IOMeter csv files, and screenshots comparing
> > similar runs of FIO and IOMeter.
> >
> 
> Thanks. I've replicated the problem and will work out what's going
> wrong.
> 
Any updates? 
Is there anything I can do to help?
I have a personal goal to convert our IOMeter testing to FIO.
--Jeff
> --
> Bruce Cran
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

On 26/09/2012 17:30, Jeffrey McVay (jmcvay) wrote:
> Any updates? Is there anything I can do to help? I have a personal 
> goal to convert our IOMeter testing to FIO.

Sorry, I've not had any time to work on this yet - I'll work on it today.

-- 
Bruce Cran

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jens Axboe]

On 2012-09-26 18:32, Bruce Cran wrote:
> On 26/09/2012 17:30, Jeffrey McVay (jmcvay) wrote:
>> Any updates? Is there anything I can do to help? I have a personal 
>> goal to convert our IOMeter testing to FIO.
> 
> Sorry, I've not had any time to work on this yet - I'll work on it today.

Any news on this? What is the issue?

-- 
Jens Axboe

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

On 29/09/2012 07:23, Jens Axboe wrote:
> On 2012-09-26 18:32, Bruce Cran wrote:
>> On 26/09/2012 17:30, Jeffrey McVay (jmcvay) wrote:
>>> Any updates? Is there anything I can do to help? I have a personal
>>> goal to convert our IOMeter testing to FIO.
>> Sorry, I've not had any time to work on this yet - I'll work on it today.
> Any news on this? What is the issue?
>

I've not found the problem yet, but I'm also seeing an assertion failure 
(io_u->flags & IO_U_F_FLIGHT).

-- 
Bruce

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jens Axboe]

On 29/09/2012, at 15.46, Bruce Cran <bruce@cran.org.uk> wrote:

> On 29/09/2012 07:23, Jens Axboe wrote:
>> On 2012-09-26 18:32, Bruce Cran wrote:
>>> On 26/09/2012 17:30, Jeffrey McVay (jmcvay) wrote:
>>>> Any updates? Is there anything I can do to help? I have a personal
>>>> goal to convert our IOMeter testing to FIO.
>>> Sorry, I've not had any time to work on this yet - I'll work on it today.
>> Any news on this? What is the issue?
> 
> I've not found the problem yet, but I'm also seeing an assertion failure (io_u->flags & IO_U_F_FLIGHT)

That'd usually an OS or fio engine bug, it's a symptom of double completions. 

RE: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jeffrey McVay (jmcvay)]

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@kernel.dk]
> Sent: Saturday, September 29, 2012 7:22 AM
> To: Bruce Cran
> Cc: Jeffrey McVay (jmcvay); fio@vger.kernel.org
> Subject: Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme
> latencies
> 
> On 29/09/2012, at 15.46, Bruce Cran <bruce@cran.org.uk> wrote:
> 
> > On 29/09/2012 07:23, Jens Axboe wrote:
> >> On 2012-09-26 18:32, Bruce Cran wrote:
> >>> On 26/09/2012 17:30, Jeffrey McVay (jmcvay) wrote:
> >>>> Any updates? Is there anything I can do to help? I have a personal
> >>>> goal to convert our IOMeter testing to FIO.
> >>> Sorry, I've not had any time to work on this yet - I'll work on it
> today.
> >> Any news on this? What is the issue?
> >
> > I've not found the problem yet, but I'm also seeing an assertion
> failure (io_u->flags & IO_U_F_FLIGHT)
> 
> That'd usually an OS or fio engine bug, it's a symptom of double
> completions.

Anything new on this issue? There is a reviewer using FIO on Windows to measure latency values. I am concerned that they may publish inaccurate data.

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

On 06/10/2012 01:25, Jeffrey McVay (jmcvay) wrote:
> Anything new on this issue? There is a reviewer using FIO on Windows to measure latency values. I am concerned that they may publish inaccurate data.

I'm working on it again today.
The problem is that windowsaio iterates over the list of outstanding IOs 
(busylist), but that list has the most recent first. The sequence of 
queue/dequeue with an iodepth of 4 is:

queue 1
queue 2
queue 3
queue 4
dequeue 4
queue 5
dequeue 5
queue 6
dequeue 6
dequeue 3
dequeue 2
dequeue 1

So the first IO therefore has an apparent latency equal to the runtime 
of fio.

-- 
Bruce Cran

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On 10/10/2012 11:07, Bruce Cran wrote:
> The problem is that windowsaio iterates over the list of outstanding 
> IOs (busylist), but that list has the most recent first.

The attached patch appears to fix the problem:

Add outstanding IOs to the tail of io_u_busylist, not the head.

-- 
Bruce Cran

[-- Attachment #2: aio_queue.diff --]
[-- Type: text/plain, Size: 356 bytes --]

diff --git a/io_u.c b/io_u.c
index a2c583d..e047677 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1096,7 +1096,7 @@ again:
 
 		io_u->error = 0;
 		flist_del(&io_u->list);
-		flist_add(&io_u->list, &td->io_u_busylist);
+		flist_add_tail(&io_u->list, &td->io_u_busylist);
 		td->cur_depth++;
 		io_u->flags |= IO_U_F_IN_CUR_DEPTH;
 	} else if (td->o.verify_async) {

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jens Axboe]

On 2012-10-10 04:41, Bruce Cran wrote:
> On 10/10/2012 11:07, Bruce Cran wrote:
>> The problem is that windowsaio iterates over the list of outstanding 
>> IOs (busylist), but that list has the most recent first.
> 
> The attached patch appears to fix the problem:
> 
> Add outstanding IOs to the tail of io_u_busylist, not the head.

So where is this bug exactly? Is Windows reaping events from the head of
this list, and thus starving indefinitely potentially?

Patch makes sense in any case, but it should not make a functional
difference. I'll apply it.

-- 
Jens Axboe

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Bruce Cran]

On 10/10/2012 15:15, Jens Axboe wrote:
> So where is this bug exactly? Is Windows reaping events from the head of
> this list, and thus starving indefinitely potentially?

Yes - both windowsaio and posixaio read from the head of the list.

-- 
Bruce Cran

Re: FIO 2.0.9 for Windows Server 2008 RC1 SP2 reports extreme latencies

[Jens Axboe]

On 2012-10-10 08:18, Bruce Cran wrote:
> On 10/10/2012 15:15, Jens Axboe wrote:
>> So where is this bug exactly? Is Windows reaping events from the head of
>> this list, and thus starving indefinitely potentially?
> 
> Yes - both windowsaio and posixaio read from the head of the list.

Gotcha, I did check windowsaio after sending the email. Patch is in, so
all should be well.

-- 
Jens Axboe