io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Lakshmi]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

Hi Jens,
   I have a job file that looks like below and it keeps giving errors
like attached in screen shot. I am using fio-2.2.4

cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
--invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
--direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"

Any idea what is causing this?

Thanks
LD

[-- Attachment #2: IO_U_F_FLIGHT error.png --]
[-- Type: image/png, Size: 10524 bytes --]

Re: io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Andrey Kuzmin]

On Fri, Feb 6, 2015 at 11:03 PM, Lakshmi <sendtold@gmail.com> wrote:
> Hi Jens,
>    I have a job file that looks like below and it keeps giving errors
> like attached in screen shot. I am using fio-2.2.4
>
> cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
> --invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
> --direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"

While t's perfectly legal, you're setting a rather large total io
depth for a single device (32 by 64), but otherwise the job seems fine
(and runs fine against a regular file with fio 2.2.5 here). If the bug
persists, you might want to rerun it under gdb to get a clue (such as
to print out the io_u that triggers the assert).

Regards,
Andrey

>
> Any idea what is causing this?
>
> Thanks
> LD

Re: io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Jens Axboe]

On 02/06/2015 01:03 PM, Lakshmi wrote:
> Hi Jens,
>    I have a job file that looks like below and it keeps giving errors
> like attached in screen shot. I am using fio-2.2.4
> 
> cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
> --invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
> --direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"
> 
> Any idea what is causing this?

Is this on ppc64?


-- 
Jens Axboe

Re: io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Of]

Yes Jens it is ppc64

Sent from my iPhone

> On Feb 6, 2015, at 3:13 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 02/06/2015 01:03 PM, Lakshmi wrote:
>> Hi Jens,
>>   I have a job file that looks like below and it keeps giving errors
>> like attached in screen shot. I am using fio-2.2.4
>> 
>> cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
>> --invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
>> --direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"
>> 
>> Any idea what is causing this?
> 
> Is this on ppc64?
> 
> 
> -- 
> Jens Axboe
> 

Re: io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Sitsofe Wheeler]

On 6 February 2015 at 20:03, Lakshmi <sendtold@gmail.com> wrote:
>
> cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
> --invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
> --direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"

I could be wrong but doesn't an iodepth of 32 and 64 jobs means that
you would try and queue up 2048 (32*64) 4K I/Os on the same disk at
once? Does /dev/sde really have a queue depth that high (cat
/sys/block/sde/queue/nr_requests will tell you the kernel side) and is
that your intent?

-- 
Sitsofe | http://sucs.org/~sits/

Re: io_u->flags & IO_U_F_FLIGHT failed errors when using libaio

[Jens Axboe]

On 02/06/2015 04:32 PM, Of wrote:
> Yes Jens it is ppc64

I'm pretty sure we have some memory ordering issues, or perhaps the 
barriers on ppc64 are just wrong. I'll have to check them.

The job:

>>> cmd="$FIO --name=test  --group_reporting=1 --ioengine=libaio
>>> --invalidate=1 --iodepth=32 --bs=4K --filename=/dev/sde --rw=randread
>>> --direct=1 --buffered=0 --time_based --runtime=300 --numjobs=64"

does indeed have 32 * 64 potentially pending, which is huge. So there 
are valid questions on whether this is really what you want. But 
regardless of that, it should not pose a problem for fio.

Is this on Linux? If so, can you send the output of /proc/cpuinfo, so I 
can see what CPU this is exactly.

-- 
Jens Axboe