Huge eta with R/W mix job and job hangs

Huge eta with R/W mix job and job hangs

[Lakshmi]

Hi Jens,
  I am using the below job file, and this works fine when the numjobs
is less(ex: 5,10) however when I specify a huge numjobs , the job
hangs and shows a huge eta as shown below.

# fio FIO_Test.job
test: (g=0): rw=randrw, bs=71K-71K/20K-20K/20K-20K, ioengine=posixaio,
iodepth=32
...
fio-2.1.10
Starting 20 processes
Killed1 (f=1): [__________________m_] [0.0% done] [0KB/0KB/0KB /s]
[0/0/0 iops] [eta 1158050441d:06h:50m:07s]


When I just run a 100% Reads (71K) or 100% writes(20K) separately
with the below below parameters it works fine, but it is only when i
do a mix of reads and writes.

Any idea what is going on and if i am missing something?
OS:RHEL7
HDDs(sdf):2x300GB Hdds on RAID10

[test]
 name=test
 ioengine=posixaio
 invalidate=1
 iodepth=32
 bs=71K,20K
 group_reporting=1
 rw=randrw
 rwmixread=96
 rwmixwrite=4
 direct=1
 buffered=0
 size=1M
 time_based
 runtime=300
 filename=/dev/sdf
 numjobs=64

Please help!!
Thanks

Fwd: Huge eta with R/W mix job and job hangs

[Lakshmi]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi Jens,
 I am using the latest version of fio 2.1.10

Thanks
ld
---------- Forwarded message ----------
From: Lakshmi <sendtold@gmail.com>
Date: Wed, Dec 17, 2014 at 2:27 PM
Subject: Huge eta with R/W mix job and job hangs
To: "fio@vger.kernel.org" <fio@vger.kernel.org>

Hi Jens,
  I am using the below job file, and this works fine when the numjobs
is less(ex: 5,10) however when I specify a huge numjobs , the job
hangs and shows a huge eta as shown below.

# fio FIO_Test.job
test: (g=0): rw=randrw, bs=71K-71K/20K-20K/20K-20K, ioengine=posixaio,
iodepth=32
...
fio-2.1.10
Starting 20 processes
Killed1 (f=1): [__________________m_] [0.0% done] [0KB/0KB/0KB /s]
[0/0/0 iops] [eta 1158050441d:06h:50m:07s]


When I just run a 100% Reads (71K) or 100% writes(20K) separately
with the below below parameters it works fine, but it is only when i
do a mix of reads and writes.

Any idea what is going on and if i am missing something?
OS:RHEL7
HDDs(sdf):2x300GB Hdds on RAID10

[test]
 name=test
 ioengine=posixaio
 invalidate=1
 iodepth=32
 bs=71K,20K
 group_reporting=1
 rw=randrw
 rwmixread=96
 rwmixwrite=4
 direct=1
 buffered=0
 size=1M
 time_based
 runtime=300
 filename=/dev/sdf
 numjobs=64

Please help!!
Thanks

[-- Attachment #2: Type: text/html, Size: 1816 bytes --]

Re: Fwd: Huge eta with R/W mix job and job hangs

[Jens Axboe]

On 12/17/2014 04:46 PM, Lakshmi wrote:
> Hi Jens,
>  I am using the latest version of fio 2.1.10

2.1.10 was released back in June, there's been 11-14 since then. Just
try 2.2.0, which was released today.

-- 
Jens Axboe

Re: Fwd: Huge eta with R/W mix job and job hangs

[Lakshmi]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

Hi Jens,
   Updated to 2.2.0 and I am still seeing the same issue.
==========================================

# fio FIO_Test.job
test: (g=0): rw=randrw, bs=71K-71K/20K-20K/20K-20K, ioengine=posixaio,
iodepth=32
...
fio-2.2.0
Starting 64 processes
Jobs: 10 (f=10):
[_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
[1.9% done] [301.4MB/0KB/0KB /s] [4425/0/]
Jobs: 10 (f=10):
[_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
[1.9% done] [0KB/0KB/0KB /s] [0/0/0 iops]
Jobs: 10 (f=10):
[_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
[2.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 26m:57s]
..................................

Jobs: 10 (f=10):
[_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
[0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]Jobs: 10 (f=10):
[_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
[0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]Killed1158050441d:06h:30m:16s]

Thanks
Ld

On Wed, Dec 17, 2014 at 7:54 PM, Jens Axboe <axboe@kernel.dk> wrote:
>
> On 12/17/2014 04:46 PM, Lakshmi wrote:
> > Hi Jens,
> >  I am using the latest version of fio 2.1.10
>
> 2.1.10 was released back in June, there's been 11-14 since then. Just
> try 2.2.0, which was released today.
>
> --
> Jens Axboe
>
>

[-- Attachment #2: Type: text/html, Size: 2041 bytes --]

Re: Fwd: Huge eta with R/W mix job and job hangs

[Jens Axboe]

On 12/18/2014 10:08 AM, Lakshmi wrote:
> Hi Jens,
>     Updated to 2.2.0 and I am still seeing the same issue.
> ==========================================
>
> # fio FIO_Test.job
> test: (g=0): rw=randrw, bs=71K-71K/20K-20K/20K-20K, ioengine=posixaio,
> iodepth=32
> ...
> fio-2.2.0
> Starting 64 processes
> Jobs: 10 (f=10):
> [_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
> [1.9% done] [301.4MB/0KB/0KB /s] [4425/0/]
> Jobs: 10 (f=10):
> [_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
> [1.9% done] [0KB/0KB/0KB /s] [0/0/0 iops]
> Jobs: 10 (f=10):
> [_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
> [2.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 26m:57s]
> ..................................
>
> Jobs: 10 (f=10):
> [_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
> [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]Jobs: 10 (f=10):
> [_(18),m(1),_(1),m(1),_(8),m(1),_(1),m(1),_(4),m(1),_(2),m(1),_(1),m(1),_(3),m(2),_(12),m(1),_(4)]
> [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops]Killed1158050441d:06h:30m:16s]

Can you send fio a SIGILL and have it dump core, and then bzip2 that 
core up and store it somewhere where I can reach it?

-- 
Jens Axboe