* Fio benchmark output "error=Too many open files
@ 2011-09-20 10:06 yan li
2011-09-20 19:55 ` Jens Axboe
0 siblings, 1 reply; 3+ messages in thread
From: yan li @ 2011-09-20 10:06 UTC (permalink / raw)
To: fio
Hi fio users and developers,
When I run fio benchmark, there are some error in the screen.
Does someone meet the same circumstances ?
Thanks, ^_^
#fio fio.cfg
......
fio: pid=29128, err=24/file:filesetup.c:529,
func=open(/mnt/rw1/rw1.168.2), error=Too many open files
fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2),
error=Too many open files
fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2),
error=Too many open files
fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2),
error=Too many open files
fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2),
error=Too many open files
......
OS:Scientific Linux release 6.0 (Carbon) 2.6.32-131.6.1.el6.x86_64 #1
SMP Tue Jul 12 17:14:50 CDT 2011 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/security/limits.conf
* hard nofile 20480
# fio -v
fio 1.57
# cat fio.cfg
[rw]
rw=randread
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt
iodepth=256
zero_buffers
timeout=500
thread
[rw1]
rw=randread
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt/rw1
iodepth=256
zero_buffers
timeout=500
thread
[rw2]
rw=randwrite
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt/rw2
iodepth=256
zero_buffers
timeout=500
thread
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fio benchmark output "error=Too many open files
2011-09-20 10:06 Fio benchmark output "error=Too many open files yan li
@ 2011-09-20 19:55 ` Jens Axboe
2011-09-21 7:34 ` yan li
0 siblings, 1 reply; 3+ messages in thread
From: Jens Axboe @ 2011-09-20 19:55 UTC (permalink / raw)
To: yan li; +Cc: fio
On 2011-09-20 12:06, yan li wrote:
> Hi fio users and developers,
> When I run fio benchmark, there are some error in the screen.
> Does someone meet the same circumstances ?
> Thanks, ^_^
>
> #fio fio.cfg
> ......
> fio: pid=29128, err=24/file:filesetup.c:529,
> func=open(/mnt/rw1/rw1.168.2), error=Too many open files
> fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2),
> error=Too many open files
> fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2),
> error=Too many open files
> fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2),
> error=Too many open files
> fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2),
> error=Too many open files
> ......
Fio keeps all of its data files open, and from the jobs below, you have
3*8*128 = 3072
in data files alone, add some house keeping, and perhaps you hit the
limit? Seems a bit odd, given that the typical max is somewhat larger,
and your selinux settings allow higher as well. What do
/proc/sys/fs/file-{nr,max}
show when this happens? You could also freeze fio at this point and ls
-l /proc/pid-of-fio/fd/ to see if something strange is being held open.
--
Jens Axboe
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fio benchmark output "error=Too many open files
2011-09-20 19:55 ` Jens Axboe
@ 2011-09-21 7:34 ` yan li
0 siblings, 0 replies; 3+ messages in thread
From: yan li @ 2011-09-21 7:34 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Hi Jens;
Thanks for your help. As you asked me, I checked again in the
benchmark system. Selinux is disabled.
Sorry about my careless, Because setting of limit.conf is not work.
When I ran fio, only 1600 files have opened.
# cat /proc/sys/fs/file-nr
576 0 390196
run fio:
#cat /proc/sys/fs/file-nr
1600 0 390196
# cat /proc/sys/fs/file-max
390196
#cd /proc/fio_pid/fd
# ls -l | grep -c rw
1003
At this time, I knew, the limit.conf setting is not work.
#ulimit -n
1024
I add another line in /etc/security/limits.conf
* soft nofile 20480
Now, it 's ok. when I run fio, there are no the error messages.
#ulimit -n
20480
--
Homer Li
2011/9/21 Jens Axboe <axboe@kernel.dk>:
> On 2011-09-20 12:06, yan li wrote:
>> Hi fio users and developers,
>> When I run fio benchmark, there are some error in the screen.
>> Does someone meet the same circumstances ?
>> Thanks, ^_^
>>
>> #fio fio.cfg
>> ......
>> fio: pid=29128, err=24/file:filesetup.c:529,
>> func=open(/mnt/rw1/rw1.168.2), error=Too many open files
>> fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2),
>> error=Too many open files
>> fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2),
>> error=Too many open files
>> fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2),
>> error=Too many open files
>> fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2),
>> error=Too many open files
>> ......
>
> Fio keeps all of its data files open, and from the jobs below, you have
>
> 3*8*128 = 3072
>
> in data files alone, add some house keeping, and perhaps you hit the
> limit? Seems a bit odd, given that the typical max is somewhat larger,
> and your selinux settings allow higher as well. What do
>
> /proc/sys/fs/file-{nr,max}
>
> show when this happens? You could also freeze fio at this point and ls
> -l /proc/pid-of-fio/fd/ to see if something strange is being held open.
>
> --
> Jens Axboe
>
>
--
Let one live alone doing no evil, care-free, like an elephant in the
elephant forest
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-09-21 7:34 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-09-20 10:06 Fio benchmark output "error=Too many open files yan li
2011-09-20 19:55 ` Jens Axboe
2011-09-21 7:34 ` yan li
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.