* Fio benchmark output "error=Too many open files
@ 2011-09-20 10:06 yan li
2011-09-20 19:55 ` Jens Axboe
0 siblings, 1 reply; 3+ messages in thread
From: yan li @ 2011-09-20 10:06 UTC (permalink / raw)
To: fio
Hi fio users and developers,
When I run fio benchmark, there are some error in the screen.
Does someone meet the same circumstances ?
Thanks, ^_^
#fio fio.cfg
......
fio: pid=29128, err=24/file:filesetup.c:529,
func=open(/mnt/rw1/rw1.168.2), error=Too many open files
fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2),
error=Too many open files
fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2),
error=Too many open files
fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2),
error=Too many open files
fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2),
error=Too many open files
......
OS:Scientific Linux release 6.0 (Carbon) 2.6.32-131.6.1.el6.x86_64 #1
SMP Tue Jul 12 17:14:50 CDT 2011 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/security/limits.conf
* hard nofile 20480
# fio -v
fio 1.57
# cat fio.cfg
[rw]
rw=randread
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt
iodepth=256
zero_buffers
timeout=500
thread
[rw1]
rw=randread
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt/rw1
iodepth=256
zero_buffers
timeout=500
thread
[rw2]
rw=randwrite
ioengine=libaio
size=32m
numjobs=128
nrfiles=8
bssplit=512/20:1k/20:2k/10:4k/40:8k/10
directory=/mnt/rw2
iodepth=256
zero_buffers
timeout=500
thread
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: Fio benchmark output "error=Too many open files 2011-09-20 10:06 Fio benchmark output "error=Too many open files yan li @ 2011-09-20 19:55 ` Jens Axboe 2011-09-21 7:34 ` yan li 0 siblings, 1 reply; 3+ messages in thread From: Jens Axboe @ 2011-09-20 19:55 UTC (permalink / raw) To: yan li; +Cc: fio On 2011-09-20 12:06, yan li wrote: > Hi fio users and developers, > When I run fio benchmark, there are some error in the screen. > Does someone meet the same circumstances ? > Thanks, ^_^ > > #fio fio.cfg > ...... > fio: pid=29128, err=24/file:filesetup.c:529, > func=open(/mnt/rw1/rw1.168.2), error=Too many open files > fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2), > error=Too many open files > fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2), > error=Too many open files > fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2), > error=Too many open files > fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2), > error=Too many open files > ...... Fio keeps all of its data files open, and from the jobs below, you have 3*8*128 = 3072 in data files alone, add some house keeping, and perhaps you hit the limit? Seems a bit odd, given that the typical max is somewhat larger, and your selinux settings allow higher as well. What do /proc/sys/fs/file-{nr,max} show when this happens? You could also freeze fio at this point and ls -l /proc/pid-of-fio/fd/ to see if something strange is being held open. -- Jens Axboe ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fio benchmark output "error=Too many open files 2011-09-20 19:55 ` Jens Axboe @ 2011-09-21 7:34 ` yan li 0 siblings, 0 replies; 3+ messages in thread From: yan li @ 2011-09-21 7:34 UTC (permalink / raw) To: Jens Axboe; +Cc: fio Hi Jens; Thanks for your help. As you asked me, I checked again in the benchmark system. Selinux is disabled. Sorry about my careless, Because setting of limit.conf is not work. When I ran fio, only 1600 files have opened. # cat /proc/sys/fs/file-nr 576 0 390196 run fio: #cat /proc/sys/fs/file-nr 1600 0 390196 # cat /proc/sys/fs/file-max 390196 #cd /proc/fio_pid/fd # ls -l | grep -c rw 1003 At this time, I knew, the limit.conf setting is not work. #ulimit -n 1024 I add another line in /etc/security/limits.conf * soft nofile 20480 Now, it 's ok. when I run fio, there are no the error messages. #ulimit -n 20480 -- Homer Li 2011/9/21 Jens Axboe <axboe@kernel.dk>: > On 2011-09-20 12:06, yan li wrote: >> Hi fio users and developers, >> When I run fio benchmark, there are some error in the screen. >> Does someone meet the same circumstances ? >> Thanks, ^_^ >> >> #fio fio.cfg >> ...... >> fio: pid=29128, err=24/file:filesetup.c:529, >> func=open(/mnt/rw1/rw1.168.2), error=Too many open files >> fio: pid=29039, err=24/file:filesetup.c:529, func=open(/mnt/rw.79.2), >> error=Too many open files >> fio: pid=29040, err=24/file:filesetup.c:529, func=open(/mnt/rw.80.2), >> error=Too many open files >> fio: pid=29074, err=24/file:filesetup.c:529, func=open(/mnt/rw.114.2), >> error=Too many open files >> fio: pid=29026, err=24/file:filesetup.c:529, func=open(/mnt/rw.66.2), >> error=Too many open files >> ...... > > Fio keeps all of its data files open, and from the jobs below, you have > > 3*8*128 = 3072 > > in data files alone, add some house keeping, and perhaps you hit the > limit? Seems a bit odd, given that the typical max is somewhat larger, > and your selinux settings allow higher as well. What do > > /proc/sys/fs/file-{nr,max} > > show when this happens? You could also freeze fio at this point and ls > -l /proc/pid-of-fio/fd/ to see if something strange is being held open. > > -- > Jens Axboe > > -- Let one live alone doing no evil, care-free, like an elephant in the elephant forest ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-09-21 7:34 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-09-20 10:06 Fio benchmark output "error=Too many open files yan li 2011-09-20 19:55 ` Jens Axboe 2011-09-21 7:34 ` yan li
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.