slow open() calls and o_nonblock

["Aaron Wiebe" <epiphani@gmail.com>]

Greetings all.  I'm not on this list, so I apologize if this subject
has been covered before.  (Also, please cc me in the response.)

I've spent the last several months trying to work around the lack of a
decent disk AIO interface.  I'm starting to wonder if one exists
anywhere.  The short version:

I have written a daemon that needs to open several thousand files a
minute and write a small amount of data to each file.  After extensive
research, I ended up going with the POSIX AIO kludgy pthreads wrapper
in glibc to handle my writes due to the time constraints of writing my
own pthreads handler into the application.

The problem with this equation is that opens, closes and non-readwrite
operations (fchmod, fcntl, etc) have no interface in posix aio.  Now I
was under the assumption that given open and close operations are
comparatively less common than the write operations, this wouldn't be
a huge problem.  My tests seemed to reflect that.

I went to production with this yesterday to discover that under
production load, our filesystems (nfs on netapps) were substantially
slower than I was expecting.  open() calls are taking upwards of 2
seconds on occation, and usually ~20ms.

Now, Netapp speed aside, O_NONBLOCK and O_DIRECT seem to make zero
difference to my open times.  Example:

open("/somefile", O_WRONLY|O_NONBLOCK|O_CREAT, 0644) = 1621 <0.415147>

Now, I'm a userspace guy so I can be pretty dense, but shouldn't a
call with a nonblocking flag return EAGAIN if its going to take
anywhere near 415ms?  Is there a way I can force opens to EAGAIN if
they take more than 10ms?

Thanks for any help you folks can offer.

-Aaron Wiebe

(ps.  having come from the socket side of the fence, its incredibly
frustrating to be unable to poll() or epoll regular file FDs --
Especially knowing that the kernel is translating them into a TCP
socket to do NFS anyway.  Please add regular files to epoll and give
me a way to do the opens in the same fasion as connects!)