Re: sleeps and waits during io_submit

[Brian Foster <bfoster@redhat.com>]

On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
> On 12/01/2015 08:51 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 06:29 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
> >>>>On 12/01/2015 06:01 PM, Brian Foster wrote:
> >>>>>On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> >>>>>>On 12/01/2015 04:56 PM, Brian Foster wrote:
> >>>>>>>On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>>>>>>>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>>>>>>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>>>>>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>>>>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>>>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
...
> >>The case of waiting for I/O is much more worrying, because I/O latency are
> >>much higher.  But it seems like most of the DIO path does not trigger
> >>locking around I/O (and we are careful to avoid the ones that do, like
> >>writing beyond eof).
> >>
> >>(sorry for repeating myself, I have the feeling we are talking past each
> >>other and want to be on the same page)
> >>
> >Yeah, my point is just that just because the thread blocked on I/O,
> >doesn't mean the cpu can't carry on with some useful work for another
> >task.
> 
> In our case, there is no other task.  We run one thread per logical core, so
> if that thread gets blocked, the cpu idles.
> 
> The whole point of io_submit() is to issue an I/O and let the caller
> continue processing immediately.  It is the equivalent of O_NONBLOCK for
> networking code.  If O_NONBLOCK did block from time to time, practically all
> modern network applications would see a huge performance drop.
> 

Ok, but my understanding is that O_NONBLOCK would return an error code
in the blocking case such that userspace can do something else or retry
from a blockable context. I think this is similar to what hch posted wrt
to the pwrite2() bits for nonblocking buffered I/O or what I was asking
about earlier on with regard to returning an error if some blocking
would otherwise occur.

> >
> >>>>>  We submit an I/O which is
> >>>>>asynchronous in nature and wait on a completion, which causes the cpu to
> >>>>>schedule and execute another task until the completion is set by I/O
> >>>>>completion (via an async callback). At that point, the issuing thread
> >>>>>continues where it left off. I suspect I'm missing something... can you
> >>>>>elaborate on what you'd do differently here (and how it helps)?
> >>>>Just apply the same technique everywhere: convert locks to trylock +
> >>>>schedule a continuation on failure.
> >>>>
> >>>I'm certainly not an expert on the kernel scheduling, locking and
> >>>serialization mechanisms, but my understanding is that most things
> >>>outside of spin locks are reschedule points. For example, the
> >>>wait_for_completion() calls XFS uses to wait on I/O boil down to
> >>>schedule_timeout() calls. Buffer locks are implemented as semaphores and
> >>>down() can end up in the same place.
> >>But, for the most part, XFS seems to be able to avoid sleeping.  The call to
> >>__blockdev_direct_IO only launches the I/O, so any locking is only around
> >>cpu operations and, unless there is contention, won't cause us to sleep in
> >>io_submit().
> >>
> >>Trying to follow the code, it looks like xfs_get_blocks_direct (and
> >>__blockdev_direct_IO's get_block parameter in general) is synchronous, so
> >>we're just lucky to have everything in cache.  If it isn't, we block right
> >>there.  I really hope I'm misreading this and some other magic is happening
> >>elsewhere instead of this.
> >>
> >Nope, it's synchronous from a code perspective. The
> >xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
> >inode bmap metadata if it hasn't been done already. Note that this
> >should only happen once as everything is stored in-core, so in most
> >cases this is skipped. It's also possible extents are read in via some
> >other path/operation on the inode before an async I/O happens to be
> >submitted (e.g., see some of the other xfs_bmapi_read() callers).
> 
> Is there (could we add) some ioctl to prime this cache?  We could call it
> from a worker thread where we don't mind blocking during open.
> 

I suppose that's possible, or the worker thread could perform some
existing operation known to prime the cache. I don't think it's worth
getting into without a concrete example, however. The extent read
example we're batting around might not ever be a problem (as you've
noted due to file size), if files are truncated and recycled, for
example.

> What is the eviction policy for this cache?   Is it simply the block
> device's page cache?
> 

IIUC the extent list stays around until the inode is reclaimed. There's
a separate buffer cache for metadata buffers. Both types of objects
would be reclaimed based on memory pressure.

> What about the write path, will we see the same problems there?  I would
> guess the problem is less severe there if the metadata is written with
> writeback policy.
> 

Metadata is modified in-core and handed off to the logging
infrastructure via a transaction. The log is flushed to disk some time
later and metadata writeback occurs asynchronously via the xfsaild
thread.

Brian

> >
> >Either way, the extents have to be read in at some point and I'd expect
> >that cpu to schedule onto some other task while that thread waits on I/O
> >to complete (read-ahead could also be a factor here, but I haven't
> >really dug into how that is triggered for buffers).
> 
> To provide an example, our application, which is a database, faces this
> problem exact at a higher level.  Data is stored in data files, and data
> items' locations are stored in index files. When we read a bit of data, we
> issue an index read, and pass it a continuation to be executed when the read
> completes.  This latter continuation parses the data and passes it to the
> code that prepares it for merging with data from other data files, and an
> eventual return to the user.
> 
> Having written code for over a year in this style, I've come to expect it to
> be used everywhere asynchronous I/O is used, but I realize it is fairly hard
> without good support from a framework that allows continuations to be
> composed in a natural way.
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs