Re: splice vs truncate lockdep splat

[Dave Chinner <david@fromorbit.com>]

On Mon, Mar 21, 2016 at 10:16:38AM +1100, Dave Chinner wrote:
> On Sun, Mar 20, 2016 at 09:43:01AM -0400, Sage Weil wrote:
> > Hi all,
> > 
> > For the last few kernel releases we've been seeing this pretty regularly:
> > 
> 
> [snip]
> 
> Yup, splice still has locking problems. Always has, and the
> write_iter rework a few releases ago just moved the warnings
> elsewhere.
> 
> >  #012-> #2 (&pipe->mutex/1){+.+.+.}:
> 
> xfs_file_splice_read
>   xfs_iolock(XFS_IOLOCK_SHARED)
>     ....
>     pipe_lock(pipe)
> 
> XFS takes the XFS_IOLOCK_SHARED here to serialise splice read
> against truncate.
> 
> >  #012-> #1 (&(&ip->i_iolock)->mr_lock){++++++}:
> 
> mutex_lock(inode->i_mutex)
> ....
>   xfs_vn_setattr
>     xfs_iolock(XFS_IOLOCK_EXCL)
> 
> Truncate lock order for IO serialisation (no pipe lock).
> 
> >  #012-> #0 (&sb->s_type->i_mutex_key#19){+.+.+.}:
> 
> iter_file_splice_write
>   pipe_lock(pipe)
>     xfs_file_buffered_aio_write
>       mutex_lock(inode->i_mutex)
>         xfs_iolock(XFS_IOLOCK_EXCL)
> 
> XFS takes i_mutex (required for buffered IO path) and
> XFS_IOLOCK_EXCL (for atomic write semantics and truncate exclusion),
> but the splice write path has put this under the pipe lock....
> 
> i.e. because the read/write path take the IOLOCK different ways,
> lockdep seems to think it's OK. Hence IO testing doesn't report
> issues, but the moment you throw in another operation that takes
> both the i_mutex and XFS_IOLOCK_EXCL, lockdep will through a lock
> order violation report.
> 
> > We're not actually doing racing truncate and splice (or any racing 
> > operations for that matter) to the same files, so we don't actually hit 
> > this, but the lockdep warning is enough to make our test runs fail, and 
> > we'd rather not whitelist this if it's something that should get fixed 
> > anyway.
> 
> We can't fix it in XFS - the high level splice infrastructure which
> that read IO serialisation is done entirely based on page cache page
> locking and so can hold the pipe lock across ->splice_read callouts.
                                                 ^^^^^^^^^^^
						vfs_iter_write()

Got my read and write callouts  mixed up there. (i.e. splice_read is
assumed to have no filesystem locking above the pipe lock, so it is
assumed to be safe to take inode locks in vfs_iter_write() whilst
under the pipe lock).

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs