[PATCH] locks: try to catch potential deadlock between file-private and classic locks from same process

[Jeff Layton <jlayton@redhat.com>]

My expectation is that programs shouldn't mix classic and file-private
locks, but Glenn Skinner pointed out to me that that may occur at times
even if the programmer isn't aware.

Suppose we have a program that uses file-private locks. That program
then links in a library that uses classic POSIX locks. If those locks
end up conflicting and one is using blocking locks, then the program
could end up deadlocked.

Try to catch this situation in posix_locks_deadlock by looking for the
case where the blocking lock was set by the same process but has a
different type, and have the kernel return EDEADLK if that occurs.

This check is not perfect. You could (in principle) have a threaded
process that is using classic locks in one thread and file-private locks
in another. That's not necessarily a deadlockable situation but this
check would cause an EDEADLK return in that case.

By the same token, you could also have a file-private lock that was
inherited across a fork(). If the inheriting process ends up blocking on
that while trying to set a classic POSIX lock then this check would miss
it and the program would deadlock.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/locks.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/fs/locks.c b/fs/locks.c
index 6fdf26a79cc8..19c0c5c24b93 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -790,7 +790,17 @@ static int posix_locks_deadlock(struct file_lock *caller_fl,
 	int i = 0;
 
 	/*
-	 * This deadlock detector can't reasonably detect deadlocks with
+	 * If one lock is file-private and the other one isn't, and these are
+	 * owned by the same process, then we may be in a situation where
+	 * a library is attempting to use a different locking flavor than the
+	 * original program.
+	 */
+	if (caller_fl->fl_pid == block_fl->fl_pid &&
+	    IS_FILE_PVT(caller_fl) != IS_FILE_PVT(block_fl))
+		return 1;
+
+	/*
+	 * This deadlock detector can't reasonably detect cyclic deadlocks with
 	 * FL_FILE_PVT locks, since they aren't owned by a process, per-se.
 	 */
 	if (IS_FILE_PVT(caller_fl))
-- 
1.8.5.3