From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Layton Subject: [PATCH] locks: try to catch potential deadlock between file-private and classic locks from same process Date: Tue, 4 Mar 2014 14:10:49 -0500 Message-ID: <1393960249-18961-1-git-send-email-jlayton@redhat.com> Cc: bfields@fieldses.org, luto@amacapital.net To: linux-fsdevel@vger.kernel.org Return-path: Received: from mail-qc0-f177.google.com ([209.85.216.177]:46791 "EHLO mail-qc0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752144AbaCDTKz (ORCPT ); Tue, 4 Mar 2014 14:10:55 -0500 Received: by mail-qc0-f177.google.com with SMTP id w7so5734673qcr.22 for ; Tue, 04 Mar 2014 11:10:54 -0800 (PST) Sender: linux-fsdevel-owner@vger.kernel.org List-ID: My expectation is that programs shouldn't mix classic and file-private locks, but Glenn Skinner pointed out to me that that may occur at times even if the programmer isn't aware. Suppose we have a program that uses file-private locks. That program then links in a library that uses classic POSIX locks. If those locks end up conflicting and one is using blocking locks, then the program could end up deadlocked. Try to catch this situation in posix_locks_deadlock by looking for the case where the blocking lock was set by the same process but has a different type, and have the kernel return EDEADLK if that occurs. This check is not perfect. You could (in principle) have a threaded process that is using classic locks in one thread and file-private locks in another. That's not necessarily a deadlockable situation but this check would cause an EDEADLK return in that case. By the same token, you could also have a file-private lock that was inherited across a fork(). If the inheriting process ends up blocking on that while trying to set a classic POSIX lock then this check would miss it and the program would deadlock. Signed-off-by: Jeff Layton --- fs/locks.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/locks.c b/fs/locks.c index 6fdf26a79cc8..19c0c5c24b93 100644 --- a/fs/locks.c +++ b/fs/locks.c @@ -790,7 +790,17 @@ static int posix_locks_deadlock(struct file_lock *caller_fl, int i = 0; /* - * This deadlock detector can't reasonably detect deadlocks with + * If one lock is file-private and the other one isn't, and these are + * owned by the same process, then we may be in a situation where + * a library is attempting to use a different locking flavor than the + * original program. + */ + if (caller_fl->fl_pid == block_fl->fl_pid && + IS_FILE_PVT(caller_fl) != IS_FILE_PVT(block_fl)) + return 1; + + /* + * This deadlock detector can't reasonably detect cyclic deadlocks with * FL_FILE_PVT locks, since they aren't owned by a process, per-se. */ if (IS_FILE_PVT(caller_fl)) -- 1.8.5.3