From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3523C10F11 for ; Thu, 11 Apr 2019 03:24:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7004D20873 for ; Thu, 11 Apr 2019 03:24:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726689AbfDKDYx (ORCPT ); Wed, 10 Apr 2019 23:24:53 -0400 Received: from youngberry.canonical.com ([91.189.89.112]:40008 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726230AbfDKDYx (ORCPT ); Wed, 10 Apr 2019 23:24:53 -0400 Received: from mail-pl1-f197.google.com ([209.85.214.197]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1hEQKY-0007KR-EP for stable@vger.kernel.org; Thu, 11 Apr 2019 03:24:50 +0000 Received: by mail-pl1-f197.google.com with SMTP id d10so3168503plo.12 for ; Wed, 10 Apr 2019 20:24:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=EpyIOZV1lBzVdbxRS7L6T6+wWMPiTnzJ+HxF3Z44pqw=; b=U4pPt4B0Oo8JD5gfRvtJNk5/cxKqkbxaaxP21uaRbHd3CmLYkx32COptgGdoExvGir J7SEwQ67GZVbcUPjMRY1+iC8T5Hux9kUK81V7pHbo4zSn5q81GbaPcfRkzy0f8XWBLRa hoidIY+lihYiHRYCEw6AwGhpD7IrsWz/RKARmsMe8cVp9DWAUCAYdMK1Mgp3jVNcSntI l4BdLCGW4/PhLnLiR99jtlXg0xJgmYPE1umVIekEsEd+tWObjNf28/96OPxzeyZOGqd6 MgCCBxZu5c+BFCbcBUB6kF1YA8CflGKliwHzuK3zn5bqeaq29JC/jdjoMM/z9psOCr40 c7zQ== X-Gm-Message-State: APjAAAXoNpv1izA88XZlv0adSNDgsyp/hAvzi9qxsmDR7SBeK9v447+t K3j4FvKinHnFfK+PC82jj98Ok1FR+5OPUGoX8CMV5tkTPcj5TSeOZ+ZleZOl/lHR+zp+1VBDBQc YbebzPysRj8ar4H6isgGHWmclyJnEz5K2kQ== X-Received: by 2002:a63:6ac1:: with SMTP id f184mr42136995pgc.25.1554953088810; Wed, 10 Apr 2019 20:24:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqwqHwKvE5L8Ue9cpzMTz/03V+YklajlXNN90jEW3ipuywDdrwYCXgy5btLr551y/bFgTL5wOw== X-Received: by 2002:a63:6ac1:: with SMTP id f184mr42136983pgc.25.1554953088510; Wed, 10 Apr 2019 20:24:48 -0700 (PDT) Received: from localhost.localdomain (222-154-101-5-fibre.sparkbb.co.nz. [222.154.101.5]) by smtp.gmail.com with ESMTPSA id s16sm50747170pgo.69.2019.04.10.20.24.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Apr 2019 20:24:47 -0700 (PDT) From: Matthew Ruffell To: stable@vger.kernel.org Subject: [PATCH 4.9 0/3] fanotify: Fix notification subsystem hang Date: Thu, 11 Apr 2019 15:24:27 +1200 Message-Id: <20190411032430.17353-1-matthew.ruffell@canonical.com> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org BugLink: https://bugs.launchpad.net/bugs/1775165 [Note to upstream] I understand that this patch is a little long for -stable, but this patch series fixes a real issue, seen by real users, is testable, and is made up from upstream commits. Please consider it. [Impact] When userspace tasks which are processing fanotify permission events act incorrectly, the fsnotify_mark_srcu SRCU is held indefinitely which causes the whole notification subsystem to hang. This has been seen in production, and it can also be seen when running the Linux Test Project testsuite, specifically fanotify07. [Fix] Instead of holding the SRCU lock while waiting for userspace to respond, which may never happen, or not in the order we are expecting, we drop the fsnotify_mark_srcu SRCU lock before waiting for userspace response, and then reacquire the lock again when userspace responds. The fixes are from a series of upstream commits: 05f0e38724e8449184acd8fbf0473ee5a07adc6c (cherry-pick) 9385a84d7e1f658bb2d96ab798393e4b16268aaa (backport) abc77577a669f424c5d0c185b9994f2621c52aa4 (backport) [Testcase] You can reproduce the problem pretty quickly with the Linux Test Project: Steps (with root): 1. sudo apt-get install git xfsprogs -y 2. git clone --depth=1 https://github.com/linux-test-project/ltp.git 3. cd ltp 4. make autotools 5. ./configure 6. make; make install 7. cd /opt/ltp 8. echo -e "fanotify07 fanotify07 \nfanotify08 fanotify08" > /tmp/jobs 9. ./runltp -f /tmp/jobs On a stock 4.9.168 kernel, the system will hang, and the testcase will look like: <<>> tag=fanotify07 stime=1554326200 cmdline="fanotify07 " contacts="" analysis=exit <<>> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Test timeouted, sending SIGKILL! Cannot kill test processes! Congratulation, likely test hit a kernel bug. Exitting uncleanly... <<>> initiation_status="ok" duration=350 termination_type=exited termination_id=1 corefile=no cutime=0 cstime=0 <<>> Looking at dmesg, we see the following call stack [ 41.648244] LTP: starting fanotify07 (fanotify07 ) [ 242.729211] INFO: task fanotify07:1511 blocked for more than 120 seconds. [ 242.729257] Not tainted 4.9.168vanilla #1 [ 242.729281] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.729320] fanotify07 D 0 1511 1510 0x00000000 [ 242.729325] ffff92faf98a64c0 ffff92faf50f8000 ffff92faf19f2d00 ffff92faf989ad00 [ 242.729329] ffff92faffc19900 ffffb2f5c0cbbc70 ffffffffae8cf2c2 ffffb2f5c0cbbd60 [ 242.729333] 00e590a200000010 ffff92faffc19900 0000000000000046 7fffffffffffffff [ 242.729336] Call Trace: [ 242.729345] [] ? __schedule+0x242/0x700 [ 242.729348] [] schedule+0x2c/0x80 [ 242.729351] [] schedule_timeout+0x1fb/0x370 [ 242.729355] [] ? add_timer+0x11e/0x290 [ 242.729358] [] wait_for_completion+0xba/0x140 [ 242.729361] [] ? wake_up_q+0x80/0x80 [ 242.729364] [] __synchronize_srcu+0xf4/0x140 [ 242.729367] [] ? trace_raw_output_rcu_utilization+0x60/0x60 [ 242.729370] [] synchronize_srcu+0x23/0x40 [ 242.729374] [] fsnotify_mark_destroy_list+0x7b/0xe0 [ 242.729377] [] fsnotify_destroy_group+0x1f/0x50 [ 242.729380] [] fanotify_release+0xd6/0x140 [ 242.729384] [] __fput+0xea/0x230 [ 242.729386] [] ____fput+0xe/0x10 [ 242.729390] [] task_work_run+0x7c/0xa0 [ 242.729394] [] exit_to_usermode_loop+0x93/0xa0 [ 242.729397] [] do_syscall_64+0xc7/0xe0 [ 242.729400] [] entry_SYSCALL_64_after_swapgs+0x58/0xc6 Note this call stack is the same as the one for 4.4, but some function names have changed due to the two commits already included in 4.9. On a patched kernel, the test will pass successfully, and there will be no messages in dmesg. [Regression Potential] This makes modifications to how locking is performed in fsnotify / fanotify and there may be some cause for regression. Running all fanotify Linux Test Project tests shows that there are no extra failures caused by the patches, and instead fewer failures are seen due to the bugfix. Running the entire Linux Test Project testsuite actually works and runs to completion, something which doesn't happen in a unpatched kernel since it will hang on the fanotify07 test. The patches are taken from upstream, and all necessary commits have been taken into account, so I am happy with the potential risks and that testing has been completed. Jan Kara (3): fsnotify: Provide framework for dropping SRCU lock in ->handle_event fsnotify: Pass fsnotify_iter_info into handle_event handler fanotify: Release SRCU lock when waiting for userspace response fs/notify/dnotify/dnotify.c | 3 +- fs/notify/fanotify/fanotify.c | 20 ++++++- fs/notify/fsnotify.c | 19 +++++-- fs/notify/fsnotify.h | 6 ++ fs/notify/group.c | 1 + fs/notify/inotify/inotify.h | 3 +- fs/notify/inotify/inotify_fsnotify.c | 3 +- fs/notify/inotify/inotify_user.c | 2 +- fs/notify/mark.c | 83 +++++++++++++++++++++++++++- include/linux/fsnotify_backend.h | 8 ++- kernel/audit_fsnotify.c | 3 +- kernel/audit_tree.c | 3 +- kernel/audit_watch.c | 3 +- 13 files changed, 139 insertions(+), 18 deletions(-) -- 2.19.1