From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B50BBCA9EC0 for ; Mon, 28 Oct 2019 12:50:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 87D8820873 for ; Mon, 28 Oct 2019 12:50:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572267047; bh=9Pff/niXX3LYQ2+AxgdRJKKA0heWGPjmAW6N2+ud4G0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=h101g2W5m6EeEyos2oc/uuqi87GgtatMmE/uYiwtx7/kDg1txL4wBNj+HEUaTqnQH prjuHQyEyxPqdleEf7RDwUYCQz83Wqob+llLU+luYvFTFYlAhXzJ1GSxkWNO+xwwiO cBllNNy9cEpjzENcL85tnpgXJOQyyhEpN8aQ45Ug= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728910AbfJ1Mur (ORCPT ); Mon, 28 Oct 2019 08:50:47 -0400 Received: from mail.kernel.org ([198.145.29.99]:42780 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726940AbfJ1Mur (ORCPT ); Mon, 28 Oct 2019 08:50:47 -0400 Received: from localhost (unknown [91.217.168.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 45FA5214E0; Mon, 28 Oct 2019 12:50:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572267045; bh=9Pff/niXX3LYQ2+AxgdRJKKA0heWGPjmAW6N2+ud4G0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=vzTEa1VUMuFa13mAI/GUbOrHX2p4cJMuUWuATodxiXG0r9nbUGRfrmbQ+9p/3VtDF RethsiYHhR8iceXL9vOlqN2NUGfrWA1DuC5RoFSlT46hoOCfiJFE6nLMBomJ320tE+ WKjl43jNtuyVDHa4wFbiXlczLTa2N2sBAPvCkiVs= Date: Mon, 28 Oct 2019 12:46:57 +0100 From: Greg KH To: Sasha Levin Cc: mpatocka@redhat.com, guru2018@gmail.com, ntsironis@arrikto.com, snitzer@redhat.com, stable@vger.kernel.org Subject: Re: FAILED: patch "[PATCH] dm snapshot: rework COW throttling to fix deadlock" failed to apply to 5.3-stable tree Message-ID: <20191028114657.GA7337@kroah.com> References: <157219064719033@kroah.com> <20191028093928.GM1560@sasha-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191028093928.GM1560@sasha-vm> User-Agent: Mutt/1.12.2 (2019-09-21) Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org On Mon, Oct 28, 2019 at 05:39:28AM -0400, Sasha Levin wrote: > On Sun, Oct 27, 2019 at 04:37:28PM +0100, gregkh@linuxfoundation.org wrote: > > > > The patch below does not apply to the 5.3-stable tree. > > If someone wants it applied there, or to any other stable or longterm > > tree, then please email the backport, including the original git commit > > id to . > > > > thanks, > > > > greg k-h > > > > ------------------ original commit in Linus's tree ------------------ > > > > > From b21555786f18cd77f2311ad89074533109ae3ffa Mon Sep 17 00:00:00 2001 > > From: Mikulas Patocka > > Date: Wed, 2 Oct 2019 06:15:53 -0400 > > Subject: [PATCH] dm snapshot: rework COW throttling to fix deadlock > > > > Commit 721b1d98fb517a ("dm snapshot: Fix excessive memory usage and > > workqueue stalls") introduced a semaphore to limit the maximum number of > > in-flight kcopyd (COW) jobs. > > > > The implementation of this throttling mechanism is prone to a deadlock: > > > > 1. One or more threads write to the origin device causing COW, which is > > performed by kcopyd. > > > > 2. At some point some of these threads might reach the s->cow_count > > semaphore limit and block in down(&s->cow_count), holding a read lock > > on _origins_lock. > > > > 3. Someone tries to acquire a write lock on _origins_lock, e.g., > > snapshot_ctr(), which blocks because the threads at step (2) already > > hold a read lock on it. > > > > 4. A COW operation completes and kcopyd runs dm-snapshot's completion > > callback, which ends up calling pending_complete(). > > pending_complete() tries to resubmit any deferred origin bios. This > > requires acquiring a read lock on _origins_lock, which blocks. > > > > This happens because the read-write semaphore implementation gives > > priority to writers, meaning that as soon as a writer tries to enter > > the critical section, no readers will be allowed in, until all > > writers have completed their work. > > > > So, pending_complete() waits for the writer at step (3) to acquire > > and release the lock. This writer waits for the readers at step (2) > > to release the read lock and those readers wait for > > pending_complete() (the kcopyd thread) to signal the s->cow_count > > semaphore: DEADLOCK. > > > > The above was thoroughly analyzed and documented by Nikos Tsironis as > > part of his initial proposal for fixing this deadlock, see: > > https://www.redhat.com/archives/dm-devel/2019-October/msg00001.html > > > > Fix this deadlock by reworking COW throttling so that it waits without > > holding any locks. Add a variable 'in_progress' that counts how many > > kcopyd jobs are running. A function wait_for_in_progress() will sleep if > > 'in_progress' is over the limit. It drops _origins_lock in order to > > avoid the deadlock. > > > > Reported-by: Guruswamy Basavaiah > > Reported-by: Nikos Tsironis > > Reviewed-by: Nikos Tsironis > > Tested-by: Nikos Tsironis > > Fixes: 721b1d98fb51 ("dm snapshot: Fix excessive memory usage and workqueue stalls") > > Cc: stable@vger.kernel.org # v5.0+ > > Depends-on: 4a3f111a73a8c ("dm snapshot: introduce account_start_copy() and account_end_copy()") > > Signed-off-by: Mikulas Patocka > > Signed-off-by: Mike Snitzer > > Grabbing the listed dependency solved it for 5.3-4.19. For 4.14 and > older I've also grabbed the semaphore->mutex conversion. Ugh, I missed that it said that there. I'll do this for 4.19, unless you have these ready to go for when the tree "opens up" again. thanks, greg k-h