From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934317AbeEWV32 (ORCPT ); Wed, 23 May 2018 17:29:28 -0400 Received: from mail-io0-f193.google.com ([209.85.223.193]:34213 "EHLO mail-io0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933195AbeEWV30 (ORCPT ); Wed, 23 May 2018 17:29:26 -0400 X-Google-Smtp-Source: AB8JxZrlTlHV2vsS0HZCEJPPXGwHvzsWHmxrkJCsFhSzf95+xcI35PbJ+erKPSDWA5xUshG7az4Wzg== Subject: Re: [PATCH] bdi: Move cgroup bdi_writeback to a dedicated low concurrency workqueue To: Tejun Heo Cc: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Jan Kara , Andrew Morton , kernel-team@fb.com References: <20180523175632.GO1718769@devbig577.frc2.facebook.com> From: Jens Axboe Message-ID: <82f07e3e-44a7-e05e-ed98-15cd7acba600@kernel.dk> Date: Wed, 23 May 2018 15:29:23 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <20180523175632.GO1718769@devbig577.frc2.facebook.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 5/23/18 11:56 AM, Tejun Heo wrote: > From 0aa2e9b921d6db71150633ff290199554f0842a8 Mon Sep 17 00:00:00 2001 > From: Tejun Heo > Date: Wed, 23 May 2018 10:29:00 -0700 > > cgwb_release() punts the actual release to cgwb_release_workfn() on > system_wq. Depending on the number of cgroups or block devices, there > can be a lot of cgwb_release_workfn() in flight at the same time. > > We're periodically seeing close to 256 kworkers getting stuck with the > following stack trace and overtime the entire system gets stuck. > > [] _synchronize_rcu_expedited.constprop.72+0x2fc/0x330 > [] synchronize_rcu_expedited+0x24/0x30 > [] bdi_unregister+0x53/0x290 > [] release_bdi+0x89/0xc0 > [] wb_exit+0x85/0xa0 > [] cgwb_release_workfn+0x54/0xb0 > [] process_one_work+0x150/0x410 > [] worker_thread+0x6d/0x520 > [] kthread+0x12c/0x160 > [] ret_from_fork+0x29/0x40 > [] 0xffffffffffffffff > > The events leading to the lockup are... > > 1. A lot of cgwb_release_workfn() is queued at the same time and all > system_wq kworkers are assigned to execute them. > > 2. They all end up calling synchronize_rcu_expedited(). One of them > wins and tries to perform the expedited synchronization. > > 3. However, that invovles queueing rcu_exp_work to system_wq and > waiting for it. Because #1 is holding all available kworkers on > system_wq, rcu_exp_work can't be executed. cgwb_release_workfn() > is waiting for synchronize_rcu_expedited() which in turn is waiting > for cgwb_release_workfn() to free up some of the kworkers. > > We shouldn't be scheduling hundreds of cgwb_release_workfn() at the > same time. There's nothing to be gained from that. This patch > updates cgwb release path to use a dedicated percpu workqueue with > @max_active of 1. > > While this resolves the problem at hand, it might be a good idea to > isolate rcu_exp_work to its own workqueue too as it can be used from > various paths and is prone to this sort of indirect A-A deadlocks. Applied, thanks. -- Jens Axboe