From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751284AbdISFhG (ORCPT ); Tue, 19 Sep 2017 01:37:06 -0400 Received: from mail-pf0-f180.google.com ([209.85.192.180]:44317 "EHLO mail-pf0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750713AbdISFhE (ORCPT ); Tue, 19 Sep 2017 01:37:04 -0400 X-Google-Smtp-Source: AOwi7QDqspCEy4j5nTa5CD8LNlj8fyj8Qgin5Mz8IlsUeLBnH+1x73NxS/FVVJoZGbviIPUuOXmsUA== Date: Tue, 19 Sep 2017 13:37:49 +0800 From: Boqun Feng To: "Paul E. McKenney" Cc: Byungchul Park , Steven Rostedt , Neeraj Upadhyay , josh@joshtriplett.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, linux-kernel@vger.kernel.org, sramana@codeaurora.org, prsood@codeaurora.org, pkondeti@codeaurora.org, markivx@codeaurora.org, peterz@infradead.org, kernel-team@lge.com Subject: Re: Query regarding synchronize_sched_expedited and resched_cpu Message-ID: <20170919053749.GA12412@tardis> References: <20170918121213.312c82b0@gandalf.local.home> <20170918162412.GM3521@linux.vnet.ibm.com> <20170918122931.0e3341f3@gandalf.local.home> <20170918165527.GN3521@linux.vnet.ibm.com> <20170918235311.GA20177@linux.vnet.ibm.com> <20170919015027.GD5994@X58A-UD3R> <20170919020610.GF5994@X58A-UD3R> <20170919023329.GA3521@linux.vnet.ibm.com> <20170919024822.GG5994@X58A-UD3R> <20170919040456.GC3521@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="TB36FDmn/VVEgNH/" Content-Disposition: inline In-Reply-To: <20170919040456.GC3521@linux.vnet.ibm.com> User-Agent: Mutt/1.9.0 (2017-09-02) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --TB36FDmn/VVEgNH/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 18, 2017 at 09:04:56PM -0700, Paul E. McKenney wrote: > On Tue, Sep 19, 2017 at 11:48:22AM +0900, Byungchul Park wrote: > > On Mon, Sep 18, 2017 at 07:33:29PM -0700, Paul E. McKenney wrote: > > > > > Hello Paul and Steven, > > > > >=20 So I think this is another false positive, and the reason is we use st->done for multiple purposes. > > > > > This is saying: > > > > >=20 > > > > > Thread A > > > > > -------- > > > > > takedown_cpu() > > > > > irq_lock_sparse() > > > > > wait_for_completion(&st->done) // Wait for completion of B Thread A wait for the idle task on the outgoing to set the st->state to=20 CPUHP_AP_IDLE_DEAD(i.e. the corresponding complete() is the one in cpuhp_complete_idle_dead()), and it happens when we try to _offline_ a cpu. > > > > > irq_unlock_sparse() > > > > >=20 > > > > > Thread B > > > > > -------- > > > > > cpuhp_invoke_callback() > > > > > irq_lock_sparse() // Wait for A to irq_unlock_sparse() irq_affinity_online_cpu() is called here, so it happens when we try to _online_ a cpu. > > > > > (on the way going to complete(&st->done)) and we are going to complete(&st->done) in a hotplug thread context to indicate the hotplug thread has finished its job(i.e. this complete() is the one in cpuhp_thread_fun()). So even though the &st->done are the same instance, the deadlock could not happen, I think, as we could not up/down a same cpu at the same time? If I'm not missing something subtle. To fix this we can either 1) have dedicated completion instances for different wait purposes in cpuhp_cpu_state. or 2) extend crossrelease to have the "subclass" concept, so that callsite of complete() and wait_for_completion() for the same completion instance but with different purposes could be differed by lockdep. Thoughts? Regards, Boqun > > > > >=20 > > > > > So, lockdep consider this as a deadlock. > > > > > Is it possible to happen? > > > >=20 > > > > In addition, if it's impossible, then we should fix lock class > > > > assignments so that the locks actually have different classes. > > >=20 > > > Interesting, and thank you for the analysis! > > >=20 > > > The strange thing is that the way you describe it, this would be a > > > deterministic deadlock. Yet CPU hotplug operations complete just fine > > > in my tests. What am I missing here? > >=20 > > Hi, :) > >=20 > > Lockdep basically reports either (1) an actual deadlock happened at the > > time or (2) a deadlock possibility, even w/o LOCKDEP_CROSSRELEASE. > >=20 > > Both are useful. But LOCKDEP_CROSSRELEASE can only do the latter. IOW, > > the deadlock would actually happen _only_ when the two threads(A and B) > > run simultaniously. > >=20 > > In your case, those two threads might run at different timings. So it's > > not an actual deadlock, but still has a possibility for the problem to > > happen later. >=20 > Fair enough, if the wakeup always happened first, deadlock might well > be avoided. If the sleep happened first, I suspect deadlock would > be deterministic in this case. >=20 > Thanx, Paul >=20 > > > > > Thanks, > > > > > Byungchul > > > > >=20 > > > > > > [ 35.313943]=20 > > > > > > [ 35.313943] 3 locks held by torture_onoff/766: > > > > > > [ 35.313943] #0: (cpu_add_remove_lock){+.+.}, at: [] do_cpu_down+0x22/0x50 > > > > > > [ 35.313943] #1: (cpu_hotplug_lock.rw_sem){++++}, at: [] percpu_down_write+0x21/0xf0 > > > > > > [ 35.313943] #2: (sparse_irq_lock){+.+.}, at: [] irq_lock_sparse+0x12/0x20 > > > > > > [ 35.313943]=20 > > > > > > [ 35.313943] stack backtrace: > > > > > > [ 35.313943] CPU: 7 PID: 766 Comm: torture_onoff Not tainted = 4.13.0-rc4+ #1 > > > > > > [ 35.313943] Hardware name: QEMU Standard PC (i440FX + PIIX, = 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 > > > > > > [ 35.313943] Call Trace: > > > > > > [ 35.313943] dump_stack+0x67/0x97 > > > > > > [ 35.313943] print_circular_bug+0x21d/0x330 > > > > > > [ 35.313943] ? add_lock_to_list.isra.31+0xc0/0xc0 > > > > > > [ 35.313943] check_prev_add+0x401/0x800 > > > > > > [ 35.313943] ? wake_up_q+0x70/0x70 > > > > > > [ 35.313943] __lock_acquire+0x1100/0x11a0 > > > > > > [ 35.313943] ? __lock_acquire+0x1100/0x11a0 > > > > > > [ 35.313943] ? add_lock_to_list.isra.31+0xc0/0xc0 > > > > > > [ 35.313943] lock_acquire+0x9e/0x1e0 > > > > > > [ 35.313943] ? takedown_cpu+0x86/0xf0 > > > > > > [ 35.313943] wait_for_completion+0x36/0x130 > > > > > > [ 35.313943] ? takedown_cpu+0x86/0xf0 > > > > > > [ 35.313943] ? stop_machine_cpuslocked+0xb9/0xd0 > > > > > > [ 35.313943] ? cpuhp_invoke_callback+0x8b0/0x8b0 > > > > > > [ 35.313943] ? cpuhp_complete_idle_dead+0x10/0x10 > > > > > > [ 35.313943] takedown_cpu+0x86/0xf0 > > > > > > [ 35.313943] cpuhp_invoke_callback+0xa7/0x8b0 > > > > > > [ 35.313943] cpuhp_down_callbacks+0x3d/0x80 > > > > > > [ 35.313943] _cpu_down+0xbb/0xf0 > > > > > > [ 35.313943] do_cpu_down+0x39/0x50 > > > > > > [ 35.313943] cpu_down+0xb/0x10 > > > > > > [ 35.313943] torture_offline+0x75/0x140 > > > > > > [ 35.313943] torture_onoff+0x102/0x1e0 > > > > > > [ 35.313943] kthread+0x142/0x180 > > > > > > [ 35.313943] ? torture_kthread_stopping+0x70/0x70 > > > > > > [ 35.313943] ? kthread_create_on_node+0x40/0x40 > > > > > > [ 35.313943] ret_from_fork+0x27/0x40 > > > >=20 > >=20 >=20 --TB36FDmn/VVEgNH/ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEj5IosQTPz8XU1wRHSXnow7UH+rgFAlnArSkACgkQSXnow7UH +rgSxwf8C8VXGQqVKimKJpOww9FzYiD89PMdatayddOh7WOPcbDn98+aMizVLNQC QIODztbBPS4xMjEOs5a64+FI0/6c2/F5jFV8Oy5iqwUdaNHdtZqsxv/mGWPYt8VZ 0AxB+ZCOphv1l9Dv5cQ9meoj0HRsGuiNFvnAhJx1WANyIUx/B3yqUxq+1/FBZ8MQ DKbW0mQTOX2WgpYSY1k5rQWXtYIX/eUuqy10Ffo2tzVbOAi0zOQ0309pAxk18/g3 FsrBLxxlMqSfkk06LO+sBaALO/Y7k4KF9nzVqpeDKEPPsXIuv4cTy3Q3JmsMvuda C2OByDJ8BJjqcYxTC8U+P5VAMTgkVw== =Bwxg -----END PGP SIGNATURE----- --TB36FDmn/VVEgNH/--