From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751912AbcA2F10 (ORCPT ); Fri, 29 Jan 2016 00:27:26 -0500 Received: from mail-pa0-f67.google.com ([209.85.220.67]:36289 "EHLO mail-pa0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750743AbcA2F1Y (ORCPT ); Fri, 29 Jan 2016 00:27:24 -0500 Date: Fri, 29 Jan 2016 14:28:38 +0900 From: Sergey Senozhatsky To: Peter Hurley Cc: Sergey Senozhatsky , Sergey Senozhatsky , Byungchul Park , akpm@linux-foundation.org, mingo@kernel.org, linux-kernel@vger.kernel.org, akinobu.mita@gmail.com, jack@suse.cz, torvalds@linux-foundation.org Subject: Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code Message-ID: <20160129052838.GD4820@swordfish> References: <20160128023750.GB1834@swordfish> <000301d15985$7f416690$7dc433b0$@lge.com> <20160128060530.GC1834@swordfish> <20160128081313.GB31266@X58A-UD3R> <20160128104137.GA610@swordfish> <20160128105342.GB610@swordfish> <20160128154257.GA564@swordfish> <56AA9F63.9070600@hurleysoftware.com> <20160129002703.GA4820@swordfish> <56AAEB50.8050801@hurleysoftware.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56AAEB50.8050801@hurleysoftware.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On (01/28/16 20:32), Peter Hurley wrote: [..] > You're assuming that Byungchul's patch is relevant to the recursion > he witnessed. There are several paths into spin_dump(). yes. I was speaking in the context of Byungchul's report. > Here's one that doesn't wait at all: > > vprintk_emit() > console_trylock() > down_trylock() > raw_spin_lock_irqsave() > ... > do_raw_spin_lock() > debug_spin_lock_before() > SPIN_BUG_ON() > spin_bug() > spin_dump() > printk() > ** RINSE AND REPEAT ** ah, yes, agree. > >> Additionally, what if the console_sem is simply corrupted? > >> A livelock with no output ever is not very helpful. > > > > if it's corrupted then this is not a spinlock debug problem. > > at all. > > I don't follow you. > indeed very misleading, sorry, almost didn't sleep last nigh. removing SPIN_BUG_ON entirely is not my logic, not all. printk locks are special, I agree. in context of the proposed patch - we can't disable spin_dump() for printk locks if they were corrupted. for printk locks it's over, nothing will be printed anymore. the only thing that _may be_ will help is zap_locks(), but not 100% guaranteed... we can panic the system, probably, if printk locks are getting corrupted, but panic() will not do the trick in general case, at this point -- console_unlock() takes the logbuf_lock, which can be corrupted. apart from that console driver can be in a weird state. I sort of proposed to update console_flush_on_panic() (called from panic()) function a while ago to do zap_locks(), so in this case declaring BUG() from spinlock debug when we see 'bad' printk-related locks will have better chances to work out (assuming that console driver(-s) is (are) not against us). [..] > This was in reference to a problem with spin lock recursion that > can't print. The spin lock recursion deadlocks, but you'll never > see the diagnostic because the driver is already holding the lock > (not from printk() but from some other code). > > The printk doesn't even need to be directly related to the > console driver itself, but some other thing that the console driver > depends on while holding the spin lock that it claims for console output. aha, ok. yes, this is certainly possible. > > this is not a case of printk recursion and it should be handled > > just fine. console drivers are called under console_sem only. > > logbuf lock is unlocked. vprintk_emit() adds message to the logbuf, > > calls console_trylock() (which of course does not lock anything) > > and returns back to console_driver code. > > Not if locks are zapped because printk() recognizes a recursion. > Note console driver's locks are not zapped. For example, yes, I proposed to add a ->reset callback to struct console a while ago, and to do a console reset loop in zap_locks() zap_locks: ... for_each_console(con) if (con->reset) con->reset(con) that would re-init console drivers (locks, etc.). IOW, panic() does zap_locks(), zap_locks() zap the locks and resets the console drivers. that's sort of what I have in my private kernels. [..] > > the only case when we really have a printk recursion is when > > someone calls printk() from within the vprintk_emit() logbuf_lock > > area. > > Not true. > > A while back, Jan Kara reworked the call site around > console_trylock_from_printk(). Hung with no output under unknown > conditions [1]. > > Never solved, but obviously means there are unhandled recursions. aha, ok. -ss