linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: Byungchul Park <byungchul.park@lge.com>,
	akpm@linux-foundation.org, mingo@kernel.org,
	linux-kernel@vger.kernel.org, akinobu.mita@gmail.com,
	jack@suse.cz, torvalds@linux-foundation.org,
	peter@hurleysoftware.com, sergey.senozhatsky@gmail.com
Subject: Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code
Date: Thu, 28 Jan 2016 19:53:42 +0900	[thread overview]
Message-ID: <20160128105342.GB610@swordfish> (raw)
In-Reply-To: <20160128104137.GA610@swordfish>

two small corrections.

On (01/28/16 19:41), Sergey Senozhatsky wrote:
[..]
> > Unfortunately, it's not reproduced anymore.
> > 
> > If it's clearly a spinlock caller's bug as you said, modifying the
> > spinlock debug code does not help it at all. But I found there's a
> > possiblity in the debug code *itself* to cause a lockup. So I tried
> > to fix it. What do you think about it?
> 
> ah... silly me... you mean the first CPU that triggers the spin_dump() will
					^^^ this, of course, is true for
					console_sem->lock and logbuf_lock
					only.

> deadlock itself, so the rest of CPUs will see endless recursive
> spin_lock()->spin_dump()->spin_lock()->spin_dump() calls?
> 
> like the one below?
> 
> 
> CPUZ is doing vprintk_emit()->spin_lock(), CPUA is the spin_lock's owner
> 
> CPUZ -> vprintk_emit()
> 	__spin_lock_debug()
> 		for (i = 0; i < `loops_per_jiffy * HZ'; i++) {   << wait for the lock
> 			if (arch_spin_trylock())
> 				return;
> 			__delay(1);
> 			}
> 		spin_dump()       << lock is still owned by CPUA
> 		{   -> vprintk_emit()
> 			__spin_lock_debug()
> 				for (...) {
> 					if (arch_spin_trylock())
> 						return;
> 					__delay(1);
> 				}
- 		<< CPUA unlocked the lock
> 				spin_dump()
> 				{   -> vprintk_emit()
> 					__spin_lock_debug()
      the "<< CPUA unlocked the lock" line better be here. to make it correct.

+		<< CPUA unlocked the lock
> 						for (...) {
> 							if (arch_spin_trylock())   << success!!
> 							/* CPUZ now owns the lock */
> 								return;
> 						}
> 				}
> 
> 			<< we return here with the spin_lock being owned by this CPUZ
> 
> 				trigger_all_cpu_backtrace()
> 
> 			<< and... now it does the arch_spin_lock()
> 				/*
> 				 * The trylock above was causing a livelock.  Give the lower level arch
> 				 * specific lock code a chance to acquire the lock. We have already
> 				 * printed a warning/backtrace at this point. The non-debug arch
> 				 * specific code might actually succeed in acquiring the lock.  If it is
> 				 * not successful, the end-result is the same - there is no forward
> 				 * progress.
> 				 */
> 				arch_spin_lock(&lock->raw_lock);
> 
> 			<< which obviously dealocks this CPU...
> 		}
> 
> 		trigger_all_cpu_backtrace()
> 
> 		arch_spin_lock()
> 
> 
> 
> 
> so
> 	"the CPUZ is now keeping the lock forever, and not going to release it"
> and
> 	"CPUA-CPUX will do vprintk_emit()->spin_lock()->spin_dump()->vprintk_emit()->..."
> 
> 
> 
> My apologies for not getting it right the first time. Sorry!
> 
> Can you please update your bug description in the commit message?
> It's the deadlock that is causing the recursion on other CPUs in the
> first place.

	-ss

  reply	other threads:[~2016-01-28 10:52 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-01-27 12:01 [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code Byungchul Park
2016-01-27 22:49 ` Peter Hurley
2016-01-28  7:15   ` Byungchul Park
2016-01-29  8:19     ` Byungchul Park
2016-01-28  1:42 ` Byungchul Park
2016-01-28  2:37   ` Sergey Senozhatsky
2016-01-28  4:36     ` byungchul.park
2016-01-28  6:05       ` Sergey Senozhatsky
2016-01-28  8:13         ` Byungchul Park
2016-01-28 10:41           ` Sergey Senozhatsky
2016-01-28 10:53             ` Sergey Senozhatsky [this message]
2016-01-28 15:42               ` Sergey Senozhatsky
2016-01-28 23:08                 ` Peter Hurley
2016-01-28 23:54                   ` Byungchul Park
2016-01-29  0:54                     ` Sergey Senozhatsky
2016-01-29  3:00                       ` Byungchul Park
2016-01-29  4:05                         ` Sergey Senozhatsky
2016-01-29 12:15                           ` Byungchul Park
2016-01-29  0:27                   ` Sergey Senozhatsky
2016-01-29  4:32                     ` Peter Hurley
2016-01-29  5:28                       ` Sergey Senozhatsky
2016-01-29  5:48                         ` Peter Hurley
2016-01-29  6:16                           ` Sergey Senozhatsky
2016-01-29  6:37                             ` Sergey Senozhatsky
2016-01-31 12:30                               ` Sergey Senozhatsky
2016-01-31 12:33                                 ` [PATCH 1/3] printk: introduce console_reset_on_panic() function Sergey Senozhatsky
2016-01-31 12:33                                   ` [PATCH 2/3] printk: introduce reset_console_drivers() Sergey Senozhatsky
2016-01-31 12:47                                     ` kbuild test robot
2016-01-31 12:33                                   ` [PATCH 3/3] spinlock_debug: panic on recursive lock spin_dump() Sergey Senozhatsky
2016-02-01 16:14                                     ` Sergey Senozhatsky
2016-02-02  7:59                                       ` Sergey Senozhatsky
2016-01-31 12:42                                   ` [PATCH 1/3] printk: introduce console_reset_on_panic() function kbuild test robot
2016-01-29  6:54                     ` [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code Byungchul Park
2016-01-29  7:13                       ` Sergey Senozhatsky
2016-01-29  8:13                         ` Byungchul Park

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160128105342.GB610@swordfish \
    --to=sergey.senozhatsky.work@gmail.com \
    --cc=akinobu.mita@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=byungchul.park@lge.com \
    --cc=jack@suse.cz \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peter@hurleysoftware.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).