From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: Byungchul Park <byungchul.park@lge.com>,
akpm@linux-foundation.org, mingo@kernel.org,
linux-kernel@vger.kernel.org, akinobu.mita@gmail.com,
jack@suse.cz, torvalds@linux-foundation.org,
peter@hurleysoftware.com, sergey.senozhatsky@gmail.com
Subject: Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code
Date: Thu, 28 Jan 2016 19:53:42 +0900 [thread overview]
Message-ID: <20160128105342.GB610@swordfish> (raw)
In-Reply-To: <20160128104137.GA610@swordfish>
two small corrections.
On (01/28/16 19:41), Sergey Senozhatsky wrote:
[..]
> > Unfortunately, it's not reproduced anymore.
> >
> > If it's clearly a spinlock caller's bug as you said, modifying the
> > spinlock debug code does not help it at all. But I found there's a
> > possiblity in the debug code *itself* to cause a lockup. So I tried
> > to fix it. What do you think about it?
>
> ah... silly me... you mean the first CPU that triggers the spin_dump() will
^^^ this, of course, is true for
console_sem->lock and logbuf_lock
only.
> deadlock itself, so the rest of CPUs will see endless recursive
> spin_lock()->spin_dump()->spin_lock()->spin_dump() calls?
>
> like the one below?
>
>
> CPUZ is doing vprintk_emit()->spin_lock(), CPUA is the spin_lock's owner
>
> CPUZ -> vprintk_emit()
> __spin_lock_debug()
> for (i = 0; i < `loops_per_jiffy * HZ'; i++) { << wait for the lock
> if (arch_spin_trylock())
> return;
> __delay(1);
> }
> spin_dump() << lock is still owned by CPUA
> { -> vprintk_emit()
> __spin_lock_debug()
> for (...) {
> if (arch_spin_trylock())
> return;
> __delay(1);
> }
- << CPUA unlocked the lock
> spin_dump()
> { -> vprintk_emit()
> __spin_lock_debug()
the "<< CPUA unlocked the lock" line better be here. to make it correct.
+ << CPUA unlocked the lock
> for (...) {
> if (arch_spin_trylock()) << success!!
> /* CPUZ now owns the lock */
> return;
> }
> }
>
> << we return here with the spin_lock being owned by this CPUZ
>
> trigger_all_cpu_backtrace()
>
> << and... now it does the arch_spin_lock()
> /*
> * The trylock above was causing a livelock. Give the lower level arch
> * specific lock code a chance to acquire the lock. We have already
> * printed a warning/backtrace at this point. The non-debug arch
> * specific code might actually succeed in acquiring the lock. If it is
> * not successful, the end-result is the same - there is no forward
> * progress.
> */
> arch_spin_lock(&lock->raw_lock);
>
> << which obviously dealocks this CPU...
> }
>
> trigger_all_cpu_backtrace()
>
> arch_spin_lock()
>
>
>
>
> so
> "the CPUZ is now keeping the lock forever, and not going to release it"
> and
> "CPUA-CPUX will do vprintk_emit()->spin_lock()->spin_dump()->vprintk_emit()->..."
>
>
>
> My apologies for not getting it right the first time. Sorry!
>
> Can you please update your bug description in the commit message?
> It's the deadlock that is causing the recursion on other CPUs in the
> first place.
-ss
next prev parent reply other threads:[~2016-01-28 10:52 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-27 12:01 [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code Byungchul Park
2016-01-27 22:49 ` Peter Hurley
2016-01-28 7:15 ` Byungchul Park
2016-01-29 8:19 ` Byungchul Park
2016-01-28 1:42 ` Byungchul Park
2016-01-28 2:37 ` Sergey Senozhatsky
2016-01-28 4:36 ` byungchul.park
2016-01-28 6:05 ` Sergey Senozhatsky
2016-01-28 8:13 ` Byungchul Park
2016-01-28 10:41 ` Sergey Senozhatsky
2016-01-28 10:53 ` Sergey Senozhatsky [this message]
2016-01-28 15:42 ` Sergey Senozhatsky
2016-01-28 23:08 ` Peter Hurley
2016-01-28 23:54 ` Byungchul Park
2016-01-29 0:54 ` Sergey Senozhatsky
2016-01-29 3:00 ` Byungchul Park
2016-01-29 4:05 ` Sergey Senozhatsky
2016-01-29 12:15 ` Byungchul Park
2016-01-29 0:27 ` Sergey Senozhatsky
2016-01-29 4:32 ` Peter Hurley
2016-01-29 5:28 ` Sergey Senozhatsky
2016-01-29 5:48 ` Peter Hurley
2016-01-29 6:16 ` Sergey Senozhatsky
2016-01-29 6:37 ` Sergey Senozhatsky
2016-01-31 12:30 ` Sergey Senozhatsky
2016-01-31 12:33 ` [PATCH 1/3] printk: introduce console_reset_on_panic() function Sergey Senozhatsky
2016-01-31 12:33 ` [PATCH 2/3] printk: introduce reset_console_drivers() Sergey Senozhatsky
2016-01-31 12:47 ` kbuild test robot
2016-01-31 12:33 ` [PATCH 3/3] spinlock_debug: panic on recursive lock spin_dump() Sergey Senozhatsky
2016-02-01 16:14 ` Sergey Senozhatsky
2016-02-02 7:59 ` Sergey Senozhatsky
2016-01-31 12:42 ` [PATCH 1/3] printk: introduce console_reset_on_panic() function kbuild test robot
2016-01-29 6:54 ` [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code Byungchul Park
2016-01-29 7:13 ` Sergey Senozhatsky
2016-01-29 8:13 ` Byungchul Park
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160128105342.GB610@swordfish \
--to=sergey.senozhatsky.work@gmail.com \
--cc=akinobu.mita@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=byungchul.park@lge.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peter@hurleysoftware.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).