[lockdep] chain_key collision check triggers

[lockdep] chain_key collision check triggers

[Heiko Carstens]

Hi all,

the new lockdep chain_key collision detection code triggers quite reliably
on s390, so it looks like we need a different iterate_chain_key()
implementation?

Actually there was already another report from Shuah Khan, which didn't get
any replies as far as I can tell:
https://lkml.org/lkml/2016/4/12/1080

Anyway, these are the chains on s390 which collided:

[ 9550.753790] ======================
[ 9550.753791] [chain_key collision ]
[ 9550.753793] 4.6.0-02644-gbf82442b7914 #2 Tainted: G        W      
[ 9550.753795] ----------------------
[ 9550.753796] crond/2973: Hash chain already cached but the contents don't match!
[ 9550.753798] Held locks:depth: 6
[ 9550.753800]  class_idx:925 -> chain_key:000000000000039d (&sig->cred_guard_mutex){+.+.+.}, at: [<0000000000346bb4>] prepare_bprm_creds+0x44/0x98
[ 9550.753807]  class_idx:944 -> chain_key:000000000073a3b0 (sb_writers#3){.+.+.+}, at: [<0000000000360aa2>] touch_atime+0x5a/0xd8
[ 9550.753814]  class_idx:1064 -> chain_key:0000000e74760428 (jbd2_handle){+.+...}, at: [<0000000000442a20>] start_this_handle+0x470/0x558
[ 9550.753823]  class_idx:843 -> chain_key:0001ce8ec085034b (&(&block->request_queue_lock)->rlock){..-.-.}, at: [<0000000000686c12>] blk_queue_bio+0x582/0x608
[ 9550.753830]  class_idx:842 -> chain_key:39d1d810a069634a (&(&block->queue_lock)->rlock){+.-.-.}, at: [<00000000007df038>] do_dasd_request+0x38/0x60
[ 9550.753836]  class_idx:850 -> chain_key:3b02140d2c694468 (dasd_global_profile.lock){..-.-.}, at: [<00000000007deeb8>] __dasd_process_request_queue+0x2a0/0x3e8
[ 9550.753841] Locks in cached chain:depth: 6
[ 9550.753844]  class_idx:926 -> chain_key:000000000000039e (&type->i_mutex_dir_key){+.+.+.}
[ 9550.753848]  class_idx:944 -> chain_key:000000000073c3b0 (sb_writers#3){.+.+.+}
[ 9550.753852]  class_idx:1064 -> chain_key:0000000e78760428 (jbd2_handle){+.+...}
[ 9550.753857]  class_idx:843 -> chain_key:0001cf0ec085034b (&(&block->request_queue_lock)->rlock){..-.-.}
[ 9550.753861]  class_idx:842 -> chain_key:39e1d810a069634a (&(&block->queue_lock)->rlock){+.-.-.}
[ 9550.753865]  class_idx:852 -> chain_key:3b02140d2c694468 (&(&device->profile.lock)->rlock){..-.-.}

Re: [lockdep] chain_key collision check triggers

[Peter Zijlstra]

On Fri, Jun 03, 2016 at 09:15:44AM +0200, Heiko Carstens wrote:
> Hi all,
> 
> the new lockdep chain_key collision detection code triggers quite reliably
> on s390, so it looks like we need a different iterate_chain_key()
> implementation?

That too, I have one queued up; but if you can realiably trigger this
that's 'good' news. Most reports so far have been one offs that people
could not reproduce.

Let me wake up a bit more, but in the meantime, the below is what I had
queued.

---
Subject: locking,lockdep: Use __jhash_mix for iterate_chain_key
From: Peter Zijlstra <peterz@infradead.org>
Date: Mon May 30 18:31:33 CEST 2016

Use __jhash_mix() to mix the class_idx into the class_key. This
function provides better mixing than the previously used, home grown
mix function.

Leave hashing to the professionals :-)

Cc: torvalds@linux-foundation.org
Suggested-by: George Spelvin <linux@sciencehorizons.net>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/locking/lockdep.c |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -46,6 +46,7 @@
 #include <linux/gfp.h>
 #include <linux/kmemcheck.h>
 #include <linux/random.h>
+#include <linux/jhash.h>
 
 #include <asm/sections.h>
 
@@ -309,10 +310,14 @@ static struct hlist_head chainhash_table
  * It's a 64-bit hash, because it's important for the keys to be
  * unique.
  */
-#define iterate_chain_key(key1, key2) \
-	(((key1) << MAX_LOCKDEP_KEYS_BITS) ^ \
-	((key1) >> (64-MAX_LOCKDEP_KEYS_BITS)) ^ \
-	(key2))
+static inline u64 iterate_chain_key(u64 key, u32 idx)
+{
+	u32 k0 = key, k1 = key >> 32;
+
+	__jhash_mix(idx, k0, k1); /* Macro that modifies arguments! */
+
+	return k0 | (u64)k1 << 32;
+}
 
 void lockdep_off(void)
 {

Re: [lockdep] chain_key collision check triggers

[Heiko Carstens]

On Fri, Jun 03, 2016 at 10:48:01AM +0200, Peter Zijlstra wrote:
> On Fri, Jun 03, 2016 at 09:15:44AM +0200, Heiko Carstens wrote:
> > Hi all,
> > 
> > the new lockdep chain_key collision detection code triggers quite reliably
> > on s390, so it looks like we need a different iterate_chain_key()
> > implementation?
> 
> That too, I have one queued up; but if you can realiably trigger this
> that's 'good' news. Most reports so far have been one offs that people
> could not reproduce.
> 
> Let me wake up a bit more, but in the meantime, the below is what I had
> queued.
> 
> ---
> Subject: locking,lockdep: Use __jhash_mix for iterate_chain_key
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Mon May 30 18:31:33 CEST 2016
> 
> Use __jhash_mix() to mix the class_idx into the class_key. This
> function provides better mixing than the previously used, home grown
> mix function.

Ok, thanks! I'll use that then. Unfortunately the new hash function
generates 28 instead of 2 instructions on s390.

Re: [lockdep] chain_key collision check triggers

[Peter Zijlstra]

On Fri, Jun 03, 2016 at 01:12:50PM +0200, Heiko Carstens wrote:
> On Fri, Jun 03, 2016 at 10:48:01AM +0200, Peter Zijlstra wrote:
> > On Fri, Jun 03, 2016 at 09:15:44AM +0200, Heiko Carstens wrote:
> > > Hi all,
> > > 
> > > the new lockdep chain_key collision detection code triggers quite reliably
> > > on s390, so it looks like we need a different iterate_chain_key()
> > > implementation?
> > 
> > That too, I have one queued up; but if you can realiably trigger this
> > that's 'good' news. Most reports so far have been one offs that people
> > could not reproduce.
> > 
> > Let me wake up a bit more, but in the meantime, the below is what I had
> > queued.
> > 
> > ---
> > Subject: locking,lockdep: Use __jhash_mix for iterate_chain_key
> > From: Peter Zijlstra <peterz@infradead.org>
> > Date: Mon May 30 18:31:33 CEST 2016
> > 
> > Use __jhash_mix() to mix the class_idx into the class_key. This
> > function provides better mixing than the previously used, home grown
> > mix function.
> 
> Ok, thanks! I'll use that then. Unfortunately the new hash function
> generates 28 instead of 2 instructions on s390.

Yeah, but I doubt you can actually measure the performance difference,
lockdep is a pig anyway.