linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* lockdep splat with v4.11 and generic/386
@ 2017-05-04 19:38 Ross Zwisler
  2017-05-05  7:59 ` Jan Kara
  0 siblings, 1 reply; 5+ messages in thread
From: Ross Zwisler @ 2017-05-04 19:38 UTC (permalink / raw)
  To: linux-ext4, Theodore Ts'o, Jan Kara

I hit the following lockdep splat during some regression testing today, and
was able to reproduce it on vanilla v4.11 without DAX.  My test setup is big
memmap PMEM device in a QEMU virtual machine, but I don't think that the block
device is important.  I was able to reproduce this easily by running
generic/386 in a loop.  Here's the failure:

  FSTYP         -- ext4
  PLATFORM      -- Linux/x86_64 lorwyn 4.11.0
  MKFS_OPTIONS  -- /dev/pmem0p2
  MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
  
  generic/386 1s ... 1s
  _check_dmesg: something found in dmesg (see /root/xfstests/results//generic/386.dmesg)
  Ran: generic/386
  Failures: generic/386
  Failed 1 of 1 tests

Here's the lockdep splat, passed through kasan_symbolize.py:

  run fstests generic/386 at 2017-05-04 13:36:35
  EXT4-fs (pmem0p2): mounted filesystem with ordered data mode. Opts: acl,user_xattr,quota
  
  ======================================================
  [ INFO: possible circular locking dependency detected ]
  4.11.0 #1 Not tainted
  -------------------------------------------------------
  mkdir/8458 is trying to acquire lock:
   (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8133a8bb>] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
  
  but task is already holding lock:
   (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
  
  which lock already depends on the new lock.
  
  
  the existing dependency chain (in reverse order) is:
  
  -> #1 (&ei->i_data_sem/2){++++..}:
        [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
        [<      none      >] down_read+0x43/0xa0 kernel/locking/rwsem.c:23
        [<      none      >] ext4_map_blocks+0x2b4/0x5e0 fs/ext4/inode.c:540
        [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
        [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
        [<      none      >] ext4_quota_read+0xce/0x110 fs/ext4/super.c:5470
        [<      none      >] read_blk+0x4c/0x60 fs/quota/quota_tree.c:63
        [<      none      >] find_tree_dqentry+0x44/0x230 fs/quota/quota_tree.c:579
        [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
        [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
        [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
        [<     inline     >] find_dqentry fs/quota/quota_tree.c:602
        [<      none      >] qtree_read_dquot+0x12e/0x260 fs/quota/quota_tree.c:622
        [<      none      >] v2_read_dquot+0x2e/0x30 fs/quota/quota_v2.c:288
        [<      none      >] dquot_acquire+0xe3/0x120 fs/quota/dquot.c:411
        [<      none      >] ext4_acquire_dquot+0x68/0xa0 fs/ext4/super.c:5227
        [<      none      >] dqget+0x305/0x470 fs/quota/dquot.c:891
        [<      none      >] __dquot_initialize+0x151/0x290 fs/quota/dquot.c:1460
        [<      none      >] dquot_initialize+0x13/0x20 fs/quota/dquot.c:1511
        [<      none      >] ext4_mkdir+0x66/0x470 fs/ext4/namei.c:2645
        [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
        [<     inline     >] SYSC_mkdirat fs/namei.c:3792
        [<     inline     >] SyS_mkdirat fs/namei.c:3776
        [<     inline     >] SYSC_mkdir fs/namei.c:3803
        [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
        [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
  
  -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
        [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
        [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
        [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
        [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
        [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
        [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
        [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
        [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
        [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
        [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
        [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
        [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
        [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
        [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
        [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
        [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
        [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
        [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
        [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
        [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
        [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
        [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
        [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
        [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
        [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
        [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
        [<     inline     >] SYSC_mkdirat fs/namei.c:3792
        [<     inline     >] SyS_mkdirat fs/namei.c:3776
        [<     inline     >] SYSC_mkdir fs/namei.c:3803
        [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
        [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
  
  other info that might help us debug this:
  
   Possible unsafe locking scenario:
  
         CPU0                    CPU1
         ----                    ----
    lock(&ei->i_data_sem/2);
                                 lock(&s->s_dquot.dqio_mutex);
                                 lock(&ei->i_data_sem/2);
    lock(&s->s_dquot.dqio_mutex);
  
   *** DEADLOCK ***
  
  5 locks held by mkdir/8458:
   #0:  (sb_writers#14){.+.+.+}, at: [<     inline     >] sb_start_write ./include/linux/fs.h:1504
   #0:  (sb_writers#14){.+.+.+}, at: [<ffffffff812e1e24>] mnt_want_write+0x24/0x50 fs/namespace.c:388
   #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<     inline     >] inode_lock_nested ./include/linux/fs.h:731
   #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<ffffffff812cb5e3>] filename_create+0x83/0x160 fs/namei.c:3594
   #2:  (jbd2_handle){++++..}, at: [<ffffffff813d31b2>] start_this_handle+0x112/0x450 fs/jbd2/transaction.c:361
   #3:  (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
   #4:  (dquot_srcu){......}, at: [<     inline     >] srcu_read_lock ./include/linux/srcu.h:237
   #4:  (dquot_srcu){......}, at: [<ffffffff8133c44b>] __dquot_alloc_space+0xbb/0x2b0 fs/quota/dquot.c:1668
  
  stack backtrace:
  CPU: 3 PID: 8458 Comm: mkdir Not tainted 4.11.0 #1
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-1.fc25 04/01/2014
  Call Trace:
  [<     inline     >] __dump_stack lib/dump_stack.c:16
  [<      none      >] dump_stack+0x86/0xc3 lib/dump_stack.c:52
  [<      none      >] print_circular_bug+0x1be/0x210 kernel/locking/lockdep.c:1204
  [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
  [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
  [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
  [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
  [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
   ?[<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
   ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
   ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
  [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
  [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
   ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
   ?[<     inline     >] spin_unlock ./include/linux/spinlock.h:339
   ?[<      none      >] dquot_mark_dquot_dirty+0x57/0xc0 fs/quota/dquot.c:355
   ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
   ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
   ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
   ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
  [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
   ?[<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
  [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
  [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
  [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
  [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
  [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
  [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
   ?[<      none      >] __this_cpu_preempt_check+0x13/0x20 lib/smp_processor_id.c:62
   ?[<      none      >] __percpu_counter_add+0x85/0xb0 lib/percpu_counter.c:90
  [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
  [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
  [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
  [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
   ?[<     inline     >] lock_is_held ./include/linux/lockdep.h:348
   ?[<      none      >] rcu_read_lock_sched_held+0x4a/0x80 kernel/rcu/update.c:114
   ?[<     inline     >] trace_kmalloc ./include/trace/events/kmem.h:45
   ?[<      none      >] __kmalloc+0x2ce/0x300 mm/slub.c:3743
   ?[<     inline     >] kmalloc ./include/linux/slab.h:495
   ?[<     inline     >] kzalloc ./include/linux/slab.h:663
   ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
   ?[<     inline     >] kmalloc ./include/linux/slab.h:495
   ?[<     inline     >] kzalloc ./include/linux/slab.h:663
   ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
  [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
   ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
   ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
  [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
  [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
  [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
  [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
  [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
  [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
  [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
  [<     inline     >] SYSC_mkdirat fs/namei.c:3792
  [<     inline     >] SyS_mkdirat fs/namei.c:3776
  [<     inline     >] SYSC_mkdir fs/namei.c:3803
  [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
  [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
  RIP: 0033:0x7fb614c15947
  RSP: 002b:00007fff31be4938 EFLAGS: 00000216 ORIG_RAX: 0000000000000053
  RAX: ffffffffffffffda RBX: 00007fff31be4c48 RCX: 00007fb614c15947
  RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007fff31be6aa5
  RBP: 0000000000000003 R08: 00000000000001ff R09: 0000564258eadac0
  R10: 000056425b0a4060 R11: 0000000000000216 R12: 0000000000000000
  R13: 0000564258eadb00 R14: 0000000000000000 R15: 0000000000000000

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: lockdep splat with v4.11 and generic/386
  2017-05-04 19:38 lockdep splat with v4.11 and generic/386 Ross Zwisler
@ 2017-05-05  7:59 ` Jan Kara
  2017-05-05  8:49   ` Jan Kara
  0 siblings, 1 reply; 5+ messages in thread
From: Jan Kara @ 2017-05-05  7:59 UTC (permalink / raw)
  To: Ross Zwisler; +Cc: linux-ext4, Theodore Ts'o, Jan Kara

On Thu 04-05-17 13:38:38, Ross Zwisler wrote:
> I hit the following lockdep splat during some regression testing today, and
> was able to reproduce it on vanilla v4.11 without DAX.  My test setup is big
> memmap PMEM device in a QEMU virtual machine, but I don't think that the block
> device is important.  I was able to reproduce this easily by running
> generic/386 in a loop.  Here's the failure:

Thanks for report! This is false positive caused by a bug in our lockdep
annotation - locks on quota files (i_data_sem in this case) rank
differently than locks on normal files - specifically the lock ordering is
i_data_sem (normal file, directory, ...) -> dqio_mutex -> i_data_sem (quota
file). Now we do take care to tell lockdep about this by calling
lockdep_set_quota_inode() in fs/ext4/super.c for quota files. However we
don't return i_data_sem back to the normal locking class on freeing inode
and when quota file inode gets reused for something else, it has a wrong
locking class set which confuses lockdep. I'll send a fix.

								Honza

> 
>   FSTYP         -- ext4
>   PLATFORM      -- Linux/x86_64 lorwyn 4.11.0
>   MKFS_OPTIONS  -- /dev/pmem0p2
>   MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
>   
>   generic/386 1s ... 1s
>   _check_dmesg: something found in dmesg (see /root/xfstests/results//generic/386.dmesg)
>   Ran: generic/386
>   Failures: generic/386
>   Failed 1 of 1 tests
> 
> Here's the lockdep splat, passed through kasan_symbolize.py:
> 
>   run fstests generic/386 at 2017-05-04 13:36:35
>   EXT4-fs (pmem0p2): mounted filesystem with ordered data mode. Opts: acl,user_xattr,quota
>   
>   ======================================================
>   [ INFO: possible circular locking dependency detected ]
>   4.11.0 #1 Not tainted
>   -------------------------------------------------------
>   mkdir/8458 is trying to acquire lock:
>    (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8133a8bb>] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>   
>   but task is already holding lock:
>    (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
>   
>   which lock already depends on the new lock.
>   
>   
>   the existing dependency chain (in reverse order) is:
>   
>   -> #1 (&ei->i_data_sem/2){++++..}:
>         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
>         [<      none      >] down_read+0x43/0xa0 kernel/locking/rwsem.c:23
>         [<      none      >] ext4_map_blocks+0x2b4/0x5e0 fs/ext4/inode.c:540
>         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
>         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
>         [<      none      >] ext4_quota_read+0xce/0x110 fs/ext4/super.c:5470
>         [<      none      >] read_blk+0x4c/0x60 fs/quota/quota_tree.c:63
>         [<      none      >] find_tree_dqentry+0x44/0x230 fs/quota/quota_tree.c:579
>         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
>         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
>         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
>         [<     inline     >] find_dqentry fs/quota/quota_tree.c:602
>         [<      none      >] qtree_read_dquot+0x12e/0x260 fs/quota/quota_tree.c:622
>         [<      none      >] v2_read_dquot+0x2e/0x30 fs/quota/quota_v2.c:288
>         [<      none      >] dquot_acquire+0xe3/0x120 fs/quota/dquot.c:411
>         [<      none      >] ext4_acquire_dquot+0x68/0xa0 fs/ext4/super.c:5227
>         [<      none      >] dqget+0x305/0x470 fs/quota/dquot.c:891
>         [<      none      >] __dquot_initialize+0x151/0x290 fs/quota/dquot.c:1460
>         [<      none      >] dquot_initialize+0x13/0x20 fs/quota/dquot.c:1511
>         [<      none      >] ext4_mkdir+0x66/0x470 fs/ext4/namei.c:2645
>         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
>         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
>         [<     inline     >] SyS_mkdirat fs/namei.c:3776
>         [<     inline     >] SYSC_mkdir fs/namei.c:3803
>         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
>         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>   
>   -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
>         [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
>         [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
>         [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
>         [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
>         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
>         [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
>         [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
>         [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
>         [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>         [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
>         [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
>         [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
>         [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
>         [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
>         [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
>         [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
>         [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
>         [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
>         [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
>         [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
>         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
>         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
>         [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
>         [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
>         [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
>         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
>         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
>         [<     inline     >] SyS_mkdirat fs/namei.c:3776
>         [<     inline     >] SYSC_mkdir fs/namei.c:3803
>         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
>         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>   
>   other info that might help us debug this:
>   
>    Possible unsafe locking scenario:
>   
>          CPU0                    CPU1
>          ----                    ----
>     lock(&ei->i_data_sem/2);
>                                  lock(&s->s_dquot.dqio_mutex);
>                                  lock(&ei->i_data_sem/2);
>     lock(&s->s_dquot.dqio_mutex);
>   
>    *** DEADLOCK ***
>   
>   5 locks held by mkdir/8458:
>    #0:  (sb_writers#14){.+.+.+}, at: [<     inline     >] sb_start_write ./include/linux/fs.h:1504
>    #0:  (sb_writers#14){.+.+.+}, at: [<ffffffff812e1e24>] mnt_want_write+0x24/0x50 fs/namespace.c:388
>    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<     inline     >] inode_lock_nested ./include/linux/fs.h:731
>    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<ffffffff812cb5e3>] filename_create+0x83/0x160 fs/namei.c:3594
>    #2:  (jbd2_handle){++++..}, at: [<ffffffff813d31b2>] start_this_handle+0x112/0x450 fs/jbd2/transaction.c:361
>    #3:  (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
>    #4:  (dquot_srcu){......}, at: [<     inline     >] srcu_read_lock ./include/linux/srcu.h:237
>    #4:  (dquot_srcu){......}, at: [<ffffffff8133c44b>] __dquot_alloc_space+0xbb/0x2b0 fs/quota/dquot.c:1668
>   
>   stack backtrace:
>   CPU: 3 PID: 8458 Comm: mkdir Not tainted 4.11.0 #1
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-1.fc25 04/01/2014
>   Call Trace:
>   [<     inline     >] __dump_stack lib/dump_stack.c:16
>   [<      none      >] dump_stack+0x86/0xc3 lib/dump_stack.c:52
>   [<      none      >] print_circular_bug+0x1be/0x210 kernel/locking/lockdep.c:1204
>   [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
>   [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
>   [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
>   [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
>   [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
>    ?[<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
>    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>   [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
>   [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
>    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>    ?[<     inline     >] spin_unlock ./include/linux/spinlock.h:339
>    ?[<      none      >] dquot_mark_dquot_dirty+0x57/0xc0 fs/quota/dquot.c:355
>    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
>    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
>    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
>    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
>   [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
>    ?[<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
>   [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
>   [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
>   [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
>   [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
>   [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
>   [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
>    ?[<      none      >] __this_cpu_preempt_check+0x13/0x20 lib/smp_processor_id.c:62
>    ?[<      none      >] __percpu_counter_add+0x85/0xb0 lib/percpu_counter.c:90
>   [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
>   [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
>   [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
>   [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
>    ?[<     inline     >] lock_is_held ./include/linux/lockdep.h:348
>    ?[<      none      >] rcu_read_lock_sched_held+0x4a/0x80 kernel/rcu/update.c:114
>    ?[<     inline     >] trace_kmalloc ./include/trace/events/kmem.h:45
>    ?[<      none      >] __kmalloc+0x2ce/0x300 mm/slub.c:3743
>    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
>    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
>    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
>    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
>    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
>    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
>   [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
>    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
>    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
>   [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
>   [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
>   [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
>   [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
>   [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
>   [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
>   [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
>   [<     inline     >] SYSC_mkdirat fs/namei.c:3792
>   [<     inline     >] SyS_mkdirat fs/namei.c:3776
>   [<     inline     >] SYSC_mkdir fs/namei.c:3803
>   [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
>   [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>   RIP: 0033:0x7fb614c15947
>   RSP: 002b:00007fff31be4938 EFLAGS: 00000216 ORIG_RAX: 0000000000000053
>   RAX: ffffffffffffffda RBX: 00007fff31be4c48 RCX: 00007fb614c15947
>   RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007fff31be6aa5
>   RBP: 0000000000000003 R08: 00000000000001ff R09: 0000564258eadac0
>   R10: 000056425b0a4060 R11: 0000000000000216 R12: 0000000000000000
>   R13: 0000564258eadb00 R14: 0000000000000000 R15: 0000000000000000
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: lockdep splat with v4.11 and generic/386
  2017-05-05  7:59 ` Jan Kara
@ 2017-05-05  8:49   ` Jan Kara
  2017-05-05 16:50     ` Darrick J. Wong
  0 siblings, 1 reply; 5+ messages in thread
From: Jan Kara @ 2017-05-05  8:49 UTC (permalink / raw)
  To: Ross Zwisler; +Cc: linux-ext4, Theodore Ts'o, Jan Kara

On Fri 05-05-17 09:59:23, Jan Kara wrote:
> On Thu 04-05-17 13:38:38, Ross Zwisler wrote:
> > I hit the following lockdep splat during some regression testing today, and
> > was able to reproduce it on vanilla v4.11 without DAX.  My test setup is big
> > memmap PMEM device in a QEMU virtual machine, but I don't think that the block
> > device is important.  I was able to reproduce this easily by running
> > generic/386 in a loop.  Here's the failure:
> 
> Thanks for report! This is false positive caused by a bug in our lockdep
> annotation - locks on quota files (i_data_sem in this case) rank
> differently than locks on normal files - specifically the lock ordering is
> i_data_sem (normal file, directory, ...) -> dqio_mutex -> i_data_sem (quota
> file). Now we do take care to tell lockdep about this by calling
> lockdep_set_quota_inode() in fs/ext4/super.c for quota files. However we
> don't return i_data_sem back to the normal locking class on freeing inode
> and when quota file inode gets reused for something else, it has a wrong
> locking class set which confuses lockdep. I'll send a fix.

Hum, I have the fix implemented but how did you make that test work for
ext4? For me it is not run because:

xfs_quota binary does not support foreign filesystems

and looking at xfs_quota source in the latest git, -f still is not
supported...

								Honza

> >   FSTYP         -- ext4
> >   PLATFORM      -- Linux/x86_64 lorwyn 4.11.0
> >   MKFS_OPTIONS  -- /dev/pmem0p2
> >   MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
> >   
> >   generic/386 1s ... 1s
> >   _check_dmesg: something found in dmesg (see /root/xfstests/results//generic/386.dmesg)
> >   Ran: generic/386
> >   Failures: generic/386
> >   Failed 1 of 1 tests
> > 
> > Here's the lockdep splat, passed through kasan_symbolize.py:
> > 
> >   run fstests generic/386 at 2017-05-04 13:36:35
> >   EXT4-fs (pmem0p2): mounted filesystem with ordered data mode. Opts: acl,user_xattr,quota
> >   
> >   ======================================================
> >   [ INFO: possible circular locking dependency detected ]
> >   4.11.0 #1 Not tainted
> >   -------------------------------------------------------
> >   mkdir/8458 is trying to acquire lock:
> >    (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8133a8bb>] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >   
> >   but task is already holding lock:
> >    (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> >   
> >   which lock already depends on the new lock.
> >   
> >   
> >   the existing dependency chain (in reverse order) is:
> >   
> >   -> #1 (&ei->i_data_sem/2){++++..}:
> >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> >         [<      none      >] down_read+0x43/0xa0 kernel/locking/rwsem.c:23
> >         [<      none      >] ext4_map_blocks+0x2b4/0x5e0 fs/ext4/inode.c:540
> >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> >         [<      none      >] ext4_quota_read+0xce/0x110 fs/ext4/super.c:5470
> >         [<      none      >] read_blk+0x4c/0x60 fs/quota/quota_tree.c:63
> >         [<      none      >] find_tree_dqentry+0x44/0x230 fs/quota/quota_tree.c:579
> >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> >         [<     inline     >] find_dqentry fs/quota/quota_tree.c:602
> >         [<      none      >] qtree_read_dquot+0x12e/0x260 fs/quota/quota_tree.c:622
> >         [<      none      >] v2_read_dquot+0x2e/0x30 fs/quota/quota_v2.c:288
> >         [<      none      >] dquot_acquire+0xe3/0x120 fs/quota/dquot.c:411
> >         [<      none      >] ext4_acquire_dquot+0x68/0xa0 fs/ext4/super.c:5227
> >         [<      none      >] dqget+0x305/0x470 fs/quota/dquot.c:891
> >         [<      none      >] __dquot_initialize+0x151/0x290 fs/quota/dquot.c:1460
> >         [<      none      >] dquot_initialize+0x13/0x20 fs/quota/dquot.c:1511
> >         [<      none      >] ext4_mkdir+0x66/0x470 fs/ext4/namei.c:2645
> >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> >   
> >   -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
> >         [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> >         [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> >         [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> >         [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> >         [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> >         [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> >         [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> >         [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >         [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> >         [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> >         [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> >         [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> >         [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> >         [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> >         [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> >         [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> >         [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> >         [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> >         [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> >         [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> >         [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> >         [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> >   
> >   other info that might help us debug this:
> >   
> >    Possible unsafe locking scenario:
> >   
> >          CPU0                    CPU1
> >          ----                    ----
> >     lock(&ei->i_data_sem/2);
> >                                  lock(&s->s_dquot.dqio_mutex);
> >                                  lock(&ei->i_data_sem/2);
> >     lock(&s->s_dquot.dqio_mutex);
> >   
> >    *** DEADLOCK ***
> >   
> >   5 locks held by mkdir/8458:
> >    #0:  (sb_writers#14){.+.+.+}, at: [<     inline     >] sb_start_write ./include/linux/fs.h:1504
> >    #0:  (sb_writers#14){.+.+.+}, at: [<ffffffff812e1e24>] mnt_want_write+0x24/0x50 fs/namespace.c:388
> >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<     inline     >] inode_lock_nested ./include/linux/fs.h:731
> >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<ffffffff812cb5e3>] filename_create+0x83/0x160 fs/namei.c:3594
> >    #2:  (jbd2_handle){++++..}, at: [<ffffffff813d31b2>] start_this_handle+0x112/0x450 fs/jbd2/transaction.c:361
> >    #3:  (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> >    #4:  (dquot_srcu){......}, at: [<     inline     >] srcu_read_lock ./include/linux/srcu.h:237
> >    #4:  (dquot_srcu){......}, at: [<ffffffff8133c44b>] __dquot_alloc_space+0xbb/0x2b0 fs/quota/dquot.c:1668
> >   
> >   stack backtrace:
> >   CPU: 3 PID: 8458 Comm: mkdir Not tainted 4.11.0 #1
> >   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-1.fc25 04/01/2014
> >   Call Trace:
> >   [<     inline     >] __dump_stack lib/dump_stack.c:16
> >   [<      none      >] dump_stack+0x86/0xc3 lib/dump_stack.c:52
> >   [<      none      >] print_circular_bug+0x1be/0x210 kernel/locking/lockdep.c:1204
> >   [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> >   [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> >   [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> >   [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> >   [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> >    ?[<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >   [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> >   [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >    ?[<     inline     >] spin_unlock ./include/linux/spinlock.h:339
> >    ?[<      none      >] dquot_mark_dquot_dirty+0x57/0xc0 fs/quota/dquot.c:355
> >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> >   [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> >    ?[<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> >   [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> >   [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> >   [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> >   [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> >   [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> >   [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> >    ?[<      none      >] __this_cpu_preempt_check+0x13/0x20 lib/smp_processor_id.c:62
> >    ?[<      none      >] __percpu_counter_add+0x85/0xb0 lib/percpu_counter.c:90
> >   [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> >   [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> >   [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> >   [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> >    ?[<     inline     >] lock_is_held ./include/linux/lockdep.h:348
> >    ?[<      none      >] rcu_read_lock_sched_held+0x4a/0x80 kernel/rcu/update.c:114
> >    ?[<     inline     >] trace_kmalloc ./include/trace/events/kmem.h:45
> >    ?[<      none      >] __kmalloc+0x2ce/0x300 mm/slub.c:3743
> >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> >   [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> >   [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> >   [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> >   [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> >   [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> >   [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> >   [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> >   [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> >   [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> >   [<     inline     >] SyS_mkdirat fs/namei.c:3776
> >   [<     inline     >] SYSC_mkdir fs/namei.c:3803
> >   [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> >   [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> >   RIP: 0033:0x7fb614c15947
> >   RSP: 002b:00007fff31be4938 EFLAGS: 00000216 ORIG_RAX: 0000000000000053
> >   RAX: ffffffffffffffda RBX: 00007fff31be4c48 RCX: 00007fb614c15947
> >   RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007fff31be6aa5
> >   RBP: 0000000000000003 R08: 00000000000001ff R09: 0000564258eadac0
> >   R10: 000056425b0a4060 R11: 0000000000000216 R12: 0000000000000000
> >   R13: 0000564258eadb00 R14: 0000000000000000 R15: 0000000000000000
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: lockdep splat with v4.11 and generic/386
  2017-05-05  8:49   ` Jan Kara
@ 2017-05-05 16:50     ` Darrick J. Wong
  2017-05-09  8:17       ` Jan Kara
  0 siblings, 1 reply; 5+ messages in thread
From: Darrick J. Wong @ 2017-05-05 16:50 UTC (permalink / raw)
  To: Jan Kara; +Cc: Ross Zwisler, linux-ext4, Theodore Ts'o

On Fri, May 05, 2017 at 10:49:50AM +0200, Jan Kara wrote:
> On Fri 05-05-17 09:59:23, Jan Kara wrote:
> > On Thu 04-05-17 13:38:38, Ross Zwisler wrote:
> > > I hit the following lockdep splat during some regression testing today, and
> > > was able to reproduce it on vanilla v4.11 without DAX.  My test setup is big
> > > memmap PMEM device in a QEMU virtual machine, but I don't think that the block
> > > device is important.  I was able to reproduce this easily by running
> > > generic/386 in a loop.  Here's the failure:
> > 
> > Thanks for report! This is false positive caused by a bug in our lockdep
> > annotation - locks on quota files (i_data_sem in this case) rank
> > differently than locks on normal files - specifically the lock ordering is
> > i_data_sem (normal file, directory, ...) -> dqio_mutex -> i_data_sem (quota
> > file). Now we do take care to tell lockdep about this by calling
> > lockdep_set_quota_inode() in fs/ext4/super.c for quota files. However we
> > don't return i_data_sem back to the normal locking class on freeing inode
> > and when quota file inode gets reused for something else, it has a wrong
> > locking class set which confuses lockdep. I'll send a fix.
> 
> Hum, I have the fix implemented but how did you make that test work for
> ext4? For me it is not run because:
> 
> xfs_quota binary does not support foreign filesystems
> 
> and looking at xfs_quota source in the latest git, -f still is not
> supported...

What git repo are you using?  xfs_quota has had -f since xfsprogs 4.8.

xfsprogs upstream lives at:
git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git

(XFS moved off SGI infrastructure last October.)

--D

> 
> 								Honza
> 
> > >   FSTYP         -- ext4
> > >   PLATFORM      -- Linux/x86_64 lorwyn 4.11.0
> > >   MKFS_OPTIONS  -- /dev/pmem0p2
> > >   MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
> > >   
> > >   generic/386 1s ... 1s
> > >   _check_dmesg: something found in dmesg (see /root/xfstests/results//generic/386.dmesg)
> > >   Ran: generic/386
> > >   Failures: generic/386
> > >   Failed 1 of 1 tests
> > > 
> > > Here's the lockdep splat, passed through kasan_symbolize.py:
> > > 
> > >   run fstests generic/386 at 2017-05-04 13:36:35
> > >   EXT4-fs (pmem0p2): mounted filesystem with ordered data mode. Opts: acl,user_xattr,quota
> > >   
> > >   ======================================================
> > >   [ INFO: possible circular locking dependency detected ]
> > >   4.11.0 #1 Not tainted
> > >   -------------------------------------------------------
> > >   mkdir/8458 is trying to acquire lock:
> > >    (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8133a8bb>] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >   
> > >   but task is already holding lock:
> > >    (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> > >   
> > >   which lock already depends on the new lock.
> > >   
> > >   
> > >   the existing dependency chain (in reverse order) is:
> > >   
> > >   -> #1 (&ei->i_data_sem/2){++++..}:
> > >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > >         [<      none      >] down_read+0x43/0xa0 kernel/locking/rwsem.c:23
> > >         [<      none      >] ext4_map_blocks+0x2b4/0x5e0 fs/ext4/inode.c:540
> > >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > >         [<      none      >] ext4_quota_read+0xce/0x110 fs/ext4/super.c:5470
> > >         [<      none      >] read_blk+0x4c/0x60 fs/quota/quota_tree.c:63
> > >         [<      none      >] find_tree_dqentry+0x44/0x230 fs/quota/quota_tree.c:579
> > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > >         [<     inline     >] find_dqentry fs/quota/quota_tree.c:602
> > >         [<      none      >] qtree_read_dquot+0x12e/0x260 fs/quota/quota_tree.c:622
> > >         [<      none      >] v2_read_dquot+0x2e/0x30 fs/quota/quota_v2.c:288
> > >         [<      none      >] dquot_acquire+0xe3/0x120 fs/quota/dquot.c:411
> > >         [<      none      >] ext4_acquire_dquot+0x68/0xa0 fs/ext4/super.c:5227
> > >         [<      none      >] dqget+0x305/0x470 fs/quota/dquot.c:891
> > >         [<      none      >] __dquot_initialize+0x151/0x290 fs/quota/dquot.c:1460
> > >         [<      none      >] dquot_initialize+0x13/0x20 fs/quota/dquot.c:1511
> > >         [<      none      >] ext4_mkdir+0x66/0x470 fs/ext4/namei.c:2645
> > >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > >   
> > >   -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
> > >         [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> > >         [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> > >         [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> > >         [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> > >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > >         [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> > >         [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> > >         [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > >         [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >         [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> > >         [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> > >         [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> > >         [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> > >         [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> > >         [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> > >         [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> > >         [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> > >         [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> > >         [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> > >         [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> > >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > >         [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> > >         [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> > >         [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> > >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > >   
> > >   other info that might help us debug this:
> > >   
> > >    Possible unsafe locking scenario:
> > >   
> > >          CPU0                    CPU1
> > >          ----                    ----
> > >     lock(&ei->i_data_sem/2);
> > >                                  lock(&s->s_dquot.dqio_mutex);
> > >                                  lock(&ei->i_data_sem/2);
> > >     lock(&s->s_dquot.dqio_mutex);
> > >   
> > >    *** DEADLOCK ***
> > >   
> > >   5 locks held by mkdir/8458:
> > >    #0:  (sb_writers#14){.+.+.+}, at: [<     inline     >] sb_start_write ./include/linux/fs.h:1504
> > >    #0:  (sb_writers#14){.+.+.+}, at: [<ffffffff812e1e24>] mnt_want_write+0x24/0x50 fs/namespace.c:388
> > >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<     inline     >] inode_lock_nested ./include/linux/fs.h:731
> > >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<ffffffff812cb5e3>] filename_create+0x83/0x160 fs/namei.c:3594
> > >    #2:  (jbd2_handle){++++..}, at: [<ffffffff813d31b2>] start_this_handle+0x112/0x450 fs/jbd2/transaction.c:361
> > >    #3:  (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> > >    #4:  (dquot_srcu){......}, at: [<     inline     >] srcu_read_lock ./include/linux/srcu.h:237
> > >    #4:  (dquot_srcu){......}, at: [<ffffffff8133c44b>] __dquot_alloc_space+0xbb/0x2b0 fs/quota/dquot.c:1668
> > >   
> > >   stack backtrace:
> > >   CPU: 3 PID: 8458 Comm: mkdir Not tainted 4.11.0 #1
> > >   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-1.fc25 04/01/2014
> > >   Call Trace:
> > >   [<     inline     >] __dump_stack lib/dump_stack.c:16
> > >   [<      none      >] dump_stack+0x86/0xc3 lib/dump_stack.c:52
> > >   [<      none      >] print_circular_bug+0x1be/0x210 kernel/locking/lockdep.c:1204
> > >   [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> > >   [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> > >   [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> > >   [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> > >   [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > >    ?[<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >   [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> > >   [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >    ?[<     inline     >] spin_unlock ./include/linux/spinlock.h:339
> > >    ?[<      none      >] dquot_mark_dquot_dirty+0x57/0xc0 fs/quota/dquot.c:355
> > >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> > >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> > >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> > >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> > >   [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > >    ?[<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > >   [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > >   [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> > >   [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> > >   [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> > >   [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> > >   [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> > >    ?[<      none      >] __this_cpu_preempt_check+0x13/0x20 lib/smp_processor_id.c:62
> > >    ?[<      none      >] __percpu_counter_add+0x85/0xb0 lib/percpu_counter.c:90
> > >   [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> > >   [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> > >   [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> > >   [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> > >    ?[<     inline     >] lock_is_held ./include/linux/lockdep.h:348
> > >    ?[<      none      >] rcu_read_lock_sched_held+0x4a/0x80 kernel/rcu/update.c:114
> > >    ?[<     inline     >] trace_kmalloc ./include/trace/events/kmem.h:45
> > >    ?[<      none      >] __kmalloc+0x2ce/0x300 mm/slub.c:3743
> > >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> > >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> > >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> > >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> > >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> > >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> > >   [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> > >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> > >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> > >   [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> > >   [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > >   [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > >   [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> > >   [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> > >   [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> > >   [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > >   [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > >   [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > >   [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > >   [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > >   [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > >   RIP: 0033:0x7fb614c15947
> > >   RSP: 002b:00007fff31be4938 EFLAGS: 00000216 ORIG_RAX: 0000000000000053
> > >   RAX: ffffffffffffffda RBX: 00007fff31be4c48 RCX: 00007fb614c15947
> > >   RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007fff31be6aa5
> > >   RBP: 0000000000000003 R08: 00000000000001ff R09: 0000564258eadac0
> > >   R10: 000056425b0a4060 R11: 0000000000000216 R12: 0000000000000000
> > >   R13: 0000564258eadb00 R14: 0000000000000000 R15: 0000000000000000
> > -- 
> > Jan Kara <jack@suse.com>
> > SUSE Labs, CR
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: lockdep splat with v4.11 and generic/386
  2017-05-05 16:50     ` Darrick J. Wong
@ 2017-05-09  8:17       ` Jan Kara
  0 siblings, 0 replies; 5+ messages in thread
From: Jan Kara @ 2017-05-09  8:17 UTC (permalink / raw)
  To: Darrick J. Wong; +Cc: Jan Kara, Ross Zwisler, linux-ext4, Theodore Ts'o

On Fri 05-05-17 09:50:35, Darrick J. Wong wrote:
> On Fri, May 05, 2017 at 10:49:50AM +0200, Jan Kara wrote:
> > On Fri 05-05-17 09:59:23, Jan Kara wrote:
> > > On Thu 04-05-17 13:38:38, Ross Zwisler wrote:
> > > > I hit the following lockdep splat during some regression testing today, and
> > > > was able to reproduce it on vanilla v4.11 without DAX.  My test setup is big
> > > > memmap PMEM device in a QEMU virtual machine, but I don't think that the block
> > > > device is important.  I was able to reproduce this easily by running
> > > > generic/386 in a loop.  Here's the failure:
> > > 
> > > Thanks for report! This is false positive caused by a bug in our lockdep
> > > annotation - locks on quota files (i_data_sem in this case) rank
> > > differently than locks on normal files - specifically the lock ordering is
> > > i_data_sem (normal file, directory, ...) -> dqio_mutex -> i_data_sem (quota
> > > file). Now we do take care to tell lockdep about this by calling
> > > lockdep_set_quota_inode() in fs/ext4/super.c for quota files. However we
> > > don't return i_data_sem back to the normal locking class on freeing inode
> > > and when quota file inode gets reused for something else, it has a wrong
> > > locking class set which confuses lockdep. I'll send a fix.
> > 
> > Hum, I have the fix implemented but how did you make that test work for
> > ext4? For me it is not run because:
> > 
> > xfs_quota binary does not support foreign filesystems
> > 
> > and looking at xfs_quota source in the latest git, -f still is not
> > supported...
> 
> What git repo are you using?  xfs_quota has had -f since xfsprogs 4.8.
> 
> xfsprogs upstream lives at:
> git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git
> 
> (XFS moved off SGI infrastructure last October.)

Ha! Good point. My git repo still points to SGI. I know the repo moved but
didn't realize it could be the problem when looking at this :) Thanks!

								Honza


> > > >   FSTYP         -- ext4
> > > >   PLATFORM      -- Linux/x86_64 lorwyn 4.11.0
> > > >   MKFS_OPTIONS  -- /dev/pmem0p2
> > > >   MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
> > > >   
> > > >   generic/386 1s ... 1s
> > > >   _check_dmesg: something found in dmesg (see /root/xfstests/results//generic/386.dmesg)
> > > >   Ran: generic/386
> > > >   Failures: generic/386
> > > >   Failed 1 of 1 tests
> > > > 
> > > > Here's the lockdep splat, passed through kasan_symbolize.py:
> > > > 
> > > >   run fstests generic/386 at 2017-05-04 13:36:35
> > > >   EXT4-fs (pmem0p2): mounted filesystem with ordered data mode. Opts: acl,user_xattr,quota
> > > >   
> > > >   ======================================================
> > > >   [ INFO: possible circular locking dependency detected ]
> > > >   4.11.0 #1 Not tainted
> > > >   -------------------------------------------------------
> > > >   mkdir/8458 is trying to acquire lock:
> > > >    (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8133a8bb>] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >   
> > > >   but task is already holding lock:
> > > >    (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> > > >   
> > > >   which lock already depends on the new lock.
> > > >   
> > > >   
> > > >   the existing dependency chain (in reverse order) is:
> > > >   
> > > >   -> #1 (&ei->i_data_sem/2){++++..}:
> > > >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > > >         [<      none      >] down_read+0x43/0xa0 kernel/locking/rwsem.c:23
> > > >         [<      none      >] ext4_map_blocks+0x2b4/0x5e0 fs/ext4/inode.c:540
> > > >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > > >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > > >         [<      none      >] ext4_quota_read+0xce/0x110 fs/ext4/super.c:5470
> > > >         [<      none      >] read_blk+0x4c/0x60 fs/quota/quota_tree.c:63
> > > >         [<      none      >] find_tree_dqentry+0x44/0x230 fs/quota/quota_tree.c:579
> > > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > > >         [<      none      >] find_tree_dqentry+0x1ad/0x230 fs/quota/quota_tree.c:590
> > > >         [<     inline     >] find_dqentry fs/quota/quota_tree.c:602
> > > >         [<      none      >] qtree_read_dquot+0x12e/0x260 fs/quota/quota_tree.c:622
> > > >         [<      none      >] v2_read_dquot+0x2e/0x30 fs/quota/quota_v2.c:288
> > > >         [<      none      >] dquot_acquire+0xe3/0x120 fs/quota/dquot.c:411
> > > >         [<      none      >] ext4_acquire_dquot+0x68/0xa0 fs/ext4/super.c:5227
> > > >         [<      none      >] dqget+0x305/0x470 fs/quota/dquot.c:891
> > > >         [<      none      >] __dquot_initialize+0x151/0x290 fs/quota/dquot.c:1460
> > > >         [<      none      >] dquot_initialize+0x13/0x20 fs/quota/dquot.c:1511
> > > >         [<      none      >] ext4_mkdir+0x66/0x470 fs/ext4/namei.c:2645
> > > >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > > >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > > >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > > >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > > >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > > >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > > >   
> > > >   -> #0 (&s->s_dquot.dqio_mutex){+.+...}:
> > > >         [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> > > >         [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> > > >         [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> > > >         [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> > > >         [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > > >         [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> > > >         [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> > > >         [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > > >         [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >         [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> > > >         [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> > > >         [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> > > >         [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> > > >         [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> > > >         [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> > > >         [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> > > >         [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> > > >         [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> > > >         [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> > > >         [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> > > >         [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > > >         [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > > >         [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> > > >         [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> > > >         [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> > > >         [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > > >         [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > > >         [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > > >         [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > > >         [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > > >         [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > > >   
> > > >   other info that might help us debug this:
> > > >   
> > > >    Possible unsafe locking scenario:
> > > >   
> > > >          CPU0                    CPU1
> > > >          ----                    ----
> > > >     lock(&ei->i_data_sem/2);
> > > >                                  lock(&s->s_dquot.dqio_mutex);
> > > >                                  lock(&ei->i_data_sem/2);
> > > >     lock(&s->s_dquot.dqio_mutex);
> > > >   
> > > >    *** DEADLOCK ***
> > > >   
> > > >   5 locks held by mkdir/8458:
> > > >    #0:  (sb_writers#14){.+.+.+}, at: [<     inline     >] sb_start_write ./include/linux/fs.h:1504
> > > >    #0:  (sb_writers#14){.+.+.+}, at: [<ffffffff812e1e24>] mnt_want_write+0x24/0x50 fs/namespace.c:388
> > > >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<     inline     >] inode_lock_nested ./include/linux/fs.h:731
> > > >    #1:  (&type->i_mutex_dir_key#5/1){+.+.+.}, at: [<ffffffff812cb5e3>] filename_create+0x83/0x160 fs/namei.c:3594
> > > >    #2:  (jbd2_handle){++++..}, at: [<ffffffff813d31b2>] start_this_handle+0x112/0x450 fs/jbd2/transaction.c:361
> > > >    #3:  (&ei->i_data_sem/2){++++..}, at: [<ffffffff81371ba1>] ext4_map_blocks+0x151/0x5e0 fs/ext4/inode.c:612
> > > >    #4:  (dquot_srcu){......}, at: [<     inline     >] srcu_read_lock ./include/linux/srcu.h:237
> > > >    #4:  (dquot_srcu){......}, at: [<ffffffff8133c44b>] __dquot_alloc_space+0xbb/0x2b0 fs/quota/dquot.c:1668
> > > >   
> > > >   stack backtrace:
> > > >   CPU: 3 PID: 8458 Comm: mkdir Not tainted 4.11.0 #1
> > > >   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-1.fc25 04/01/2014
> > > >   Call Trace:
> > > >   [<     inline     >] __dump_stack lib/dump_stack.c:16
> > > >   [<      none      >] dump_stack+0x86/0xc3 lib/dump_stack.c:52
> > > >   [<      none      >] print_circular_bug+0x1be/0x210 kernel/locking/lockdep.c:1204
> > > >   [<     inline     >] check_prev_add kernel/locking/lockdep.c:1830
> > > >   [<     inline     >] check_prevs_add kernel/locking/lockdep.c:1940
> > > >   [<     inline     >] validate_chain kernel/locking/lockdep.c:2267
> > > >   [<      none      >] __lock_acquire+0x12c2/0x1450 kernel/locking/lockdep.c:3347
> > > >   [<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > > >    ?[<      none      >] lock_acquire+0xea/0x1f0 kernel/locking/lockdep.c:3762
> > > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >   [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:756
> > > >   [<      none      >] __mutex_lock+0x8d/0xa80 kernel/locking/mutex.c:893
> > > >    ?[<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >    ?[<     inline     >] spin_unlock ./include/linux/spinlock.h:339
> > > >    ?[<      none      >] dquot_mark_dquot_dirty+0x57/0xc0 fs/quota/dquot.c:355
> > > >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> > > >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> > > >    ?[<     inline     >] __ext4_journal_start fs/ext4/ext4_jbd2.h:318
> > > >    ?[<      none      >] ext4_write_dquot+0x5c/0xa0 fs/ext4/super.c:5207
> > > >   [<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > > >    ?[<      none      >] mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:908
> > > >   [<      none      >] dquot_commit+0x2b/0xd0 fs/quota/dquot.c:453
> > > >   [<      none      >] ext4_write_dquot+0x6f/0xa0 fs/ext4/super.c:5211
> > > >   [<      none      >] ext4_mark_dquot_dirty+0x3f/0x60 fs/ext4/super.c:5262
> > > >   [<     inline     >] mark_dquot_dirty fs/quota/dquot.c:337
> > > >   [<     inline     >] mark_all_dquot_dirty fs/quota/dquot.c:369
> > > >   [<      none      >] __dquot_alloc_space+0x275/0x2b0 fs/quota/dquot.c:1693
> > > >    ?[<      none      >] __this_cpu_preempt_check+0x13/0x20 lib/smp_processor_id.c:62
> > > >    ?[<      none      >] __percpu_counter_add+0x85/0xb0 lib/percpu_counter.c:90
> > > >   [<     inline     >] dquot_alloc_space_nodirty ./include/linux/quotaops.h:284
> > > >   [<     inline     >] dquot_alloc_space ./include/linux/quotaops.h:297
> > > >   [<     inline     >] dquot_alloc_block ./include/linux/quotaops.h:321
> > > >   [<      none      >] ext4_mb_new_blocks+0x113/0x1110 fs/ext4/mballoc.c:4487
> > > >    ?[<     inline     >] lock_is_held ./include/linux/lockdep.h:348
> > > >    ?[<      none      >] rcu_read_lock_sched_held+0x4a/0x80 kernel/rcu/update.c:114
> > > >    ?[<     inline     >] trace_kmalloc ./include/trace/events/kmem.h:45
> > > >    ?[<      none      >] __kmalloc+0x2ce/0x300 mm/slub.c:3743
> > > >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> > > >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> > > >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> > > >    ?[<     inline     >] kmalloc ./include/linux/slab.h:495
> > > >    ?[<     inline     >] kzalloc ./include/linux/slab.h:663
> > > >    ?[<      none      >] ext4_find_extent+0x295/0x2d0 fs/ext4/extents.c:894
> > > >   [<      none      >] ext4_ext_map_blocks+0xe2f/0x20d0 fs/ext4/extents.c:4478
> > > >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> > > >    ?[<      none      >] ext4_map_blocks+0x331/0x5e0 fs/ext4/inode.c:571
> > > >   [<      none      >] ext4_map_blocks+0x175/0x5e0 fs/ext4/inode.c:619
> > > >   [<      none      >] ext4_getblk+0x51/0x1a0 fs/ext4/inode.c:949
> > > >   [<      none      >] ext4_bread+0x22/0xc0 fs/ext4/inode.c:999
> > > >   [<      none      >] ext4_append+0x4d/0xe0 fs/ext4/namei.c:64
> > > >   [<     inline     >] ext4_init_new_dir fs/ext4/namei.c:2615
> > > >   [<      none      >] ext4_mkdir+0x266/0x470 fs/ext4/namei.c:2662
> > > >   [<      none      >] vfs_mkdir+0x119/0x1c0 fs/namei.c:3769
> > > >   [<     inline     >] SYSC_mkdirat fs/namei.c:3792
> > > >   [<     inline     >] SyS_mkdirat fs/namei.c:3776
> > > >   [<     inline     >] SYSC_mkdir fs/namei.c:3803
> > > >   [<      none      >] SyS_mkdir+0x7a/0xf0 fs/namei.c:3801
> > > >   [<      none      >] entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
> > > >   RIP: 0033:0x7fb614c15947
> > > >   RSP: 002b:00007fff31be4938 EFLAGS: 00000216 ORIG_RAX: 0000000000000053
> > > >   RAX: ffffffffffffffda RBX: 00007fff31be4c48 RCX: 00007fb614c15947
> > > >   RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007fff31be6aa5
> > > >   RBP: 0000000000000003 R08: 00000000000001ff R09: 0000564258eadac0
> > > >   R10: 000056425b0a4060 R11: 0000000000000216 R12: 0000000000000000
> > > >   R13: 0000564258eadb00 R14: 0000000000000000 R15: 0000000000000000
> > > -- 
> > > Jan Kara <jack@suse.com>
> > > SUSE Labs, CR
> > -- 
> > Jan Kara <jack@suse.com>
> > SUSE Labs, CR
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2017-05-09  8:25 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-05-04 19:38 lockdep splat with v4.11 and generic/386 Ross Zwisler
2017-05-05  7:59 ` Jan Kara
2017-05-05  8:49   ` Jan Kara
2017-05-05 16:50     ` Darrick J. Wong
2017-05-09  8:17       ` Jan Kara

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).