* [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock)
@ 2014-06-10 12:04 Tetsuo Handa
2014-06-10 14:34 ` Paul E. McKenney
0 siblings, 1 reply; 5+ messages in thread
From: Tetsuo Handa @ 2014-06-10 12:04 UTC (permalink / raw)
To: viro, paulmck; +Cc: torvalds, linux-fsdevel, linux-kernel
Hello.
I noticed that commit 59a3d4c363 "Merge branch 'core-rcu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next" causes
an oops when rebooting a system. Do you have any clue?
Config is at http://I-love.SAKURA.ne.jp/tmp/config-pre-3.16-rc1 .
---------- console log start ----------
EXT4-fs (sda1): re-mounted. Opts: (null)
BUG: unable to handle kernel paging request at ffff88004c0d2f40
IP: [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
PGD 2d32067 PUD 2d35067 PMD 7fb83067 PTE 800000004c0d2060
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: fuse ipv6 vhost_net macvtap macvlan vhost tun ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore e1000 dm_mod sg i2c_piix4 i2c_core parport_pc parport shpchp ext4(E) jbd2(E) mbcache(E) crc16(E) sd_mod(E) crc_t10dif(E) sr_mod(E) cdrom(E) mptspi(E) mptscsih(E) mptbase(E) scsi_transport_spi(E) pata_acpi(E) ata_generic(E) ata_piix(E)
CPU: 2 PID: 2277 Comm: mount Tainted: G E 3.15.0-rc8-04322-g59a3d4c #378
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff880046d904d0 ti: ffff8800415a0000 task.ti: ffff8800415a0000
RIP: 0010:[<ffffffff810bc269>] [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
RSP: 0018:ffff8800415a3d38 EFLAGS: 00010282
RAX: ffff88007962c080 RBX: ffff88004c0d2f40 RCX: 0000000000000002
RDX: 0000000000000018 RSI: ffff88004c0d4f58 RDI: ffff88004c0d2f40
RBP: ffff8800415a3d38 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88004c0d4eb0
R13: ffff88004c0d4fb0 R14: ffff8800415a3dd8 R15: ffff88004c0d4f40
FS: 00007fe45be4a7e0(0000) GS:ffff880079600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff88004c0d2f40 CR3: 00000000494b6000 CR4: 00000000000407e0
Stack:
ffff8800415a3d68 ffffffff816380ee ffffffff811e3a64 ffffffff813117c9
ffff88004d88deb0 ffff88004c0d2eb0 ffff8800415a3db8 ffffffff811e3a80
ffff88004c0dafb0 ffff8800708e8fc8 ffff880046d904d0 0000000000000400
Call Trace:
[<ffffffff816380ee>] _raw_spin_trylock+0x1e/0x70
[<ffffffff811e3a64>] ? shrink_dentry_list+0x44/0x370
[<ffffffff813117c9>] ? lockref_put_or_lock+0x29/0x40
[<ffffffff811e3a80>] shrink_dentry_list+0x60/0x370
[<ffffffff811e3ec9>] shrink_dcache_sb+0xd9/0x120
[<ffffffff811cc252>] do_remount_sb+0x52/0x1a0
[<ffffffff811edf8d>] ? do_remount+0x9d/0x130
[<ffffffff811ee015>] do_remount+0x125/0x130
[<ffffffff811f13f2>] do_mount+0x1b2/0x210
[<ffffffff8117e92b>] ? strndup_user+0x5b/0x80
[<ffffffff811f14e0>] SyS_mount+0x90/0xe0
[<ffffffff81642992>] system_call_fastpath+0x16/0x1b
Code: ff ff ff 31 ff e8 28 ef 01 00 e9 41 ff ff ff 4c 89 ef e8 cb aa fa ff e9 44 ff ff ff 90 90 90 90 90 90 55 48 89 e5 66 66 66 66 90 <8b> 17 89 d0 c1 e8 10 66 39 c2 74 0b 31 c0 c9 c3 0f 1f 80 00 00
RIP [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
RSP <ffff8800415a3d38>
CR2: ffff88004c0d2f40
---[ end trace 98409f38f5f1bb7f ]---
BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
in_atomic(): 1, irqs_disabled(): 1, pid: 2277, name: mount
INFO: lockdep is turned off.
irq event stamp: 69452
hardirqs last enabled at (69451): [<ffffffff810d9c6c>] __call_rcu.clone.0+0x14c/0x330
hardirqs last disabled at (69452): [<ffffffff81639753>] error_sti+0x5/0x6
softirqs last enabled at (68978): [<ffffffff8105b691>] __do_softirq+0x1e1/0x420
softirqs last disabled at (68969): [<ffffffff8105ba05>] irq_exit+0xc5/0xd0
CPU: 2 PID: 2277 Comm: mount Tainted: G D E 3.15.0-rc8-04322-g59a3d4c #378
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
ffffffff81a0d7a3 ffff8800415a3998 ffffffff8163284a ffff8800415a3998
ffff880046d904d0 ffff8800415a39c8 ffffffff8108ce85 ffff880046d904d0
ffff880047415eb8 0000000000000009 0000000000000009 ffff8800415a39f8
Call Trace:
[<ffffffff8163284a>] dump_stack+0x51/0x6f
[<ffffffff8108ce85>] __might_sleep+0x175/0x230
[<ffffffff81637026>] down_read+0x26/0xa0
[<ffffffff8106aa44>] exit_signals+0x24/0x140
[<ffffffff81058bb5>] do_exit+0xb5/0x570
[<ffffffff8163a161>] oops_end+0xa1/0xf0
[<ffffffff81045f5e>] no_context+0x12e/0x200
[<ffffffff8104615d>] __bad_area_nosemaphore+0x12d/0x230
[<ffffffff810b70b5>] ? __lock_acquire+0x445/0x600
[<ffffffff81046273>] bad_area_nosemaphore+0x13/0x20
[<ffffffff8163cdc1>] __do_page_fault+0xe1/0x4c0
[<ffffffff81638d90>] ? _raw_spin_unlock_irqrestore+0x40/0x70
[<ffffffff8131145d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[<ffffffff8163d1fc>] do_page_fault+0xc/0x10
[<ffffffff81639552>] page_fault+0x22/0x30
[<ffffffff810bc269>] ? do_raw_spin_trylock+0x9/0x50
[<ffffffff816380ee>] _raw_spin_trylock+0x1e/0x70
[<ffffffff811e3a64>] ? shrink_dentry_list+0x44/0x370
[<ffffffff813117c9>] ? lockref_put_or_lock+0x29/0x40
[<ffffffff811e3a80>] shrink_dentry_list+0x60/0x370
[<ffffffff811e3ec9>] shrink_dcache_sb+0xd9/0x120
[<ffffffff811cc252>] do_remount_sb+0x52/0x1a0
[<ffffffff811edf8d>] ? do_remount+0x9d/0x130
[<ffffffff811ee015>] do_remount+0x125/0x130
[<ffffffff811f13f2>] do_mount+0x1b2/0x210
[<ffffffff8117e92b>] ? strndup_user+0x5b/0x80
[<ffffffff811f14e0>] SyS_mount+0x90/0xe0
[<ffffffff81642992>] system_call_fastpath+0x16/0x1b
note: mount[2277] exited with preempt_count 2
---------- console log end ----------
---------- bisect log start ----------
# bad: [776edb59317ada867dfcddde40b55648beeb0078] Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next
# good: [26fc9cd200ec839e0b3095e05ae018f27314e7aa] kernfs: move the last knowledge of sysfs out from kernfs
# good: [f9c78b2be2cac2a7a397d489275e7d9f9ae785f2] block: move bio.c and bio-integrity.c from fs/ to block/
# good: [681a2895486243a82547d8c9f53043eb54b53da0] Merge branch 'for-3.16/core' of git://git.kernel.dk/linux-block into next
# good: [5da77761e6fd51f633b4f31051c4f839e01c29c0] Merge tag 'driver-core-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core into next
# good: [4e857c58efeb99393cba5a5d0d8ec7117183137c] arch: Mass conversion of smp_mb__*()
# good: [bf8102228a8bf053051f311e5486042fe0542894] intel_pstate: Improve initial busy calculation
# good: [192a98e280e560510a62aca8cfa83b4ae7c095bb] ALSA: hda/realtek - Fix COEF widget NID for ALC260 replacer fixup
# good: [1860e379875dfe7271c649058aeddffe5afd9d0d] Linux 3.15
# good: [455c6fdbd219161bd09b1165f11699d6d73de11c] Linux 3.14
# good: [d8ec26d7f8287f5788a494f56e8814210f0e64be] Linux 3.13
# good: [5e01dc7b26d9f24f39abace5da98ccbd6a5ceb52] Linux 3.12
# good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
# good: [8bb495e3f02401ee6f76d1b1d77f3ac9f079e376] Linux 3.10
# good: [c1be5a5b1b355d40e6cf79cc979eb66dafa24ad1] Linux 3.9
# good: [19f949f52599ba7c3f67a5897ac6be14bfcb1200] Linux 3.8
# good: [29594404d7fe73cd80eaa4ee8c43dcc53970c60e] Linux 3.7
# good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
# good: [c16fa4f2ad19908a47c63d8fa436a1178438c7e7] Linux 3.3
# good: [805a6af8dba5dfdd35ec35dc52ec0122400b2610] Linux 3.2
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1
# good: [02f8c6aee8df3cdc935e9bdd4f2d020306035dbe] Linux 3.0
git bisect start '776edb59317ada867dfcddde40b55648beeb0078' '26fc9cd200ec839e0b3095e05ae018f27314e7aa' 'f9c78b2be2cac2a7a397d489275e7d9f9ae785f2' '681a2895486243a82547d8c9f53043eb54b53da0' '5da77761e6fd51f633b4f31051c4f839e01c29c0' '4e857c58efeb99393cba5a5d0d8ec7117183137c' 'bf8102228a8bf053051f311e5486042fe0542894' '192a98e280e560510a62aca8cfa83b4ae7c095bb' 'v3.15' 'v3.14' 'v3.13' 'v3.12' 'v3.11' 'v3.10' 'v3.9' 'v3.8' 'v3.7' 'v3.6' 'v3.5' 'v3.4' 'v3.3' 'v3.2' 'v3.1' 'v3.0'
# good: [539b4f72d0da83e4d20df7cfc27dc49c00316940] staging/rtl8192u: Put EXPORT_SYMBOL just after the exported function
git bisect good 539b4f72d0da83e4d20df7cfc27dc49c00316940
# good: [49eb7b0750d9483c74e9c14ae6ea1e9d62481c3c] Merge tag 'tty-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty into next
git bisect good 49eb7b0750d9483c74e9c14ae6ea1e9d62481c3c
# good: [8f5759aeb88a47448cd92ab55a016d013b154a98] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux into next
git bisect good 8f5759aeb88a47448cd92ab55a016d013b154a98
# good: [de6b25de4584febbe1808bc782734ae163b22c26] Merge tag 'regmap-v3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap into next
git bisect good de6b25de4584febbe1808bc782734ae163b22c26
# good: [312c76f1a3989b8d0c0c13fee765bb2c41f2d114] Merge tag 'regulator-v3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator into next
git bisect good 312c76f1a3989b8d0c0c13fee765bb2c41f2d114
# good: [48d684fdad83d7525a557e6ff9c37811b6a9947b] rcutorture: Run rcu_torture_writer at normal priority
git bisect good 48d684fdad83d7525a557e6ff9c37811b6a9947b
# good: [a5d6d3a1b00a0ad88f07c3a727c79b27915278e3] softirq: A single rcu_bh_qs() call per softirq set is enough
git bisect good a5d6d3a1b00a0ad88f07c3a727c79b27915278e3
# good: [61f38db3e3c0e4c3be0858750e2cabeadaecac0c] rcu: Provide API to suppress stall warnings while sysrc runs
git bisect good 61f38db3e3c0e4c3be0858750e2cabeadaecac0c
# good: [f70e3c4f8b6ab61f713e040822ec51f5de498146] CMA: correct unlock target
git bisect good f70e3c4f8b6ab61f713e040822ec51f5de498146
# bad: [59a3d4c3631e553357b7305dc09db1990aa6757c] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next
git bisect bad 59a3d4c3631e553357b7305dc09db1990aa6757c
# good: [e14505a8d50882ff3bdd4b791b14d90a0881fa4d] Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
git bisect good e14505a8d50882ff3bdd4b791b14d90a0881fa4d
# good: [ff806d034ef8e9a95ff0b0532104dd65332e446b] Merge branch 'for-v3.16' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping into next
git bisect good ff806d034ef8e9a95ff0b0532104dd65332e446b
---------- bisect log end ----------
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock)
2014-06-10 12:04 [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock) Tetsuo Handa
@ 2014-06-10 14:34 ` Paul E. McKenney
2014-06-11 11:35 ` Tetsuo Handa
0 siblings, 1 reply; 5+ messages in thread
From: Paul E. McKenney @ 2014-06-10 14:34 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: viro, torvalds, linux-fsdevel, linux-kernel
On Tue, Jun 10, 2014 at 09:04:51PM +0900, Tetsuo Handa wrote:
> Hello.
>
> I noticed that commit 59a3d4c363 "Merge branch 'core-rcu-for-linus' of
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next" causes
> an oops when rebooting a system. Do you have any clue?
Can't say that I do. However, the commits that might have made some
difference in how RCU operates are the following. I have my doubts
(listed below) about their being the cause, but if the oops is reproducible,
why not try reverting each and see what happens? Or just bisect within
the core-rcu-for-linus branches?
365187fbc04f: Update cpu_needs_another_gp() for futures from non-NOCB CPUs
(Seems unlikely, but you never know...)
48a7639ce80c: Make callers awaken grace-period kthread
(I could see this maybe causing a hang, but not a panic.)
595f3900f6b4: Replace NR_CPUS with nr_cpu_ids
(Hard to imagine, but if nr_cpu_ids gets set late or something...)
495aa969dbae: Consolidate kfree_call_rcu() to use rcu_state pointer
(Seems unlikely, but you never know...)
a381d757d93f: Merge rcu_sched_force_quiescent_state() with rcu_force_quiescent_state()
(Seems unlikely, but you never know...)
5057f55e543b: Bind RCU grace-period kthreads if NO_HZ_FULL
(Would seem more likely if NO_HZ_FULL=y in your .config.)
becb41bfe054: Make large and small sysidle systems use same state machine
(Would seem more likely if NO_HZ_FULL_SYSIDLE=y in your .config.)
fa07a58f71ee: Replace __this_cpu_ptr() uses with raw_cpu_ptr()
(Seems unlikely, but you never know...)
a5d6d3a1b00a: A single rcu_bh_qs() call per softirq set is enough
(Seems unlikely, but you never know...)
Thanx, Paul
> Config is at http://I-love.SAKURA.ne.jp/tmp/config-pre-3.16-rc1 .
>
> ---------- console log start ----------
> EXT4-fs (sda1): re-mounted. Opts: (null)
> BUG: unable to handle kernel paging request at ffff88004c0d2f40
> IP: [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
> PGD 2d32067 PUD 2d35067 PMD 7fb83067 PTE 800000004c0d2060
> Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
> Modules linked in: fuse ipv6 vhost_net macvtap macvlan vhost tun ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore e1000 dm_mod sg i2c_piix4 i2c_core parport_pc parport shpchp ext4(E) jbd2(E) mbcache(E) crc16(E) sd_mod(E) crc_t10dif(E) sr_mod(E) cdrom(E) mptspi(E) mptscsih(E) mptbase(E) scsi_transport_spi(E) pata_acpi(E) ata_generic(E) ata_piix(E)
> CPU: 2 PID: 2277 Comm: mount Tainted: G E 3.15.0-rc8-04322-g59a3d4c #378
> Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
> task: ffff880046d904d0 ti: ffff8800415a0000 task.ti: ffff8800415a0000
> RIP: 0010:[<ffffffff810bc269>] [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
> RSP: 0018:ffff8800415a3d38 EFLAGS: 00010282
> RAX: ffff88007962c080 RBX: ffff88004c0d2f40 RCX: 0000000000000002
> RDX: 0000000000000018 RSI: ffff88004c0d4f58 RDI: ffff88004c0d2f40
> RBP: ffff8800415a3d38 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff88004c0d4eb0
> R13: ffff88004c0d4fb0 R14: ffff8800415a3dd8 R15: ffff88004c0d4f40
> FS: 00007fe45be4a7e0(0000) GS:ffff880079600000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffff88004c0d2f40 CR3: 00000000494b6000 CR4: 00000000000407e0
> Stack:
> ffff8800415a3d68 ffffffff816380ee ffffffff811e3a64 ffffffff813117c9
> ffff88004d88deb0 ffff88004c0d2eb0 ffff8800415a3db8 ffffffff811e3a80
> ffff88004c0dafb0 ffff8800708e8fc8 ffff880046d904d0 0000000000000400
> Call Trace:
> [<ffffffff816380ee>] _raw_spin_trylock+0x1e/0x70
> [<ffffffff811e3a64>] ? shrink_dentry_list+0x44/0x370
> [<ffffffff813117c9>] ? lockref_put_or_lock+0x29/0x40
> [<ffffffff811e3a80>] shrink_dentry_list+0x60/0x370
> [<ffffffff811e3ec9>] shrink_dcache_sb+0xd9/0x120
> [<ffffffff811cc252>] do_remount_sb+0x52/0x1a0
> [<ffffffff811edf8d>] ? do_remount+0x9d/0x130
> [<ffffffff811ee015>] do_remount+0x125/0x130
> [<ffffffff811f13f2>] do_mount+0x1b2/0x210
> [<ffffffff8117e92b>] ? strndup_user+0x5b/0x80
> [<ffffffff811f14e0>] SyS_mount+0x90/0xe0
> [<ffffffff81642992>] system_call_fastpath+0x16/0x1b
> Code: ff ff ff 31 ff e8 28 ef 01 00 e9 41 ff ff ff 4c 89 ef e8 cb aa fa ff e9 44 ff ff ff 90 90 90 90 90 90 55 48 89 e5 66 66 66 66 90 <8b> 17 89 d0 c1 e8 10 66 39 c2 74 0b 31 c0 c9 c3 0f 1f 80 00 00
> RIP [<ffffffff810bc269>] do_raw_spin_trylock+0x9/0x50
> RSP <ffff8800415a3d38>
> CR2: ffff88004c0d2f40
> ---[ end trace 98409f38f5f1bb7f ]---
> BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
> in_atomic(): 1, irqs_disabled(): 1, pid: 2277, name: mount
> INFO: lockdep is turned off.
> irq event stamp: 69452
> hardirqs last enabled at (69451): [<ffffffff810d9c6c>] __call_rcu.clone.0+0x14c/0x330
> hardirqs last disabled at (69452): [<ffffffff81639753>] error_sti+0x5/0x6
> softirqs last enabled at (68978): [<ffffffff8105b691>] __do_softirq+0x1e1/0x420
> softirqs last disabled at (68969): [<ffffffff8105ba05>] irq_exit+0xc5/0xd0
> CPU: 2 PID: 2277 Comm: mount Tainted: G D E 3.15.0-rc8-04322-g59a3d4c #378
> Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
> ffffffff81a0d7a3 ffff8800415a3998 ffffffff8163284a ffff8800415a3998
> ffff880046d904d0 ffff8800415a39c8 ffffffff8108ce85 ffff880046d904d0
> ffff880047415eb8 0000000000000009 0000000000000009 ffff8800415a39f8
> Call Trace:
> [<ffffffff8163284a>] dump_stack+0x51/0x6f
> [<ffffffff8108ce85>] __might_sleep+0x175/0x230
> [<ffffffff81637026>] down_read+0x26/0xa0
> [<ffffffff8106aa44>] exit_signals+0x24/0x140
> [<ffffffff81058bb5>] do_exit+0xb5/0x570
> [<ffffffff8163a161>] oops_end+0xa1/0xf0
> [<ffffffff81045f5e>] no_context+0x12e/0x200
> [<ffffffff8104615d>] __bad_area_nosemaphore+0x12d/0x230
> [<ffffffff810b70b5>] ? __lock_acquire+0x445/0x600
> [<ffffffff81046273>] bad_area_nosemaphore+0x13/0x20
> [<ffffffff8163cdc1>] __do_page_fault+0xe1/0x4c0
> [<ffffffff81638d90>] ? _raw_spin_unlock_irqrestore+0x40/0x70
> [<ffffffff8131145d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
> [<ffffffff8163d1fc>] do_page_fault+0xc/0x10
> [<ffffffff81639552>] page_fault+0x22/0x30
> [<ffffffff810bc269>] ? do_raw_spin_trylock+0x9/0x50
> [<ffffffff816380ee>] _raw_spin_trylock+0x1e/0x70
> [<ffffffff811e3a64>] ? shrink_dentry_list+0x44/0x370
> [<ffffffff813117c9>] ? lockref_put_or_lock+0x29/0x40
> [<ffffffff811e3a80>] shrink_dentry_list+0x60/0x370
> [<ffffffff811e3ec9>] shrink_dcache_sb+0xd9/0x120
> [<ffffffff811cc252>] do_remount_sb+0x52/0x1a0
> [<ffffffff811edf8d>] ? do_remount+0x9d/0x130
> [<ffffffff811ee015>] do_remount+0x125/0x130
> [<ffffffff811f13f2>] do_mount+0x1b2/0x210
> [<ffffffff8117e92b>] ? strndup_user+0x5b/0x80
> [<ffffffff811f14e0>] SyS_mount+0x90/0xe0
> [<ffffffff81642992>] system_call_fastpath+0x16/0x1b
> note: mount[2277] exited with preempt_count 2
> ---------- console log end ----------
>
> ---------- bisect log start ----------
> # bad: [776edb59317ada867dfcddde40b55648beeb0078] Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next
> # good: [26fc9cd200ec839e0b3095e05ae018f27314e7aa] kernfs: move the last knowledge of sysfs out from kernfs
> # good: [f9c78b2be2cac2a7a397d489275e7d9f9ae785f2] block: move bio.c and bio-integrity.c from fs/ to block/
> # good: [681a2895486243a82547d8c9f53043eb54b53da0] Merge branch 'for-3.16/core' of git://git.kernel.dk/linux-block into next
> # good: [5da77761e6fd51f633b4f31051c4f839e01c29c0] Merge tag 'driver-core-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core into next
> # good: [4e857c58efeb99393cba5a5d0d8ec7117183137c] arch: Mass conversion of smp_mb__*()
> # good: [bf8102228a8bf053051f311e5486042fe0542894] intel_pstate: Improve initial busy calculation
> # good: [192a98e280e560510a62aca8cfa83b4ae7c095bb] ALSA: hda/realtek - Fix COEF widget NID for ALC260 replacer fixup
> # good: [1860e379875dfe7271c649058aeddffe5afd9d0d] Linux 3.15
> # good: [455c6fdbd219161bd09b1165f11699d6d73de11c] Linux 3.14
> # good: [d8ec26d7f8287f5788a494f56e8814210f0e64be] Linux 3.13
> # good: [5e01dc7b26d9f24f39abace5da98ccbd6a5ceb52] Linux 3.12
> # good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
> # good: [8bb495e3f02401ee6f76d1b1d77f3ac9f079e376] Linux 3.10
> # good: [c1be5a5b1b355d40e6cf79cc979eb66dafa24ad1] Linux 3.9
> # good: [19f949f52599ba7c3f67a5897ac6be14bfcb1200] Linux 3.8
> # good: [29594404d7fe73cd80eaa4ee8c43dcc53970c60e] Linux 3.7
> # good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
> # good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
> # good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
> # good: [c16fa4f2ad19908a47c63d8fa436a1178438c7e7] Linux 3.3
> # good: [805a6af8dba5dfdd35ec35dc52ec0122400b2610] Linux 3.2
> # good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1
> # good: [02f8c6aee8df3cdc935e9bdd4f2d020306035dbe] Linux 3.0
> git bisect start '776edb59317ada867dfcddde40b55648beeb0078' '26fc9cd200ec839e0b3095e05ae018f27314e7aa' 'f9c78b2be2cac2a7a397d489275e7d9f9ae785f2' '681a2895486243a82547d8c9f53043eb54b53da0' '5da77761e6fd51f633b4f31051c4f839e01c29c0' '4e857c58efeb99393cba5a5d0d8ec7117183137c' 'bf8102228a8bf053051f311e5486042fe0542894' '192a98e280e560510a62aca8cfa83b4ae7c095bb' 'v3.15' 'v3.14' 'v3.13' 'v3.12' 'v3.11' 'v3.10' 'v3.9' 'v3.8' 'v3.7' 'v3.6' 'v3.5' 'v3.4' 'v3.3' 'v3.2' 'v3.1' 'v3.0'
> # good: [539b4f72d0da83e4d20df7cfc27dc49c00316940] staging/rtl8192u: Put EXPORT_SYMBOL just after the exported function
> git bisect good 539b4f72d0da83e4d20df7cfc27dc49c00316940
> # good: [49eb7b0750d9483c74e9c14ae6ea1e9d62481c3c] Merge tag 'tty-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty into next
> git bisect good 49eb7b0750d9483c74e9c14ae6ea1e9d62481c3c
> # good: [8f5759aeb88a47448cd92ab55a016d013b154a98] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux into next
> git bisect good 8f5759aeb88a47448cd92ab55a016d013b154a98
> # good: [de6b25de4584febbe1808bc782734ae163b22c26] Merge tag 'regmap-v3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap into next
> git bisect good de6b25de4584febbe1808bc782734ae163b22c26
> # good: [312c76f1a3989b8d0c0c13fee765bb2c41f2d114] Merge tag 'regulator-v3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator into next
> git bisect good 312c76f1a3989b8d0c0c13fee765bb2c41f2d114
> # good: [48d684fdad83d7525a557e6ff9c37811b6a9947b] rcutorture: Run rcu_torture_writer at normal priority
> git bisect good 48d684fdad83d7525a557e6ff9c37811b6a9947b
> # good: [a5d6d3a1b00a0ad88f07c3a727c79b27915278e3] softirq: A single rcu_bh_qs() call per softirq set is enough
> git bisect good a5d6d3a1b00a0ad88f07c3a727c79b27915278e3
> # good: [61f38db3e3c0e4c3be0858750e2cabeadaecac0c] rcu: Provide API to suppress stall warnings while sysrc runs
> git bisect good 61f38db3e3c0e4c3be0858750e2cabeadaecac0c
> # good: [f70e3c4f8b6ab61f713e040822ec51f5de498146] CMA: correct unlock target
> git bisect good f70e3c4f8b6ab61f713e040822ec51f5de498146
> # bad: [59a3d4c3631e553357b7305dc09db1990aa6757c] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next
> git bisect bad 59a3d4c3631e553357b7305dc09db1990aa6757c
> # good: [e14505a8d50882ff3bdd4b791b14d90a0881fa4d] Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
> git bisect good e14505a8d50882ff3bdd4b791b14d90a0881fa4d
> # good: [ff806d034ef8e9a95ff0b0532104dd65332e446b] Merge branch 'for-v3.16' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping into next
> git bisect good ff806d034ef8e9a95ff0b0532104dd65332e446b
> ---------- bisect log end ----------
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock)
2014-06-10 14:34 ` Paul E. McKenney
@ 2014-06-11 11:35 ` Tetsuo Handa
2014-06-11 16:01 ` Al Viro
0 siblings, 1 reply; 5+ messages in thread
From: Tetsuo Handa @ 2014-06-11 11:35 UTC (permalink / raw)
To: paulmck, viro; +Cc: torvalds, linux-fsdevel, linux-kernel
Paul E. McKenney wrote:
> On Tue, Jun 10, 2014 at 09:04:51PM +0900, Tetsuo Handa wrote:
> > Hello.
> >
> > I noticed that commit 59a3d4c363 "Merge branch 'core-rcu-for-linus' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next" causes
> > an oops when rebooting a system. Do you have any clue?
>
> Can't say that I do. However, the commits that might have made some
> difference in how RCU operates are the following. I have my doubts
> (listed below) about their being the cause, but if the oops is reproducible,
> why not try reverting each and see what happens? Or just bisect within
> the core-rcu-for-linus branches?
Thank you. This oops seems to be 100% reproducible. I tried
for i in 365187fbc04f 48a7639ce80c 595f3900f6b4 495aa969dbae a381d757d93f 5057f55e543b becb41bfe054 fa07a58f71ee a5d6d3a1b00a
do
git reset --hard $i; yes | make -sj4; make -s modules_install install
git reset --hard $i~1; yes | make -sj4; make -s modules_install install
done
on tip/tip.git but none of them built the oops kernel.
I also confirmed that the command line which triggers this oops is
mount -n -o ro,remount sysfs /sys
in /etc/init.d/halt script in CentOS 6.5.
Manually executing the command line from login shell triggers this oops.
I don't know why but commit 59a3d4c363 revealed a refcount bug
in sysfs (or maybe in vfs)?
---------- quote from /etc/init.d/halt start ----------
# Remount read only anything that's left mounted.
# echo $"Remounting remaining filesystems readonly"
mount | awk '{ print $1,$3 }' | while read dev dir; do
fstab-decode mount -n -o ro,remount $dev $dir
done
---------- quote from /etc/init.d/halt end ----------
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock)
2014-06-11 11:35 ` Tetsuo Handa
@ 2014-06-11 16:01 ` Al Viro
2014-06-12 11:39 ` Tetsuo Handa
0 siblings, 1 reply; 5+ messages in thread
From: Al Viro @ 2014-06-11 16:01 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: paulmck, torvalds, linux-fsdevel, linux-kernel
On Wed, Jun 11, 2014 at 08:35:44PM +0900, Tetsuo Handa wrote:
> I also confirmed that the command line which triggers this oops is
>
> mount -n -o ro,remount sysfs /sys
>
> in /etc/init.d/halt script in CentOS 6.5.
> Manually executing the command line from login shell triggers this oops.
>
> I don't know why but commit 59a3d4c363 revealed a refcount bug
> in sysfs (or maybe in vfs)?
Try to check if this helps:
diff --git a/fs/dcache.c b/fs/dcache.c
index be2bea8..e99c6f5 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -532,10 +532,12 @@ static inline struct dentry *lock_parent(struct dentry *dentry)
struct dentry *parent = dentry->d_parent;
if (IS_ROOT(dentry))
return NULL;
+ if (unlikely((int)dentry->d_lockref.count < 0))
+ return NULL;
if (likely(spin_trylock(&parent->d_lock)))
return parent;
- spin_unlock(&dentry->d_lock);
rcu_read_lock();
+ spin_unlock(&dentry->d_lock);
again:
parent = ACCESS_ONCE(dentry->d_parent);
spin_lock(&parent->d_lock);
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock)
2014-06-11 16:01 ` Al Viro
@ 2014-06-12 11:39 ` Tetsuo Handa
0 siblings, 0 replies; 5+ messages in thread
From: Tetsuo Handa @ 2014-06-12 11:39 UTC (permalink / raw)
To: viro; +Cc: paulmck, torvalds, linux-fsdevel, linux-kernel
Al Viro wrote:
> Try to check if this helps:
Yes, that helps. Thank you.
I used below version and the oops no longer occur.
---------- patch start ----------
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -532,10 +532,15 @@ static inline struct dentry *lock_parent(struct dentry *dentry)
struct dentry *parent = dentry->d_parent;
if (IS_ROOT(dentry))
return NULL;
+ if (WARN_ON_ONCE((int)dentry->d_lockref.count < 0)) {
+ pr_warn("dentry->d_lockref.count=%d dentry->d_sb->s_magic=%lx\n",
+ dentry->d_lockref.count, dentry->d_sb->s_magic);
+ return NULL;
+ }
if (likely(spin_trylock(&parent->d_lock)))
return parent;
- spin_unlock(&dentry->d_lock);
rcu_read_lock();
+ spin_unlock(&dentry->d_lock);
again:
parent = ACCESS_ONCE(dentry->d_parent);
spin_lock(&parent->d_lock);
---------- patch end ----------
ef53 is ext4, 9fa0 is proc, 62656572 is sysfs.
I don't know why only sysfs triggered the oops.
---------- console log on boot ----------
[ 6.987663] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 7.071152] dracut: Mounted root filesystem /dev/sda1
[ 7.275840] random: nonblocking pool is initialized
[ 7.414446] load_policy (374) used greatest stack depth: 10832 bytes left
[ 7.417446] dracut: /sbin/load_policy: Can't load policy: No such device
[ 7.630160] dracut: Switching root
[ 8.333437] ------------[ cut here ]------------
[ 8.333737] WARNING: CPU: 1 PID: 394 at fs/dcache.c:535 shrink_dentry_list+0x388/0x400()
[ 8.334546] Modules linked in: ext4(E) jbd2(E) mbcache(E) crc16(E) sd_mod(E) crc_t10dif(E) sr_mod(E) cdrom(E) mptspi(E) mptscsih(E) mptbase(E) scsi_transport_spi(E) pata_acpi(E) ata_generic(E) ata_piix(E)
[ 8.336012] CPU: 1 PID: 394 Comm: mount Tainted: G E 3.15.0-08974-g3c81bdd-dirty #390
[ 8.337616] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[ 8.339667] 0000000000000217 ffff88004b20bd18 ffffffff8163844d ffffffff81a23cef
[ 8.340469] 0000000000000000 ffff88004b20bd58 ffffffff81054d9c ffff880048ca6fb0
[ 8.340908] ffff880078c06eb0 ffff880048ca6eb0 ffff880048ca6fb0 ffff88004b20bdd8
[ 8.341841] Call Trace:
[ 8.342440] [<ffffffff8163844d>] dump_stack+0x51/0x6c
[ 8.343495] [<ffffffff81054d9c>] warn_slowpath_common+0x8c/0xc0
[ 8.344556] [<ffffffff81054dea>] warn_slowpath_null+0x1a/0x20
[ 8.344764] [<ffffffff811e71b8>] shrink_dentry_list+0x388/0x400
[ 8.345536] [<ffffffff811e7369>] shrink_dcache_sb+0xd9/0x120
[ 8.346501] [<ffffffff811f14fd>] ? do_remount+0x9d/0x130
[ 8.347564] [<ffffffff811cf6e2>] do_remount_sb+0x52/0x1a0
[ 8.348506] [<ffffffff811f14fd>] ? do_remount+0x9d/0x130
[ 8.349520] [<ffffffff811f1585>] do_remount+0x125/0x130
[ 8.350552] [<ffffffff811f4962>] do_mount+0x1b2/0x210
[ 8.351496] [<ffffffff811810fb>] ? strndup_user+0x5b/0x80
[ 8.352501] [<ffffffff811f4a50>] SyS_mount+0x90/0xe0
[ 8.353500] [<ffffffff816483a2>] system_call_fastpath+0x16/0x1b
[ 8.354533] ---[ end trace eddfc2138fdcf65f ]---
[ 8.355486] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=9fa0
(...snipped...)
[ 8.431502] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=9fa0
Welcome to CentOS
---------- console log on boot ----------
---------- console log on shutdown ----------
init: Re-executing /sbin/init
[ 76.646596] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=ef53
(...snipped...)
[ 76.701837] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=ef53
[ 76.718264] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 76.962112] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=9fa0
(...snipped...)
[ 77.100116] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=9fa0
[ 77.110684] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=62656572
(...snipped...)
[ 77.424319] dentry->d_lockref.count=-128 dentry->d_sb->s_magic=62656572
Please stand by while rebooting the system...
[ 77.556052] reboot: Restarting system
[ 77.556942] reboot: machine restart
---------- console log on shutdown ----------
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-06-12 11:39 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-06-10 12:04 [pre 3.16-rc1] Oops at spin_trylock(&inode->i_lock) Tetsuo Handa
2014-06-10 14:34 ` Paul E. McKenney
2014-06-11 11:35 ` Tetsuo Handa
2014-06-11 16:01 ` Al Viro
2014-06-12 11:39 ` Tetsuo Handa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).