From: Sasha Levin <levinsasha928@gmail.com>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: linux-kernel <linux-kernel@vger.kernel.org>
Subject: INFO: task rcuc/0:7 blocked for more than 120 seconds.
Date: Mon, 26 Dec 2011 14:16:43 +0200 [thread overview]
Message-ID: <1324901803.31721.4.camel@lappy> (raw)
Hi Paul,
I've recently got the following panic which was caused by khungtask:
[ 1921.589512] INFO: task rcuc/0:7 blocked for more than 120 seconds.
[ 1921.590370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1921.597103] rcuc/0 D ffff880012f61630 4400 7 2 0x00000000
[ 1921.598646] ffff880012f6b980 0000000000000086 ffff880012f6bfd8 00000000001d4740
[ 1921.600289] ffff880012f6bfd8 ffff880012f61630 ffff880012f6bfd8 ffff880012f6a000
[ 1921.601707] 00000000001d4800 ffff880012f6a000 ffff880012f6bfd8 00000000001d4800
[ 1921.603258] Call Trace:
[ 1921.603703] [<ffffffff8255eefa>] schedule+0x3a/0x50
[ 1921.605462] [<ffffffff8255cd65>] schedule_timeout+0x255/0x4d0
[ 1921.606540] [<ffffffff8112a25e>] ? mark_held_locks+0x6e/0x130
[ 1921.607633] [<ffffffff811277b2>] ? lock_release_holdtime+0xb2/0x160
[ 1921.608798] [<ffffffff825602bb>] ? _raw_spin_unlock_irq+0x2b/0x70
[ 1921.610154] [<ffffffff8255f630>] wait_for_common+0x120/0x170
[ 1921.617878] [<ffffffff81104f30>] ? try_to_wake_up+0x2f0/0x2f0
[ 1921.618949] [<ffffffff811754d0>] ? __call_rcu+0x3c0/0x3c0
[ 1921.621405] [<ffffffff8255f728>] wait_for_completion+0x18/0x20
[ 1921.623622] [<ffffffff810ee0b9>] wait_rcu_gp+0x59/0x80
[ 1921.626789] [<ffffffff810ec0c0>] ? perf_trace_rcu_batch_end+0x120/0x120
[ 1921.629440] [<ffffffff8255f554>] ? wait_for_common+0x44/0x170
[ 1921.632445] [<ffffffff81179d3c>] synchronize_rcu+0x1c/0x20
[ 1921.635455] [<ffffffff810f8980>] atomic_notifier_chain_unregister+0x60/0x80
[ 1921.638550] [<ffffffff8111bab3>] task_handoff_unregister+0x13/0x20
[ 1921.641271] [<ffffffff8211342f>] task_notify_func+0x2f/0x40
[ 1921.643894] [<ffffffff810f8817>] notifier_call_chain+0x67/0x110
[ 1921.646580] [<ffffffff810f8a14>] __atomic_notifier_call_chain+0x74/0x110
[ 1921.654064] [<ffffffff810f89d9>] ? __atomic_notifier_call_chain+0x39/0x110
[ 1921.656034] [<ffffffff810c1a30>] ? __put_task_struct+0xc0/0x120
[ 1921.658625] [<ffffffff810c71a0>] ? will_become_orphaned_pgrp+0x100/0x100
[ 1921.661527] [<ffffffff810f8ac1>] atomic_notifier_call_chain+0x11/0x20
[ 1921.664295] [<ffffffff8111ba55>] profile_handoff_task+0x15/0x20
[ 1921.666885] [<ffffffff810c19eb>] __put_task_struct+0x7b/0x120
[ 1921.669492] [<ffffffff810c71d5>] delayed_put_task_struct+0x35/0x140
[ 1921.672129] [<ffffffff81175994>] rcu_do_batch.clone.12+0x224/0xab0
[ 1921.674697] [<ffffffff8255ea62>] ? __schedule+0x432/0x890
[ 1921.677028] [<ffffffff811783fa>] rcu_cpu_kthread+0x48a/0xa40
[ 1921.682719] [<ffffffff82560245>] ? _raw_spin_unlock_irqrestore+0x55/0xa0
[ 1921.684913] [<ffffffff81177f70>] ? rcu_idle_enter+0xb0/0xb0
[ 1921.686641] [<ffffffff810f11f6>] kthread+0xb6/0xc0
[ 1921.689956] [<ffffffff82562cf4>] kernel_thread_helper+0x4/0x10
[ 1921.692815] [<ffffffff810fe170>] ? finish_task_switch+0x80/0x110
[ 1921.695613] [<ffffffff82561038>] ? retint_restore_args+0x13/0x13
[ 1921.698409] [<ffffffff810f1140>] ? kthread_flush_work_fn+0x10/0x10
[ 1921.701199] [<ffffffff82562cf0>] ? gs_change+0x13/0x13
[ 1921.703680] 1 lock held by rcuc/0/7:
[ 1921.705325] #0: (rcu_read_lock){.+.+..}, at: [<ffffffff810f89d9>] __atomic_notifier_call_chain+0x39/0x110
[ 1921.710297] Kernel panic - not syncing: hung_task: blocked tasks
[ 1921.715383] Rebooting in 1 seconds..
This is the RCU related .config:
# cat .config | grep RCU
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_TRACE=y
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
CONFIG_PROVE_RCU=y
CONFIG_PROVE_RCU_REPEATEDLY=y
CONFIG_SPARSE_RCU_POINTER=y
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
Please let me know if I can help with anything else.
--
Sasha.
next reply other threads:[~2011-12-26 12:16 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-26 12:16 Sasha Levin [this message]
2011-12-26 16:31 ` INFO: task rcuc/0:7 blocked for more than 120 seconds Paul E. McKenney
2011-12-26 16:37 ` Frederic Weisbecker
2011-12-26 19:56 ` Paul E. McKenney
2012-01-04 19:03 ` [PATCH] rcu: Improve detection of illegal synchronize_rcu() call from RCU read side Frederic Weisbecker
2012-01-04 21:30 ` Paul E. McKenney
2012-01-05 1:45 ` Frederic Weisbecker
2012-01-05 2:01 ` Paul E. McKenney
2012-01-05 2:06 ` Frederic Weisbecker
2012-01-05 2:17 ` Paul E. McKenney
2011-12-27 9:13 ` INFO: task rcuc/0:7 blocked for more than 120 seconds Sasha Levin
2011-12-28 4:29 ` Paul E. McKenney
2012-01-03 20:27 ` Paul E. McKenney
2012-01-03 20:37 ` Greg KH
2012-01-03 21:38 ` Paul E. McKenney
2012-01-03 21:50 ` Greg KH
2012-01-03 22:26 ` Paul E. McKenney
2012-01-03 22:33 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1324901803.31721.4.camel@lappy \
--to=levinsasha928@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).