From: Richard Yao <ryao@cs.stonybrook.edu>
To: Kernel development list <linux-kernel@vger.kernel.org>
Subject: I need tips on how to debug a deadlock involving swap
Date: Mon, 7 May 2012 11:53:55 -0400 [thread overview]
Message-ID: <4FA7F013.8020002@cs.stonybrook.edu> (raw)
[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]
I have a deadlock that occurs when I swap to a virtual block device. The
driver is out-of-tree and it processes IO requests in worker threads.
Setting PF_MEMALLOC will prevent the deadlock, but it has the side
effect of grabbing pages from ZONE_DMA, which is bad.
I believe that direct reclaim is being triggered when swap occurs,
causing swap operations holding locks to depend on swap operations that
require those locks, but I am having trouble identifying how that happens.
The deadlock occurs in the IO worker threads, but the hung task timeout
provides a backtrace for the thread that triggered the IO request, which
is not helpful:
[ 218.252066] INFO: task python2.7:7027 blocked for more than 15 seconds.
[ 218.252070] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 218.252073] python2.7 D ffffffff814051e0 0 7027 7022
0x00000000
[ 218.252079] ffff8801b4c73798 0000000000000086 ffff8801b4c73758
ffff8801b4c73758
[ 218.252085] ffff880224b84a40 ffff8801b4c73fd8 ffff8801b4c73fd8
ffff8801b4c73fd8
[ 218.252091] ffff8802268aca40 ffff880224b84a40 ffff8801b4c73768
ffff88022fc91738
[ 218.252097] Call Trace:
[ 218.252105] [<ffffffff810b8300>] ? __lock_page+0x70/0x70
[ 218.252111] [<ffffffff8133241a>] schedule+0x3a/0x50
[ 218.252114] [<ffffffff813324ba>] io_schedule+0x8a/0xd0
[ 218.252118] [<ffffffff810b8309>] sleep_on_page+0x9/0x10
[ 218.252121] [<ffffffff81330547>] __wait_on_bit+0x57/0x80
[ 218.252131] [<ffffffff810c097e>] ? account_page_writeback+0xe/0x10
[ 218.252134] [<ffffffff810b84e0>] wait_on_page_bit+0x70/0x80
[ 218.252137] [<ffffffff81052da0>] ? autoremove_wake_function+0x40/0x40
[ 218.252141] [<ffffffff810c7245>] shrink_page_list+0x465/0x8f0
[ 218.252144] [<ffffffff810c7cf9>] shrink_inactive_list+0x379/0x470
[ 218.252147] [<ffffffff81336c2d>] ? sub_preempt_count+0x9d/0xd0
[ 218.252150] [<ffffffff810c8261>] shrink_mem_cgroup_zone+0x471/0x570
[ 218.252153] [<ffffffff810c8e0b>] do_try_to_free_pages+0xfb/0x420
[ 218.252156] [<ffffffff810c9251>] try_to_free_pages+0x71/0x80
[ 218.252159] [<ffffffff810c04f9>] __alloc_pages_nodemask+0x469/0x7a0
[ 218.252162] [<ffffffff810c3750>] ? __put_single_page+0x30/0x30
[ 218.252166] [<ffffffff810fa36c>] do_huge_pmd_anonymous_page+0x14c/0x350
[ 218.252170] [<ffffffff810d69cf>] handle_mm_fault+0x13f/0x2f0
[ 218.252172] [<ffffffff8133662e>] do_page_fault+0x14e/0x590
[ 218.252176] [<ffffffff81061739>] ? set_next_entity+0x39/0x80
[ 218.252179] [<ffffffff81062a8b>] ? pick_next_task_fair+0x6b/0x150
[ 218.252181] [<ffffffff8105dcf1>] ? get_parent_ip+0x11/0x50
[ 218.252184] [<ffffffff81336c2d>] ? sub_preempt_count+0x9d/0xd0
[ 218.252186] [<ffffffff81331fb8>] ? __schedule+0x2f8/0x6c0
[ 218.252189] [<ffffffff81333a75>] page_fault+0x25/0x30
Is there any way that I can ask the kernel to print stack traces of the
worker threads on demand?
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]
reply other threads:[~2012-05-07 15:54 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FA7F013.8020002@cs.stonybrook.edu \
--to=ryao@cs.stonybrook.edu \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox