From: Masayoshi Mizuma <m.mizuma@jp.fujitsu.com>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs@oss.sgi.com, linux-mm@kvack.org
Subject: xfs: two deadlock problems occur when kswapd writebacks XFS pages.
Date: Tue, 17 Jun 2014 17:50:02 +0900 [thread overview]
Message-ID: <53A0013A.1010100@jp.fujitsu.com> (raw)
I found two deadlock problems occur when kswapd writebacks XFS pages.
I detected these problems on RHEL kernel actually, and I suppose these
also happen on upstream kernel (3.16-rc1).
1.
A process (processA) has acquired read semaphore "xfs_cil.xc_ctx_lock"
at xfs_log_commit_cil() and it is waiting for the kswapd. Then, a
kworker has issued xlog_cil_push_work() and it is waiting for acquiring
the write semaphore. kswapd is waiting for acquiring the read semaphore
at xfs_log_commit_cil() because the kworker has been waiting before for
acquiring the write semaphore at xlog_cil_push(). Therefore, a deadlock
happens.
The deadlock flow is as follows.
processA | kworker | kswapd
----------------------+--------------------------+----------------------
| xfs_trans_commit | |
| xfs_log_commit_cil | |
| down_read(xc_ctx_lock)| |
| xlog_cil_insert_items | |
| xlog_cil_insert_format_items |
| kmem_alloc | |
| : | |
| shrink_inactive_list | |
| congestion_wait | |
| # waiting for kswapd..| |
| | xlog_cil_push_work |
| | xlog_cil_push |
| | xfs_trans_commit |
| | down_write(xc_ctx_lock) |
| | # waiting for processA...|
| | | shrink_page_list
| | | xfs_vm_writepage
| | | xfs_map_blocks
| | | xfs_iomap_write_allocate
| | | xfs_trans_commit
| | | xfs_log_commit_cil
| | | down_read(xc_ctx_lock)
V(time) | | # waiting for kworker...
----------------------+--------------------------+-----------------------
To fix this, should we up the read semaphore before calling kmem_alloc()
at xlog_cil_insert_format_items() to avoid blocking the kworker? Or,
should we the second argument of kmem_alloc() from KM_SLEEP|KM_NOFS
to KM_NOSLEEP to avoid waiting for the kswapd. Or...
2.
A kworker (kworkerA), whish is a writeback thread, is waiting for
the XFS allocation thread (kworkerB) while it writebacks XFS pages.
kworkerB has started the allocation and it is waiting for kswapd to
allocate free pages. kswapd has started writeback XFS pages and
it is waiting for more log space. The reason why exhaustion of the
log space is both the writeback thread and kswapd are stuck, so
some processes, who have allocated the log space and are requesting
free pages, are also stuck.
The deadlock flow is as follows.
kworkerA | kworkerB | kswapd
----------------------+--------------------------+-----------------------
| wb_writeback | |
| : | |
| xfs_vm_writepage | |
| xfs_map_blocks | |
| xfs_iomap_write_allocate |
| xfs_bmapi_write | |
| xfs_bmapi_allocate | |
| wait_for_completion | |
| # waiting for kworkerB... |
| | xfs_bmapi_allocate_worker|
| | : |
| | xfs_buf_get_map |
| | xfs_buf_allocate_memory |
| | alloc_pages_current |
| | : |
| | shrink_inactive_list |
| | congestion_wait |
| | # waiting for kswapd... |
| | | shrink_page_list
| | | xfs_vm_writepage
| | | :
| | | xfs_log_reserve
| | | :
| | | xlog_grant_head_check
| | | xlog_grant_head_wait
| | | # waiting for more
| | | # space...
V(time) | |
----------------------+--------------------------+-----------------------
I don't have any ideas to fix this...
Thanks,
Masayoshi Mizuma
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next reply other threads:[~2014-06-17 8:51 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-06-17 8:50 Masayoshi Mizuma [this message]
2014-06-17 13:26 ` xfs: two deadlock problems occur when kswapd writebacks XFS pages Dave Chinner
2014-06-18 9:37 ` Masayoshi Mizuma
2014-06-18 11:48 ` Dave Chinner
[not found] ` <53A7D6CC.1040605@jp.fujitsu.com>
2014-06-24 22:05 ` Dave Chinner
2014-07-14 11:00 ` Masayoshi Mizuma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53A0013A.1010100@jp.fujitsu.com \
--to=m.mizuma@jp.fujitsu.com \
--cc=david@fromorbit.com \
--cc=linux-mm@kvack.org \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).