linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: NeilBrown <neilb-l3A5Bk7waGM@public.gmane.org>
To: Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Cc: linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: rpciod deadlock issue
Date: Mon, 25 Aug 2014 16:05:01 +1000	[thread overview]
Message-ID: <20140825160501.433b3e9e@notabene.brown> (raw)
In-Reply-To: <53F6F772.6020708-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

[-- Attachment #1: Type: text/plain, Size: 4144 bytes --]

On Fri, 22 Aug 2014 15:55:30 +0800 Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:

> Hi All,
> 
> I got an nfs hung issue, looks like "rpciod" run into deadlock. Bug is
> reported on 2.6.32, but seems mainline also suffers this bug from the
> source code.
> 
> See the following rpciod trace. rpciod allocated memory using GFP_KERNEL
> in xs_setup_xprt(). That triggered direct reclaim when available memory
> was not enough, where it waited an write-back page done, but that page
> was a nfs page, and it depended on rpciod to write back. So this caused
> a deadlock.
> 
> I am not sure how to fix this issue. Replace GFP_KERNEL with GFP_NOFS in
> xs_setup_xprt() can fix this trace, but there are other place allocating
> memory with GFP_KERNEL in rpciod, like
> xs_tcp_setup_socket()->xs_create_sock()->__sock_create()->sock_alloc(),
> there is no way to pass GFP_NOFS to network command code. Also mainline
> has changed to not care ___GFP_FS before waiting page write back done.
> Upstream commit 5cf02d0 (nfs: skip commit in releasepage if we're
> freeing memory for fs-related reasons) uses PF_FSTRANS to avoid another
> deadlock when direct reclaim, i am thinking whether we can check
> PF_FSTRANS flag in shrink_page_list(), if this flag is set, it will not
> wait any page write back done? I saw this flag is also used by xfs, not
> sure whether this will affect xfs.
> 
> Any advices is appreciated.

This problem shouldn't affect mainline.

Since Linux 3.2, "direct reclaim" never wait for writeback - that is left for
kswapd to do. (See "A pivotal patch" in https://lwn.net/Articles/595652/)
So this deadlock cannot happen.

Probably the simplest fix for your deadlock would be:
- in shrink_page_list, clear may_enter_fs if PF_FSTRANS is set.
- in rpc_async_schedule, set PF_FSTRANS before calling __rpc_execute, and
  clear it again afterwards.

NeilBrown

> 
> @ crash> bt 1539
> @ PID: 1539   TASK: ffff88178f64a040  CPU: 1   COMMAND: "rpciod/1"
> @  #0 [ffff88178f64d2c0] schedule at ffffffff8145833a
> @  #1 [ffff88178f64d348] io_schedule at ffffffff8145842c
> @  #2 [ffff88178f64d368] sync_page at ffffffff810d8161
> @  #3 [ffff88178f64d378] __wait_on_bit at ffffffff8145895b
> @  #4 [ffff88178f64d3b8] wait_on_page_bit at ffffffff810d82fe
> @  #5 [ffff88178f64d418] wait_on_page_writeback at ffffffff810e2a1a
> @  #6 [ffff88178f64d438] shrink_page_list at ffffffff810e34e1
> @  #7 [ffff88178f64d588] shrink_list at ffffffff810e3dbe
> @  #8 [ffff88178f64d6f8] shrink_zone at ffffffff810e425e
> @  #9 [ffff88178f64d7b8] do_try_to_free_pages at ffffffff810e4978
> @ #10 [ffff88178f64d828] try_to_free_pages at ffffffff810e4c31
> @ #11 [ffff88178f64d8c8] __alloc_pages_nodemask at ffffffff810de370
> @ #12 [ffff88178f64d978] kmem_getpages at ffffffff8110e18b
> @ #13 [ffff88178f64d9a8] fallback_alloc at ffffffff8110e35e
> @ #14 [ffff88178f64da08] ____cache_alloc_node at ffffffff8110e51f
> @ #15 [ffff88178f64da48] __kmalloc at ffffffff8110efba
> @ #16 [ffff88178f64da98] xs_setup_xprt at ffffffffa00a563f [sunrpc]
> @ #17 [ffff88178f64dad8] xs_setup_tcp at ffffffffa00a7648 [sunrpc]
> @ #18 [ffff88178f64daf8] xprt_create_transport at ffffffffa00a478f [sunrpc]
> @ #19 [ffff88178f64db18] rpc_create at ffffffffa00a2d7a [sunrpc]
> @ #20 [ffff88178f64dbf8] rpcb_create at ffffffffa00b026b [sunrpc]
> @ #21 [ffff88178f64dc98] rpcb_getport_async at ffffffffa00b0c94 [sunrpc]
> @ #22 [ffff88178f64ddf8] call_bind at ffffffffa00a11f8 [sunrpc]
> @ #23 [ffff88178f64de18] __rpc_execute at ffffffffa00a88ef [sunrpc]
> @ #24 [ffff88178f64de58] rpc_async_schedule at ffffffffa00a9187 [sunrpc]
> @ #25 [ffff88178f64de78] worker_thread at ffffffff81072ed2
> @ #26 [ffff88178f64dee8] kthread at ffffffff81076df3
> @ #27 [ffff88178f64df48] kernel_thread at ffffffff81012e2a
> @ crash>
> 
> Thanks,
> Junxiao.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

  parent reply	other threads:[~2014-08-25  6:05 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-08-22  7:55 rpciod deadlock issue Junxiao Bi
     [not found] ` <53F6F772.6020708-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2014-08-22 22:49   ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Trond Myklebust
2014-08-22 22:49     ` [PATCH v2 2/2] NFS: Ensure that rpciod does not trigger reclaim writebacks Trond Myklebust
     [not found]     ` <1408747772-37938-1-git-send-email-trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
2014-08-25  5:34       ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Junxiao Bi
2014-08-25  6:48       ` NeilBrown
     [not found]         ` <20140825164852.50723141-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
2014-08-26  5:43           ` Junxiao Bi
2014-08-26  6:21             ` NeilBrown
2014-08-26  6:49               ` Junxiao Bi
2014-08-26  7:04                 ` NeilBrown
     [not found]                   ` <20140826170410.20560764-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
2014-08-26  7:23                     ` Junxiao Bi
2014-08-26 10:53           ` Mel Gorman
2014-08-26 12:58             ` Trond Myklebust
2014-08-26 13:26               ` Mel Gorman
     [not found]                 ` <20140826132624.GU17696-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org>
2014-08-26 23:19                   ` Johannes Weiner
     [not found]                     ` <20140826231938.GA13889-druUgvl0LCNAfugRpC6u6w@public.gmane.org>
2014-08-26 23:51                       ` Trond Myklebust
     [not found]                         ` <CAHQdGtRPsVFVfph5OcsZk_+WYPPJ-MpE2myZfXAb3jq6fuM4zw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-08-27  0:00                           ` Trond Myklebust
2014-08-27 15:36                             ` Mel Gorman
     [not found]                               ` <20140827153644.GF12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org>
2014-08-27 16:15                                 ` Trond Myklebust
2014-08-28  8:30                                   ` Mel Gorman
     [not found]                                     ` <20140828083053.GJ12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org>
2014-08-28  8:49                                       ` Junxiao Bi
2014-08-28  9:25                                         ` Mel Gorman
2014-09-04 13:54                                 ` Michal Hocko
2014-09-09  2:33                                   ` NeilBrown
2014-09-10 13:48                                     ` Michal Hocko
     [not found]                                       ` <20140910134842.GG25219-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2014-09-10 23:57                                         ` NeilBrown
     [not found]                                           ` <20140911095743.1ed87519-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
2014-09-11  8:50                                             ` Michal Hocko
     [not found]                                               ` <20140911085046.GC22042-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2014-09-11 10:53                                                 ` NeilBrown
2014-08-27  1:43                       ` NeilBrown
2014-08-25  6:05   ` NeilBrown [this message]
     [not found]     ` <20140825160501.433b3e9e-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
2014-08-25  6:15       ` rpciod deadlock issue NeilBrown

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20140825160501.433b3e9e@notabene.brown \
    --to=neilb-l3a5bk7wagm@public.gmane.org \
    --cc=junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \
    --cc=linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).