From: "Michael S. Tsirkin" <mst@redhat.com>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: linux-mm@kvack.org, virtualization@lists.linux-foundation.org
Subject: Re: mm, virtio: possible OOM lockup at virtballoon_oom_notify()
Date: Fri, 29 Sep 2017 07:00:05 +0300 [thread overview]
Message-ID: <20170929065654-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <201709111927.IDD00574.tFVJHLOSOOMQFF@I-love.SAKURA.ne.jp>
On Mon, Sep 11, 2017 at 07:27:19PM +0900, Tetsuo Handa wrote:
> Hello.
>
> I noticed that virtio_balloon is using register_oom_notifier() and
> leak_balloon() from virtballoon_oom_notify() might depend on
> __GFP_DIRECT_RECLAIM memory allocation.
>
> In leak_balloon(), mutex_lock(&vb->balloon_lock) is called in order to
> serialize against fill_balloon(). But in fill_balloon(),
> alloc_page(GFP_HIGHUSER[_MOVABLE] | __GFP_NOMEMALLOC | __GFP_NORETRY) is
> called with vb->balloon_lock mutex held. Since GFP_HIGHUSER[_MOVABLE] implies
> __GFP_DIRECT_RECLAIM | __GFP_IO | __GFP_FS, this allocation attempt might
> depend on somebody else's __GFP_DIRECT_RECLAIM | !__GFP_NORETRY memory
> allocation. Such __GFP_DIRECT_RECLAIM | !__GFP_NORETRY allocation can reach
> __alloc_pages_may_oom() and hold oom_lock mutex and call out_of_memory().
> And leak_balloon() is called by virtballoon_oom_notify() via
> blocking_notifier_call_chain() callback when vb->balloon_lock mutex is already
> held by fill_balloon(). As a result, despite __GFP_NORETRY is specified,
> fill_balloon() can indirectly get stuck waiting for vb->balloon_lock mutex
> at leak_balloon().
That would be tricky to fix. I guess we'll need to drop the lock
while allocating memory - not an easy fix.
> Also, in leak_balloon(), virtqueue_add_outbuf(GFP_KERNEL) is called via
> tell_host(). Reaching __alloc_pages_may_oom() from this virtqueue_add_outbuf()
> request from leak_balloon() from virtballoon_oom_notify() from
> blocking_notifier_call_chain() from out_of_memory() leads to OOM lockup
> because oom_lock mutex is already held before calling out_of_memory().
I guess we should just do
GFP_KERNEL & ~__GFP_DIRECT_RECLAIM there then?
>
> OOM notifier callback should not (directly or indirectly) depend on
> __GFP_DIRECT_RECLAIM memory allocation attempt. Can you fix this dependency?
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: jasowang@redhat.com, virtualization@lists.linux-foundation.org,
linux-mm@kvack.org
Subject: Re: mm, virtio: possible OOM lockup at virtballoon_oom_notify()
Date: Fri, 29 Sep 2017 07:00:05 +0300 [thread overview]
Message-ID: <20170929065654-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <201709111927.IDD00574.tFVJHLOSOOMQFF@I-love.SAKURA.ne.jp>
On Mon, Sep 11, 2017 at 07:27:19PM +0900, Tetsuo Handa wrote:
> Hello.
>
> I noticed that virtio_balloon is using register_oom_notifier() and
> leak_balloon() from virtballoon_oom_notify() might depend on
> __GFP_DIRECT_RECLAIM memory allocation.
>
> In leak_balloon(), mutex_lock(&vb->balloon_lock) is called in order to
> serialize against fill_balloon(). But in fill_balloon(),
> alloc_page(GFP_HIGHUSER[_MOVABLE] | __GFP_NOMEMALLOC | __GFP_NORETRY) is
> called with vb->balloon_lock mutex held. Since GFP_HIGHUSER[_MOVABLE] implies
> __GFP_DIRECT_RECLAIM | __GFP_IO | __GFP_FS, this allocation attempt might
> depend on somebody else's __GFP_DIRECT_RECLAIM | !__GFP_NORETRY memory
> allocation. Such __GFP_DIRECT_RECLAIM | !__GFP_NORETRY allocation can reach
> __alloc_pages_may_oom() and hold oom_lock mutex and call out_of_memory().
> And leak_balloon() is called by virtballoon_oom_notify() via
> blocking_notifier_call_chain() callback when vb->balloon_lock mutex is already
> held by fill_balloon(). As a result, despite __GFP_NORETRY is specified,
> fill_balloon() can indirectly get stuck waiting for vb->balloon_lock mutex
> at leak_balloon().
That would be tricky to fix. I guess we'll need to drop the lock
while allocating memory - not an easy fix.
> Also, in leak_balloon(), virtqueue_add_outbuf(GFP_KERNEL) is called via
> tell_host(). Reaching __alloc_pages_may_oom() from this virtqueue_add_outbuf()
> request from leak_balloon() from virtballoon_oom_notify() from
> blocking_notifier_call_chain() from out_of_memory() leads to OOM lockup
> because oom_lock mutex is already held before calling out_of_memory().
I guess we should just do
GFP_KERNEL & ~__GFP_DIRECT_RECLAIM there then?
>
> OOM notifier callback should not (directly or indirectly) depend on
> __GFP_DIRECT_RECLAIM memory allocation attempt. Can you fix this dependency?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-09-29 4:00 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-11 10:27 mm, virtio: possible OOM lockup at virtballoon_oom_notify() Tetsuo Handa
2017-09-29 4:00 ` Michael S. Tsirkin [this message]
2017-09-29 4:00 ` Michael S. Tsirkin
2017-09-29 4:44 ` Tetsuo Handa
2017-10-01 5:44 ` [RFC] [PATCH] mm, oom: Offload OOM notify callback to a kernel thread Tetsuo Handa
2017-10-02 3:59 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 3:59 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 3:59 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 9:06 ` Michal Hocko
2017-10-02 11:33 ` [RFC] [PATCH] mm, oom: " Tetsuo Handa
2017-10-02 11:50 ` [RFC] [PATCH] mm,oom: " Michal Hocko
2017-10-02 11:50 ` [RFC] [PATCH] mm, oom: " Michal Hocko
2017-10-02 13:05 ` [RFC] [PATCH] mm,oom: " Tetsuo Handa
2017-10-02 13:13 ` Michal Hocko
2017-10-02 13:52 ` Tetsuo Handa
2017-10-02 14:23 ` Michael S. Tsirkin
2017-10-02 14:44 ` Tetsuo Handa
2017-10-07 11:30 ` Tetsuo Handa
2017-10-09 7:46 ` Michal Hocko
2017-10-09 8:06 ` Tetsuo Handa
2017-10-09 12:28 ` Michal Hocko
2017-10-09 13:31 ` Tetsuo Handa
2017-10-09 13:37 ` Michal Hocko
2017-10-09 14:24 ` Michael S. Tsirkin
2017-10-02 14:15 ` Michael S. Tsirkin
2017-10-02 14:11 ` Michael S. Tsirkin
2017-10-02 14:11 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 14:11 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 14:19 ` [RFC] [PATCH] mm, oom: " Michal Hocko
2017-10-02 14:19 ` [RFC] [PATCH] mm,oom: " Michal Hocko
2017-10-02 14:29 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 14:29 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 14:31 ` Michal Hocko
2017-10-02 14:31 ` Michal Hocko
2017-10-02 14:29 ` Michael S. Tsirkin
2017-10-02 14:19 ` Michal Hocko
2017-10-02 9:06 ` Michal Hocko
2017-09-29 4:44 ` mm, virtio: possible OOM lockup at virtballoon_oom_notify() Tetsuo Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170929065654-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=linux-mm@kvack.org \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.