Re: How to handle TIF_MEMDIE stalls?

Tetsuo Handa
2015-02-16T12:22:50Z

Michal Hocko wrote:
> > but I think we need to be prepared for cases where sending SIGKILL to
> > all threads sharing the same memory does not help.
>
> Sure, unkillable tasks are a problem which we have to handle. Having
> GFP_KERNEL allocations looping without way out contributes to this which
> is sad but your current data just show that sometimes it might take ages
> to finish even without that going on.

Hello. Can we resume TIF_MEMDIE stall discussion?

I'd like to propose

(1) Make several locks killable.

(2) Implement TIF_MEMDIE timeout.

(3) Replace kmalloc() with kmalloc_nofail() and kmalloc_noretry().

for handling TIF_MEMDIE stall problems.

(1) Make several locks killable.

On Linux 3.19, running below command line as an unprivileged user
on a system with 4 CPUs / 2GB RAM / no swap can make the system unusable.

$ for i in `seq 1 100`; do dd if=/dev/zero of=/tmp/file bs=104857600 count=100 & done

---------- An example with ext4 partition ----------
(...snipped...)
[ 369.902616] dd D ffff88007fc12d00 0 9113 6418 0x00000080
[ 369.904867] ffff88007b460890 0000000000012d00 ffff88007b28ffd8 0000000000012d00
[ 369.907254] ffff88007b460890 ffff88007fc12d80 ffff88007a6eb360 0000000000000001
[ 369.909855] ffffffff810946cb 00000000000025f6 ffffffff8108ef1d 0000000000000000
[ 369.912054] Call Trace:
[ 369.913175] [<ffffffff810946cb>] ? put_prev_entity+0x5b/0x2c0
[ 369.914960] [<ffffffff8108ef1d>] ? pick_next_entity+0x9d/0x170
[ 369.916778] [<ffffffff8109157e>] ? set_next_entity+0x4e/0x60
[ 369.918634] [<ffffffff81097953>] ? pick_next_task_fair+0x453/0x520
[ 369.920530] [<ffffffff8100c6e0>] ? __switch_to+0x240/0x570
[ 369.922263] [<ffffffff815799f9>] ? schedule_preempt_disabled+0x9/0x10
[ 369.924161] [<ffffffff8157af25>] ? __mutex_lock_slowpath+0xb5/0x120
[ 369.926106] [<ffffffff8157afa6>] ? mutex_lock+0x16/0x25
[ 369.927800] [<ffffffffa01f3acc>] ? ext4_file_write_iter+0x7c/0x3a0 [ext4]
[ 369.929778] [<ffffffff81280fbc>] ? __clear_user+0x1c/0x40
[ 369.931491] [<ffffffff8112c876>] ? iov_iter_zero+0x66/0x2d0
[ 369.933235] [<ffffffff811732a3>] ? new_sync_write+0x83/0xd0
[ 369.934977] [<ffffffff8117397d>] ? vfs_write+0xad/0x1f0
[ 369.936703] [<ffffffff8101b57b>] ? syscall_trace_enter_phase1+0x19b/0x1b0
[ 369.938674] [<ffffffff8117459d>] ? SyS_write+0x4d/0xc0
[ 369.940336] [<ffffffff8157d329>] ? system_call_fastpath+0x12/0x17
(...snipped...)
[ 498.421741] SysRq : Manual OOM execution
[ 498.423627] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[ 498.952807] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[ 498.954450] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB
(...snipped...)
[ 502.068921] SysRq : Manual OOM execution
[ 502.070825] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[ 502.618222] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[ 502.620016] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB
(...snipped...)
[ 503.900554] SysRq : Manual OOM execution
[ 503.902387] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[ 504.410444] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[ 504.412221] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB
(...snipped...)
---------- An example with ext4 partition ----------

---------- An example with xfs partition ----------
(...snipped...)
[ 127.135041] Out of memory: Kill process 2505 (dd) score 59 or sacrifice child
[ 127.136460] Killed process 2505 (dd) total-vm:210340kB, anon-rss:102464kB, file-rss:1728kB
(...snipped...)
[ 243.672302] dd D ffff88005bd27cb8 12776 2505 2386 0x00100084
[ 243.674066] ffff88005bd27cb8 ffff88005bd27c98 ffff88007850c740 0000000000014080
[ 243.676005] 0000000000000000 ffff88005bd27fd8 0000000000014080 ffff88005835d740
[ 243.677916] ffff88007850c740 0000000000000014 ffff8800669bee50 ffff8800669bee54
[ 243.679823] Call Trace:
[ 243.680478] [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[ 243.682047] [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[ 243.683548] [<ffffffff816b83e8>] ? page_fault+0x28/0x30
[ 243.684875] [<ffffffff816b4463>] mutex_lock+0x23/0x37
[ 243.686146] [<ffffffff8129df6c>] xfs_file_buffered_aio_write+0x6c/0x240
[ 243.687791] [<ffffffff813497b5>] ? __clear_user+0x25/0x50
[ 243.689121] [<ffffffff8117294d>] ? iov_iter_zero+0x6d/0x2e0
[ 243.690511] [<ffffffff8129e1b8>] xfs_file_write_iter+0x78/0x110
[ 243.691990] [<ffffffff811beb31>] new_sync_write+0x81/0xb0
[ 243.693329] [<ffffffff811bf2a7>] vfs_write+0xb7/0x1f0
[ 243.694581] [<ffffffff811bfeb6>] SyS_write+0x46/0xb0
[ 243.695834] [<ffffffff81109196>] ? __audit_syscall_exit+0x236/0x2e0
[ 243.697376] [<ffffffff816b64a9>] system_call_fastpath+0x12/0x17
(...snipped...)
[ 291.433296] dd D ffff88005bd27cb8 12776 2505 2386 0x00100084
[ 291.433297] ffff88005bd27cb8 ffff88005bd27c98 ffff88007850c740 0000000000014080
[ 291.433298] 0000000000000000 ffff88005bd27fd8 0000000000014080 ffff88005835d740
[ 291.433298] ffff88007850c740 0000000000000014 ffff8800669bee50 ffff8800669bee54
[ 291.433299] Call Trace:
[ 291.433300] [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[ 291.433301] [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[ 291.433302] [<ffffffff816b83e8>] ? page_fault+0x28/0x30
[ 291.433303] [<ffffffff816b4463>] mutex_lock+0x23/0x37
[ 291.433304] [<ffffffff8129df6c>] xfs_file_buffered_aio_write+0x6c/0x240
[ 291.433306] [<ffffffff813497b5>] ? __clear_user+0x25/0x50
[ 291.433307] [<ffffffff8117294d>] ? iov_iter_zero+0x6d/0x2e0
[ 291.433308] [<ffffffff8129e1b8>] xfs_file_write_iter+0x78/0x110
[ 291.433309] [<ffffffff811beb31>] new_sync_write+0x81/0xb0
[ 291.433311] [<ffffffff811bf2a7>] vfs_write+0xb7/0x1f0
[ 291.433312] [<ffffffff811bfeb6>] SyS_write+0x46/0xb0
[ 291.433313] [<ffffffff81109196>] ? __audit_syscall_exit+0x236/0x2e0
[ 291.433314] [<ffffffff816b64a9>] system_call_fastpath+0x12/0x17
(...snipped...)
---------- An example with xfs partition ----------

This is because the OOM killer happily tries to kill a process which is
blocked at unkillable mutex_lock(). If locks shown above were killable,
we can reduce the possibility of getting stuck.

I didn't check whether it has livelocked or not. But too slow to wait is
not acceptable. Oh, why every thread trying to allocate memory has to repeat
the loop that might defer somebody who can make progress if CPU time was
given? I wish only somebody like kswapd repeats the loop on behalf of all
threads waiting at memory allocation slowpath...

(2) Implement TIF_MEMDIE timeout.

While the command line shown above is an artificial stresstest, I'm seeing
troubles on real KVM systems where the guests hang entirely with many
processes being blocked at jbd2_journal_commit_transaction() or
jbd2_journal_get_write_access(). The root cause of guest's stall is not yet
identified but is at least independent with TIF_MEMDIE. However, cron jobs
which are blocked at those functions after I/O stall begins exhaust all of
the system's memory and make the situation worse (e.g. load average exceeded
7000 on a guest with 2 CPUs as of occurrence of the OOM killer livelock).

Unkillable locks in non-critical paths can be replaced with killable locks.
But there are critical paths where fail-on-SIGKILL can lead to unwanted
results (e.g. filesystem's error action such as remount as r/o or call
panic() being taken), there are locks (e.g. rw_semaphore used by mmap_sem)
where killable version does not exist, and there are wait_for_completion()
calls where killable version does not worth complicating the code.

If TIF_MEMDIE timeout were implemented, we can cope with the OOM killer
livelock problem by choosing more OOM victims (for survive strategy) or
calling panic() (for debug and reboot strategy).

(3) Replace kmalloc() with kmalloc_nofail() and kmalloc_noretry().

Currently small allocations are implicitly treated like __GFP_NOFAIL
unless TIF_MEMDIE is set. But silently changing small allocations like
__GFP_NORETRY will cause obscure bugs. If TIF_MEMDIE timeout is implemented,
we will no longer worry about unkillable tasks which is retrying forever at
memory allocation; instead we kill more OOM victims and satisfy the request.
Therefore, we could introduce kmalloc_nofail(size, gfp) which does
kmalloc(size, gfp | __GFP_NOFAIL) (i.e. invoke the OOM killer) and
kmalloc_noretry(size, gfp) which does kmalloc(size, gfp | __GFP_NORETRY)
(i.e. do not invoke the OOM killer), and switch from kmalloc() to either
kmalloc_noretry() or kmalloc_nofail(). Those who are doing smaller than
PAGE_SIZE bytes allocations would wish to switch from kmalloc() to
kmalloc_nofail() and eliminate untested memory allocation failure paths.
Those who are well prepared for memory allocation failures would wish to
switch from kmalloc() to kmalloc_noretry(). Eventually, kmalloc() which is
implicitly treating small allocations like __GFP_NOFAIL and invoking the
OOM killer will be abolished.