Re: [PATCH 0/2] coredump: make SIGNAL_GROUP_COREDUMP more friendly to oom-killer

[Oleg Nesterov <oleg@redhat.com>]

On 09/30, Tetsuo Handa wrote:
>
> Oleg Nesterov wrote:
> > Just in case, this doesn't depend on the previous series I sent.
> >
> > Tetsuo, iirc we already discussed the change in 1/2 some time ago,
> > could you review?
> >
> > Oleg.
>
> I tested patch 1/2 and 2/2 on next-20150929 using reproducer at
> http://lkml.kernel.org/r/201503150240.GII00591.OVSFtQLOFOHJMF@I-love.SAKURA.ne.jp .
>
>   $ while :; do ./a.out; done
>
> Unfortunately, since hangup on coredump to pipe occurs sometimes,
> I can't tell whether this patchset solves hangup on coredump to pipe.

Obviously it doesn't. There are a lot more problems here.

It is hardly possible to enumerate them, but let me quote the changelog
from d003f371b27016354c

    Note: this is only the first step, this patch doesn't try to solve other
    problems.  The SIGNAL_GROUP_COREDUMP check is obviously racy, a task can
    participate in coredump after it was already observed in PF_EXITING state,
    so TIF_MEMDIE (which also blocks oom-killer) still can be wrongly set.
    fatal_signal_pending() can be true because of SIGNAL_GROUP_COREDUMP so
    out_of_memory() and mem_cgroup_out_of_memory() shouldn't blindly trust it.
    And even the name/usage of the new helper is confusing, an exiting thread
    can only free its ->mm if it is the only/last task in thread group.

This patch just makes the SIGNAL_GROUP_COREDUMP check in task_will_free_mem()
a bit more correct wrt CLONE_VM tasks, nothing more.

Oleg.