From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC077C433EF for ; Wed, 30 Mar 2022 21:36:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A7C88D0001; Wed, 30 Mar 2022 17:36:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 457BD6B0073; Wed, 30 Mar 2022 17:36:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D2398D0001; Wed, 30 Mar 2022 17:36:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 188B16B0072 for ; Wed, 30 Mar 2022 17:36:33 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C87B91828AC88 for ; Wed, 30 Mar 2022 21:36:32 +0000 (UTC) X-FDA: 79302361824.31.321C5DA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 4E3E7C0005 for ; Wed, 30 Mar 2022 21:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648676191; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nq2QaWKmjRi7ediHTAyvB1XNhkWF7u/Z5LX0xWbEYz0=; b=QIuJ2LwDEgrhaNk79yDwbPlylrpJlNymCWT5WlqvOH95Wmk8mw+4BB8yJXzWDmqETT5onK 0YB2m66L6mjLjhbMDfr+OCHn2bHBoVMtlnxHsUPN8u/pLJUcu427f3elN+pvds33wu0JsC lzPDvr1Ixcgi4CX0+xtt3EYY0gWn5xw= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-548-nZZOPP2KN3Gigv1_gFzphA-1; Wed, 30 Mar 2022 17:36:28 -0400 X-MC-Unique: nZZOPP2KN3Gigv1_gFzphA-1 Received: by mail-io1-f70.google.com with SMTP id u10-20020a5ec00a000000b00648e5804d5bso15231110iol.12 for ; Wed, 30 Mar 2022 14:36:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:from:to:cc:references:in-reply-to :content-transfer-encoding; bh=nq2QaWKmjRi7ediHTAyvB1XNhkWF7u/Z5LX0xWbEYz0=; b=GDuZiNHbCW7PGze9V+5Nsk1K0n9mH9zmI3U+ScX/lMQfUgbsw5R+S8qc2B64/ZYTRC naqjoXtmJ+aZwOotVHhRxNf6pFnTCZ5y09BdBBnUBKcr3nht0TSlT4/FXnZ+4bcw4bdN 3IneUvtidJLN8/1HMYGnPbn6Ujb6Eu+hS0Oh9Rkw9f+b03FdQODGRYxg0OUdpRo4jH9n 1NqWV/LA5w82+dOZv2c/tAl1IXFKDz1OR8DQOzaUIOHpIy+wtp6aqPDLKPTjA+3Od1UI QY9yC1uGoKQZV5NodKL5AbK26vfA5BTyoj0PhrTuTgK4NlR1aE6pYyuJHqrF16qn0lYw L7Hg== X-Gm-Message-State: AOAM53101S+JQK4NpT01NgwudrVnZxqpdNY4U7463oMn3/tYHPcuVICF dqsUNZzmN1oq/MDkLxSQUY/sHc6eR4Sl17Nn+XhMNQKNIZjH03y1/qi+zS62u+/Qjj0xmTUMXow o6EUlgUCYY4A= X-Received: by 2002:a05:6638:d0c:b0:31a:5d8a:c013 with SMTP id q12-20020a0566380d0c00b0031a5d8ac013mr1095309jaj.132.1648676187678; Wed, 30 Mar 2022 14:36:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxYtx5RIUIAL5R23SCsLiL/45ug3XfeCC3Y2ZFaEe+SFSXjnE7oqU9hxSRpEVlxEqP4tjYw4w== X-Received: by 2002:a05:6638:d0c:b0:31a:5d8a:c013 with SMTP id q12-20020a0566380d0c00b0031a5d8ac013mr1095294jaj.132.1648676187476; Wed, 30 Mar 2022 14:36:27 -0700 (PDT) Received: from ?IPV6:2601:280:4400:a2e0:7336:512c:930d:4f0e? ([2601:280:4400:a2e0:7336:512c:930d:4f0e]) by smtp.gmail.com with ESMTPSA id ay18-20020a5d9d92000000b0064c77f6aaecsm6121018iob.3.2022.03.30.14.36.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 30 Mar 2022 14:36:27 -0700 (PDT) Message-ID: <0991b55e-3d69-a591-9bf4-26013b6ba843@redhat.com> Date: Wed, 30 Mar 2022 15:36:25 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: [PATCH v5] mm/oom_kill.c: futex: Close a race between do_exit and the oom_reaper From: Nico Pache To: Michal Hocko , Thomas Gleixner Cc: Davidlohr Bueso , linux-mm@kvack.org, Andrea Arcangeli , Joel Savitz , Andrew Morton , linux-kernel@vger.kernel.org, Rafael Aquini , Waiman Long , Baoquan He , Christoph von Recklinghausen , Don Dutile , "Herton R . Krzesinski" , Ingo Molnar , Peter Zijlstra , Darren Hart , Andre Almeida , David Rientjes References: <20220318033621.626006-1-npache@redhat.com> <20220322004231.rwmnbjpq4ms6fnbi@offworld> <20220322025724.j3japdo5qocwgchz@offworld> <87bkxyaufi.ffs@tglx> <87zglha9rt.ffs@tglx> In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QIuJ2LwD; spf=none (imf22.hostedemail.com: domain of npache@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=npache@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4E3E7C0005 X-Stat-Signature: y38rzkebh4kfg8389f97zrqt59wjpdbe X-HE-Tag: 1648676192-81089 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 3/30/22 12:18, Nico Pache wrote: > > > On 3/30/22 03:18, Michal Hocko wrote: >> Nico, >> >> On Wed 23-03-22 10:17:29, Michal Hocko wrote: >>> Let me skip over futex part which I need to digest and only focus on the >>> oom side of the things for clarification. >>> >>> On Tue 22-03-22 23:43:18, Thomas Gleixner wrote: >> [...] >>>> You can easily validate that by doing: >>>> >>>> wake_oom_reaper(task) >>>> task->reap_time = jiffies + HZ; >>>> queue_task(task); >>>> wakeup(reaper); >>>> >>>> and then: >>>> >>>> oom_reap_task(task) >>>> now = READ_ONCE(jiffies); >>>> if (time_before(now, task->reap_time) >>>> schedule_timeout_idle(task->reap_time - now); >>>> >>>> before trying to actually reap the mm. >>>> >>>> That will prevent the enforced race in most cases and allow the exiting >>>> and/or killed processes to cleanup themself. Not pretty, but it should >>>> reduce the chance of the reaper to win the race with the exiting and/or >>>> killed process significantly. >>>> >>>> It's not going to work when the problem is combined with a heavy VM >>>> overload situation which keeps a guest (or one/some it's vCPUs) away >>>> from being scheduled. See below for a discussion of guarantees. >>>> >>>> If it failed to do so when the sleep returns, then you still can reap >>>> it. >>> >>> Yes, this is certainly an option. Please note that the oom_reaper is not >>> the only way to trigger this. process_mrelease syscall performs the same >>> operation from the userspace. Arguably process_mrelease could be used >>> sanely/correctly because the userspace oom killer can do pro-cleanup >>> steps before going to final SIGKILL & process_mrelease. One way would be >>> to send SIGTERM in the first step and allow the victim to perform its >>> cleanup. >> >> are you working on another version of the fix/workaround based on the >> discussion so far? > > We are indeed! Sorry for the delay we've been taking the time to do our due > diligence on some of the claims made. We are also spending time rewriting the > reproducer to include more test cases that Thomas brought up. > > Ill summarize here, and reply to the original emails in more detail.... > > Firstly, we have implemented & tested the VMA skipping... it does fix our case. > Thomas brought up a few good points about the robust list head and the potential > waiters being in different VMAs; however, I think its a moot point, given that > the locks will only be reaped if allocated as ((private|anon)|| !shared). Sorry... not completely moot. As Thomas pointed out, a robust list with the following structure will probably fail to recover its waiters: TLS (robust head, skip)* --> private lock (reaped) --> shared lock (not reaped) We are working on getting a test case with multiple locks and mixed mapping types to prove this. Skipping the robust list head VMA will be beneficial in cases were the robust list is full of shared locks: TLS (robust head, skip)* --> shared lock(not reaped) --> shared lock(not reaped) -- Nico