From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B7C8C636CC for ; Thu, 16 Feb 2023 14:41:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73F1A6B0078; Thu, 16 Feb 2023 09:41:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EE46280002; Thu, 16 Feb 2023 09:41:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58ED9280001; Thu, 16 Feb 2023 09:41:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4AF746B0078 for ; Thu, 16 Feb 2023 09:41:30 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1CB601A1017 for ; Thu, 16 Feb 2023 14:41:30 +0000 (UTC) X-FDA: 80473418340.15.BFDA951 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id BDED0A001A for ; Thu, 16 Feb 2023 14:41:27 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H9sDj7Kr; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676558487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s9RDP1c6xkNkVax9cafkp6rZ/0yeZhIak6lzwPuSvqM=; b=fNkmzhCZ+Z0AzvVYzbHYFqrQ6WRGCDSEWd8hd12iUCS02kRLEuBBKh7V41z3DW/eQDlNCt G0Ofy5aZ3Ax/GB/v0N54gWiW3H3P5f7MO9yNpB+quA5sD/2v71J29OzjZHdc4d7OSucnHR bQS05q/MNhTC46Obr7A9YZuCeC5I8RY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H9sDj7Kr; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676558487; a=rsa-sha256; cv=none; b=gK1BwQ1ovrwswjs/kQMiRjRY0TE3bJ6QePFYraB6AISvv1uqM2DmwiTIAkLzSg1xB74SzV IKz2CKCmWrTG3Ru8Nj380VAgT7z9F3LqoPATViUXHex6uyCESErqs+t+w43T3Eb8TuDkIM vCxTvvGJxqFOSPwFZqpk452DKtmVekA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676558486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=s9RDP1c6xkNkVax9cafkp6rZ/0yeZhIak6lzwPuSvqM=; b=H9sDj7KrU+nGtRjzleOXeqGJr+H4Z05bS1IWQKl6FIehULczBwZ/pAYlnh4Zj4Dn2KqnfJ yxBnG/v09Ao4yqxY0NS7J0s01uLVJ6C1HX27l5yXgt9cGMOM0ATQx/fRSXgtZoC2X302XM 9+OmQQcT+FtjW/VEqAYVIkSQ+urvbP8= Received: from mail-il1-f198.google.com (mail-il1-f198.google.com [209.85.166.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-54--h9okGgqOxuaTD_4f3NInA-1; Thu, 16 Feb 2023 09:41:25 -0500 X-MC-Unique: -h9okGgqOxuaTD_4f3NInA-1 Received: by mail-il1-f198.google.com with SMTP id o10-20020a056e02102a00b003006328df7bso1360317ilj.17 for ; Thu, 16 Feb 2023 06:41:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=s9RDP1c6xkNkVax9cafkp6rZ/0yeZhIak6lzwPuSvqM=; b=ZRB7P9o6+EHaZO2KTHar8+X3caeBOH6JVzZBrKUuaRn6/kxJgv/txo2XaO6Qe/oODB xbH/dvndwzYaQhhLeUhx1ur4oW98UvvJgW3IJVfJl+mlzzQordMkKHzlfEX/k4oorusH HxryvcXzpOPE3lDhrpprmE5EV+p/13qD3/G0Jr4PwIBdvDqhIDiLPQvB2Fq+sY8cPAB9 TTd3pmWGB9Kj5cLy+BNETmiOjLR9GE4ojLwl45KCIC7T0WstfnR4jiHr0TpkZbPd7ekv geI6HjiZxGpG19IL8xnX39rzpobBE/m/kuG649IL2G36khXyBSbs8vCqgEi5XN/ErLVB fPIQ== X-Gm-Message-State: AO0yUKUUK5C5hplcbEWdETXl3ab59PPmFdlZLF6nfsvjaZTStRw5fKSz CedUblJ+yG6XNY5TyDHFoJwN26URdyuC/p4SZrtOrRpxT+60LAC+vo4vuvIjqO75SmdmnKZOq9d tTpKs43Nk1Xk= X-Received: by 2002:a05:6602:5cd:b0:73a:6c75:5a85 with SMTP id w13-20020a05660205cd00b0073a6c755a85mr3626516iox.0.1676558484712; Thu, 16 Feb 2023 06:41:24 -0800 (PST) X-Google-Smtp-Source: AK7set+98roKKBnuZxMyibOAjamWK9+xx6bFibIU+6FWg5RGiFXyilSLEuIP/FrNR9EITJLB0rpQUA== X-Received: by 2002:a05:6602:5cd:b0:73a:6c75:5a85 with SMTP id w13-20020a05660205cd00b0073a6c755a85mr3626503iox.0.1676558484380; Thu, 16 Feb 2023 06:41:24 -0800 (PST) Received: from x1n (bras-base-aurron9127w-grc-56-70-30-145-63.dsl.bell.ca. [70.30.145.63]) by smtp.gmail.com with ESMTPSA id b9-20020a6be709000000b0074549126e97sm166725ioh.1.2023.02.16.06.41.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Feb 2023 06:41:23 -0800 (PST) Date: Thu, 16 Feb 2023 09:41:22 -0500 From: Peter Xu To: David Stevens Cc: linux-mm@kvack.org, Matthew Wilcox , Andrew Morton , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , linux-kernel@vger.kernel.org Subject: Re: [PATCH 2/2] mm/khugepaged: skip shmem with userfaultfd Message-ID: References: <20230214075710.2401855-1-stevensd@google.com> <20230214075710.2401855-2-stevensd@google.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Stat-Signature: d6f5x3aahhm6se6qy5uga5cd7adwykoa X-Rspam-User: X-Rspamd-Queue-Id: BDED0A001A X-Rspamd-Server: rspam06 X-HE-Tag: 1676558487-261258 X-HE-Meta: U2FsdGVkX1/lGjCegNbuIt4PWGtHlQBDUHYdGTcDuTlFVe/igfcd0Vzl2ZoikfF1+KnNHpbu/be+pCBViMmW5C6lVaK4tm0rjjsFWqSQyzjsC13+0ucQUTHJs8Z7m6ujQDdhshux6YfpDDVk8GfRBq9tljbjVufa5HiLsObf8HkoSI15O6xx2H67zwVVjTHuPNnEfOVNK0qcqj6JSwpsNTs58QIxsDyFjp2+PPgA1gx911PnUDRwsdQJQwvHzrmGTI0nLHJlZa2l4WbzhJIxAPDSrkJUuPqxjcdjXQkXAqq43aINTN6TlMJE4ZT2K14aMom8S73nWj/S6wUnzMKJzhSW6Mr1SS2Bm0HJImpT/Sl2Rh4+IPZD9WnTLhf5tpJKKkgqwCTm5YdSRwyeYdxuVZgUddUhMYSuJqgeF54xv+VYRnRU0josH3MLfty1ACsliScXccIZZx+3809PSKkYgSlSbhSJ/USFmJCHcYAEvqe9I2MSYR5wWkhgEa3C0npFxUlSyJvnXHp+UR8BXFVZLujtl6EashsJD5K8dFZdfG2A4NHE5h0tavBdnoHcbSA+C310hzieXEBdb6UYjIXawksAXjk3SlPVb+rFdPUoSm8FmOuAGo2fLZurWb9BIhvJrVDWYEqBixdZ3xKNThhw5EDS9lFMnPMvhlRV3GmmkU5B0Jiz1Ia2puJfInuJXzWeb+6PpcGBZAdVaBPNvuKo4n48pFseNYXgBcZiVx1iC4sQKC179ZGtoWb0WJyZEDv3c5/iaJ4pSEY9j8QIPBZwtOx4EdfzMSuHdNgERllG7N4CCLR54ffBmjc/MOCilthbG2IdCPTkXedMe1UDoq6zejLeN5LM1qoKlIXbRK2Rs8mYbgPRAwe8n2YIuXNrJpiFn3pg4/hixy7JI3i7/MFHrocrkNp2I6NYurAJmfE5Bk+WtU3I5WGjUT93Wqfjn3zCwkhjF54+wfTISpQO0Ao m65EOiTf IbHcsjyw8KbiaO5OgT1zlgaIe7sdacthE5WpqaFGvBxS3HP3gl2sv/qKv1Xh8tMULEX9TbYDK/dvBkAWNtehLuoHpUqYkwiKBb6MMLCZnpi72UsMLByuBc7TuGlR7Cffwga0E6DA5F1ktGepYU43SfRPIDBhoIjldJmAQh11uLVZUhEHvooi2o5TpQcMi4qrL66wHsFzsH67BrYGwlaVOTeYVpsgYLWnCHAGP59z61dF9xVP3WygPkL0rAFd+N1/88dHTKD3nNCGZkSI18GBXhtgyMxylewEpdsQvwptbPR7vZlLJGRnh8Cktq2RgnIsdKI95svnHG65t83adfykKz021ik5YUSH4ysoz+BarI4kupIv8PAA30B+lMaUBy96tKhGr6pl5fwCndeRBeK4sIMrQRFu1ljemkA/rLfxZ5a3iXXdflBzxro1Kb55jRt6S3OIRvtfppKPNpnc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 16, 2023 at 10:37:47AM +0900, David Stevens wrote: > On Thu, Feb 16, 2023 at 7:48 AM Peter Xu wrote: > > > > On Tue, Feb 14, 2023 at 04:57:10PM +0900, David Stevens wrote: > > > From: David Stevens > > > > > > Make sure that collapse_file respects any userfaultfds registered with > > > MODE_MISSING. If userspace has any such userfaultfds registered, then > > > for any page which it knows to be missing, it may expect a > > > UFFD_EVENT_PAGEFAULT. This means collapse_file needs to take care when > > > collapsing a shmem range would result in replacing an empty page with a > > > THP, so that it doesn't break userfaultfd. > > > > > > Synchronization when checking for userfaultfds in collapse_file is > > > tricky because the mmap locks can't be used to prevent races with the > > > registration of new userfaultfds. Instead, we provide synchronization by > > > ensuring that userspace cannot observe the fact that pages are missing > > > before we check for userfaultfds. Although this allows registration of a > > > userfaultfd to race with collapse_file, it ensures that userspace cannot > > > observe any pages transition from missing to present after such a race. > > > This makes such a race indistinguishable to the collapse occurring > > > immediately before the userfaultfd registration. > > > > > > The first step to provide this synchronization is to stop filling gaps > > > during the loop iterating over the target range, since the page cache > > > lock can be dropped during that loop. The second step is to fill the > > > gaps with XA_RETRY_ENTRY after the page cache lock is acquired the final > > > time, to avoid races with accesses to the page cache that only take the > > > RCU read lock. > > > > > > This fix is targeted at khugepaged, but the change also applies to > > > MADV_COLLAPSE. MADV_COLLAPSE on a range with a userfaultfd will now > > > return EBUSY if there are any missing pages (instead of succeeding on > > > shmem and returning EINVAL on anonymous memory). There is also now a > > > window during MADV_COLLAPSE where a fault on a missing page will cause > > > the syscall to fail with EAGAIN. > > > > > > The fact that intermediate page cache state can no longer be observed > > > before the rollback of a failed collapse is also technically a > > > userspace-visible change (via at least SEEK_DATA and SEEK_END), but it > > > is exceedingly unlikely that anything relies on being able to observe > > > that transient state. > > > > > > Signed-off-by: David Stevens > > > --- > > > mm/khugepaged.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > > > 1 file changed, 58 insertions(+), 8 deletions(-) > > > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > index b648f1053d95..8c2e2349e883 100644 > > > --- a/mm/khugepaged.c > > > +++ b/mm/khugepaged.c > > > @@ -55,6 +55,7 @@ enum scan_result { > > > SCAN_CGROUP_CHARGE_FAIL, > > > SCAN_TRUNCATED, > > > SCAN_PAGE_HAS_PRIVATE, > > > + SCAN_PAGE_FILLED, > > > > PS: You may want to also touch SCAN_STATUS in huge_memory.h next time. > > > > > }; > > > > > > #define CREATE_TRACE_POINTS > > > @@ -1725,8 +1726,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, > > > * - allocate and lock a new huge page; > > > * - scan page cache replacing old pages with the new one > > > * + swap/gup in pages if necessary; > > > - * + fill in gaps; > > > > IIUC it's not a complete removal, but just moved downwards: > > > > > * + keep old pages around in case rollback is required; > > > + * - finalize updates to the page cache; > > > > + fill in gaps with RETRY entries > > + detect race conditions with userfaultfds > > > > > * - if replacing succeeds: > > > * + copy data over; > > > * + free old pages; > > > @@ -1805,13 +1806,12 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, > > > result = SCAN_TRUNCATED; > > > goto xa_locked; > > > } > > > - xas_set(&xas, index); > > > + xas_set(&xas, index + 1); > > > } > > > if (!shmem_charge(mapping->host, 1)) { > > > result = SCAN_FAIL; > > > goto xa_locked; > > > } > > > - xas_store(&xas, hpage); > > > nr_none++; > > > continue; > > > } > > > @@ -1970,6 +1970,56 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, > > > put_page(page); > > > goto xa_unlocked; > > > } > > > + > > > + if (nr_none) { > > > + struct vm_area_struct *vma; > > > + int nr_none_check = 0; > > > + > > > + xas_unlock_irq(&xas); > > > + i_mmap_lock_read(mapping); > > > + xas_lock_irq(&xas); > > > + > > > + xas_set(&xas, start); > > > + for (index = start; index < end; index++) { > > > + if (!xas_next(&xas)) { > > > + xas_store(&xas, XA_RETRY_ENTRY); > > > + nr_none_check++; > > > + } > > > + } > > > + > > > + if (nr_none != nr_none_check) { > > > + result = SCAN_PAGE_FILLED; > > > + goto immap_locked; > > > + } > > > + > > > + /* > > > + * If userspace observed a missing page in a VMA with an armed > > > + * userfaultfd, then it might expect a UFFD_EVENT_PAGEFAULT for > > > + * that page, so we need to roll back to avoid suppressing such > > > + * an event. Any userfaultfds armed after this point will not be > > > + * able to observe any missing pages due to the previously > > > + * inserted retry entries. > > > + */ > > > + vma_interval_tree_foreach(vma, &mapping->i_mmap, start, start) { > > > + if (userfaultfd_missing(vma)) { > > > + result = SCAN_EXCEED_NONE_PTE; > > > + goto immap_locked; > > > + } > > > + } > > > + > > > +immap_locked: > > > + i_mmap_unlock_read(mapping); > > > + if (result != SCAN_SUCCEED) { > > > + xas_set(&xas, start); > > > + for (index = start; index < end; index++) { > > > + if (xas_next(&xas) == XA_RETRY_ENTRY) > > > + xas_store(&xas, NULL); > > > + } > > > + > > > + goto xa_locked; > > > + } > > > + } > > > + > > > > Until here, all look fine to me (ignoring patch 1 for now; assuming the > > hpage is always uptodate). > > > > My question is after here we'll release page cache lock again before > > try_to_unmap_flush(), but is it safe to keep RETRY entries after releasing > > page cache lock? It means other threads can be spinning. I assume page > > lock is always safe and sleepable, but not sure about the page cache lock > > here. > > We insert the multi-index entry for hpage before releasing the page > cache lock, which should replace all of the XA_RETRY_ENTRYs. So the > page cache will be fully up to date when we release the lock, at least > in terms of which pages it contains. IIUC we released it before copying the pages: xa_locked: xas_unlock_irq(&xas); <-------------------------------- here xa_unlocked: /* * If collapse is successful, flush must be done now before copying. * If collapse is unsuccessful, does flush actually need to be done? * Do it anyway, to clear the state. */ try_to_unmap_flush(); Before insertion of the multi-index: /* Join all the small entries into a single multi-index entry. */ xas_set_order(&xas, start, HPAGE_PMD_ORDER); xas_store(&xas, hpage); Thanks, -- Peter Xu