From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1950C433E6 for ; Tue, 9 Feb 2021 21:17:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 207D164E85 for ; Tue, 9 Feb 2021 21:17:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 207D164E85 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 59AD06B0005; Tue, 9 Feb 2021 16:17:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 548E06B006C; Tue, 9 Feb 2021 16:17:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45DFF6B006E; Tue, 9 Feb 2021 16:17:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 2FB7C6B0005 for ; Tue, 9 Feb 2021 16:17:50 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EC5B8185726E0 for ; Tue, 9 Feb 2021 21:17:49 +0000 (UTC) X-FDA: 77799991458.02.crate67_25058aa2760a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id D271310097AA2 for ; Tue, 9 Feb 2021 21:17:49 +0000 (UTC) X-HE-Tag: crate67_25058aa2760a X-Filterd-Recvd-Size: 5356 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Tue, 9 Feb 2021 21:17:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612905469; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uskw3PbEX7wEnO/ex/Xsz3m8Nxa2KyupOaIbM7L2BOs=; b=cHs+oSZdgZRQ4ln/bBunaKHmmfXdbzTHhopG8CqnL/KpDy7VdZwdAx0jxY9qUKoeHsq0IK 3NkhTtE2WezWxBEDXre/tkcTC97V9HilWuFkdmSrC57NjzM8IQUcn02ROK7T5e05Fltawj 5jZ0YWw8wsWLBunfavpglUlPx8Czjpc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-343-k2Jf9BXbPxKvPSd8tNYV6w-1; Tue, 09 Feb 2021 16:17:45 -0500 X-MC-Unique: k2Jf9BXbPxKvPSd8tNYV6w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AEB8F192CC44; Tue, 9 Feb 2021 21:17:42 +0000 (UTC) Received: from redhat.com (ovpn-115-63.rdu2.redhat.com [10.10.115.63]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AE56A60D11; Tue, 9 Feb 2021 21:17:40 +0000 (UTC) Date: Tue, 9 Feb 2021 16:17:38 -0500 From: Jerome Glisse To: Jason Gunthorpe Cc: Alistair Popple , Daniel Vetter , Linux MM , Nouveau Dev , Ben Skeggs , Andrew Morton , Linux Doc Mailing List , Linux Kernel Mailing List , kvm-ppc@vger.kernel.org, dri-devel , John Hubbard , Ralph Campbell Subject: Re: [PATCH 0/9] Add support for SVM atomics in Nouveau Message-ID: <20210209211738.GA834106@redhat.com> References: <20210209010722.13839-1-apopple@nvidia.com> <3426910.QXTomnrpqD@nvdebian> <20210209133520.GB4718@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20210209133520.GB4718@ziepe.ca> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 09, 2021 at 09:35:20AM -0400, Jason Gunthorpe wrote: > On Tue, Feb 09, 2021 at 11:57:28PM +1100, Alistair Popple wrote: > > On Tuesday, 9 February 2021 9:27:05 PM AEDT Daniel Vetter wrote: > > > > > > > > Recent changes to pin_user_pages() prevent the creation of pinned= pages in > > > > ZONE_MOVABLE. This series allows pinned pages to be created in=20 > > ZONE_MOVABLE > > > > as attempts to migrate may fail which would be fatal to userspace= . > > > > > > > > In this case migration of the pinned page is unnecessary as the p= age can=20 > > be > > > > unpinned at anytime by having the driver revoke atomic permission= as it > > > > does for the migrate_to_ram() callback. However a method of calli= ng this > > > > when memory needs to be moved has yet to be resolved so any discu= ssion is > > > > welcome. > > >=20 > > > Why do we need to pin for gpu atomics? You still have the callback = for > > > cpu faults, so you > > > can move the page as needed, and hence a long-term pin sounds like = the > > > wrong approach. > >=20 > > Technically a real long term unmoveable pin isn't required, because a= s you say=20 > > the page can be moved as needed at any time. However I needed some wa= y of=20 > > stopping the CPU page from being freed once the userspace mappings fo= r it had=20 > > been removed.=20 >=20 > The issue is you took the page out of the PTE it belongs to, which > makes it orphaned and unlocatable by the rest of the mm? >=20 > Ideally this would leave the PTE in place so everything continues to > work, just disable CPU access to it. >=20 > Maybe some kind of special swap entry? >=20 > I also don't much like the use of ZONE_DEVICE here, that should only > be used for actual device memory, not as a temporary proxy for CPU > pages.. Having two struct pages refer to the same physical memory is > pretty ugly. >=20 > > The normal solution of registering an MMU notifier to unpin the page = when it=20 > > needs to be moved also doesn't work as the CPU page tables now point = to the > > device-private page and hence the migration code won't call any inval= idate=20 > > notifiers for the CPU page. >=20 > The fact the page is lost from the MM seems to be the main issue here. >=20 > > Yes, I would like to avoid the long term pin constraints as well if p= ossible I=20 > > just haven't found a solution yet. Are you suggesting it might be pos= sible to=20 > > add a callback in the page migration logic to specially deal with mov= ing these=20 > > pages? >=20 > How would migration even find the page? Migration can scan memory from physical address (isolate_migratepages_ran= ge()) So the CPU mapping is not the only path to get to a page. Cheers, J=E9r=F4me