From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C99C9CA9ED4 for ; Mon, 4 Nov 2019 19:53:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A3558206BA for ; Mon, 4 Nov 2019 19:53:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="h7mbt8Qf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728741AbfKDTxH (ORCPT ); Mon, 4 Nov 2019 14:53:07 -0500 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:22435 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728766AbfKDTxH (ORCPT ); Mon, 4 Nov 2019 14:53:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1572897186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5YqXY05LcZm45Xoi2n7UBvKIcNMM7euzmr8OM5rBxks=; b=h7mbt8Qf6MfxBGX7EepWQ0FVQUf9WO9sNbsB6pRtLGeVtPF3DJ0Uqn+X5BAheanx/TzY6v FTMWPuDQfCOZ+4asJLcjY2aqxihbLcCYbkndNhy9aEgVMyJVlGRXp5aiFTFpG+Kz9/Ey00 64vj0Y5khi/vDXfazj/do4qP0HdUKOw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-92-ZQL91USHOcmTk___p7nkiA-1; Mon, 04 Nov 2019 14:53:03 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 760D1107ACC2; Mon, 4 Nov 2019 19:52:58 +0000 (UTC) Received: from redhat.com (unknown [10.20.6.178]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E8A5B1FC; Mon, 4 Nov 2019 19:52:49 +0000 (UTC) Date: Mon, 4 Nov 2019 14:52:48 -0500 From: Jerome Glisse To: John Hubbard Cc: Andrew Morton , Al Viro , Alex Williamson , Benjamin Herrenschmidt , =?iso-8859-1?Q?Bj=F6rn_T=F6pel?= , Christoph Hellwig , Dan Williams , Daniel Vetter , Dave Chinner , David Airlie , "David S . Miller" , Ira Weiny , Jan Kara , Jason Gunthorpe , Jens Axboe , Jonathan Corbet , Magnus Karlsson , Mauro Carvalho Chehab , Michael Ellerman , Michal Hocko , Mike Kravetz , Paul Mackerras , Shuah Khan , Vlastimil Babka , bpf@vger.kernel.org, dri-devel@lists.freedesktop.org, kvm@vger.kernel.org, linux-block@vger.kernel.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, netdev@vger.kernel.org, linux-mm@kvack.org, LKML Subject: Re: [PATCH v2 05/18] mm/gup: introduce pin_user_pages*() and FOLL_PIN Message-ID: <20191104195248.GA7731@redhat.com> References: <20191103211813.213227-1-jhubbard@nvidia.com> <20191103211813.213227-6-jhubbard@nvidia.com> <20191104173325.GD5134@redhat.com> <20191104191811.GI5134@redhat.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-MC-Unique: ZQL91USHOcmTk___p7nkiA-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org On Mon, Nov 04, 2019 at 11:30:32AM -0800, John Hubbard wrote: > On 11/4/19 11:18 AM, Jerome Glisse wrote: > > On Mon, Nov 04, 2019 at 11:04:38AM -0800, John Hubbard wrote: > >> On 11/4/19 9:33 AM, Jerome Glisse wrote: > >> ... > >>> > >>> Few nitpick belows, nonetheless: > >>> > >>> Reviewed-by: J=E9r=F4me Glisse > >>> [...] > >>>> + > >>>> +CASE 3: ODP > >>>> +----------- > >>>> +(Mellanox/Infiniband On Demand Paging: the hardware supports > >>>> +replayable page faulting). There are GUP references to pages servin= g as DMA > >>>> +buffers. For ODP, MMU notifiers are used to synchronize with page_m= kclean() > >>>> +and munmap(). Therefore, normal GUP calls are sufficient, so neithe= r flag > >>>> +needs to be set. > >>> > >>> I would not include ODP or anything like it here, they do not use > >>> GUP anymore and i believe it is more confusing here. I would how- > >>> ever include some text in this documentation explaining that hard- > >>> ware that support page fault is superior as it does not incur any > >>> of the issues described here. > >> > >> OK, agreed, here's a new write up that I'll put in v3: > >> > >> > >> CASE 3: ODP > >> ----------- > >=20 > > ODP is RDMA, maybe Hardware with page fault support instead > >=20 > >> Advanced, but non-CPU (DMA) hardware that supports replayable page fau= lts. >=20 > OK, so: >=20 > "RDMA hardware with page faulting support." >=20 > for the first sentence. I would drop RDMA completely, RDMA is just one example, they are GPU, FPGA = and others that are in that category. See below >=20 >=20 > >> Here, a well-written driver doesn't normally need to pin pages at all.= However, > >> if the driver does choose to do so, it can register MMU notifiers for = the range, > >> and will be called back upon invalidation. Either way (avoiding page p= inning, or > >> using MMU notifiers to unpin upon request), there is proper synchroniz= ation with=20 > >> both filesystem and mm (page_mkclean(), munmap(), etc). > >> > >> Therefore, neither flag needs to be set. > >=20 > > In fact GUP should never be use with those. >=20 >=20 > Yes. The next paragraph says that, but maybe not strong enough. >=20 >=20 > >> > >> It's worth mentioning here that pinning pages should not be the first = design > >> choice. If page fault capable hardware is available, then the software= should > >> be written so that it does not pin pages. This allows mm and filesyste= ms to > >> operate more efficiently and reliably. >=20 > Here's what we have after the above changes: >=20 > CASE 3: ODP > ----------- > RDMA hardware with page faulting support. Here, a well-written driver doe= sn't CASE3: Hardware with page fault support --------------------------------------- Here, a well-written .... > normally need to pin pages at all. However, if the driver does choose to = do so, > it can register MMU notifiers for the range, and will be called back upon > invalidation. Either way (avoiding page pinning, or using MMU notifiers t= o unpin > upon request), there is proper synchronization with both filesystem and m= m > (page_mkclean(), munmap(), etc). >=20 > Therefore, neither flag needs to be set. >=20 > In this case, ideally, neither get_user_pages() nor pin_user_pages() shou= ld be=20 > called. Instead, the software should be written so that it does not pin p= ages.=20 > This allows mm and filesystems to operate more efficiently and reliably. >=20 > >>> [...] > >>> > >>>> @@ -1014,7 +1018,16 @@ static __always_inline long __get_user_pages_= locked(struct task_struct *tsk, > >>>> =09=09BUG_ON(*locked !=3D 1); > >>>> =09} > >>>> =20 > >>>> -=09if (pages) > >>>> +=09/* > >>>> +=09 * FOLL_PIN and FOLL_GET are mutually exclusive. Traditional beh= avior > >>>> +=09 * is to set FOLL_GET if the caller wants pages[] filled in (but= has > >>>> +=09 * carelessly failed to specify FOLL_GET), so keep doing that, b= ut only > >>>> +=09 * for FOLL_GET, not for the newer FOLL_PIN. > >>>> +=09 * > >>>> +=09 * FOLL_PIN always expects pages to be non-null, but no need to = assert > >>>> +=09 * that here, as any failures will be obvious enough. > >>>> +=09 */ > >>>> +=09if (pages && !(flags & FOLL_PIN)) > >>>> =09=09flags |=3D FOLL_GET; > >>> > >>> Did you look at user that have pages and not FOLL_GET set ? > >>> I believe it would be better to first fix them to end up > >>> with FOLL_GET set and then error out if pages is !=3D NULL but > >>> nor FOLL_GET or FOLL_PIN is set. > >>> > >> > >> I was perhaps overly cautious, and didn't go there. However, it's prob= ably > >> doable, given that there was already the following in __get_user_pages= (): > >> > >> VM_BUG_ON(!!pages !=3D !!(gup_flags & FOLL_GET)); > >> > >> ...which will have conditioned people and code to set FOLL_GET togethe= r with > >> pages. So I agree that the time is right. > >> > >> In order to make bisecting future failures simpler, I can insert a pat= ch right=20 > >> before this one, that changes the FOLL_GET setting into an assert, lik= e this: > >> > >> diff --git a/mm/gup.c b/mm/gup.c > >> index 8f236a335ae9..be338961e80d 100644 > >> --- a/mm/gup.c > >> +++ b/mm/gup.c > >> @@ -1014,8 +1014,8 @@ static __always_inline long __get_user_pages_loc= ked(struct task_struct *tsk, > >> BUG_ON(*locked !=3D 1); > >> } > >> =20 > >> - if (pages) > >> - flags |=3D FOLL_GET; > >> + if (pages && WARN_ON_ONCE(!(gup_flags & FOLL_GET))) > >> + return -EINVAL; > >> =20 > >> pages_done =3D 0; > >> lock_dropped =3D false; > >> > >> > >> ...and then add in FOLL_PIN, with this patch. > >=20 > > looks good but double check that it should not happens, i will try > > to check on my side too. >=20 > Yes, I'll look. >=20 > ... > >>>> +=09 */ > >>>> +=09gup_flags |=3D FOLL_REMOTE | FOLL_PIN; > >>> > >>> Wouldn't it be better to not add pin_longterm_pages_remote() until > >>> it can be properly implemented ? > >>> > >> > >> Well, the problem is that I need each call site that requires FOLL_PIN > >> to use a proper wrapper. It's the FOLL_PIN that is the focus here, bec= ause > >> there is a hard, bright rule, which is: if and only if a caller sets > >> FOLL_PIN, then the dma-page tracking happens, and put_user_page() must > >> be called. > >> > >> So this leaves me with only two reasonable choices: > >> > >> a) Convert the call site as above: pin_longterm_pages_remote(), which = sets > >> FOLL_PIN (the key point!), and leaves the FOLL_LONGTERM situation exac= tly > >> as it has been so far. When the FOLL_LONGTERM situation is fixed, the = call > >> site *might* not need any changes to adopt the working gup.c code. > >> > >> b) Convert the call site to pin_user_pages_remote(), which also sets > >> FOLL_PIN, and also leaves the FOLL_LONGTERM situation exactly as befor= e. > >> There would also be a comment at the call site, to the effect of, "thi= s > >> is the wrong call to make: it really requires FOLL_LONGTERM behavior". > >> > >> When the FOLL_LONGTERM situation is fixed, the call site will need to = be > >> changed to pin_longterm_pages_remote(). > >> > >> So you can probably see why I picked (a). > >=20 > > But right now nobody has FOLL_LONGTERM and FOLL_REMOTE. So you should > > never have the need for pin_longterm_pages_remote(). My fear is that > > longterm has implication and it would be better to not drop this implic= ation > > by adding a wrapper that does not do what the name says. > >=20 > > So do not introduce pin_longterm_pages_remote() until its first user > > happens. This is option c) > >=20 >=20 > Almost forgot, though: there is already another user: Infiniband: >=20 > drivers/infiniband/core/umem_odp.c:646: npages =3D pin_longterm_p= ages_remote(owning_process, owning_mm, odp do not need that, i thought the HMM convertion was already upstream but seems not, in any case odp do not need the longterm case it only so best is to revert that user to gup_fast or something until it get converted to HMM. Cheers, J=E9r=F4me From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5057ECA9EB5 for ; Mon, 4 Nov 2019 19:54:53 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BBD5C206BA for ; Mon, 4 Nov 2019 19:54:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="h7mbt8Qf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBD5C206BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 476Nmp5SFDzF453 for ; Tue, 5 Nov 2019 06:54:50 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=205.139.110.120; helo=us-smtp-1.mimecast.com; envelope-from=jglisse@redhat.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="h7mbt8Qf"; dkim-atps=neutral Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 476Nkt5PzqzF3YQ for ; Tue, 5 Nov 2019 06:53:09 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1572897186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5YqXY05LcZm45Xoi2n7UBvKIcNMM7euzmr8OM5rBxks=; b=h7mbt8Qf6MfxBGX7EepWQ0FVQUf9WO9sNbsB6pRtLGeVtPF3DJ0Uqn+X5BAheanx/TzY6v FTMWPuDQfCOZ+4asJLcjY2aqxihbLcCYbkndNhy9aEgVMyJVlGRXp5aiFTFpG+Kz9/Ey00 64vj0Y5khi/vDXfazj/do4qP0HdUKOw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-92-ZQL91USHOcmTk___p7nkiA-1; Mon, 04 Nov 2019 14:53:03 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 760D1107ACC2; Mon, 4 Nov 2019 19:52:58 +0000 (UTC) Received: from redhat.com (unknown [10.20.6.178]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E8A5B1FC; Mon, 4 Nov 2019 19:52:49 +0000 (UTC) Date: Mon, 4 Nov 2019 14:52:48 -0500 From: Jerome Glisse To: John Hubbard Subject: Re: [PATCH v2 05/18] mm/gup: introduce pin_user_pages*() and FOLL_PIN Message-ID: <20191104195248.GA7731@redhat.com> References: <20191103211813.213227-1-jhubbard@nvidia.com> <20191103211813.213227-6-jhubbard@nvidia.com> <20191104173325.GD5134@redhat.com> <20191104191811.GI5134@redhat.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-MC-Unique: ZQL91USHOcmTk___p7nkiA-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Jan Kara , kvm@vger.kernel.org, linux-doc@vger.kernel.org, David Airlie , Dave Chinner , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, Paul Mackerras , linux-kselftest@vger.kernel.org, Ira Weiny , Jonathan Corbet , linux-rdma@vger.kernel.org, Christoph Hellwig , Jason Gunthorpe , Vlastimil Babka , =?iso-8859-1?Q?Bj=F6rn_T=F6pel?= , linux-media@vger.kernel.org, Shuah Khan , linux-block@vger.kernel.org, Alex Williamson , Al Viro , Dan Williams , Mauro Carvalho Chehab , bpf@vger.kernel.org, Magnus Karlsson , Jens Axboe , netdev@vger.kernel.org, LKML , Daniel Vetter , linux-fsdevel@vger.kernel.org, Andrew Morton , linuxppc-dev@lists.ozlabs.org, "David S . Miller" , Mike Kravetz Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, Nov 04, 2019 at 11:30:32AM -0800, John Hubbard wrote: > On 11/4/19 11:18 AM, Jerome Glisse wrote: > > On Mon, Nov 04, 2019 at 11:04:38AM -0800, John Hubbard wrote: > >> On 11/4/19 9:33 AM, Jerome Glisse wrote: > >> ... > >>> > >>> Few nitpick belows, nonetheless: > >>> > >>> Reviewed-by: J=E9r=F4me Glisse > >>> [...] > >>>> + > >>>> +CASE 3: ODP > >>>> +----------- > >>>> +(Mellanox/Infiniband On Demand Paging: the hardware supports > >>>> +replayable page faulting). There are GUP references to pages servin= g as DMA > >>>> +buffers. For ODP, MMU notifiers are used to synchronize with page_m= kclean() > >>>> +and munmap(). Therefore, normal GUP calls are sufficient, so neithe= r flag > >>>> +needs to be set. > >>> > >>> I would not include ODP or anything like it here, they do not use > >>> GUP anymore and i believe it is more confusing here. I would how- > >>> ever include some text in this documentation explaining that hard- > >>> ware that support page fault is superior as it does not incur any > >>> of the issues described here. > >> > >> OK, agreed, here's a new write up that I'll put in v3: > >> > >> > >> CASE 3: ODP > >> ----------- > >=20 > > ODP is RDMA, maybe Hardware with page fault support instead > >=20 > >> Advanced, but non-CPU (DMA) hardware that supports replayable page fau= lts. >=20 > OK, so: >=20 > "RDMA hardware with page faulting support." >=20 > for the first sentence. I would drop RDMA completely, RDMA is just one example, they are GPU, FPGA = and others that are in that category. See below >=20 >=20 > >> Here, a well-written driver doesn't normally need to pin pages at all.= However, > >> if the driver does choose to do so, it can register MMU notifiers for = the range, > >> and will be called back upon invalidation. Either way (avoiding page p= inning, or > >> using MMU notifiers to unpin upon request), there is proper synchroniz= ation with=20 > >> both filesystem and mm (page_mkclean(), munmap(), etc). > >> > >> Therefore, neither flag needs to be set. > >=20 > > In fact GUP should never be use with those. >=20 >=20 > Yes. The next paragraph says that, but maybe not strong enough. >=20 >=20 > >> > >> It's worth mentioning here that pinning pages should not be the first = design > >> choice. If page fault capable hardware is available, then the software= should > >> be written so that it does not pin pages. This allows mm and filesyste= ms to > >> operate more efficiently and reliably. >=20 > Here's what we have after the above changes: >=20 > CASE 3: ODP > ----------- > RDMA hardware with page faulting support. Here, a well-written driver doe= sn't CASE3: Hardware with page fault support --------------------------------------- Here, a well-written .... > normally need to pin pages at all. However, if the driver does choose to = do so, > it can register MMU notifiers for the range, and will be called back upon > invalidation. Either way (avoiding page pinning, or using MMU notifiers t= o unpin > upon request), there is proper synchronization with both filesystem and m= m > (page_mkclean(), munmap(), etc). >=20 > Therefore, neither flag needs to be set. >=20 > In this case, ideally, neither get_user_pages() nor pin_user_pages() shou= ld be=20 > called. Instead, the software should be written so that it does not pin p= ages.=20 > This allows mm and filesystems to operate more efficiently and reliably. >=20 > >>> [...] > >>> > >>>> @@ -1014,7 +1018,16 @@ static __always_inline long __get_user_pages_= locked(struct task_struct *tsk, > >>>> =09=09BUG_ON(*locked !=3D 1); > >>>> =09} > >>>> =20 > >>>> -=09if (pages) > >>>> +=09/* > >>>> +=09 * FOLL_PIN and FOLL_GET are mutually exclusive. Traditional beh= avior > >>>> +=09 * is to set FOLL_GET if the caller wants pages[] filled in (but= has > >>>> +=09 * carelessly failed to specify FOLL_GET), so keep doing that, b= ut only > >>>> +=09 * for FOLL_GET, not for the newer FOLL_PIN. > >>>> +=09 * > >>>> +=09 * FOLL_PIN always expects pages to be non-null, but no need to = assert > >>>> +=09 * that here, as any failures will be obvious enough. > >>>> +=09 */ > >>>> +=09if (pages && !(flags & FOLL_PIN)) > >>>> =09=09flags |=3D FOLL_GET; > >>> > >>> Did you look at user that have pages and not FOLL_GET set ? > >>> I believe it would be better to first fix them to end up > >>> with FOLL_GET set and then error out if pages is !=3D NULL but > >>> nor FOLL_GET or FOLL_PIN is set. > >>> > >> > >> I was perhaps overly cautious, and didn't go there. However, it's prob= ably > >> doable, given that there was already the following in __get_user_pages= (): > >> > >> VM_BUG_ON(!!pages !=3D !!(gup_flags & FOLL_GET)); > >> > >> ...which will have conditioned people and code to set FOLL_GET togethe= r with > >> pages. So I agree that the time is right. > >> > >> In order to make bisecting future failures simpler, I can insert a pat= ch right=20 > >> before this one, that changes the FOLL_GET setting into an assert, lik= e this: > >> > >> diff --git a/mm/gup.c b/mm/gup.c > >> index 8f236a335ae9..be338961e80d 100644 > >> --- a/mm/gup.c > >> +++ b/mm/gup.c > >> @@ -1014,8 +1014,8 @@ static __always_inline long __get_user_pages_loc= ked(struct task_struct *tsk, > >> BUG_ON(*locked !=3D 1); > >> } > >> =20 > >> - if (pages) > >> - flags |=3D FOLL_GET; > >> + if (pages && WARN_ON_ONCE(!(gup_flags & FOLL_GET))) > >> + return -EINVAL; > >> =20 > >> pages_done =3D 0; > >> lock_dropped =3D false; > >> > >> > >> ...and then add in FOLL_PIN, with this patch. > >=20 > > looks good but double check that it should not happens, i will try > > to check on my side too. >=20 > Yes, I'll look. >=20 > ... > >>>> +=09 */ > >>>> +=09gup_flags |=3D FOLL_REMOTE | FOLL_PIN; > >>> > >>> Wouldn't it be better to not add pin_longterm_pages_remote() until > >>> it can be properly implemented ? > >>> > >> > >> Well, the problem is that I need each call site that requires FOLL_PIN > >> to use a proper wrapper. It's the FOLL_PIN that is the focus here, bec= ause > >> there is a hard, bright rule, which is: if and only if a caller sets > >> FOLL_PIN, then the dma-page tracking happens, and put_user_page() must > >> be called. > >> > >> So this leaves me with only two reasonable choices: > >> > >> a) Convert the call site as above: pin_longterm_pages_remote(), which = sets > >> FOLL_PIN (the key point!), and leaves the FOLL_LONGTERM situation exac= tly > >> as it has been so far. When the FOLL_LONGTERM situation is fixed, the = call > >> site *might* not need any changes to adopt the working gup.c code. > >> > >> b) Convert the call site to pin_user_pages_remote(), which also sets > >> FOLL_PIN, and also leaves the FOLL_LONGTERM situation exactly as befor= e. > >> There would also be a comment at the call site, to the effect of, "thi= s > >> is the wrong call to make: it really requires FOLL_LONGTERM behavior". > >> > >> When the FOLL_LONGTERM situation is fixed, the call site will need to = be > >> changed to pin_longterm_pages_remote(). > >> > >> So you can probably see why I picked (a). > >=20 > > But right now nobody has FOLL_LONGTERM and FOLL_REMOTE. So you should > > never have the need for pin_longterm_pages_remote(). My fear is that > > longterm has implication and it would be better to not drop this implic= ation > > by adding a wrapper that does not do what the name says. > >=20 > > So do not introduce pin_longterm_pages_remote() until its first user > > happens. This is option c) > >=20 >=20 > Almost forgot, though: there is already another user: Infiniband: >=20 > drivers/infiniband/core/umem_odp.c:646: npages =3D pin_longterm_p= ages_remote(owning_process, owning_mm, odp do not need that, i thought the HMM convertion was already upstream but seems not, in any case odp do not need the longterm case it only so best is to revert that user to gup_fast or something until it get converted to HMM. Cheers, J=E9r=F4me From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH v2 05/18] mm/gup: introduce pin_user_pages*() and FOLL_PIN Date: Mon, 4 Nov 2019 14:52:48 -0500 Message-ID: <20191104195248.GA7731@redhat.com> References: <20191103211813.213227-1-jhubbard@nvidia.com> <20191103211813.213227-6-jhubbard@nvidia.com> <20191104173325.GD5134@redhat.com> <20191104191811.GI5134@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A4566E855 for ; Mon, 4 Nov 2019 19:53:08 +0000 (UTC) In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: John Hubbard Cc: Michal Hocko , Jan Kara , kvm@vger.kernel.org, linux-doc@vger.kernel.org, David Airlie , Dave Chinner , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, Paul Mackerras , linux-kselftest@vger.kernel.org, Ira Weiny , Jonathan Corbet , linux-rdma@vger.kernel.org, Michael Ellerman , Christoph Hellwig , Jason Gunthorpe , Vlastimil Babka , =?iso-8859-1?Q?Bj=F6rn_T=F6pel?= , linux-media@vger.kernel.org, Shuah Khan , linux-block@vger.kernel.org, Alex Williamson , Al Viro , Dan Williams , Mauro Carvalho Chehab , bpf@vger.kernel.org, Magnus List-Id: dri-devel@lists.freedesktop.org T24gTW9uLCBOb3YgMDQsIDIwMTkgYXQgMTE6MzA6MzJBTSAtMDgwMCwgSm9obiBIdWJiYXJkIHdy b3RlOgo+IE9uIDExLzQvMTkgMTE6MTggQU0sIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBN b24sIE5vdiAwNCwgMjAxOSBhdCAxMTowNDozOEFNIC0wODAwLCBKb2huIEh1YmJhcmQgd3JvdGU6 Cj4gPj4gT24gMTEvNC8xOSA5OjMzIEFNLCBKZXJvbWUgR2xpc3NlIHdyb3RlOgo+ID4+IC4uLgo+ ID4+Pgo+ID4+PiBGZXcgbml0cGljayBiZWxvd3MsIG5vbmV0aGVsZXNzOgo+ID4+Pgo+ID4+PiBS ZXZpZXdlZC1ieTogSsOpcsO0bWUgR2xpc3NlIDxqZ2xpc3NlQHJlZGhhdC5jb20+Cj4gPj4+IFsu Li5dCj4gPj4+PiArCj4gPj4+PiArQ0FTRSAzOiBPRFAKPiA+Pj4+ICstLS0tLS0tLS0tLQo+ID4+ Pj4gKyhNZWxsYW5veC9JbmZpbmliYW5kIE9uIERlbWFuZCBQYWdpbmc6IHRoZSBoYXJkd2FyZSBz dXBwb3J0cwo+ID4+Pj4gK3JlcGxheWFibGUgcGFnZSBmYXVsdGluZykuIFRoZXJlIGFyZSBHVVAg cmVmZXJlbmNlcyB0byBwYWdlcyBzZXJ2aW5nIGFzIERNQQo+ID4+Pj4gK2J1ZmZlcnMuIEZvciBP RFAsIE1NVSBub3RpZmllcnMgYXJlIHVzZWQgdG8gc3luY2hyb25pemUgd2l0aCBwYWdlX21rY2xl YW4oKQo+ID4+Pj4gK2FuZCBtdW5tYXAoKS4gVGhlcmVmb3JlLCBub3JtYWwgR1VQIGNhbGxzIGFy ZSBzdWZmaWNpZW50LCBzbyBuZWl0aGVyIGZsYWcKPiA+Pj4+ICtuZWVkcyB0byBiZSBzZXQuCj4g Pj4+Cj4gPj4+IEkgd291bGQgbm90IGluY2x1ZGUgT0RQIG9yIGFueXRoaW5nIGxpa2UgaXQgaGVy ZSwgdGhleSBkbyBub3QgdXNlCj4gPj4+IEdVUCBhbnltb3JlIGFuZCBpIGJlbGlldmUgaXQgaXMg bW9yZSBjb25mdXNpbmcgaGVyZS4gSSB3b3VsZCBob3ctCj4gPj4+IGV2ZXIgaW5jbHVkZSBzb21l IHRleHQgaW4gdGhpcyBkb2N1bWVudGF0aW9uIGV4cGxhaW5pbmcgdGhhdCBoYXJkLQo+ID4+PiB3 YXJlIHRoYXQgc3VwcG9ydCBwYWdlIGZhdWx0IGlzIHN1cGVyaW9yIGFzIGl0IGRvZXMgbm90IGlu Y3VyIGFueQo+ID4+PiBvZiB0aGUgaXNzdWVzIGRlc2NyaWJlZCBoZXJlLgo+ID4+Cj4gPj4gT0ss IGFncmVlZCwgaGVyZSdzIGEgbmV3IHdyaXRlIHVwIHRoYXQgSSdsbCBwdXQgaW4gdjM6Cj4gPj4K PiA+Pgo+ID4+IENBU0UgMzogT0RQCj4gPj4gLS0tLS0tLS0tLS0KPiA+IAo+ID4gT0RQIGlzIFJE TUEsIG1heWJlIEhhcmR3YXJlIHdpdGggcGFnZSBmYXVsdCBzdXBwb3J0IGluc3RlYWQKPiA+IAo+ ID4+IEFkdmFuY2VkLCBidXQgbm9uLUNQVSAoRE1BKSBoYXJkd2FyZSB0aGF0IHN1cHBvcnRzIHJl cGxheWFibGUgcGFnZSBmYXVsdHMuCj4gCj4gT0ssIHNvOgo+IAo+ICAgICAiUkRNQSBoYXJkd2Fy ZSB3aXRoIHBhZ2UgZmF1bHRpbmcgc3VwcG9ydC4iCj4gCj4gZm9yIHRoZSBmaXJzdCBzZW50ZW5j ZS4KCkkgd291bGQgZHJvcCBSRE1BIGNvbXBsZXRlbHksIFJETUEgaXMganVzdCBvbmUgZXhhbXBs ZSwgdGhleSBhcmUgR1BVLCBGUEdBIGFuZApvdGhlcnMgdGhhdCBhcmUgaW4gdGhhdCBjYXRlZ29y eS4gU2VlIGJlbG93Cgo+IAo+IAo+ID4+IEhlcmUsIGEgd2VsbC13cml0dGVuIGRyaXZlciBkb2Vz bid0IG5vcm1hbGx5IG5lZWQgdG8gcGluIHBhZ2VzIGF0IGFsbC4gSG93ZXZlciwKPiA+PiBpZiB0 aGUgZHJpdmVyIGRvZXMgY2hvb3NlIHRvIGRvIHNvLCBpdCBjYW4gcmVnaXN0ZXIgTU1VIG5vdGlm aWVycyBmb3IgdGhlIHJhbmdlLAo+ID4+IGFuZCB3aWxsIGJlIGNhbGxlZCBiYWNrIHVwb24gaW52 YWxpZGF0aW9uLiBFaXRoZXIgd2F5IChhdm9pZGluZyBwYWdlIHBpbm5pbmcsIG9yCj4gPj4gdXNp bmcgTU1VIG5vdGlmaWVycyB0byB1bnBpbiB1cG9uIHJlcXVlc3QpLCB0aGVyZSBpcyBwcm9wZXIg c3luY2hyb25pemF0aW9uIHdpdGggCj4gPj4gYm90aCBmaWxlc3lzdGVtIGFuZCBtbSAocGFnZV9t a2NsZWFuKCksIG11bm1hcCgpLCBldGMpLgo+ID4+Cj4gPj4gVGhlcmVmb3JlLCBuZWl0aGVyIGZs YWcgbmVlZHMgdG8gYmUgc2V0Lgo+ID4gCj4gPiBJbiBmYWN0IEdVUCBzaG91bGQgbmV2ZXIgYmUg dXNlIHdpdGggdGhvc2UuCj4gCj4gCj4gWWVzLiBUaGUgbmV4dCBwYXJhZ3JhcGggc2F5cyB0aGF0 LCBidXQgbWF5YmUgbm90IHN0cm9uZyBlbm91Z2guCj4gCj4gCj4gPj4KPiA+PiBJdCdzIHdvcnRo IG1lbnRpb25pbmcgaGVyZSB0aGF0IHBpbm5pbmcgcGFnZXMgc2hvdWxkIG5vdCBiZSB0aGUgZmly c3QgZGVzaWduCj4gPj4gY2hvaWNlLiBJZiBwYWdlIGZhdWx0IGNhcGFibGUgaGFyZHdhcmUgaXMg YXZhaWxhYmxlLCB0aGVuIHRoZSBzb2Z0d2FyZSBzaG91bGQKPiA+PiBiZSB3cml0dGVuIHNvIHRo YXQgaXQgZG9lcyBub3QgcGluIHBhZ2VzLiBUaGlzIGFsbG93cyBtbSBhbmQgZmlsZXN5c3RlbXMg dG8KPiA+PiBvcGVyYXRlIG1vcmUgZWZmaWNpZW50bHkgYW5kIHJlbGlhYmx5Lgo+IAo+IEhlcmUn cyB3aGF0IHdlIGhhdmUgYWZ0ZXIgdGhlIGFib3ZlIGNoYW5nZXM6Cj4gCj4gQ0FTRSAzOiBPRFAK PiAtLS0tLS0tLS0tLQo+IFJETUEgaGFyZHdhcmUgd2l0aCBwYWdlIGZhdWx0aW5nIHN1cHBvcnQu IEhlcmUsIGEgd2VsbC13cml0dGVuIGRyaXZlciBkb2Vzbid0CgpDQVNFMzogSGFyZHdhcmUgd2l0 aCBwYWdlIGZhdWx0IHN1cHBvcnQKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tCgpIZXJlLCBhIHdlbGwtd3JpdHRlbiAuLi4uCgoKPiBub3JtYWxseSBuZWVkIHRvIHBpbiBw YWdlcyBhdCBhbGwuIEhvd2V2ZXIsIGlmIHRoZSBkcml2ZXIgZG9lcyBjaG9vc2UgdG8gZG8gc28s Cj4gaXQgY2FuIHJlZ2lzdGVyIE1NVSBub3RpZmllcnMgZm9yIHRoZSByYW5nZSwgYW5kIHdpbGwg YmUgY2FsbGVkIGJhY2sgdXBvbgo+IGludmFsaWRhdGlvbi4gRWl0aGVyIHdheSAoYXZvaWRpbmcg cGFnZSBwaW5uaW5nLCBvciB1c2luZyBNTVUgbm90aWZpZXJzIHRvIHVucGluCj4gdXBvbiByZXF1 ZXN0KSwgdGhlcmUgaXMgcHJvcGVyIHN5bmNocm9uaXphdGlvbiB3aXRoIGJvdGggZmlsZXN5c3Rl bSBhbmQgbW0KPiAocGFnZV9ta2NsZWFuKCksIG11bm1hcCgpLCBldGMpLgo+IAo+IFRoZXJlZm9y ZSwgbmVpdGhlciBmbGFnIG5lZWRzIHRvIGJlIHNldC4KPiAKPiBJbiB0aGlzIGNhc2UsIGlkZWFs bHksIG5laXRoZXIgZ2V0X3VzZXJfcGFnZXMoKSBub3IgcGluX3VzZXJfcGFnZXMoKSBzaG91bGQg YmUgCj4gY2FsbGVkLiBJbnN0ZWFkLCB0aGUgc29mdHdhcmUgc2hvdWxkIGJlIHdyaXR0ZW4gc28g dGhhdCBpdCBkb2VzIG5vdCBwaW4gcGFnZXMuIAo+IFRoaXMgYWxsb3dzIG1tIGFuZCBmaWxlc3lz dGVtcyB0byBvcGVyYXRlIG1vcmUgZWZmaWNpZW50bHkgYW5kIHJlbGlhYmx5Lgo+IAo+ID4+PiBb Li4uXQo+ID4+Pgo+ID4+Pj4gQEAgLTEwMTQsNyArMTAxOCwxNiBAQCBzdGF0aWMgX19hbHdheXNf aW5saW5lIGxvbmcgX19nZXRfdXNlcl9wYWdlc19sb2NrZWQoc3RydWN0IHRhc2tfc3RydWN0ICp0 c2ssCj4gPj4+PiAgCQlCVUdfT04oKmxvY2tlZCAhPSAxKTsKPiA+Pj4+ICAJfQo+ID4+Pj4gIAo+ ID4+Pj4gLQlpZiAocGFnZXMpCj4gPj4+PiArCS8qCj4gPj4+PiArCSAqIEZPTExfUElOIGFuZCBG T0xMX0dFVCBhcmUgbXV0dWFsbHkgZXhjbHVzaXZlLiBUcmFkaXRpb25hbCBiZWhhdmlvcgo+ID4+ Pj4gKwkgKiBpcyB0byBzZXQgRk9MTF9HRVQgaWYgdGhlIGNhbGxlciB3YW50cyBwYWdlc1tdIGZp bGxlZCBpbiAoYnV0IGhhcwo+ID4+Pj4gKwkgKiBjYXJlbGVzc2x5IGZhaWxlZCB0byBzcGVjaWZ5 IEZPTExfR0VUKSwgc28ga2VlcCBkb2luZyB0aGF0LCBidXQgb25seQo+ID4+Pj4gKwkgKiBmb3Ig Rk9MTF9HRVQsIG5vdCBmb3IgdGhlIG5ld2VyIEZPTExfUElOLgo+ID4+Pj4gKwkgKgo+ID4+Pj4g KwkgKiBGT0xMX1BJTiBhbHdheXMgZXhwZWN0cyBwYWdlcyB0byBiZSBub24tbnVsbCwgYnV0IG5v IG5lZWQgdG8gYXNzZXJ0Cj4gPj4+PiArCSAqIHRoYXQgaGVyZSwgYXMgYW55IGZhaWx1cmVzIHdp bGwgYmUgb2J2aW91cyBlbm91Z2guCj4gPj4+PiArCSAqLwo+ID4+Pj4gKwlpZiAocGFnZXMgJiYg IShmbGFncyAmIEZPTExfUElOKSkKPiA+Pj4+ICAJCWZsYWdzIHw9IEZPTExfR0VUOwo+ID4+Pgo+ ID4+PiBEaWQgeW91IGxvb2sgYXQgdXNlciB0aGF0IGhhdmUgcGFnZXMgYW5kIG5vdCBGT0xMX0dF VCBzZXQgPwo+ID4+PiBJIGJlbGlldmUgaXQgd291bGQgYmUgYmV0dGVyIHRvIGZpcnN0IGZpeCB0 aGVtIHRvIGVuZCB1cAo+ID4+PiB3aXRoIEZPTExfR0VUIHNldCBhbmQgdGhlbiBlcnJvciBvdXQg aWYgcGFnZXMgaXMgIT0gTlVMTCBidXQKPiA+Pj4gbm9yIEZPTExfR0VUIG9yIEZPTExfUElOIGlz IHNldC4KPiA+Pj4KPiA+Pgo+ID4+IEkgd2FzIHBlcmhhcHMgb3Zlcmx5IGNhdXRpb3VzLCBhbmQg ZGlkbid0IGdvIHRoZXJlLiBIb3dldmVyLCBpdCdzIHByb2JhYmx5Cj4gPj4gZG9hYmxlLCBnaXZl biB0aGF0IHRoZXJlIHdhcyBhbHJlYWR5IHRoZSBmb2xsb3dpbmcgaW4gX19nZXRfdXNlcl9wYWdl cygpOgo+ID4+Cj4gPj4gICAgIFZNX0JVR19PTighIXBhZ2VzICE9ICEhKGd1cF9mbGFncyAmIEZP TExfR0VUKSk7Cj4gPj4KPiA+PiAuLi53aGljaCB3aWxsIGhhdmUgY29uZGl0aW9uZWQgcGVvcGxl IGFuZCBjb2RlIHRvIHNldCBGT0xMX0dFVCB0b2dldGhlciB3aXRoCj4gPj4gcGFnZXMuIFNvIEkg YWdyZWUgdGhhdCB0aGUgdGltZSBpcyByaWdodC4KPiA+Pgo+ID4+IEluIG9yZGVyIHRvIG1ha2Ug YmlzZWN0aW5nIGZ1dHVyZSBmYWlsdXJlcyBzaW1wbGVyLCBJIGNhbiBpbnNlcnQgYSBwYXRjaCBy aWdodCAKPiA+PiBiZWZvcmUgdGhpcyBvbmUsIHRoYXQgY2hhbmdlcyB0aGUgRk9MTF9HRVQgc2V0 dGluZyBpbnRvIGFuIGFzc2VydCwgbGlrZSB0aGlzOgo+ID4+Cj4gPj4gZGlmZiAtLWdpdCBhL21t L2d1cC5jIGIvbW0vZ3VwLmMKPiA+PiBpbmRleCA4ZjIzNmEzMzVhZTkuLmJlMzM4OTYxZTgwZCAx MDA2NDQKPiA+PiAtLS0gYS9tbS9ndXAuYwo+ID4+ICsrKyBiL21tL2d1cC5jCj4gPj4gQEAgLTEw MTQsOCArMTAxNCw4IEBAIHN0YXRpYyBfX2Fsd2F5c19pbmxpbmUgbG9uZyBfX2dldF91c2VyX3Bh Z2VzX2xvY2tlZChzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRzaywKPiA+PiAgICAgICAgICAgICAgICAg QlVHX09OKCpsb2NrZWQgIT0gMSk7Cj4gPj4gICAgICAgICB9Cj4gPj4gIAo+ID4+IC0gICAgICAg aWYgKHBhZ2VzKQo+ID4+IC0gICAgICAgICAgICAgICBmbGFncyB8PSBGT0xMX0dFVDsKPiA+PiAr ICAgICAgIGlmIChwYWdlcyAmJiBXQVJOX09OX09OQ0UoIShndXBfZmxhZ3MgJiBGT0xMX0dFVCkp KQo+ID4+ICsgICAgICAgICAgICAgICByZXR1cm4gLUVJTlZBTDsKPiA+PiAgCj4gPj4gICAgICAg ICBwYWdlc19kb25lID0gMDsKPiA+PiAgICAgICAgIGxvY2tfZHJvcHBlZCA9IGZhbHNlOwo+ID4+ Cj4gPj4KPiA+PiAuLi5hbmQgdGhlbiBhZGQgaW4gRk9MTF9QSU4sIHdpdGggdGhpcyBwYXRjaC4K PiA+IAo+ID4gbG9va3MgZ29vZCBidXQgZG91YmxlIGNoZWNrIHRoYXQgaXQgc2hvdWxkIG5vdCBo YXBwZW5zLCBpIHdpbGwgdHJ5Cj4gPiB0byBjaGVjayBvbiBteSBzaWRlIHRvby4KPiAKPiBZZXMs IEknbGwgbG9vay4KPiAKPiAuLi4KPiA+Pj4+ICsJICovCj4gPj4+PiArCWd1cF9mbGFncyB8PSBG T0xMX1JFTU9URSB8IEZPTExfUElOOwo+ID4+Pgo+ID4+PiBXb3VsZG4ndCBpdCBiZSBiZXR0ZXIg dG8gbm90IGFkZCBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKCkgdW50aWwKPiA+Pj4gaXQgY2Fu IGJlIHByb3Blcmx5IGltcGxlbWVudGVkID8KPiA+Pj4KPiA+Pgo+ID4+IFdlbGwsIHRoZSBwcm9i bGVtIGlzIHRoYXQgSSBuZWVkIGVhY2ggY2FsbCBzaXRlIHRoYXQgcmVxdWlyZXMgRk9MTF9QSU4K PiA+PiB0byB1c2UgYSBwcm9wZXIgd3JhcHBlci4gSXQncyB0aGUgRk9MTF9QSU4gdGhhdCBpcyB0 aGUgZm9jdXMgaGVyZSwgYmVjYXVzZQo+ID4+IHRoZXJlIGlzIGEgaGFyZCwgYnJpZ2h0IHJ1bGUs IHdoaWNoIGlzOiBpZiBhbmQgb25seSBpZiBhIGNhbGxlciBzZXRzCj4gPj4gRk9MTF9QSU4sIHRo ZW4gdGhlIGRtYS1wYWdlIHRyYWNraW5nIGhhcHBlbnMsIGFuZCBwdXRfdXNlcl9wYWdlKCkgbXVz dAo+ID4+IGJlIGNhbGxlZC4KPiA+Pgo+ID4+IFNvIHRoaXMgbGVhdmVzIG1lIHdpdGggb25seSB0 d28gcmVhc29uYWJsZSBjaG9pY2VzOgo+ID4+Cj4gPj4gYSkgQ29udmVydCB0aGUgY2FsbCBzaXRl IGFzIGFib3ZlOiBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKCksIHdoaWNoIHNldHMKPiA+PiBG T0xMX1BJTiAodGhlIGtleSBwb2ludCEpLCBhbmQgbGVhdmVzIHRoZSBGT0xMX0xPTkdURVJNIHNp dHVhdGlvbiBleGFjdGx5Cj4gPj4gYXMgaXQgaGFzIGJlZW4gc28gZmFyLiBXaGVuIHRoZSBGT0xM X0xPTkdURVJNIHNpdHVhdGlvbiBpcyBmaXhlZCwgdGhlIGNhbGwKPiA+PiBzaXRlICptaWdodCog bm90IG5lZWQgYW55IGNoYW5nZXMgdG8gYWRvcHQgdGhlIHdvcmtpbmcgZ3VwLmMgY29kZS4KPiA+ Pgo+ID4+IGIpIENvbnZlcnQgdGhlIGNhbGwgc2l0ZSB0byBwaW5fdXNlcl9wYWdlc19yZW1vdGUo KSwgd2hpY2ggYWxzbyBzZXRzCj4gPj4gRk9MTF9QSU4sIGFuZCBhbHNvIGxlYXZlcyB0aGUgRk9M TF9MT05HVEVSTSBzaXR1YXRpb24gZXhhY3RseSBhcyBiZWZvcmUuCj4gPj4gVGhlcmUgd291bGQg YWxzbyBiZSBhIGNvbW1lbnQgYXQgdGhlIGNhbGwgc2l0ZSwgdG8gdGhlIGVmZmVjdCBvZiwgInRo aXMKPiA+PiBpcyB0aGUgd3JvbmcgY2FsbCB0byBtYWtlOiBpdCByZWFsbHkgcmVxdWlyZXMgRk9M TF9MT05HVEVSTSBiZWhhdmlvciIuCj4gPj4KPiA+PiBXaGVuIHRoZSBGT0xMX0xPTkdURVJNIHNp dHVhdGlvbiBpcyBmaXhlZCwgdGhlIGNhbGwgc2l0ZSB3aWxsIG5lZWQgdG8gYmUKPiA+PiBjaGFu Z2VkIHRvIHBpbl9sb25ndGVybV9wYWdlc19yZW1vdGUoKS4KPiA+Pgo+ID4+IFNvIHlvdSBjYW4g cHJvYmFibHkgc2VlIHdoeSBJIHBpY2tlZCAoYSkuCj4gPiAKPiA+IEJ1dCByaWdodCBub3cgbm9i b2R5IGhhcyBGT0xMX0xPTkdURVJNIGFuZCBGT0xMX1JFTU9URS4gU28geW91IHNob3VsZAo+ID4g bmV2ZXIgaGF2ZSB0aGUgbmVlZCBmb3IgcGluX2xvbmd0ZXJtX3BhZ2VzX3JlbW90ZSgpLiBNeSBm ZWFyIGlzIHRoYXQKPiA+IGxvbmd0ZXJtIGhhcyBpbXBsaWNhdGlvbiBhbmQgaXQgd291bGQgYmUg YmV0dGVyIHRvIG5vdCBkcm9wIHRoaXMgaW1wbGljYXRpb24KPiA+IGJ5IGFkZGluZyBhIHdyYXBw ZXIgdGhhdCBkb2VzIG5vdCBkbyB3aGF0IHRoZSBuYW1lIHNheXMuCj4gPiAKPiA+IFNvIGRvIG5v dCBpbnRyb2R1Y2UgcGluX2xvbmd0ZXJtX3BhZ2VzX3JlbW90ZSgpIHVudGlsIGl0cyBmaXJzdCB1 c2VyCj4gPiBoYXBwZW5zLiBUaGlzIGlzIG9wdGlvbiBjKQo+ID4gCj4gCj4gQWxtb3N0IGZvcmdv dCwgdGhvdWdoOiB0aGVyZSBpcyBhbHJlYWR5IGFub3RoZXIgdXNlcjogSW5maW5pYmFuZDoKPiAK PiBkcml2ZXJzL2luZmluaWJhbmQvY29yZS91bWVtX29kcC5jOjY0NjogICAgICAgICBucGFnZXMg PSBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKG93bmluZ19wcm9jZXNzLCBvd25pbmdfbW0sCgpv ZHAgZG8gbm90IG5lZWQgdGhhdCwgaSB0aG91Z2h0IHRoZSBITU0gY29udmVydGlvbiB3YXMgYWxy ZWFkeSB1cHN0cmVhbQpidXQgc2VlbXMgbm90LCBpbiBhbnkgY2FzZSBvZHAgZG8gbm90IG5lZWQg dGhlIGxvbmd0ZXJtIGNhc2UgaXQgb25seQpzbyBiZXN0IGlzIHRvIHJldmVydCB0aGF0IHVzZXIg dG8gZ3VwX2Zhc3Qgb3Igc29tZXRoaW5nIHVudGlsIGl0IGdldApjb252ZXJ0ZWQgdG8gSE1NLgoK Q2hlZXJzLApKw6lyw7RtZQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0 b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJp LWRldmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01550CA9EB5 for ; Mon, 4 Nov 2019 19:53:10 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D83692089C for ; Mon, 4 Nov 2019 19:53:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D83692089C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 44A7D6E855; Mon, 4 Nov 2019 19:53:09 +0000 (UTC) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A4566E855 for ; Mon, 4 Nov 2019 19:53:08 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-92-ZQL91USHOcmTk___p7nkiA-1; Mon, 04 Nov 2019 14:53:03 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 760D1107ACC2; Mon, 4 Nov 2019 19:52:58 +0000 (UTC) Received: from redhat.com (unknown [10.20.6.178]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E8A5B1FC; Mon, 4 Nov 2019 19:52:49 +0000 (UTC) Date: Mon, 4 Nov 2019 14:52:48 -0500 From: Jerome Glisse To: John Hubbard Subject: Re: [PATCH v2 05/18] mm/gup: introduce pin_user_pages*() and FOLL_PIN Message-ID: <20191104195248.GA7731@redhat.com> References: <20191103211813.213227-1-jhubbard@nvidia.com> <20191103211813.213227-6-jhubbard@nvidia.com> <20191104173325.GD5134@redhat.com> <20191104191811.GI5134@redhat.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-MC-Unique: ZQL91USHOcmTk___p7nkiA-1 X-Mimecast-Spam-Score: 0 Content-Disposition: inline X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1572897187; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5YqXY05LcZm45Xoi2n7UBvKIcNMM7euzmr8OM5rBxks=; b=LUW4vh9DupPlwYYAvMF55kZVVVFWE5LTVgcI6qTLMGAv0rM1uq6knqm6oneXGw1pmdE9yO 2gxroMmOMOeP2KxyWk4bUFpYENWCTdD45CJ5A/zESEu9E4VCB0GZIOIPckGwcTXSZgBkwZ vGXCQuhogjiN32Ln2RlaMOAF6laugS4= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Jan Kara , kvm@vger.kernel.org, linux-doc@vger.kernel.org, David Airlie , Dave Chinner , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, Paul Mackerras , linux-kselftest@vger.kernel.org, Ira Weiny , Jonathan Corbet , linux-rdma@vger.kernel.org, Michael Ellerman , Christoph Hellwig , Jason Gunthorpe , Vlastimil Babka , =?iso-8859-1?Q?Bj=F6rn_T=F6pel?= , linux-media@vger.kernel.org, Shuah Khan , linux-block@vger.kernel.org, Alex Williamson , Al Viro , Dan Williams , Mauro Carvalho Chehab , bpf@vger.kernel.org, Magnus Karlsson , Jens Axboe , netdev@vger.kernel.org, LKML , linux-fsdevel@vger.kernel.org, Andrew Morton , linuxppc-dev@lists.ozlabs.org, "David S . Miller" , Mike Kravetz Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191104195248.PHlnsSWawiyp2SyCs526ooEovT1olPyJVvVLcDarW9c@z> T24gTW9uLCBOb3YgMDQsIDIwMTkgYXQgMTE6MzA6MzJBTSAtMDgwMCwgSm9obiBIdWJiYXJkIHdy b3RlOgo+IE9uIDExLzQvMTkgMTE6MTggQU0sIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBN b24sIE5vdiAwNCwgMjAxOSBhdCAxMTowNDozOEFNIC0wODAwLCBKb2huIEh1YmJhcmQgd3JvdGU6 Cj4gPj4gT24gMTEvNC8xOSA5OjMzIEFNLCBKZXJvbWUgR2xpc3NlIHdyb3RlOgo+ID4+IC4uLgo+ ID4+Pgo+ID4+PiBGZXcgbml0cGljayBiZWxvd3MsIG5vbmV0aGVsZXNzOgo+ID4+Pgo+ID4+PiBS ZXZpZXdlZC1ieTogSsOpcsO0bWUgR2xpc3NlIDxqZ2xpc3NlQHJlZGhhdC5jb20+Cj4gPj4+IFsu Li5dCj4gPj4+PiArCj4gPj4+PiArQ0FTRSAzOiBPRFAKPiA+Pj4+ICstLS0tLS0tLS0tLQo+ID4+ Pj4gKyhNZWxsYW5veC9JbmZpbmliYW5kIE9uIERlbWFuZCBQYWdpbmc6IHRoZSBoYXJkd2FyZSBz dXBwb3J0cwo+ID4+Pj4gK3JlcGxheWFibGUgcGFnZSBmYXVsdGluZykuIFRoZXJlIGFyZSBHVVAg cmVmZXJlbmNlcyB0byBwYWdlcyBzZXJ2aW5nIGFzIERNQQo+ID4+Pj4gK2J1ZmZlcnMuIEZvciBP RFAsIE1NVSBub3RpZmllcnMgYXJlIHVzZWQgdG8gc3luY2hyb25pemUgd2l0aCBwYWdlX21rY2xl YW4oKQo+ID4+Pj4gK2FuZCBtdW5tYXAoKS4gVGhlcmVmb3JlLCBub3JtYWwgR1VQIGNhbGxzIGFy ZSBzdWZmaWNpZW50LCBzbyBuZWl0aGVyIGZsYWcKPiA+Pj4+ICtuZWVkcyB0byBiZSBzZXQuCj4g Pj4+Cj4gPj4+IEkgd291bGQgbm90IGluY2x1ZGUgT0RQIG9yIGFueXRoaW5nIGxpa2UgaXQgaGVy ZSwgdGhleSBkbyBub3QgdXNlCj4gPj4+IEdVUCBhbnltb3JlIGFuZCBpIGJlbGlldmUgaXQgaXMg bW9yZSBjb25mdXNpbmcgaGVyZS4gSSB3b3VsZCBob3ctCj4gPj4+IGV2ZXIgaW5jbHVkZSBzb21l IHRleHQgaW4gdGhpcyBkb2N1bWVudGF0aW9uIGV4cGxhaW5pbmcgdGhhdCBoYXJkLQo+ID4+PiB3 YXJlIHRoYXQgc3VwcG9ydCBwYWdlIGZhdWx0IGlzIHN1cGVyaW9yIGFzIGl0IGRvZXMgbm90IGlu Y3VyIGFueQo+ID4+PiBvZiB0aGUgaXNzdWVzIGRlc2NyaWJlZCBoZXJlLgo+ID4+Cj4gPj4gT0ss IGFncmVlZCwgaGVyZSdzIGEgbmV3IHdyaXRlIHVwIHRoYXQgSSdsbCBwdXQgaW4gdjM6Cj4gPj4K PiA+Pgo+ID4+IENBU0UgMzogT0RQCj4gPj4gLS0tLS0tLS0tLS0KPiA+IAo+ID4gT0RQIGlzIFJE TUEsIG1heWJlIEhhcmR3YXJlIHdpdGggcGFnZSBmYXVsdCBzdXBwb3J0IGluc3RlYWQKPiA+IAo+ ID4+IEFkdmFuY2VkLCBidXQgbm9uLUNQVSAoRE1BKSBoYXJkd2FyZSB0aGF0IHN1cHBvcnRzIHJl cGxheWFibGUgcGFnZSBmYXVsdHMuCj4gCj4gT0ssIHNvOgo+IAo+ICAgICAiUkRNQSBoYXJkd2Fy ZSB3aXRoIHBhZ2UgZmF1bHRpbmcgc3VwcG9ydC4iCj4gCj4gZm9yIHRoZSBmaXJzdCBzZW50ZW5j ZS4KCkkgd291bGQgZHJvcCBSRE1BIGNvbXBsZXRlbHksIFJETUEgaXMganVzdCBvbmUgZXhhbXBs ZSwgdGhleSBhcmUgR1BVLCBGUEdBIGFuZApvdGhlcnMgdGhhdCBhcmUgaW4gdGhhdCBjYXRlZ29y eS4gU2VlIGJlbG93Cgo+IAo+IAo+ID4+IEhlcmUsIGEgd2VsbC13cml0dGVuIGRyaXZlciBkb2Vz bid0IG5vcm1hbGx5IG5lZWQgdG8gcGluIHBhZ2VzIGF0IGFsbC4gSG93ZXZlciwKPiA+PiBpZiB0 aGUgZHJpdmVyIGRvZXMgY2hvb3NlIHRvIGRvIHNvLCBpdCBjYW4gcmVnaXN0ZXIgTU1VIG5vdGlm aWVycyBmb3IgdGhlIHJhbmdlLAo+ID4+IGFuZCB3aWxsIGJlIGNhbGxlZCBiYWNrIHVwb24gaW52 YWxpZGF0aW9uLiBFaXRoZXIgd2F5IChhdm9pZGluZyBwYWdlIHBpbm5pbmcsIG9yCj4gPj4gdXNp bmcgTU1VIG5vdGlmaWVycyB0byB1bnBpbiB1cG9uIHJlcXVlc3QpLCB0aGVyZSBpcyBwcm9wZXIg c3luY2hyb25pemF0aW9uIHdpdGggCj4gPj4gYm90aCBmaWxlc3lzdGVtIGFuZCBtbSAocGFnZV9t a2NsZWFuKCksIG11bm1hcCgpLCBldGMpLgo+ID4+Cj4gPj4gVGhlcmVmb3JlLCBuZWl0aGVyIGZs YWcgbmVlZHMgdG8gYmUgc2V0Lgo+ID4gCj4gPiBJbiBmYWN0IEdVUCBzaG91bGQgbmV2ZXIgYmUg dXNlIHdpdGggdGhvc2UuCj4gCj4gCj4gWWVzLiBUaGUgbmV4dCBwYXJhZ3JhcGggc2F5cyB0aGF0 LCBidXQgbWF5YmUgbm90IHN0cm9uZyBlbm91Z2guCj4gCj4gCj4gPj4KPiA+PiBJdCdzIHdvcnRo IG1lbnRpb25pbmcgaGVyZSB0aGF0IHBpbm5pbmcgcGFnZXMgc2hvdWxkIG5vdCBiZSB0aGUgZmly c3QgZGVzaWduCj4gPj4gY2hvaWNlLiBJZiBwYWdlIGZhdWx0IGNhcGFibGUgaGFyZHdhcmUgaXMg YXZhaWxhYmxlLCB0aGVuIHRoZSBzb2Z0d2FyZSBzaG91bGQKPiA+PiBiZSB3cml0dGVuIHNvIHRo YXQgaXQgZG9lcyBub3QgcGluIHBhZ2VzLiBUaGlzIGFsbG93cyBtbSBhbmQgZmlsZXN5c3RlbXMg dG8KPiA+PiBvcGVyYXRlIG1vcmUgZWZmaWNpZW50bHkgYW5kIHJlbGlhYmx5Lgo+IAo+IEhlcmUn cyB3aGF0IHdlIGhhdmUgYWZ0ZXIgdGhlIGFib3ZlIGNoYW5nZXM6Cj4gCj4gQ0FTRSAzOiBPRFAK PiAtLS0tLS0tLS0tLQo+IFJETUEgaGFyZHdhcmUgd2l0aCBwYWdlIGZhdWx0aW5nIHN1cHBvcnQu IEhlcmUsIGEgd2VsbC13cml0dGVuIGRyaXZlciBkb2Vzbid0CgpDQVNFMzogSGFyZHdhcmUgd2l0 aCBwYWdlIGZhdWx0IHN1cHBvcnQKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tCgpIZXJlLCBhIHdlbGwtd3JpdHRlbiAuLi4uCgoKPiBub3JtYWxseSBuZWVkIHRvIHBpbiBw YWdlcyBhdCBhbGwuIEhvd2V2ZXIsIGlmIHRoZSBkcml2ZXIgZG9lcyBjaG9vc2UgdG8gZG8gc28s Cj4gaXQgY2FuIHJlZ2lzdGVyIE1NVSBub3RpZmllcnMgZm9yIHRoZSByYW5nZSwgYW5kIHdpbGwg YmUgY2FsbGVkIGJhY2sgdXBvbgo+IGludmFsaWRhdGlvbi4gRWl0aGVyIHdheSAoYXZvaWRpbmcg cGFnZSBwaW5uaW5nLCBvciB1c2luZyBNTVUgbm90aWZpZXJzIHRvIHVucGluCj4gdXBvbiByZXF1 ZXN0KSwgdGhlcmUgaXMgcHJvcGVyIHN5bmNocm9uaXphdGlvbiB3aXRoIGJvdGggZmlsZXN5c3Rl bSBhbmQgbW0KPiAocGFnZV9ta2NsZWFuKCksIG11bm1hcCgpLCBldGMpLgo+IAo+IFRoZXJlZm9y ZSwgbmVpdGhlciBmbGFnIG5lZWRzIHRvIGJlIHNldC4KPiAKPiBJbiB0aGlzIGNhc2UsIGlkZWFs bHksIG5laXRoZXIgZ2V0X3VzZXJfcGFnZXMoKSBub3IgcGluX3VzZXJfcGFnZXMoKSBzaG91bGQg YmUgCj4gY2FsbGVkLiBJbnN0ZWFkLCB0aGUgc29mdHdhcmUgc2hvdWxkIGJlIHdyaXR0ZW4gc28g dGhhdCBpdCBkb2VzIG5vdCBwaW4gcGFnZXMuIAo+IFRoaXMgYWxsb3dzIG1tIGFuZCBmaWxlc3lz dGVtcyB0byBvcGVyYXRlIG1vcmUgZWZmaWNpZW50bHkgYW5kIHJlbGlhYmx5Lgo+IAo+ID4+PiBb Li4uXQo+ID4+Pgo+ID4+Pj4gQEAgLTEwMTQsNyArMTAxOCwxNiBAQCBzdGF0aWMgX19hbHdheXNf aW5saW5lIGxvbmcgX19nZXRfdXNlcl9wYWdlc19sb2NrZWQoc3RydWN0IHRhc2tfc3RydWN0ICp0 c2ssCj4gPj4+PiAgCQlCVUdfT04oKmxvY2tlZCAhPSAxKTsKPiA+Pj4+ICAJfQo+ID4+Pj4gIAo+ ID4+Pj4gLQlpZiAocGFnZXMpCj4gPj4+PiArCS8qCj4gPj4+PiArCSAqIEZPTExfUElOIGFuZCBG T0xMX0dFVCBhcmUgbXV0dWFsbHkgZXhjbHVzaXZlLiBUcmFkaXRpb25hbCBiZWhhdmlvcgo+ID4+ Pj4gKwkgKiBpcyB0byBzZXQgRk9MTF9HRVQgaWYgdGhlIGNhbGxlciB3YW50cyBwYWdlc1tdIGZp bGxlZCBpbiAoYnV0IGhhcwo+ID4+Pj4gKwkgKiBjYXJlbGVzc2x5IGZhaWxlZCB0byBzcGVjaWZ5 IEZPTExfR0VUKSwgc28ga2VlcCBkb2luZyB0aGF0LCBidXQgb25seQo+ID4+Pj4gKwkgKiBmb3Ig Rk9MTF9HRVQsIG5vdCBmb3IgdGhlIG5ld2VyIEZPTExfUElOLgo+ID4+Pj4gKwkgKgo+ID4+Pj4g KwkgKiBGT0xMX1BJTiBhbHdheXMgZXhwZWN0cyBwYWdlcyB0byBiZSBub24tbnVsbCwgYnV0IG5v IG5lZWQgdG8gYXNzZXJ0Cj4gPj4+PiArCSAqIHRoYXQgaGVyZSwgYXMgYW55IGZhaWx1cmVzIHdp bGwgYmUgb2J2aW91cyBlbm91Z2guCj4gPj4+PiArCSAqLwo+ID4+Pj4gKwlpZiAocGFnZXMgJiYg IShmbGFncyAmIEZPTExfUElOKSkKPiA+Pj4+ICAJCWZsYWdzIHw9IEZPTExfR0VUOwo+ID4+Pgo+ ID4+PiBEaWQgeW91IGxvb2sgYXQgdXNlciB0aGF0IGhhdmUgcGFnZXMgYW5kIG5vdCBGT0xMX0dF VCBzZXQgPwo+ID4+PiBJIGJlbGlldmUgaXQgd291bGQgYmUgYmV0dGVyIHRvIGZpcnN0IGZpeCB0 aGVtIHRvIGVuZCB1cAo+ID4+PiB3aXRoIEZPTExfR0VUIHNldCBhbmQgdGhlbiBlcnJvciBvdXQg aWYgcGFnZXMgaXMgIT0gTlVMTCBidXQKPiA+Pj4gbm9yIEZPTExfR0VUIG9yIEZPTExfUElOIGlz IHNldC4KPiA+Pj4KPiA+Pgo+ID4+IEkgd2FzIHBlcmhhcHMgb3Zlcmx5IGNhdXRpb3VzLCBhbmQg ZGlkbid0IGdvIHRoZXJlLiBIb3dldmVyLCBpdCdzIHByb2JhYmx5Cj4gPj4gZG9hYmxlLCBnaXZl biB0aGF0IHRoZXJlIHdhcyBhbHJlYWR5IHRoZSBmb2xsb3dpbmcgaW4gX19nZXRfdXNlcl9wYWdl cygpOgo+ID4+Cj4gPj4gICAgIFZNX0JVR19PTighIXBhZ2VzICE9ICEhKGd1cF9mbGFncyAmIEZP TExfR0VUKSk7Cj4gPj4KPiA+PiAuLi53aGljaCB3aWxsIGhhdmUgY29uZGl0aW9uZWQgcGVvcGxl IGFuZCBjb2RlIHRvIHNldCBGT0xMX0dFVCB0b2dldGhlciB3aXRoCj4gPj4gcGFnZXMuIFNvIEkg YWdyZWUgdGhhdCB0aGUgdGltZSBpcyByaWdodC4KPiA+Pgo+ID4+IEluIG9yZGVyIHRvIG1ha2Ug YmlzZWN0aW5nIGZ1dHVyZSBmYWlsdXJlcyBzaW1wbGVyLCBJIGNhbiBpbnNlcnQgYSBwYXRjaCBy aWdodCAKPiA+PiBiZWZvcmUgdGhpcyBvbmUsIHRoYXQgY2hhbmdlcyB0aGUgRk9MTF9HRVQgc2V0 dGluZyBpbnRvIGFuIGFzc2VydCwgbGlrZSB0aGlzOgo+ID4+Cj4gPj4gZGlmZiAtLWdpdCBhL21t L2d1cC5jIGIvbW0vZ3VwLmMKPiA+PiBpbmRleCA4ZjIzNmEzMzVhZTkuLmJlMzM4OTYxZTgwZCAx MDA2NDQKPiA+PiAtLS0gYS9tbS9ndXAuYwo+ID4+ICsrKyBiL21tL2d1cC5jCj4gPj4gQEAgLTEw MTQsOCArMTAxNCw4IEBAIHN0YXRpYyBfX2Fsd2F5c19pbmxpbmUgbG9uZyBfX2dldF91c2VyX3Bh Z2VzX2xvY2tlZChzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRzaywKPiA+PiAgICAgICAgICAgICAgICAg QlVHX09OKCpsb2NrZWQgIT0gMSk7Cj4gPj4gICAgICAgICB9Cj4gPj4gIAo+ID4+IC0gICAgICAg aWYgKHBhZ2VzKQo+ID4+IC0gICAgICAgICAgICAgICBmbGFncyB8PSBGT0xMX0dFVDsKPiA+PiAr ICAgICAgIGlmIChwYWdlcyAmJiBXQVJOX09OX09OQ0UoIShndXBfZmxhZ3MgJiBGT0xMX0dFVCkp KQo+ID4+ICsgICAgICAgICAgICAgICByZXR1cm4gLUVJTlZBTDsKPiA+PiAgCj4gPj4gICAgICAg ICBwYWdlc19kb25lID0gMDsKPiA+PiAgICAgICAgIGxvY2tfZHJvcHBlZCA9IGZhbHNlOwo+ID4+ Cj4gPj4KPiA+PiAuLi5hbmQgdGhlbiBhZGQgaW4gRk9MTF9QSU4sIHdpdGggdGhpcyBwYXRjaC4K PiA+IAo+ID4gbG9va3MgZ29vZCBidXQgZG91YmxlIGNoZWNrIHRoYXQgaXQgc2hvdWxkIG5vdCBo YXBwZW5zLCBpIHdpbGwgdHJ5Cj4gPiB0byBjaGVjayBvbiBteSBzaWRlIHRvby4KPiAKPiBZZXMs IEknbGwgbG9vay4KPiAKPiAuLi4KPiA+Pj4+ICsJICovCj4gPj4+PiArCWd1cF9mbGFncyB8PSBG T0xMX1JFTU9URSB8IEZPTExfUElOOwo+ID4+Pgo+ID4+PiBXb3VsZG4ndCBpdCBiZSBiZXR0ZXIg dG8gbm90IGFkZCBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKCkgdW50aWwKPiA+Pj4gaXQgY2Fu IGJlIHByb3Blcmx5IGltcGxlbWVudGVkID8KPiA+Pj4KPiA+Pgo+ID4+IFdlbGwsIHRoZSBwcm9i bGVtIGlzIHRoYXQgSSBuZWVkIGVhY2ggY2FsbCBzaXRlIHRoYXQgcmVxdWlyZXMgRk9MTF9QSU4K PiA+PiB0byB1c2UgYSBwcm9wZXIgd3JhcHBlci4gSXQncyB0aGUgRk9MTF9QSU4gdGhhdCBpcyB0 aGUgZm9jdXMgaGVyZSwgYmVjYXVzZQo+ID4+IHRoZXJlIGlzIGEgaGFyZCwgYnJpZ2h0IHJ1bGUs IHdoaWNoIGlzOiBpZiBhbmQgb25seSBpZiBhIGNhbGxlciBzZXRzCj4gPj4gRk9MTF9QSU4sIHRo ZW4gdGhlIGRtYS1wYWdlIHRyYWNraW5nIGhhcHBlbnMsIGFuZCBwdXRfdXNlcl9wYWdlKCkgbXVz dAo+ID4+IGJlIGNhbGxlZC4KPiA+Pgo+ID4+IFNvIHRoaXMgbGVhdmVzIG1lIHdpdGggb25seSB0 d28gcmVhc29uYWJsZSBjaG9pY2VzOgo+ID4+Cj4gPj4gYSkgQ29udmVydCB0aGUgY2FsbCBzaXRl IGFzIGFib3ZlOiBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKCksIHdoaWNoIHNldHMKPiA+PiBG T0xMX1BJTiAodGhlIGtleSBwb2ludCEpLCBhbmQgbGVhdmVzIHRoZSBGT0xMX0xPTkdURVJNIHNp dHVhdGlvbiBleGFjdGx5Cj4gPj4gYXMgaXQgaGFzIGJlZW4gc28gZmFyLiBXaGVuIHRoZSBGT0xM X0xPTkdURVJNIHNpdHVhdGlvbiBpcyBmaXhlZCwgdGhlIGNhbGwKPiA+PiBzaXRlICptaWdodCog bm90IG5lZWQgYW55IGNoYW5nZXMgdG8gYWRvcHQgdGhlIHdvcmtpbmcgZ3VwLmMgY29kZS4KPiA+ Pgo+ID4+IGIpIENvbnZlcnQgdGhlIGNhbGwgc2l0ZSB0byBwaW5fdXNlcl9wYWdlc19yZW1vdGUo KSwgd2hpY2ggYWxzbyBzZXRzCj4gPj4gRk9MTF9QSU4sIGFuZCBhbHNvIGxlYXZlcyB0aGUgRk9M TF9MT05HVEVSTSBzaXR1YXRpb24gZXhhY3RseSBhcyBiZWZvcmUuCj4gPj4gVGhlcmUgd291bGQg YWxzbyBiZSBhIGNvbW1lbnQgYXQgdGhlIGNhbGwgc2l0ZSwgdG8gdGhlIGVmZmVjdCBvZiwgInRo aXMKPiA+PiBpcyB0aGUgd3JvbmcgY2FsbCB0byBtYWtlOiBpdCByZWFsbHkgcmVxdWlyZXMgRk9M TF9MT05HVEVSTSBiZWhhdmlvciIuCj4gPj4KPiA+PiBXaGVuIHRoZSBGT0xMX0xPTkdURVJNIHNp dHVhdGlvbiBpcyBmaXhlZCwgdGhlIGNhbGwgc2l0ZSB3aWxsIG5lZWQgdG8gYmUKPiA+PiBjaGFu Z2VkIHRvIHBpbl9sb25ndGVybV9wYWdlc19yZW1vdGUoKS4KPiA+Pgo+ID4+IFNvIHlvdSBjYW4g cHJvYmFibHkgc2VlIHdoeSBJIHBpY2tlZCAoYSkuCj4gPiAKPiA+IEJ1dCByaWdodCBub3cgbm9i b2R5IGhhcyBGT0xMX0xPTkdURVJNIGFuZCBGT0xMX1JFTU9URS4gU28geW91IHNob3VsZAo+ID4g bmV2ZXIgaGF2ZSB0aGUgbmVlZCBmb3IgcGluX2xvbmd0ZXJtX3BhZ2VzX3JlbW90ZSgpLiBNeSBm ZWFyIGlzIHRoYXQKPiA+IGxvbmd0ZXJtIGhhcyBpbXBsaWNhdGlvbiBhbmQgaXQgd291bGQgYmUg YmV0dGVyIHRvIG5vdCBkcm9wIHRoaXMgaW1wbGljYXRpb24KPiA+IGJ5IGFkZGluZyBhIHdyYXBw ZXIgdGhhdCBkb2VzIG5vdCBkbyB3aGF0IHRoZSBuYW1lIHNheXMuCj4gPiAKPiA+IFNvIGRvIG5v dCBpbnRyb2R1Y2UgcGluX2xvbmd0ZXJtX3BhZ2VzX3JlbW90ZSgpIHVudGlsIGl0cyBmaXJzdCB1 c2VyCj4gPiBoYXBwZW5zLiBUaGlzIGlzIG9wdGlvbiBjKQo+ID4gCj4gCj4gQWxtb3N0IGZvcmdv dCwgdGhvdWdoOiB0aGVyZSBpcyBhbHJlYWR5IGFub3RoZXIgdXNlcjogSW5maW5pYmFuZDoKPiAK PiBkcml2ZXJzL2luZmluaWJhbmQvY29yZS91bWVtX29kcC5jOjY0NjogICAgICAgICBucGFnZXMg PSBwaW5fbG9uZ3Rlcm1fcGFnZXNfcmVtb3RlKG93bmluZ19wcm9jZXNzLCBvd25pbmdfbW0sCgpv ZHAgZG8gbm90IG5lZWQgdGhhdCwgaSB0aG91Z2h0IHRoZSBITU0gY29udmVydGlvbiB3YXMgYWxy ZWFkeSB1cHN0cmVhbQpidXQgc2VlbXMgbm90LCBpbiBhbnkgY2FzZSBvZHAgZG8gbm90IG5lZWQg dGhlIGxvbmd0ZXJtIGNhc2UgaXQgb25seQpzbyBiZXN0IGlzIHRvIHJldmVydCB0aGF0IHVzZXIg dG8gZ3VwX2Zhc3Qgb3Igc29tZXRoaW5nIHVudGlsIGl0IGdldApjb252ZXJ0ZWQgdG8gSE1NLgoK Q2hlZXJzLApKw6lyw7RtZQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0 b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJp LWRldmVs