From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH hmm 00/15] Consolidate the mmu notifier interval_tree and locking Date: Wed, 23 Oct 2019 16:26:13 -0400 Message-ID: <20191023202539.GA3200@redhat.com> References: <20191015181242.8343-1-jgg@ziepe.ca> <20191021184041.GF3177@redhat.com> <20191021190556.GI6285@mellanox.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20191021190556.GI6285-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Content-Disposition: inline List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "amd-gfx" To: Jason Gunthorpe Cc: Andrea Arcangeli , Ralph Campbell , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , John Hubbard , "Felix.Kuehling-5C7GfCeVMHo@public.gmane.org" , "dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org" , "linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org" , "amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org" , Ben Skeggs T24gTW9uLCBPY3QgMjEsIDIwMTkgYXQgMDc6MDY6MDBQTSArMDAwMCwgSmFzb24gR3VudGhvcnBl IHdyb3RlOgo+IE9uIE1vbiwgT2N0IDIxLCAyMDE5IGF0IDAyOjQwOjQxUE0gLTA0MDAsIEplcm9t ZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBUdWUsIE9jdCAxNSwgMjAxOSBhdCAwMzoxMjoyN1BNIC0w MzAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6Cj4gPiA+IEZyb206IEphc29uIEd1bnRob3JwZSA8 amdnQG1lbGxhbm94LmNvbT4KPiA+ID4gCj4gPiA+IDggb2YgdGhlIG1tdV9ub3RpZmllciB1c2lu ZyBkcml2ZXJzIChpOTE1X2dlbSwgcmFkZW9uX21uLCB1bWVtX29kcCwgaGZpMSwKPiA+ID4gc2Np Zl9kbWEsIHZob3N0LCBnbnRkZXYsIGhtbSkgZHJpdmVycyBhcmUgdXNpbmcgYSBjb21tb24gcGF0 dGVybiB3aGVyZQo+ID4gPiB0aGV5IG9ubHkgdXNlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQvZW5k IGFuZCBpbW1lZGlhdGVseSBjaGVjayB0aGUKPiA+ID4gaW52YWxpZGF0aW5nIHJhbmdlIGFnYWlu c3Qgc29tZSBkcml2ZXIgZGF0YSBzdHJ1Y3R1cmUgdG8gdGVsbCBpZiB0aGUKPiA+ID4gZHJpdmVy IGlzIGludGVyZXN0ZWQuIEhhbGYgb2YgdGhlbSB1c2UgYW4gaW50ZXJ2YWxfdHJlZSwgdGhlIG90 aGVycyBhcmUKPiA+ID4gc2ltcGxlIGxpbmVhciBzZWFyY2ggbGlzdHMuCj4gPiA+IAo+ID4gPiBP ZiB0aGUgb25lcyBJIGNoZWNrZWQgdGhleSBsYXJnZWx5IHNlZW0gdG8gaGF2ZSB2YXJpb3VzIGtp bmRzIG9mIHJhY2VzLAo+ID4gPiBidWdzIGFuZCBwb29yIGltcGxlbWVudGF0aW9uLiBUaGlzIGlz IGEgcmVzdWx0IG9mIHRoZSBjb21wbGV4aXR5IGluIGhvdwo+ID4gPiB0aGUgbm90aWZpZXIgaW50 ZXJhY3RzIHdpdGggZ2V0X3VzZXJfcGFnZXMoKS4gSXQgaXMgZXh0cmVtZWx5IGRpZmZpY3VsdCB0 bwo+ID4gPiB1c2UgaXQgY29ycmVjdGx5Lgo+ID4gPiAKPiA+ID4gQ29uc29saWRhdGUgYWxsIG9m IHRoaXMgY29kZSB0b2dldGhlciBpbnRvIHRoZSBjb3JlIG1tdV9ub3RpZmllciBhbmQKPiA+ID4g cHJvdmlkZSBhIGxvY2tpbmcgc2NoZW1lIHNpbWlsYXIgdG8gaG1tX21pcnJvciB0aGF0IGFsbG93 cyB0aGUgdXNlciB0bwo+ID4gPiBzYWZlbHkgdXNlIGdldF91c2VyX3BhZ2VzKCkgYW5kIHJlbGlh Ymx5IGtub3cgaWYgdGhlIHBhZ2UgbGlzdCBzdGlsbAo+ID4gPiBtYXRjaGVzIHRoZSBtbS4KPiA+ ID4gCj4gPiA+IFRoaXMgbmV3IGFycmFuZ21lbnQgcGxheXMgbmljZWx5IHdpdGggdGhlICFibG9j a2FibGUgbW9kZSBmb3IKPiA+ID4gT09NLiBTY2FubmluZyB0aGUgaW50ZXJ2YWwgdHJlZSBpcyBk b25lIHN1Y2ggdGhhdCB0aGUgaW50ZXJzZWN0aW9uIHRlc3QKPiA+ID4gd2lsbCBhbHdheXMgc3Vj Y2VlZCwgYW5kIHNpbmNlIHRoZXJlIGlzIG5vIGludmFsaWRhdGVfcmFuZ2VfZW5kIGV4cG9zZWQg dG8KPiA+ID4gZHJpdmVycyB0aGUgc2NoZW1lIHNhZmVseSBhbGxvd3MgbXVsdGlwbGUgZHJpdmVy cyB0byBiZSBzdWJzY3JpYmVkLgo+ID4gPiAKPiA+ID4gRm91ciBwbGFjZXMgYXJlIGNvbnZlcnRl ZCBhcyBhbiBleGFtcGxlIG9mIGhvdyB0aGUgbmV3IEFQSSBpcyB1c2VkLgo+ID4gPiBGb3VyIGFy ZSBsZWZ0IGZvciBmdXR1cmUgcGF0Y2hlczoKPiA+ID4gIC0gaTkxNV9nZW0gaGFzIGNvbXBsZXgg bG9ja2luZyBhcm91bmQgZGVzdHJ1Y3Rpb24gb2YgYSByZWdpc3RyYXRpb24sCj4gPiA+ICAgIG5l ZWRzIG1vcmUgc3R1ZHkKPiA+ID4gIC0gaGZpMSAoMm5kIHVzZXIpIG5lZWRzIGFjY2VzcyB0byB0 aGUgcmJ0cmVlCj4gPiA+ICAtIHNjaWZfZG1hIGhhcyBhIGNvbXBsaWNhdGVkIGxvZ2ljIGZsb3cK PiA+ID4gIC0gdmhvc3QncyBtbXUgbm90aWZpZXJzIGFyZSBhbHJlYWR5IGJlaW5nIHJld3JpdHRl bgo+ID4gPiAKPiA+ID4gVGhpcyBpcyBzdGlsbCBiZWluZyB0ZXN0ZWQsIGJ1dCBJIGZpZ3VyZWQg dG8gc2VuZCBpdCB0byBzdGFydCBnZXR0aW5nIGhlbHAKPiA+ID4gZnJvbSB0aGUgeGVuLCBhbWQg YW5kIGhmaSBkcml2ZXJzIHdoaWNoIEkgY2Fubm90IHRlc3QgaGVyZS4KPiA+IAo+ID4gSXQgbWln aHQgYmUgYSBnb29kIG9wb3J0dW5pdHkgdG8gYWxzbyBzd2l0Y2ggdGhvc2UgdXNlcnMgdG8KPiA+ IGhtbV9yYW5nZV9mYXVsdCgpIGluc3RlYWQgb2YgR1VQIGFzIEdVUCBpcyBwb2ludGxlc3MgZm9y IHRob3NlCj4gPiB1c2Vycy4gSW4gZmFjdCB0aGUgR1VQIGlzIGFuIGltcGVkaW1lbnQgdG8gbm9y bWFsIG1tIG9wZXJhdGlvbnMuCj4gCj4gSSB0aGluayB2aG9zdCBjYW4gdXNlIGhtbV9yYW5nZV9m YXVsdAo+IAo+IGhmaTEgZG9lcyBhY3R1YWxseSBuZWVkIHRvIGhhdmUgdGhlIHBhZ2UgcGluLCBp dCBkb2Vzbid0IGZlbmNlIERNQQo+IGR1cmluZyBpbnZhbGlkYXRlLgo+IAo+IGk5MTVfZ2VtIGZl ZWxzIGFsb3QgbGlrZSBhbWRncHUsIHNvIHByb2JhYmx5IGl0IHdvdWxkIGJlbmVmaXQKPiAKPiBO byBpZGVhIGFib3V0IHNjaWZfZG1hCj4gCj4gPiBJIHdpbGwgdGVzdCBvbiBub3V2ZWF1Lgo+IAo+ IFRoYW5rcywgaG9wZWZ1bGx5IGl0IHN0aWxsIHdvcmtzLCBJIHRoaW5rIFJhbHBoIHdhcyBhYmxl IHRvIGRvIHNvbWUKPiBiYXNpYyBjaGVja3MuIEJ1dCBpdCBpcyBhIHByZXR0eSBjb21wbGljYXRl ZCBzZXJpZXMsIEkgcHJvYmFibHkgbWFkZQo+IHNvbWUgbWlzdGFrZXMuCgpTbyBpdCBzZWVtcyB0 byB3b3JrIG9rIHdpdGggbm91dmVhdSwgd2lsbCBsZXQgdGVzdHMgcnVuIGluIGxvb3AgdGhvdWdo dAp0aGVyZSBhcmUgbm90IHZlcnkgYWR2YW5jZSB0ZXN0LgoKPiAKPiBGV0lXLCBJIGtub3cgdGhh dCBub3V2ZWF1IGdldHMgYSBsb2NrZGVwIHNwbGF0IG5vdyBmcm9tIERhbmllbAo+IFZldHRlcidz IHJlY2VudCBjaGFuZ2VzLCBpdCB0cmllcyB0byBkbyBHRlBfS0VSRU5FTCBhbGxvY2F0aW9ucyB1 bmRlcgo+IGEgbG9jayBhbHNvIGhlbGQgYnkgdGhlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQgcGF0 aC4KCkkgaGF2ZSBub3Qgc2VlbiBhbnkgc3BsYXQgc28gZmFyLCBpcyBpdCB0aHJvdWdodCBzb21l IG5ldyBrZXJuZWwgY29uZmlnID8KCkNoZWVycywKSsOpcsO0bWUKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmFtZC1nZnggbWFpbGluZyBsaXN0CmFtZC1n ZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21h aWxtYW4vbGlzdGluZm8vYW1kLWdmeA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65A80CA9EBC for ; Wed, 23 Oct 2019 20:26:24 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 43E0421872 for ; Wed, 23 Oct 2019 20:26:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43E0421872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F408F6EB83; Wed, 23 Oct 2019 20:26:23 +0000 (UTC) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by gabe.freedesktop.org (Postfix) with ESMTPS id 02D716EB8B for ; Wed, 23 Oct 2019 20:26:22 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-32-Mi54ZwgNMmyd4SWvphQ_uA-1; Wed, 23 Oct 2019 16:26:18 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 01D3C800D5A; Wed, 23 Oct 2019 20:26:17 +0000 (UTC) Received: from redhat.com (ovpn-123-188.rdu2.redhat.com [10.10.123.188]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9A79610027A4; Wed, 23 Oct 2019 20:26:15 +0000 (UTC) Date: Wed, 23 Oct 2019 16:26:13 -0400 From: Jerome Glisse To: Jason Gunthorpe Subject: Re: [PATCH hmm 00/15] Consolidate the mmu notifier interval_tree and locking Message-ID: <20191023202539.GA3200@redhat.com> References: <20191015181242.8343-1-jgg@ziepe.ca> <20191021184041.GF3177@redhat.com> <20191021190556.GI6285@mellanox.com> MIME-Version: 1.0 In-Reply-To: <20191021190556.GI6285@mellanox.com> User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-MC-Unique: Mi54ZwgNMmyd4SWvphQ_uA-1 X-Mimecast-Spam-Score: 0 Content-Disposition: inline X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1571862381; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g1XeLbQwzCck5Lma54SyUL7NHxZOnECT3DsLOWtBh4k=; b=flLdISb/pLQJ0TRAqBSa1m3RAYv04lBBYV4NjaKYxYdXmPi1gpyKN+bEe/XAR8Bu1Oppyu mesM6VI4bkJcoUCAc6kDo9rBvBANy5tMwiBNlfzYYVdBo/5Q04KNz3AZnL2zNMAQPb9DXs 9I8NYjuvwyGPwFE79GWfyY0+9b6gPLQ= X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrea Arcangeli , Ralph Campbell , "linux-rdma@vger.kernel.org" , John Hubbard , "Felix.Kuehling@amd.com" , "dri-devel@lists.freedesktop.org" , "linux-mm@kvack.org" , "amd-gfx@lists.freedesktop.org" , Ben Skeggs Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" Message-ID: <20191023202613.8cpVfgGOA498n7-dkVlxLrFbOTKaV-bPQ49PLiRwIgU@z> T24gTW9uLCBPY3QgMjEsIDIwMTkgYXQgMDc6MDY6MDBQTSArMDAwMCwgSmFzb24gR3VudGhvcnBl IHdyb3RlOgo+IE9uIE1vbiwgT2N0IDIxLCAyMDE5IGF0IDAyOjQwOjQxUE0gLTA0MDAsIEplcm9t ZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBUdWUsIE9jdCAxNSwgMjAxOSBhdCAwMzoxMjoyN1BNIC0w MzAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6Cj4gPiA+IEZyb206IEphc29uIEd1bnRob3JwZSA8 amdnQG1lbGxhbm94LmNvbT4KPiA+ID4gCj4gPiA+IDggb2YgdGhlIG1tdV9ub3RpZmllciB1c2lu ZyBkcml2ZXJzIChpOTE1X2dlbSwgcmFkZW9uX21uLCB1bWVtX29kcCwgaGZpMSwKPiA+ID4gc2Np Zl9kbWEsIHZob3N0LCBnbnRkZXYsIGhtbSkgZHJpdmVycyBhcmUgdXNpbmcgYSBjb21tb24gcGF0 dGVybiB3aGVyZQo+ID4gPiB0aGV5IG9ubHkgdXNlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQvZW5k IGFuZCBpbW1lZGlhdGVseSBjaGVjayB0aGUKPiA+ID4gaW52YWxpZGF0aW5nIHJhbmdlIGFnYWlu c3Qgc29tZSBkcml2ZXIgZGF0YSBzdHJ1Y3R1cmUgdG8gdGVsbCBpZiB0aGUKPiA+ID4gZHJpdmVy IGlzIGludGVyZXN0ZWQuIEhhbGYgb2YgdGhlbSB1c2UgYW4gaW50ZXJ2YWxfdHJlZSwgdGhlIG90 aGVycyBhcmUKPiA+ID4gc2ltcGxlIGxpbmVhciBzZWFyY2ggbGlzdHMuCj4gPiA+IAo+ID4gPiBP ZiB0aGUgb25lcyBJIGNoZWNrZWQgdGhleSBsYXJnZWx5IHNlZW0gdG8gaGF2ZSB2YXJpb3VzIGtp bmRzIG9mIHJhY2VzLAo+ID4gPiBidWdzIGFuZCBwb29yIGltcGxlbWVudGF0aW9uLiBUaGlzIGlz IGEgcmVzdWx0IG9mIHRoZSBjb21wbGV4aXR5IGluIGhvdwo+ID4gPiB0aGUgbm90aWZpZXIgaW50 ZXJhY3RzIHdpdGggZ2V0X3VzZXJfcGFnZXMoKS4gSXQgaXMgZXh0cmVtZWx5IGRpZmZpY3VsdCB0 bwo+ID4gPiB1c2UgaXQgY29ycmVjdGx5Lgo+ID4gPiAKPiA+ID4gQ29uc29saWRhdGUgYWxsIG9m IHRoaXMgY29kZSB0b2dldGhlciBpbnRvIHRoZSBjb3JlIG1tdV9ub3RpZmllciBhbmQKPiA+ID4g cHJvdmlkZSBhIGxvY2tpbmcgc2NoZW1lIHNpbWlsYXIgdG8gaG1tX21pcnJvciB0aGF0IGFsbG93 cyB0aGUgdXNlciB0bwo+ID4gPiBzYWZlbHkgdXNlIGdldF91c2VyX3BhZ2VzKCkgYW5kIHJlbGlh Ymx5IGtub3cgaWYgdGhlIHBhZ2UgbGlzdCBzdGlsbAo+ID4gPiBtYXRjaGVzIHRoZSBtbS4KPiA+ ID4gCj4gPiA+IFRoaXMgbmV3IGFycmFuZ21lbnQgcGxheXMgbmljZWx5IHdpdGggdGhlICFibG9j a2FibGUgbW9kZSBmb3IKPiA+ID4gT09NLiBTY2FubmluZyB0aGUgaW50ZXJ2YWwgdHJlZSBpcyBk b25lIHN1Y2ggdGhhdCB0aGUgaW50ZXJzZWN0aW9uIHRlc3QKPiA+ID4gd2lsbCBhbHdheXMgc3Vj Y2VlZCwgYW5kIHNpbmNlIHRoZXJlIGlzIG5vIGludmFsaWRhdGVfcmFuZ2VfZW5kIGV4cG9zZWQg dG8KPiA+ID4gZHJpdmVycyB0aGUgc2NoZW1lIHNhZmVseSBhbGxvd3MgbXVsdGlwbGUgZHJpdmVy cyB0byBiZSBzdWJzY3JpYmVkLgo+ID4gPiAKPiA+ID4gRm91ciBwbGFjZXMgYXJlIGNvbnZlcnRl ZCBhcyBhbiBleGFtcGxlIG9mIGhvdyB0aGUgbmV3IEFQSSBpcyB1c2VkLgo+ID4gPiBGb3VyIGFy ZSBsZWZ0IGZvciBmdXR1cmUgcGF0Y2hlczoKPiA+ID4gIC0gaTkxNV9nZW0gaGFzIGNvbXBsZXgg bG9ja2luZyBhcm91bmQgZGVzdHJ1Y3Rpb24gb2YgYSByZWdpc3RyYXRpb24sCj4gPiA+ICAgIG5l ZWRzIG1vcmUgc3R1ZHkKPiA+ID4gIC0gaGZpMSAoMm5kIHVzZXIpIG5lZWRzIGFjY2VzcyB0byB0 aGUgcmJ0cmVlCj4gPiA+ICAtIHNjaWZfZG1hIGhhcyBhIGNvbXBsaWNhdGVkIGxvZ2ljIGZsb3cK PiA+ID4gIC0gdmhvc3QncyBtbXUgbm90aWZpZXJzIGFyZSBhbHJlYWR5IGJlaW5nIHJld3JpdHRl bgo+ID4gPiAKPiA+ID4gVGhpcyBpcyBzdGlsbCBiZWluZyB0ZXN0ZWQsIGJ1dCBJIGZpZ3VyZWQg dG8gc2VuZCBpdCB0byBzdGFydCBnZXR0aW5nIGhlbHAKPiA+ID4gZnJvbSB0aGUgeGVuLCBhbWQg YW5kIGhmaSBkcml2ZXJzIHdoaWNoIEkgY2Fubm90IHRlc3QgaGVyZS4KPiA+IAo+ID4gSXQgbWln aHQgYmUgYSBnb29kIG9wb3J0dW5pdHkgdG8gYWxzbyBzd2l0Y2ggdGhvc2UgdXNlcnMgdG8KPiA+ IGhtbV9yYW5nZV9mYXVsdCgpIGluc3RlYWQgb2YgR1VQIGFzIEdVUCBpcyBwb2ludGxlc3MgZm9y IHRob3NlCj4gPiB1c2Vycy4gSW4gZmFjdCB0aGUgR1VQIGlzIGFuIGltcGVkaW1lbnQgdG8gbm9y bWFsIG1tIG9wZXJhdGlvbnMuCj4gCj4gSSB0aGluayB2aG9zdCBjYW4gdXNlIGhtbV9yYW5nZV9m YXVsdAo+IAo+IGhmaTEgZG9lcyBhY3R1YWxseSBuZWVkIHRvIGhhdmUgdGhlIHBhZ2UgcGluLCBp dCBkb2Vzbid0IGZlbmNlIERNQQo+IGR1cmluZyBpbnZhbGlkYXRlLgo+IAo+IGk5MTVfZ2VtIGZl ZWxzIGFsb3QgbGlrZSBhbWRncHUsIHNvIHByb2JhYmx5IGl0IHdvdWxkIGJlbmVmaXQKPiAKPiBO byBpZGVhIGFib3V0IHNjaWZfZG1hCj4gCj4gPiBJIHdpbGwgdGVzdCBvbiBub3V2ZWF1Lgo+IAo+ IFRoYW5rcywgaG9wZWZ1bGx5IGl0IHN0aWxsIHdvcmtzLCBJIHRoaW5rIFJhbHBoIHdhcyBhYmxl IHRvIGRvIHNvbWUKPiBiYXNpYyBjaGVja3MuIEJ1dCBpdCBpcyBhIHByZXR0eSBjb21wbGljYXRl ZCBzZXJpZXMsIEkgcHJvYmFibHkgbWFkZQo+IHNvbWUgbWlzdGFrZXMuCgpTbyBpdCBzZWVtcyB0 byB3b3JrIG9rIHdpdGggbm91dmVhdSwgd2lsbCBsZXQgdGVzdHMgcnVuIGluIGxvb3AgdGhvdWdo dAp0aGVyZSBhcmUgbm90IHZlcnkgYWR2YW5jZSB0ZXN0LgoKPiAKPiBGV0lXLCBJIGtub3cgdGhh dCBub3V2ZWF1IGdldHMgYSBsb2NrZGVwIHNwbGF0IG5vdyBmcm9tIERhbmllbAo+IFZldHRlcidz IHJlY2VudCBjaGFuZ2VzLCBpdCB0cmllcyB0byBkbyBHRlBfS0VSRU5FTCBhbGxvY2F0aW9ucyB1 bmRlcgo+IGEgbG9jayBhbHNvIGhlbGQgYnkgdGhlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQgcGF0 aC4KCkkgaGF2ZSBub3Qgc2VlbiBhbnkgc3BsYXQgc28gZmFyLCBpcyBpdCB0aHJvdWdodCBzb21l IG5ldyBrZXJuZWwgY29uZmlnID8KCkNoZWVycywKSsOpcsO0bWUKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmFtZC1nZnggbWFpbGluZyBsaXN0CmFtZC1n ZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21h aWxtYW4vbGlzdGluZm8vYW1kLWdmeA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12857CA9EB9 for ; Wed, 23 Oct 2019 20:26:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D80082064A for ; Wed, 23 Oct 2019 20:26:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="flLdISb/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729191AbfJWU0X (ORCPT ); Wed, 23 Oct 2019 16:26:23 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:31990 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725874AbfJWU0X (ORCPT ); Wed, 23 Oct 2019 16:26:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1571862381; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g1XeLbQwzCck5Lma54SyUL7NHxZOnECT3DsLOWtBh4k=; b=flLdISb/pLQJ0TRAqBSa1m3RAYv04lBBYV4NjaKYxYdXmPi1gpyKN+bEe/XAR8Bu1Oppyu mesM6VI4bkJcoUCAc6kDo9rBvBANy5tMwiBNlfzYYVdBo/5Q04KNz3AZnL2zNMAQPb9DXs 9I8NYjuvwyGPwFE79GWfyY0+9b6gPLQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-32-Mi54ZwgNMmyd4SWvphQ_uA-1; Wed, 23 Oct 2019 16:26:18 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 01D3C800D5A; Wed, 23 Oct 2019 20:26:17 +0000 (UTC) Received: from redhat.com (ovpn-123-188.rdu2.redhat.com [10.10.123.188]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9A79610027A4; Wed, 23 Oct 2019 20:26:15 +0000 (UTC) Date: Wed, 23 Oct 2019 16:26:13 -0400 From: Jerome Glisse To: Jason Gunthorpe Cc: Ralph Campbell , John Hubbard , "Felix.Kuehling@amd.com" , "linux-rdma@vger.kernel.org" , "linux-mm@kvack.org" , Andrea Arcangeli , "dri-devel@lists.freedesktop.org" , "amd-gfx@lists.freedesktop.org" , Ben Skeggs Subject: Re: [PATCH hmm 00/15] Consolidate the mmu notifier interval_tree and locking Message-ID: <20191023202539.GA3200@redhat.com> References: <20191015181242.8343-1-jgg@ziepe.ca> <20191021184041.GF3177@redhat.com> <20191021190556.GI6285@mellanox.com> MIME-Version: 1.0 In-Reply-To: <20191021190556.GI6285@mellanox.com> User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-MC-Unique: Mi54ZwgNMmyd4SWvphQ_uA-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Mon, Oct 21, 2019 at 07:06:00PM +0000, Jason Gunthorpe wrote: > On Mon, Oct 21, 2019 at 02:40:41PM -0400, Jerome Glisse wrote: > > On Tue, Oct 15, 2019 at 03:12:27PM -0300, Jason Gunthorpe wrote: > > > From: Jason Gunthorpe > > >=20 > > > 8 of the mmu_notifier using drivers (i915_gem, radeon_mn, umem_odp, h= fi1, > > > scif_dma, vhost, gntdev, hmm) drivers are using a common pattern wher= e > > > they only use invalidate_range_start/end and immediately check the > > > invalidating range against some driver data structure to tell if the > > > driver is interested. Half of them use an interval_tree, the others a= re > > > simple linear search lists. > > >=20 > > > Of the ones I checked they largely seem to have various kinds of race= s, > > > bugs and poor implementation. This is a result of the complexity in h= ow > > > the notifier interacts with get_user_pages(). It is extremely difficu= lt to > > > use it correctly. > > >=20 > > > Consolidate all of this code together into the core mmu_notifier and > > > provide a locking scheme similar to hmm_mirror that allows the user t= o > > > safely use get_user_pages() and reliably know if the page list still > > > matches the mm. > > >=20 > > > This new arrangment plays nicely with the !blockable mode for > > > OOM. Scanning the interval tree is done such that the intersection te= st > > > will always succeed, and since there is no invalidate_range_end expos= ed to > > > drivers the scheme safely allows multiple drivers to be subscribed. > > >=20 > > > Four places are converted as an example of how the new API is used. > > > Four are left for future patches: > > > - i915_gem has complex locking around destruction of a registration, > > > needs more study > > > - hfi1 (2nd user) needs access to the rbtree > > > - scif_dma has a complicated logic flow > > > - vhost's mmu notifiers are already being rewritten > > >=20 > > > This is still being tested, but I figured to send it to start getting= help > > > from the xen, amd and hfi drivers which I cannot test here. > >=20 > > It might be a good oportunity to also switch those users to > > hmm_range_fault() instead of GUP as GUP is pointless for those > > users. In fact the GUP is an impediment to normal mm operations. >=20 > I think vhost can use hmm_range_fault >=20 > hfi1 does actually need to have the page pin, it doesn't fence DMA > during invalidate. >=20 > i915_gem feels alot like amdgpu, so probably it would benefit >=20 > No idea about scif_dma >=20 > > I will test on nouveau. >=20 > Thanks, hopefully it still works, I think Ralph was able to do some > basic checks. But it is a pretty complicated series, I probably made > some mistakes. So it seems to work ok with nouveau, will let tests run in loop thought there are not very advance test. >=20 > FWIW, I know that nouveau gets a lockdep splat now from Daniel > Vetter's recent changes, it tries to do GFP_KERENEL allocations under > a lock also held by the invalidate_range_start path. I have not seen any splat so far, is it throught some new kernel config ? Cheers, J=E9r=F4me From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 251FBCA9EBB for ; Wed, 23 Oct 2019 20:26:25 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 042772064A for ; Wed, 23 Oct 2019 20:26:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 042772064A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 40A656EB8B; Wed, 23 Oct 2019 20:26:24 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id E60C76EB83 for ; Wed, 23 Oct 2019 20:26:22 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-32-Mi54ZwgNMmyd4SWvphQ_uA-1; Wed, 23 Oct 2019 16:26:18 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 01D3C800D5A; Wed, 23 Oct 2019 20:26:17 +0000 (UTC) Received: from redhat.com (ovpn-123-188.rdu2.redhat.com [10.10.123.188]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9A79610027A4; Wed, 23 Oct 2019 20:26:15 +0000 (UTC) Date: Wed, 23 Oct 2019 16:26:13 -0400 From: Jerome Glisse To: Jason Gunthorpe Subject: Re: [PATCH hmm 00/15] Consolidate the mmu notifier interval_tree and locking Message-ID: <20191023202539.GA3200@redhat.com> References: <20191015181242.8343-1-jgg@ziepe.ca> <20191021184041.GF3177@redhat.com> <20191021190556.GI6285@mellanox.com> MIME-Version: 1.0 In-Reply-To: <20191021190556.GI6285@mellanox.com> User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-MC-Unique: Mi54ZwgNMmyd4SWvphQ_uA-1 X-Mimecast-Spam-Score: 0 Content-Disposition: inline X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1571862381; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g1XeLbQwzCck5Lma54SyUL7NHxZOnECT3DsLOWtBh4k=; b=flLdISb/pLQJ0TRAqBSa1m3RAYv04lBBYV4NjaKYxYdXmPi1gpyKN+bEe/XAR8Bu1Oppyu mesM6VI4bkJcoUCAc6kDo9rBvBANy5tMwiBNlfzYYVdBo/5Q04KNz3AZnL2zNMAQPb9DXs 9I8NYjuvwyGPwFE79GWfyY0+9b6gPLQ= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrea Arcangeli , Ralph Campbell , "linux-rdma@vger.kernel.org" , John Hubbard , "Felix.Kuehling@amd.com" , "dri-devel@lists.freedesktop.org" , "linux-mm@kvack.org" , "amd-gfx@lists.freedesktop.org" , Ben Skeggs Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191023202613.aOe7YlJdRqhw41pe8SZLq4Ch73B-nKn6GNzOxUs2KbU@z> T24gTW9uLCBPY3QgMjEsIDIwMTkgYXQgMDc6MDY6MDBQTSArMDAwMCwgSmFzb24gR3VudGhvcnBl IHdyb3RlOgo+IE9uIE1vbiwgT2N0IDIxLCAyMDE5IGF0IDAyOjQwOjQxUE0gLTA0MDAsIEplcm9t ZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBUdWUsIE9jdCAxNSwgMjAxOSBhdCAwMzoxMjoyN1BNIC0w MzAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6Cj4gPiA+IEZyb206IEphc29uIEd1bnRob3JwZSA8 amdnQG1lbGxhbm94LmNvbT4KPiA+ID4gCj4gPiA+IDggb2YgdGhlIG1tdV9ub3RpZmllciB1c2lu ZyBkcml2ZXJzIChpOTE1X2dlbSwgcmFkZW9uX21uLCB1bWVtX29kcCwgaGZpMSwKPiA+ID4gc2Np Zl9kbWEsIHZob3N0LCBnbnRkZXYsIGhtbSkgZHJpdmVycyBhcmUgdXNpbmcgYSBjb21tb24gcGF0 dGVybiB3aGVyZQo+ID4gPiB0aGV5IG9ubHkgdXNlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQvZW5k IGFuZCBpbW1lZGlhdGVseSBjaGVjayB0aGUKPiA+ID4gaW52YWxpZGF0aW5nIHJhbmdlIGFnYWlu c3Qgc29tZSBkcml2ZXIgZGF0YSBzdHJ1Y3R1cmUgdG8gdGVsbCBpZiB0aGUKPiA+ID4gZHJpdmVy IGlzIGludGVyZXN0ZWQuIEhhbGYgb2YgdGhlbSB1c2UgYW4gaW50ZXJ2YWxfdHJlZSwgdGhlIG90 aGVycyBhcmUKPiA+ID4gc2ltcGxlIGxpbmVhciBzZWFyY2ggbGlzdHMuCj4gPiA+IAo+ID4gPiBP ZiB0aGUgb25lcyBJIGNoZWNrZWQgdGhleSBsYXJnZWx5IHNlZW0gdG8gaGF2ZSB2YXJpb3VzIGtp bmRzIG9mIHJhY2VzLAo+ID4gPiBidWdzIGFuZCBwb29yIGltcGxlbWVudGF0aW9uLiBUaGlzIGlz IGEgcmVzdWx0IG9mIHRoZSBjb21wbGV4aXR5IGluIGhvdwo+ID4gPiB0aGUgbm90aWZpZXIgaW50 ZXJhY3RzIHdpdGggZ2V0X3VzZXJfcGFnZXMoKS4gSXQgaXMgZXh0cmVtZWx5IGRpZmZpY3VsdCB0 bwo+ID4gPiB1c2UgaXQgY29ycmVjdGx5Lgo+ID4gPiAKPiA+ID4gQ29uc29saWRhdGUgYWxsIG9m IHRoaXMgY29kZSB0b2dldGhlciBpbnRvIHRoZSBjb3JlIG1tdV9ub3RpZmllciBhbmQKPiA+ID4g cHJvdmlkZSBhIGxvY2tpbmcgc2NoZW1lIHNpbWlsYXIgdG8gaG1tX21pcnJvciB0aGF0IGFsbG93 cyB0aGUgdXNlciB0bwo+ID4gPiBzYWZlbHkgdXNlIGdldF91c2VyX3BhZ2VzKCkgYW5kIHJlbGlh Ymx5IGtub3cgaWYgdGhlIHBhZ2UgbGlzdCBzdGlsbAo+ID4gPiBtYXRjaGVzIHRoZSBtbS4KPiA+ ID4gCj4gPiA+IFRoaXMgbmV3IGFycmFuZ21lbnQgcGxheXMgbmljZWx5IHdpdGggdGhlICFibG9j a2FibGUgbW9kZSBmb3IKPiA+ID4gT09NLiBTY2FubmluZyB0aGUgaW50ZXJ2YWwgdHJlZSBpcyBk b25lIHN1Y2ggdGhhdCB0aGUgaW50ZXJzZWN0aW9uIHRlc3QKPiA+ID4gd2lsbCBhbHdheXMgc3Vj Y2VlZCwgYW5kIHNpbmNlIHRoZXJlIGlzIG5vIGludmFsaWRhdGVfcmFuZ2VfZW5kIGV4cG9zZWQg dG8KPiA+ID4gZHJpdmVycyB0aGUgc2NoZW1lIHNhZmVseSBhbGxvd3MgbXVsdGlwbGUgZHJpdmVy cyB0byBiZSBzdWJzY3JpYmVkLgo+ID4gPiAKPiA+ID4gRm91ciBwbGFjZXMgYXJlIGNvbnZlcnRl ZCBhcyBhbiBleGFtcGxlIG9mIGhvdyB0aGUgbmV3IEFQSSBpcyB1c2VkLgo+ID4gPiBGb3VyIGFy ZSBsZWZ0IGZvciBmdXR1cmUgcGF0Y2hlczoKPiA+ID4gIC0gaTkxNV9nZW0gaGFzIGNvbXBsZXgg bG9ja2luZyBhcm91bmQgZGVzdHJ1Y3Rpb24gb2YgYSByZWdpc3RyYXRpb24sCj4gPiA+ICAgIG5l ZWRzIG1vcmUgc3R1ZHkKPiA+ID4gIC0gaGZpMSAoMm5kIHVzZXIpIG5lZWRzIGFjY2VzcyB0byB0 aGUgcmJ0cmVlCj4gPiA+ICAtIHNjaWZfZG1hIGhhcyBhIGNvbXBsaWNhdGVkIGxvZ2ljIGZsb3cK PiA+ID4gIC0gdmhvc3QncyBtbXUgbm90aWZpZXJzIGFyZSBhbHJlYWR5IGJlaW5nIHJld3JpdHRl bgo+ID4gPiAKPiA+ID4gVGhpcyBpcyBzdGlsbCBiZWluZyB0ZXN0ZWQsIGJ1dCBJIGZpZ3VyZWQg dG8gc2VuZCBpdCB0byBzdGFydCBnZXR0aW5nIGhlbHAKPiA+ID4gZnJvbSB0aGUgeGVuLCBhbWQg YW5kIGhmaSBkcml2ZXJzIHdoaWNoIEkgY2Fubm90IHRlc3QgaGVyZS4KPiA+IAo+ID4gSXQgbWln aHQgYmUgYSBnb29kIG9wb3J0dW5pdHkgdG8gYWxzbyBzd2l0Y2ggdGhvc2UgdXNlcnMgdG8KPiA+ IGhtbV9yYW5nZV9mYXVsdCgpIGluc3RlYWQgb2YgR1VQIGFzIEdVUCBpcyBwb2ludGxlc3MgZm9y IHRob3NlCj4gPiB1c2Vycy4gSW4gZmFjdCB0aGUgR1VQIGlzIGFuIGltcGVkaW1lbnQgdG8gbm9y bWFsIG1tIG9wZXJhdGlvbnMuCj4gCj4gSSB0aGluayB2aG9zdCBjYW4gdXNlIGhtbV9yYW5nZV9m YXVsdAo+IAo+IGhmaTEgZG9lcyBhY3R1YWxseSBuZWVkIHRvIGhhdmUgdGhlIHBhZ2UgcGluLCBp dCBkb2Vzbid0IGZlbmNlIERNQQo+IGR1cmluZyBpbnZhbGlkYXRlLgo+IAo+IGk5MTVfZ2VtIGZl ZWxzIGFsb3QgbGlrZSBhbWRncHUsIHNvIHByb2JhYmx5IGl0IHdvdWxkIGJlbmVmaXQKPiAKPiBO byBpZGVhIGFib3V0IHNjaWZfZG1hCj4gCj4gPiBJIHdpbGwgdGVzdCBvbiBub3V2ZWF1Lgo+IAo+ IFRoYW5rcywgaG9wZWZ1bGx5IGl0IHN0aWxsIHdvcmtzLCBJIHRoaW5rIFJhbHBoIHdhcyBhYmxl IHRvIGRvIHNvbWUKPiBiYXNpYyBjaGVja3MuIEJ1dCBpdCBpcyBhIHByZXR0eSBjb21wbGljYXRl ZCBzZXJpZXMsIEkgcHJvYmFibHkgbWFkZQo+IHNvbWUgbWlzdGFrZXMuCgpTbyBpdCBzZWVtcyB0 byB3b3JrIG9rIHdpdGggbm91dmVhdSwgd2lsbCBsZXQgdGVzdHMgcnVuIGluIGxvb3AgdGhvdWdo dAp0aGVyZSBhcmUgbm90IHZlcnkgYWR2YW5jZSB0ZXN0LgoKPiAKPiBGV0lXLCBJIGtub3cgdGhh dCBub3V2ZWF1IGdldHMgYSBsb2NrZGVwIHNwbGF0IG5vdyBmcm9tIERhbmllbAo+IFZldHRlcidz IHJlY2VudCBjaGFuZ2VzLCBpdCB0cmllcyB0byBkbyBHRlBfS0VSRU5FTCBhbGxvY2F0aW9ucyB1 bmRlcgo+IGEgbG9jayBhbHNvIGhlbGQgYnkgdGhlIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQgcGF0 aC4KCkkgaGF2ZSBub3Qgc2VlbiBhbnkgc3BsYXQgc28gZmFyLCBpcyBpdCB0aHJvdWdodCBzb21l IG5ldyBrZXJuZWwgY29uZmlnID8KCkNoZWVycywKSsOpcsO0bWUKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJp LWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9y Zy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbA==