From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE671C433E0 for ; Thu, 28 Jan 2021 15:47:08 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4E62864DF5 for ; Thu, 28 Jan 2021 15:47:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E62864DF5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 438E66E1D8; Thu, 28 Jan 2021 15:47:07 +0000 (UTC) Received: from NAM04-CO1-obe.outbound.protection.outlook.com (mail-eopbgr690086.outbound.protection.outlook.com [40.107.69.86]) by gabe.freedesktop.org (Postfix) with ESMTPS id 30F408990D for ; Thu, 28 Jan 2021 15:47:06 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EnNDrftnqM8q+b/SIjNvsyXzpNoDB/sEe+Ctd31rSKSSdOpAAwjB6bWKuI5hzIKgQInPLI9ZK873pooaKAVaHVUW87TnyVARKfNM+1csBo0pXVYEtVa7l00WMoOpbvRQC7OuiwqchB9c95CPgo8qU3jdvaSBTMRyeVDL8ObKiPpDnL770xGFXGEnPxrPlvsy9nrjPC+pdi/G/b0SlkObPlkuj9kp9LeDFf7rlhOB0ZlzdZ47UuziQJoPL903k87mMHGdQmeiwu7mxiyqssV7OJXlSIEtxRXCuTbaU4mC0Zw2nXnYGKB+FzwjWc1uFiGTxi8HozmTTD8R8rYit6BQ6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6YGUdpB46iYQO/O9eu++i+D7oar6n4WR1pI/yUXhcw0=; b=nwAV5Exeo6vqXsapxKBoyzcLwXvuhE6z1RW7/g/KR6kBA5eme/4fCqHVOhT838kTbz3ioo8s0K4ucOQpMxBVBHEZqsxSHcw7sTrw36jXJvgrQ3xcUSS1mTANDEMHZmU96bHBHou7ydyplLyQlkLn3ArB1b+NjCh3DCIUuKzCN8iMTrKgM9CmRNsnSlKPDekywMjGxfKISny0bbI3jIyobDyWcTVzdTty+76g7efgp18Pkw/TBaeo/r7hj0/6AQkrabVUGjEJAofl0oqPCnPUb+7iKOq+dimD4l264ac4QGZ9/P/XntBerOFn0FDDZc2cBHT2OJ0AOJVV2cfJI+Y9+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6YGUdpB46iYQO/O9eu++i+D7oar6n4WR1pI/yUXhcw0=; b=x2G2I0ad//xZrFGJ8v6iwEzFiC3uQPOOQvg+a1chYUDvEwA9XV9nj2o978KQjNbzQ7nj443unbVqi+SswszhzLxutzJYRspAZRH4rgusiOexJno21iI3P8R9V2CABDLtSVP6gzHTwFK7Lid9bI7bXLOt3ufGUw+yxCvrNwltb1U= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) by MN2PR12MB4357.namprd12.prod.outlook.com (2603:10b6:208:262::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.13; Thu, 28 Jan 2021 15:47:03 +0000 Received: from MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::44f:9f01:ece7:f0e5]) by MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::44f:9f01:ece7:f0e5%3]) with mapi id 15.20.3784.019; Thu, 28 Jan 2021 15:47:03 +0000 Subject: Re: [Linaro-mm-sig] [PATCH] RFC: dma-fence: Document recoverable page fault implications To: Felix Kuehling , Maarten Lankhorst , Daniel Vetter , DRI Development References: <20210121194056.1734409-1-daniel.vetter@ffwll.ch> <6d373177-2645-1d67-9c14-dcad87c4f4d9@amd.com> <68740fcf-530e-b929-1c98-5810fc97ed23@linux.intel.com> <1e38efbc-ec52-e436-21e4-49a0d074b57b@amd.com> <18e7efbd-3d10-5ad1-49c9-7e26f0a27ef2@amd.com> <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Thu, 28 Jan 2021 16:46:55 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 In-Reply-To: <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> Content-Language: en-US X-Originating-IP: [2a02:908:1252:fb60:be8a:bd56:1f94:86e7] X-ClientProxiedBy: AM3PR04CA0144.eurprd04.prod.outlook.com (2603:10a6:207::28) To MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7] (2a02:908:1252:fb60:be8a:bd56:1f94:86e7) by AM3PR04CA0144.eurprd04.prod.outlook.com (2603:10a6:207::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Thu, 28 Jan 2021 15:47:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 45d0ccb9-e853-47f4-c44c-08d8c3a3f529 X-MS-TrafficTypeDiagnostic: MN2PR12MB4357: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qeoARvnhkew/Mql3yT8WA6X3ku3NdgUyvd/HUNviImUzJ0eLjDjKJpc4OWAqDL9OxwcSHTfRUbdT4Bf0l9GGegw13DanGrMwx1k9g72thHJNslpg7zTdy/EQpzEwWnQyGBNUzcCAUb8Yn8PdvPKA0Nbl+th1YF/FuP+e0hb5cv5ByUPJb1aOWaop6+4Ytvg5pb5g2VvrxJB55b1O7n9zMFQoVPf7kVLo7lHA7+qRcpaDcn7J/0GeEXrvfzifvAA5ebwjBPvm8OxDAI0oOT7Z1B0iVt/NyzRqNGBIEIrrYCfb++6djEcd9svWIOmU36HO7aHZANNFnSmOfIozxwwmYu2U5cCeQTAKy4EiPqO1by/VxLE/9FB4xwIwjvXFnxnTIUjoYc5dlA+k3UZT6cFWb1MKC1tVhHQ5adFANLd8dGEFiSx9P+sTyQf55AgsYbwWzO//V27KCT0jo/azPX7krPrUqSXKCUQBJFKNygpVLrZLSNlOxjWXEH+mZaDlLm6LDxEj8ji4P8lpyOth5q6Y+UPZZWUwlqmS86X2BsHHiYiNrEI4gpGkr+3Nq+Z4XNrABKDgLwwYqmudmx6T5p4foa0IvbBpwHFeFKFkjWNLS11wYeA7Vj0EeV58NDQHL+mSeVuaal4oaJVTDKSEDEPg93mPilL6JKk05cRIJTO6MHVbKZdhj8COH3GoXwOMVsMF X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB3775.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(396003)(366004)(136003)(39860400002)(346002)(2906002)(8676002)(86362001)(30864003)(45080400002)(54906003)(110136005)(4326008)(5660300002)(966005)(31696002)(478600001)(6666004)(52116002)(66574015)(2616005)(316002)(36756003)(6486002)(186003)(16526019)(66556008)(31686004)(8936002)(83380400001)(66476007)(66946007)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?RlYzOFI4WjhKZUF4elJSZUVGK1NpTWc1SUx4aGZDQ3IwYUt3YXVGTWRseVp5?= =?utf-8?B?VndkZ3BjN0d3TmxOZWp3MmJTbmtHaW9YbFRNZ2tHYjVWZmtsY1AycTdwcSt6?= =?utf-8?B?NlhJQmlwUU13VlJUUlZ5RjV1N2NSeEtDc1c3czZxV2hnN0Q5RGtORXkvbjdO?= =?utf-8?B?MTQvSm5UY1VtUmhHbVB4VjFleFVkSDVDOG40Z2VVd1NlcVVld25YU3JQN3E1?= =?utf-8?B?VWR3c1BpQTdYckpDa0JyY2lZRi9iTlJtM2lFUlp3TXAvQkdvTzR4VE8yeW40?= =?utf-8?B?UXBJNitsM0RFdWZKY3lueHYxQjB0RldmS3l4UkhlY2JIeFJ0WmpKODJVOU9v?= =?utf-8?B?NjVBZm9aMkw1MU5mR3RUR1o3NjA5bWlPdXY1OHlWSXlqUWFnNnppN21BLzNC?= =?utf-8?B?MXM0TlJyc2JsNGFpUlZTK1hqS0NWVlpiTDZNalRhd3RLRm41Rzl2d0MxK1Ir?= =?utf-8?B?WEdHa3VuNjNhcTRHeCtrRjA3NXhxRHZJeFZEbFdZODhuQWVsVllvMWtRYVJE?= =?utf-8?B?dVQyRVhsek55NHE0eGJZRk1vRUNLWmVGOS8vbEczMjYyRzYrRFdWQ29PbWMv?= =?utf-8?B?UGVidXZnM3dXOFBiQ0xqU1dmWVdRS0NXLzNXUGptQTl5L0RDdzhVYXBZVEQ1?= =?utf-8?B?eGxjTXZvUUs1ZHhJRm9ja01ZdlFSVUEzRVF6NXV4VjlIT01TcHZJbmtOS1p3?= =?utf-8?B?eUh6Y2s1WHVkYkROMGo5TlJKVVpRNlFOK1ZFVHl4ait0N3YzYUtVdU5MYXg5?= =?utf-8?B?SmFMc2JJRzZtcTRJVDVDNXBYR0dNS3FadDRKV01hRHp0R3hDTWxrNGNKY1Q5?= =?utf-8?B?VVN0SkZheG9PUXp3TVJ2V0FlN2NFTEdlNXl1cEg2YUpRWWhyUFBjcEdVSE1q?= =?utf-8?B?cmU2YjU3eXQzZE80NlkzV1pwQVpYS3VMMGNhL1AwWEhpZ1pjbUlUUm9qOTZZ?= =?utf-8?B?eVhBSjByV2prVlhYeGVGaHFKWVV6UGZTZThWWnhxRGEyYWZGYmNDQlN1bjIr?= =?utf-8?B?bkVHVFVKZmRqVGZpY0ZOUXZXY2tVLzNGbzI5Z1MrRVpXQUY3MGFVRW03Y3Nq?= =?utf-8?B?Tmhab2ZONFNJY3UwVWdQbXBRRkU1eHdUS1N1L2xGak1Sc0c3ZDJQeW96M2pn?= =?utf-8?B?cmxlTGlheG9QTDFrQnVOVm55WWcxc3RZbVNwUGJtcjRtTGlhWEdHM2w0bkZ3?= =?utf-8?B?VUtGc1lGVG14ajZIVUNpVWM0NVcycDBlWHY2T0pjYlZFWmpsekpLK3ZLVVBF?= =?utf-8?B?akRRRFVrOTJScGg4c1hLSU5iYXdmclBGeFJVem90aG5jTG53OEJkMXJaMSt5?= =?utf-8?B?K2h2Mk1MRjUvcXlJMXFqSytoaWZKZHF4L0oyREVtWHM1OEJWWnBOYVNzUk90?= =?utf-8?B?TlhvZ0JFWStndVpNbGVGTS80UkQ0VURpZm5JSkIzUGl6Z0FCaitMZ1pUb2F3?= =?utf-8?B?bGI1aVZjQTFRYUZpSXVidmxHVVFhdXd1eUh2bDlIejN0dEFHOFZRVGtiWU5u?= =?utf-8?Q?vbsBvo4hjLaXIq1OA0kjEczxxPo?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45d0ccb9-e853-47f4-c44c-08d8c3a3f529 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3775.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jan 2021 15:47:03.8432 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eOvPXibaq2dS2hWW/C5N3NAAsYb3qsnSkQNY4ZmRodGkcqSLsQXTKkTDkXqoR1yf X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4357 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linaro-mm-sig@lists.linaro.org, Daniel Vetter , Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , linux-media@vger.kernel.org Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" QW0gMjguMDEuMjEgdW0gMTY6Mzkgc2NocmllYiBGZWxpeCBLdWVobGluZzoKPiBBbSAyMDIxLTAx LTI4IHVtIDI6MzkgYS5tLiBzY2hyaWViIENocmlzdGlhbiBLw7ZuaWc6Cj4+IEFtIDI3LjAxLjIx IHVtIDIzOjAwIHNjaHJpZWIgRmVsaXggS3VlaGxpbmc6Cj4+PiBBbSAyMDIxLTAxLTI3IHVtIDc6 MTYgYS5tLiBzY2hyaWViIENocmlzdGlhbiBLw7ZuaWc6Cj4+Pj4gQW0gMjcuMDEuMjEgdW0gMTM6 MTEgc2NocmllYiBNYWFydGVuIExhbmtob3JzdDoKPj4+Pj4gT3AgMjctMDEtMjAyMSBvbSAwMToy MiBzY2hyZWVmIEZlbGl4IEt1ZWhsaW5nOgo+Pj4+Pj4gQW0gMjAyMS0wMS0yMSB1bSAyOjQwIHAu bS4gc2NocmllYiBEYW5pZWwgVmV0dGVyOgo+Pj4+Pj4+IFJlY2VudGx5IHRoZXJlIHdhcyBhIGZh aXJseSBsb25nIHRocmVhZCBhYm91dCByZWNvcmVhYmxlIGhhcmR3YXJlCj4+Pj4+Pj4gcGFnZQo+ Pj4+Pj4+IGZhdWx0cywgaG93IHRoZXkgY2FuIGRlYWRsb2NrLCBhbmQgd2hhdCB0byBkbyBhYm91 dCB0aGF0Lgo+Pj4+Pj4+Cj4+Pj4+Pj4gV2hpbGUgdGhlIGRpc2N1c3Npb24gaXMgc3RpbGwgZnJl c2ggSSBmaWd1cmVkIGdvb2QgdGltZSB0byB0cnkgYW5kCj4+Pj4+Pj4gZG9jdW1lbnQgdGhlIGNv bmNsdXNpb25zIGEgYml0Lgo+Pj4+Pj4+Cj4+Pj4+Pj4gUmVmZXJlbmNlczoKPj4+Pj4+PiBodHRw czovL25hbTExLnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9aHR0cHMlM0El MkYlMkZsb3JlLmtlcm5lbC5vcmclMkZkcmktZGV2ZWwlMkYyMDIxMDEwNzAzMDEyNy4yMDM5My0x LUZlbGl4Lkt1ZWhsaW5nJTQwYW1kLmNvbSUyRiZhbXA7ZGF0YT0wNCU3QzAxJTdDZmVsaXgua3Vl aGxpbmclNDBhbWQuY29tJTdDNGU0ODg0YmU1NWQ3NGM0ZGRhMTQwOGQ4YzM1ZmQwYWIlN0MzZGQ4 OTYxZmU0ODg0ZTYwOGUxMWE4MmQ5OTRlMTgzZCU3QzAlN0MwJTdDNjM3NDc0MTYzNTkyMjYwNTUy JTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJam9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1 TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzEwMDAmYW1wO3NkYXRhPXkyVnpD NHZiZk1pMGN0eWVyQUhmcU9EWjZ0dGh6MUZVRHdwTUNwMFBJclElM0QmYW1wO3Jlc2VydmVkPTAK Pj4+Pj4+Pgo+Pj4+Pj4+IENjOiBNYWFydGVuIExhbmtob3JzdCA8bWFhcnRlbi5sYW5raG9yc3RA bGludXguaW50ZWwuY29tPgo+Pj4+Pj4+IENjOiBUaG9tYXMgSGVsbHN0csO2bSA8dGhvbWFzLmhl bGxzdHJvbUBpbnRlbC5jb20+Cj4+Pj4+Pj4gQ2M6ICJDaHJpc3RpYW4gS8O2bmlnIiA8Y2hyaXN0 aWFuLmtvZW5pZ0BhbWQuY29tPgo+Pj4+Pj4+IENjOiBKZXJvbWUgR2xpc3NlIDxqZ2xpc3NlQHJl ZGhhdC5jb20+Cj4+Pj4+Pj4gQ2M6IEZlbGl4IEt1ZWhsaW5nIDxmZWxpeC5rdWVobGluZ0BhbWQu Y29tPgo+Pj4+Pj4+IFNpZ25lZC1vZmYtYnk6IERhbmllbCBWZXR0ZXIgPGRhbmllbC52ZXR0ZXJA aW50ZWwuY29tPgo+Pj4+Pj4+IENjOiBTdW1pdCBTZW13YWwgPHN1bWl0LnNlbXdhbEBsaW5hcm8u b3JnPgo+Pj4+Pj4+IENjOiBsaW51eC1tZWRpYUB2Z2VyLmtlcm5lbC5vcmcKPj4+Pj4+PiBDYzog bGluYXJvLW1tLXNpZ0BsaXN0cy5saW5hcm8ub3JnCj4+Pj4+Pj4gLS0gCj4+Pj4+Pj4gSSdsbCBi ZSBhd2F5IG5leHQgd2VlaywgYnV0IGZpZ3VyZWQgSSdsbCB0eXBlIHRoaXMgdXAgcXVpY2tseSBm b3IKPj4+Pj4+PiBzb21lCj4+Pj4+Pj4gY29tbWVudHMgYW5kIHRvIGNoZWNrIHdoZXRoZXIgSSBn b3QgdGhpcyBhbGwgcm91Z2hseSByaWdodC4KPj4+Pj4+Pgo+Pj4+Pj4+IENyaXRpcXVlIHZlcnkg bXVjaCB3YW50ZWQgb24gdGhpcywgc28gdGhhdCB3ZSBjYW4gbWFrZSBzdXJlIGh3IHdoaWNoCj4+ Pj4+Pj4gY2FuJ3QgcHJlZW1wdCAod2l0aCBwYWdlZmF1bHRzIHBlbmRpbmcpIGxpa2UgZ2Z4MTAg aGFzIGEgY2xlYXIKPj4+Pj4+PiBwYXRoIHRvCj4+Pj4+Pj4gc3VwcG9ydCBwYWdlIGZhdWx0cyBp biB1cHN0cmVhbS4gU28gYW55dGhpbmcgSSBtaXNzZWQsIGdvdCB3cm9uZyBvcgo+Pj4+Pj4+IGxp a2UgdGhhdCB3b3VsZCBiZSBnb29kLgo+Pj4+Pj4+IC1EYW5pZWwKPj4+Pj4+PiAtLS0KPj4+Pj4+ PiAgwqDCoCBEb2N1bWVudGF0aW9uL2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QgfCA2Ngo+Pj4+Pj4+ ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysKPj4+Pj4+PiAgwqDCoCAxIGZpbGUgY2hhbmdl ZCwgNjYgaW5zZXJ0aW9ucygrKQo+Pj4+Pj4+Cj4+Pj4+Pj4gZGlmZiAtLWdpdCBhL0RvY3VtZW50 YXRpb24vZHJpdmVyLWFwaS9kbWEtYnVmLnJzdAo+Pj4+Pj4+IGIvRG9jdW1lbnRhdGlvbi9kcml2 ZXItYXBpL2RtYS1idWYucnN0Cj4+Pj4+Pj4gaW5kZXggYTIxMzNkNjk4NzJjLi5lOTI0YzFlNGY3 YTMgMTAwNjQ0Cj4+Pj4+Pj4gLS0tIGEvRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2RtYS1idWYu cnN0Cj4+Pj4+Pj4gKysrIGIvRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2RtYS1idWYucnN0Cj4+ Pj4+Pj4gQEAgLTI1NywzICsyNTcsNjkgQEAgZmVuY2VzIGluIHRoZSBrZXJuZWwuIFRoaXMgbWVh bnM6Cj4+Pj4+Pj4gIMKgwqDCoMKgIHVzZXJzcGFjZSBpcyBhbGxvd2VkIHRvIHVzZSB1c2Vyc3Bh Y2UgZmVuY2luZyBvciBsb25nIHJ1bm5pbmcKPj4+Pj4+PiBjb21wdXRlCj4+Pj4+Pj4gIMKgwqDC oMKgIHdvcmtsb2Fkcy4gVGhpcyBhbHNvIG1lYW5zIG5vIGltcGxpY2l0IGZlbmNpbmcgZm9yIHNo YXJlZAo+Pj4+Pj4+IGJ1ZmZlcnMgaW4gdGhlc2UKPj4+Pj4+PiAgwqDCoMKgwqAgY2FzZXMuCj4+ Pj4+Pj4gKwo+Pj4+Pj4+ICtSZWNvdmVyYWJsZSBIYXJkd2FyZSBQYWdlIEZhdWx0cyBJbXBsaWNh dGlvbnMKPj4+Pj4+PiArfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+ fn5+Cj4+Pj4+Pj4gKwo+Pj4+Pj4+ICtNb2Rlcm4gaGFyZHdhcmUgc3VwcG9ydHMgcmVjb3ZlcmFi bGUgcGFnZSBmYXVsdHMsIHdoaWNoIGhhcyBhCj4+Pj4+Pj4gbG90IG9mCj4+Pj4+Pj4gK2ltcGxp Y2F0aW9ucyBmb3IgRE1BIGZlbmNlcy4KPj4+Pj4+PiArCj4+Pj4+Pj4gK0ZpcnN0LCBhIHBlbmRp bmcgcGFnZSBmYXVsdCBvYnZpb3VzbHkgaG9sZHMgdXAgdGhlIHdvcmsgdGhhdCdzCj4+Pj4+Pj4g cnVubmluZyBvbiB0aGUKPj4+Pj4+PiArYWNjZWxlcmF0b3IgYW5kIGEgbWVtb3J5IGFsbG9jYXRp b24gaXMgdXN1YWxseSByZXF1aXJlZCB0byByZXNvbHZlCj4+Pj4+Pj4gdGhlIGZhdWx0Lgo+Pj4+ Pj4+ICtCdXQgbWVtb3J5IGFsbG9jYXRpb25zIGFyZSBub3QgYWxsb3dlZCB0byBnYXRlIGNvbXBs ZXRpb24gb2YgRE1BCj4+Pj4+Pj4gZmVuY2VzLCB3aGljaAo+Pj4+Pj4+ICttZWFucyBhbnkgd29y a2xvYWQgdXNpbmcgcmVjb3ZlcmFibGUgcGFnZSBmYXVsdHMgY2Fubm90IHVzZSBETUEKPj4+Pj4+ PiBmZW5jZXMgZm9yCj4+Pj4+Pj4gK3N5bmNocm9uaXphdGlvbi4gU3luY2hyb25pemF0aW9uIGZl bmNlcyBjb250cm9sbGVkIGJ5IHVzZXJzcGFjZQo+Pj4+Pj4+IG11c3QgYmUgdXNlZAo+Pj4+Pj4+ ICtpbnN0ZWFkLgo+Pj4+Pj4+ICsKPj4+Pj4+PiArT24gR1BVcyB0aGlzIHBvc2VzIGEgcHJvYmxl bSwgYmVjYXVzZSBjdXJyZW50IGRlc2t0b3AgY29tcG9zaXRvcgo+Pj4+Pj4+IHByb3RvY29scyBv bgo+Pj4+Pj4+ICtMaW51cyByZWx5IG9uIERNQSBmZW5jZXMsIHdoaWNoIG1lYW5zIHdpdGhvdXQg YW4gZW50aXJlbHkgbmV3Cj4+Pj4+Pj4gdXNlcnNwYWNlIHN0YWNrCj4+Pj4+Pj4gK2J1aWx0IG9u IHRvcCBvZiB1c2Vyc3BhY2UgZmVuY2VzLCB0aGV5IGNhbm5vdCBiZW5lZml0IGZyb20KPj4+Pj4+ PiByZWNvdmVyYWJsZSBwYWdlCj4+Pj4+Pj4gK2ZhdWx0cy4gVGhlIGV4Y2VwdGlvbiBpcyB3aGVu IHBhZ2UgZmF1bHRzIGFyZSBvbmx5IHVzZWQgYXMKPj4+Pj4+PiBtaWdyYXRpb24gaGludHMgYW5k Cj4+Pj4+Pj4gK25ldmVyIHRvIG9uLWRlbWFuZCBmaWxsIGEgbWVtb3J5IHJlcXVlc3QuIEZvciBu b3cgdGhpcyBtZWFucwo+Pj4+Pj4+IHJlY292ZXJhYmxlIHBhZ2UKPj4+Pj4+PiArZmF1bHRzIG9u IEdQVXMgYXJlIGxpbWl0ZWQgdG8gcHVyZSBjb21wdXRlIHdvcmtsb2Fkcy4KPj4+Pj4+PiArCj4+ Pj4+Pj4gK0Z1cnRoZXJtb3JlIEdQVXMgdXN1YWxseSBoYXZlIHNoYXJlZCByZXNvdXJjZXMgYmV0 d2VlbiB0aGUgM0QKPj4+Pj4+PiByZW5kZXJpbmcgYW5kCj4+Pj4+Pj4gK2NvbXB1dGUgc2lkZSwg bGlrZSBjb21wdXRlIHVuaXRzIG9yIGNvbW1hbmQgc3VibWlzc2lvbiBlbmdpbmVzLiBJZgo+Pj4+ Pj4+IGJvdGggYSAzRAo+Pj4+Pj4+ICtqb2Igd2l0aCBhIERNQSBmZW5jZSBhbmQgYSBjb21wdXRl IHdvcmtsb2FkIHVzaW5nIHJlY292ZXJhYmxlIHBhZ2UKPj4+Pj4+PiBmYXVsdHMgYXJlCj4+Pj4+ Pj4gK3BlbmRpbmcgdGhleSBjb3VsZCBkZWFkbG9jazoKPj4+Pj4+PiArCj4+Pj4+Pj4gKy0gVGhl IDNEIHdvcmtsb2FkIG1pZ2h0IG5lZWQgdG8gd2FpdCBmb3IgdGhlIGNvbXB1dGUgam9iIHRvIGZp bmlzaAo+Pj4+Pj4+IGFuZCByZWxlYXNlCj4+Pj4+Pj4gK8KgIGhhcmR3YXJlIHJlc291cmNlcyBm aXJzdC4KPj4+Pj4+PiArCj4+Pj4+Pj4gKy0gVGhlIGNvbXB1dGUgd29ya2xvYWQgbWlnaHQgYmUg c3R1Y2sgaW4gYSBwYWdlIGZhdWx0LCBiZWNhdXNlIHRoZQo+Pj4+Pj4+IG1lbW9yeQo+Pj4+Pj4+ ICvCoCBhbGxvY2F0aW9uIGlzIHdhaXRpbmcgZm9yIHRoZSBETUEgZmVuY2Ugb2YgdGhlIDNEIHdv cmtsb2FkIHRvCj4+Pj4+Pj4gY29tcGxldGUuCj4+Pj4+Pj4gKwo+Pj4+Pj4+ICtUaGVyZSBhcmUg YSBmZXcgd2F5cyB0byBwcmV2ZW50IHRoaXMgcHJvYmxlbToKPj4+Pj4+PiArCj4+Pj4+Pj4gKy0g Q29tcHV0ZSB3b3JrbG9hZHMgY2FuIGFsd2F5cyBiZSBwcmVlbXB0ZWQsIGV2ZW4gd2hlbiBhIHBh Z2UKPj4+Pj4+PiBmYXVsdCBpcyBwZW5kaW5nCj4+Pj4+Pj4gK8KgIGFuZCBub3QgeWV0IHJlcGFp cmVkLiBOb3QgYWxsIGhhcmR3YXJlIHN1cHBvcnRzIHRoaXMuCj4+Pj4+Pj4gKwo+Pj4+Pj4+ICst IERNQSBmZW5jZSB3b3JrbG9hZHMgYW5kIHdvcmtsb2FkcyB3aGljaCBuZWVkIHBhZ2UgZmF1bHQg aGFuZGxpbmcKPj4+Pj4+PiBoYXZlCj4+Pj4+Pj4gK8KgIGluZGVwZW5kZW50IGhhcmR3YXJlIHJl c291cmNlcyB0byBndWFyYW50ZWUgZm9yd2FyZCBwcm9ncmVzcy4KPj4+Pj4+PiBUaGlzIGNvdWxk IGJlCj4+Pj4+Pj4gK8KgIGFjaGlldmVkIHRocm91Z2ggZS5nLiB0aHJvdWdoIGRlZGljYXRlZCBl bmdpbmVzIGFuZCBtaW5pbWFsCj4+Pj4+Pj4gY29tcHV0ZSB1bml0Cj4+Pj4+Pj4gK8KgIHJlc2Vy dmF0aW9ucyBmb3IgRE1BIGZlbmNlIHdvcmtsb2Fkcy4KPj4+Pj4+PiArCj4+Pj4+Pj4gKy0gVGhl IHJlc2VydmF0aW9uIGFwcHJvYWNoIGNvdWxkIGJlIGZ1cnRoZXIgcmVmaW5lZCBieSBvbmx5Cj4+ Pj4+Pj4gcmVzZXJ2aW5nIHRoZQo+Pj4+Pj4+ICvCoCBoYXJkd2FyZSByZXNvdXJjZXMgZm9yIERN QSBmZW5jZSB3b3JrbG9hZHMgd2hlbiB0aGV5IGFyZQo+Pj4+Pj4+IGluLWZsaWdodC4gVGhpcyBt dXN0Cj4+Pj4+Pj4gK8KgIGNvdmVyIHRoZSB0aW1lIGZyb20gd2hlbiB0aGUgRE1BIGZlbmNlIGlz IHZpc2libGUgdG8gb3RoZXIKPj4+Pj4+PiB0aHJlYWRzIHVwIHRvCj4+Pj4+Pj4gK8KgIG1vbWVu dCB3aGVuIGZlbmNlIGlzIGNvbXBsZXRlZCB0aHJvdWdoIGRtYV9mZW5jZV9zaWduYWwoKS4KPj4+ Pj4+PiArCj4+Pj4+Pj4gKy0gQXMgYSBsYXN0IHJlc29ydCwgaWYgdGhlIGhhcmR3YXJlIHByb3Zp ZGVzIG5vIHVzZWZ1bCByZXNlcnZhdGlvbgo+Pj4+Pj4+IG1lY2hhbmljcywKPj4+Pj4+PiArwqAg YWxsIHdvcmtsb2FkcyBtdXN0IGJlIGZsdXNoZWQgZnJvbSB0aGUgR1BVIHdoZW4gc3dpdGNoaW5n Cj4+Pj4+Pj4gYmV0d2VlbiBqb2JzCj4+Pj4+Pj4gK8KgIHJlcXVpcmluZyBETUEgZmVuY2VzIG9y IGpvYnMgcmVxdWlyaW5nIHBhZ2UgZmF1bHQgaGFuZGxpbmc6IFRoaXMKPj4+Pj4+PiBtZWFucyBh bGwgRE1BCj4+Pj4+Pj4gK8KgIGZlbmNlcyBtdXN0IGNvbXBsZXRlIGJlZm9yZSBhIGNvbXB1dGUg am9iIHdpdGggcGFnZSBmYXVsdAo+Pj4+Pj4+IGhhbmRsaW5nIGNhbiBiZQo+Pj4+Pj4+ICvCoCBp bnNlcnRlZCBpbnRvIHRoZSBzY2hlZHVsZXIgcXVldWUuIEFuZCB2aWNlIHZlcnNhLCBiZWZvcmUg YSBETUEKPj4+Pj4+PiBmZW5jZSBjYW4gYmUKPj4+Pj4+PiArwqAgbWFkZSB2aXNpYmxlIGFueXdo ZXJlIGluIHRoZSBzeXN0ZW0sIGFsbCBjb21wdXRlIHdvcmtsb2FkcyBtdXN0Cj4+Pj4+Pj4gYmUg cHJlZW1wdGVkCj4+Pj4+Pj4gK8KgIHRvIGd1YXJhbnRlZSBhbGwgcGVuZGluZyBHUFUgcGFnZSBm YXVsdHMgYXJlIGZsdXNoZWQuCj4+Pj4+PiBJIHRob3VnaHQgb2YgYW5vdGhlciBwb3NzaWJsZSB3 b3JrYXJvdW5kOgo+Pj4+Pj4KPj4+Pj4+ICDCoMKgwqAgKiBQYXJ0aXRpb24gdGhlIG1lbW9yeS4g U2VydmljaW5nIG9mIHBhZ2UgZmF1bHRzIHdpbGwgdXNlIGEKPj4+Pj4+IHNlcGFyYXRlCj4+Pj4+ PiAgwqDCoMKgwqDCoCBtZW1vcnkgcG9vbCB0aGF0IGNhbiBhbHdheXMgYmUgYWxsb2NhdGVkIGZy b20gd2l0aG91dAo+Pj4+Pj4gd2FpdGluZyBmb3IKPj4+Pj4+ICDCoMKgwqDCoMKgIGZlbmNlcy4g VGhpcyBpbmNsdWRlcyBtZW1vcnkgZm9yIHBhZ2UgdGFibGVzIGFuZCBtZW1vcnkgZm9yCj4+Pj4+ PiAgwqDCoMKgwqDCoCBtaWdyYXRpbmcgZGF0YSB0by4gWW91IG1heSBzdGVhbCBtZW1vcnkgZnJv bSBvdGhlciBwcm9jZXNzZXMKPj4+Pj4+IHRoYXQKPj4+Pj4+ICDCoMKgwqDCoMKgIGNhbiBwYWdl IGZhdWx0LCBzbyBubyBmZW5jZSB3YWl0aW5nIGlzIG5lY2Vzc2FyeS4gQmVpbmcgYWJsZSB0bwo+ Pj4+Pj4gIMKgwqDCoMKgwqAgc3RlYWwgbWVtb3J5IGF0IGFueSB0aW1lIGFsc28gbWVhbnMgdGhl cmUgYXJlIGJhc2ljYWxseSBubwo+Pj4+Pj4gIMKgwqDCoMKgwqAgb3V0LW9mLW1lbW9yeSBzaXR1 YXRpb25zIHlvdSBuZWVkIHRvIHdvcnJ5IGFib3V0LiBFdmVuIHBhZ2UKPj4+Pj4+IHRhYmxlcwo+ Pj4+Pj4gIMKgwqDCoMKgwqAgKGV4Y2VwdCB0aGUgcm9vdCBwYWdlIGRpcmVjdG9yeSBvZiBlYWNo IHByb2Nlc3MpIGNhbiBiZQo+Pj4+Pj4gc3RvbGVuIGluCj4+Pj4+PiAgwqDCoMKgwqDCoCB0aGUg d29yc3QgY2FzZS4KPj4+Pj4gSSB0aGluayAnb3ZlcmNvbW1pdCcgd291bGQgYmUgYSBuaWNlIHdh eSB0byBkZXNjcmliZSB0aGlzLiBCdXQgSSdtIG5vdAo+Pj4+PiBzdXJlIGhvdyBlYXN5IHRoaXMg aXMgdG8gaW1wbGVtZW50IGluIHByYWN0aWNlLiBZb3Ugd291bGQgYmFzaWNhbGx5Cj4+Pj4+IG5l ZWQKPj4+Pj4gdG8gY3JlYXRlIHlvdXIgb3duIG1lbW9yeSBtYW5hZ2VyIGZvciB0aGlzLgo+Pj4+ IFdlbGwgeW91IHdvdWxkIG5lZWQgYSBjb21wbGV0ZWx5IHNlcGFyYXRlIHBvb2wgZm9yIGJvdGgg ZGV2aWNlIGFzIHdlbGwKPj4+PiBhcyBzeXN0ZW0gbWVtb3J5Lgo+Pj4+Cj4+Pj4gRS5nLiBvbiBi b290IHdlIHNheSB3ZSBzdGVhbCBYIEdCIHN5c3RlbSBtZW1vcnkgb25seSBmb3IgSE1NLgo+Pj4g V2h5PyBUaGUgR1BVIGRyaXZlciBkb2Vzbid0IG5lZWQgdG8gYWxsb2NhdGUgc3lzdGVtIG1lbW9y eSBmb3IgSE1NLgo+Pj4gTWlncmF0aW9ucyB0byBzeXN0ZW0gbWVtb3J5IGFyZSBoYW5kbGVkIGJ5 IHRoZSBrZXJuZWwncyBoYW5kbGVfbW1fZmF1bHQKPj4+IGFuZCBwYWdlIGFsbG9jYXRvciBhbmQg c3dhcCBsb2dpYy4KPj4gQW5kIHRoYXQgb25lIGRlcGVuZHMgb24gZG1hX2ZlbmNlIGNvbXBsZXRp b24gYmVjYXVzZSB5b3UgY2FuIGVhc2lseQo+PiBuZWVkIHRvIHdhaXQgZm9yIGFuIE1NVSBub3Rp ZmllciBjYWxsYmFjay4KPiBJIHNlZSwgdGhlIEdGWCBNTVUgbm90aWZpZXIgZm9yIHVzZXJwb2lu dGVycyBpbiBhbWRncHUgY3VycmVudGx5IHdhaXRzCj4gZm9yIGZlbmNlcy4gRm9yIHRoZSBLRkQg TU1VIG5vdGlmaWVyIEkgYW0gcGxhbm5pbmcgdG8gZml4IHRoaXMgYnkKPiBjYXVzaW5nIEdQVSBw YWdlIGZhdWx0cyBpbnN0ZWFkIG9mIHByZWVtcHRpbmcgdGhlIHF1ZXVlcy4gQ2FuIHdlIGxpbWl0 Cj4gdXNlcnB0cnMgaW4gYW1kZ3B1IHRvIGVuZ2luZXMgdGhhdCBjYW4gcGFnZSBmYXVsdC4gQmFz aWNhbGx5IG1ha2UgaXQKPiBpbGxlZ2FsIHRvIGF0dGFjaCB1c2VycHRyIEJPcyB0byBncmFwaGlj cyBDUyBCTyBsaXN0cywgc28gdGhleSBjYW4gb25seQo+IGJlIHVzZWQgaW4gdXNlciBtb2RlIGNv bW1hbmQgc3VibWlzc2lvbnMsIHdoaWNoIGNhbiBwYWdlIGZhdWx0LiBUaGVuIHRoZQo+IEdGWCBN TVUgbm90aWZpZXIgY291bGQgaW52YWxpZGF0ZSBQVEVzIGFuZCB3b3VsZCBub3QgaGF2ZSB0byB3 YWl0IGZvcgo+IGZlbmNlcy4KCkl0J3Mgbm90IG9ubHkgdGhlIE1NVSBub3RpZmllciwgdGhlIFRU TSBzaHJpbmtlciBJJ20gYWRkaW5nIG5lZWRzIHRvIAp3YWl0IGZvciBkbWFfZmVuY2VzIGFzIHdl bGwuCgpBbmQgYXBhcnQgZnJvbSB0aGF0IHdlIGNhbid0IGxpbWl0IHVzZXJwdHJzIHNpbmNlIHRo ZXkgYXJlIHBhcnQgb2YgdGhlIApVQVBJIGFuZCBWdWxrYW4vT3BlbkdMLgoKPj4gQXMgTWFhcnRl biB3cm90ZSB3aGVuIHlvdSB3YW50IHRvIGdvIGRvd24gdGhpcyByb3V0ZSB5b3UgbmVlZCBhCj4+ IGNvbXBsZXRlIHNlcGFyYXRlIG1lbW9yeSBtYW5hZ2VtZW50IHBhcmFsbGVsIHRvIHRoZSBvbmUg b2YgdGhlIGtlcm5lbC4KPiBOb3QgcmVhbGx5LiBJJ20gdHJ5aW5nIHRvIG1ha2UgdGhlIEdQVSBt ZW1vcnkgbWFuYWdlbWVudCBtb3JlIHNpbWlsYXIgdG8KPiB3aGF0IHRoZSBrZXJuZWwgZG9lcyBm b3Igc3lzdGVtIG1lbW9yeS4KPgo+IEkgdW5kZXJzdG9vZCBNYWFydGVuJ3MgY29tbWVudCBhcyAi SSdtIGNyZWF0aW5nIGEgbmV3IG1lbW9yeSBtYW5hZ2VyIGFuZAo+IG5vdCB1c2luZyBUVE0gYW55 IG1vcmUiLiBUaGlzIGlzIHRydWUuIFRoZSBpZGVhIGlzIHRoYXQgdGhpcyBwb3J0aW9uIG9mCj4g VlJBTSB3b3VsZCBiZSBtYW5hZ2VkIG1vcmUgbGlrZSBzeXN0ZW0gbWVtb3J5LgoKSSBkb24ndCB0 aGluayB0aGF0IHdpbGwgZmx5LiBXZSBjYW4gaGF2ZSB0aGUgYmFja2luZyBzdG9yZSB3aGljaCBU VE0gCnVzZXMgZm9yIGFsbG9jYXRpb24gc2hhcmVkIHdpdGggSE1NLgoKQnV0IGVzc2VudGlhbGx5 IGJvdGggVFRNIGFsbG9jYXRpb25zIG5lZWRzIHRvIGJlIGFibGUgdG8gcHV0IHByZXNzdXJlIG9u IApITU0gYWxsb2NhdGlvbnMgYXMgd2VsbCBhcyB0aGUgb3RoZXIgd2F5IGFyb3VuZC4KClJlZ2Fy ZHMsCkNocmlzdGlhbi4KCj4KPiBSZWdhcmRzLAo+ICDCoCBGZWxpeAo+Cj4KPj4gUmVnYXJkcywK Pj4gQ2hyaXN0aWFuLgo+Pgo+Pj4gIMKgIEl0IGRvZXNuJ3QgZGVwZW5kIG9uIGFueSBmZW5jZXMs IHNvCj4+PiBpdCBjYW5ub3QgZGVhZGxvY2sgd2l0aCBhbnkgR1BVIGRyaXZlci1tYW5hZ2VkIG1l bW9yeS4gVGhlIEdQVSBkcml2ZXIKPj4+IGdldHMgaW52b2x2ZWQgaW4gdGhlIE1NVSBub3RpZmll ciB0byBpbnZhbGlkYXRlIGRldmljZSBwYWdlIHRhYmxlcy4gQnV0Cj4+PiB0aGF0IGFsc28gZG9l c24ndCBuZWVkIHRvIHdhaXQgZm9yIGFueSBmZW5jZXMuCj4+Pgo+Pj4gQW5kIGlmIHRoZSBrZXJu ZWwgcnVucyBvdXQgb2YgcGFnZWFibGUgbWVtb3J5LCB5b3UncmUgaW4gdHJvdWJsZSBhbnl3YXku Cj4+PiBUaGUgT09NIGtpbGxlciB3aWxsIHN0ZXAgaW4sIG5vdGhpbmcgbmV3IHRoZXJlLgo+Pj4K Pj4+IFJlZ2FyZHMsCj4+PiAgwqDCoCBGZWxpeAo+Pj4KPj4+Cj4+Pj4+IEJ1dCBmcm9tIGEgZGVz aWduIHBvaW50IG9mIHZpZXcsIGRlZmluaXRlbHkgYSB2YWxpZCBzb2x1dGlvbi4KPj4+PiBJIHRo aW5rIHRoZSByZXN0cmljdGlvbiBhYm92ZSBtYWtlcyBpdCBwcmV0dHkgbXVjaCB1bnVzYWJsZS4K Pj4+Pgo+Pj4+PiBCdXQgdGhpcyBsb29rcyBnb29kLCB0aG9zZSBzb2x1dGlvbnMgYXJlIGRlZmlu aXRlbHkgdGhlIHZhbGlkCj4+Pj4+IG9wdGlvbnMgd2UKPj4+Pj4gY2FuIGNob29zZSBmcm9tLgo+ Pj4+IEl0J3MgY2VydGFpbmx5IHdvcnRoIG5vdGluZywgeWVzLiBBbmQganVzdCB0byBtYWtlIHN1 cmUgdGhhdCBub2JvZHkKPj4+PiBoYXMgdGhlIGlkZWEgdG8gcmVzZXJ2ZSBvbmx5IGRldmljZSBt ZW1vcnkuCj4+Pj4KPj4+PiBDaHJpc3RpYW4uCj4+Pj4KPj4+Pj4gfk1hYXJ0ZW4KPj4+Pj4KPj4+ IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+PiBMaW5h cm8tbW0tc2lnIG1haWxpbmcgbGlzdAo+Pj4gTGluYXJvLW1tLXNpZ0BsaXN0cy5saW5hcm8ub3Jn Cj4+PiBodHRwczovL25hbTExLnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9 aHR0cHMlM0ElMkYlMkZsaXN0cy5saW5hcm8ub3JnJTJGbWFpbG1hbiUyRmxpc3RpbmZvJTJGbGlu YXJvLW1tLXNpZyZhbXA7ZGF0YT0wNCU3QzAxJTdDZmVsaXgua3VlaGxpbmclNDBhbWQuY29tJTdD NGU0ODg0YmU1NWQ3NGM0ZGRhMTQwOGQ4YzM1ZmQwYWIlN0MzZGQ4OTYxZmU0ODg0ZTYwOGUxMWE4 MmQ5OTRlMTgzZCU3QzAlN0MwJTdDNjM3NDc0MTYzNTkyMjYwNTUyJTdDVW5rbm93biU3Q1RXRnBi R1pzYjNkOGV5SldJam9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFX d2lMQ0pYVkNJNk1uMCUzRCU3QzEwMDAmYW1wO3NkYXRhPWdRajUxZURLOE9VV29RY2JZbGlZNjM5 ak9QbGVSanlMWTNRMTZuajJQTDAlM0QmYW1wO3Jlc2VydmVkPTAKPj4+CgpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0 CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3Rv cC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8266C433DB for ; Thu, 28 Jan 2021 15:52:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6287264DEF for ; Thu, 28 Jan 2021 15:52:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231210AbhA1Pwe (ORCPT ); Thu, 28 Jan 2021 10:52:34 -0500 Received: from mail-dm6nam12on2067.outbound.protection.outlook.com ([40.107.243.67]:50593 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231204AbhA1Pwc (ORCPT ); Thu, 28 Jan 2021 10:52:32 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EnNDrftnqM8q+b/SIjNvsyXzpNoDB/sEe+Ctd31rSKSSdOpAAwjB6bWKuI5hzIKgQInPLI9ZK873pooaKAVaHVUW87TnyVARKfNM+1csBo0pXVYEtVa7l00WMoOpbvRQC7OuiwqchB9c95CPgo8qU3jdvaSBTMRyeVDL8ObKiPpDnL770xGFXGEnPxrPlvsy9nrjPC+pdi/G/b0SlkObPlkuj9kp9LeDFf7rlhOB0ZlzdZ47UuziQJoPL903k87mMHGdQmeiwu7mxiyqssV7OJXlSIEtxRXCuTbaU4mC0Zw2nXnYGKB+FzwjWc1uFiGTxi8HozmTTD8R8rYit6BQ6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6YGUdpB46iYQO/O9eu++i+D7oar6n4WR1pI/yUXhcw0=; b=nwAV5Exeo6vqXsapxKBoyzcLwXvuhE6z1RW7/g/KR6kBA5eme/4fCqHVOhT838kTbz3ioo8s0K4ucOQpMxBVBHEZqsxSHcw7sTrw36jXJvgrQ3xcUSS1mTANDEMHZmU96bHBHou7ydyplLyQlkLn3ArB1b+NjCh3DCIUuKzCN8iMTrKgM9CmRNsnSlKPDekywMjGxfKISny0bbI3jIyobDyWcTVzdTty+76g7efgp18Pkw/TBaeo/r7hj0/6AQkrabVUGjEJAofl0oqPCnPUb+7iKOq+dimD4l264ac4QGZ9/P/XntBerOFn0FDDZc2cBHT2OJ0AOJVV2cfJI+Y9+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6YGUdpB46iYQO/O9eu++i+D7oar6n4WR1pI/yUXhcw0=; b=x2G2I0ad//xZrFGJ8v6iwEzFiC3uQPOOQvg+a1chYUDvEwA9XV9nj2o978KQjNbzQ7nj443unbVqi+SswszhzLxutzJYRspAZRH4rgusiOexJno21iI3P8R9V2CABDLtSVP6gzHTwFK7Lid9bI7bXLOt3ufGUw+yxCvrNwltb1U= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) by MN2PR12MB4357.namprd12.prod.outlook.com (2603:10b6:208:262::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.13; Thu, 28 Jan 2021 15:47:03 +0000 Received: from MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::44f:9f01:ece7:f0e5]) by MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::44f:9f01:ece7:f0e5%3]) with mapi id 15.20.3784.019; Thu, 28 Jan 2021 15:47:03 +0000 Subject: Re: [Linaro-mm-sig] [PATCH] RFC: dma-fence: Document recoverable page fault implications To: Felix Kuehling , Maarten Lankhorst , Daniel Vetter , DRI Development Cc: linaro-mm-sig@lists.linaro.org, Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , Daniel Vetter , linux-media@vger.kernel.org References: <20210121194056.1734409-1-daniel.vetter@ffwll.ch> <6d373177-2645-1d67-9c14-dcad87c4f4d9@amd.com> <68740fcf-530e-b929-1c98-5810fc97ed23@linux.intel.com> <1e38efbc-ec52-e436-21e4-49a0d074b57b@amd.com> <18e7efbd-3d10-5ad1-49c9-7e26f0a27ef2@amd.com> <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Thu, 28 Jan 2021 16:46:55 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 In-Reply-To: <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: [2a02:908:1252:fb60:be8a:bd56:1f94:86e7] X-ClientProxiedBy: AM3PR04CA0144.eurprd04.prod.outlook.com (2603:10a6:207::28) To MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7] (2a02:908:1252:fb60:be8a:bd56:1f94:86e7) by AM3PR04CA0144.eurprd04.prod.outlook.com (2603:10a6:207::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Thu, 28 Jan 2021 15:47:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 45d0ccb9-e853-47f4-c44c-08d8c3a3f529 X-MS-TrafficTypeDiagnostic: MN2PR12MB4357: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qeoARvnhkew/Mql3yT8WA6X3ku3NdgUyvd/HUNviImUzJ0eLjDjKJpc4OWAqDL9OxwcSHTfRUbdT4Bf0l9GGegw13DanGrMwx1k9g72thHJNslpg7zTdy/EQpzEwWnQyGBNUzcCAUb8Yn8PdvPKA0Nbl+th1YF/FuP+e0hb5cv5ByUPJb1aOWaop6+4Ytvg5pb5g2VvrxJB55b1O7n9zMFQoVPf7kVLo7lHA7+qRcpaDcn7J/0GeEXrvfzifvAA5ebwjBPvm8OxDAI0oOT7Z1B0iVt/NyzRqNGBIEIrrYCfb++6djEcd9svWIOmU36HO7aHZANNFnSmOfIozxwwmYu2U5cCeQTAKy4EiPqO1by/VxLE/9FB4xwIwjvXFnxnTIUjoYc5dlA+k3UZT6cFWb1MKC1tVhHQ5adFANLd8dGEFiSx9P+sTyQf55AgsYbwWzO//V27KCT0jo/azPX7krPrUqSXKCUQBJFKNygpVLrZLSNlOxjWXEH+mZaDlLm6LDxEj8ji4P8lpyOth5q6Y+UPZZWUwlqmS86X2BsHHiYiNrEI4gpGkr+3Nq+Z4XNrABKDgLwwYqmudmx6T5p4foa0IvbBpwHFeFKFkjWNLS11wYeA7Vj0EeV58NDQHL+mSeVuaal4oaJVTDKSEDEPg93mPilL6JKk05cRIJTO6MHVbKZdhj8COH3GoXwOMVsMF X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB3775.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(376002)(396003)(366004)(136003)(39860400002)(346002)(2906002)(8676002)(86362001)(30864003)(45080400002)(54906003)(110136005)(4326008)(5660300002)(966005)(31696002)(478600001)(6666004)(52116002)(66574015)(2616005)(316002)(36756003)(6486002)(186003)(16526019)(66556008)(31686004)(8936002)(83380400001)(66476007)(66946007)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?RlYzOFI4WjhKZUF4elJSZUVGK1NpTWc1SUx4aGZDQ3IwYUt3YXVGTWRseVp5?= =?utf-8?B?VndkZ3BjN0d3TmxOZWp3MmJTbmtHaW9YbFRNZ2tHYjVWZmtsY1AycTdwcSt6?= =?utf-8?B?NlhJQmlwUU13VlJUUlZ5RjV1N2NSeEtDc1c3czZxV2hnN0Q5RGtORXkvbjdO?= =?utf-8?B?MTQvSm5UY1VtUmhHbVB4VjFleFVkSDVDOG40Z2VVd1NlcVVld25YU3JQN3E1?= =?utf-8?B?VWR3c1BpQTdYckpDa0JyY2lZRi9iTlJtM2lFUlp3TXAvQkdvTzR4VE8yeW40?= =?utf-8?B?UXBJNitsM0RFdWZKY3lueHYxQjB0RldmS3l4UkhlY2JIeFJ0WmpKODJVOU9v?= =?utf-8?B?NjVBZm9aMkw1MU5mR3RUR1o3NjA5bWlPdXY1OHlWSXlqUWFnNnppN21BLzNC?= =?utf-8?B?MXM0TlJyc2JsNGFpUlZTK1hqS0NWVlpiTDZNalRhd3RLRm41Rzl2d0MxK1Ir?= =?utf-8?B?WEdHa3VuNjNhcTRHeCtrRjA3NXhxRHZJeFZEbFdZODhuQWVsVllvMWtRYVJE?= =?utf-8?B?dVQyRVhsek55NHE0eGJZRk1vRUNLWmVGOS8vbEczMjYyRzYrRFdWQ29PbWMv?= =?utf-8?B?UGVidXZnM3dXOFBiQ0xqU1dmWVdRS0NXLzNXUGptQTl5L0RDdzhVYXBZVEQ1?= =?utf-8?B?eGxjTXZvUUs1ZHhJRm9ja01ZdlFSVUEzRVF6NXV4VjlIT01TcHZJbmtOS1p3?= =?utf-8?B?eUh6Y2s1WHVkYkROMGo5TlJKVVpRNlFOK1ZFVHl4ait0N3YzYUtVdU5MYXg5?= =?utf-8?B?SmFMc2JJRzZtcTRJVDVDNXBYR0dNS3FadDRKV01hRHp0R3hDTWxrNGNKY1Q5?= =?utf-8?B?VVN0SkZheG9PUXp3TVJ2V0FlN2NFTEdlNXl1cEg2YUpRWWhyUFBjcEdVSE1q?= =?utf-8?B?cmU2YjU3eXQzZE80NlkzV1pwQVpYS3VMMGNhL1AwWEhpZ1pjbUlUUm9qOTZZ?= =?utf-8?B?eVhBSjByV2prVlhYeGVGaHFKWVV6UGZTZThWWnhxRGEyYWZGYmNDQlN1bjIr?= =?utf-8?B?bkVHVFVKZmRqVGZpY0ZOUXZXY2tVLzNGbzI5Z1MrRVpXQUY3MGFVRW03Y3Nq?= =?utf-8?B?Tmhab2ZONFNJY3UwVWdQbXBRRkU1eHdUS1N1L2xGak1Sc0c3ZDJQeW96M2pn?= =?utf-8?B?cmxlTGlheG9QTDFrQnVOVm55WWcxc3RZbVNwUGJtcjRtTGlhWEdHM2w0bkZ3?= =?utf-8?B?VUtGc1lGVG14ajZIVUNpVWM0NVcycDBlWHY2T0pjYlZFWmpsekpLK3ZLVVBF?= =?utf-8?B?akRRRFVrOTJScGg4c1hLSU5iYXdmclBGeFJVem90aG5jTG53OEJkMXJaMSt5?= =?utf-8?B?K2h2Mk1MRjUvcXlJMXFqSytoaWZKZHF4L0oyREVtWHM1OEJWWnBOYVNzUk90?= =?utf-8?B?TlhvZ0JFWStndVpNbGVGTS80UkQ0VURpZm5JSkIzUGl6Z0FCaitMZ1pUb2F3?= =?utf-8?B?bGI1aVZjQTFRYUZpSXVidmxHVVFhdXd1eUh2bDlIejN0dEFHOFZRVGtiWU5u?= =?utf-8?Q?vbsBvo4hjLaXIq1OA0kjEczxxPo?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45d0ccb9-e853-47f4-c44c-08d8c3a3f529 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3775.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jan 2021 15:47:03.8432 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eOvPXibaq2dS2hWW/C5N3NAAsYb3qsnSkQNY4ZmRodGkcqSLsQXTKkTDkXqoR1yf X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4357 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Am 28.01.21 um 16:39 schrieb Felix Kuehling: > Am 2021-01-28 um 2:39 a.m. schrieb Christian König: >> Am 27.01.21 um 23:00 schrieb Felix Kuehling: >>> Am 2021-01-27 um 7:16 a.m. schrieb Christian König: >>>> Am 27.01.21 um 13:11 schrieb Maarten Lankhorst: >>>>> Op 27-01-2021 om 01:22 schreef Felix Kuehling: >>>>>> Am 2021-01-21 um 2:40 p.m. schrieb Daniel Vetter: >>>>>>> Recently there was a fairly long thread about recoreable hardware >>>>>>> page >>>>>>> faults, how they can deadlock, and what to do about that. >>>>>>> >>>>>>> While the discussion is still fresh I figured good time to try and >>>>>>> document the conclusions a bit. >>>>>>> >>>>>>> References: >>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fdri-devel%2F20210107030127.20393-1-Felix.Kuehling%40amd.com%2F&data=04%7C01%7Cfelix.kuehling%40amd.com%7C4e4884be55d74c4dda1408d8c35fd0ab%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637474163592260552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=y2VzC4vbfMi0ctyerAHfqODZ6tthz1FUDwpMCp0PIrQ%3D&reserved=0 >>>>>>> >>>>>>> Cc: Maarten Lankhorst >>>>>>> Cc: Thomas Hellström >>>>>>> Cc: "Christian König" >>>>>>> Cc: Jerome Glisse >>>>>>> Cc: Felix Kuehling >>>>>>> Signed-off-by: Daniel Vetter >>>>>>> Cc: Sumit Semwal >>>>>>> Cc: linux-media@vger.kernel.org >>>>>>> Cc: linaro-mm-sig@lists.linaro.org >>>>>>> -- >>>>>>> I'll be away next week, but figured I'll type this up quickly for >>>>>>> some >>>>>>> comments and to check whether I got this all roughly right. >>>>>>> >>>>>>> Critique very much wanted on this, so that we can make sure hw which >>>>>>> can't preempt (with pagefaults pending) like gfx10 has a clear >>>>>>> path to >>>>>>> support page faults in upstream. So anything I missed, got wrong or >>>>>>> like that would be good. >>>>>>> -Daniel >>>>>>> --- >>>>>>>    Documentation/driver-api/dma-buf.rst | 66 >>>>>>> ++++++++++++++++++++++++++++ >>>>>>>    1 file changed, 66 insertions(+) >>>>>>> >>>>>>> diff --git a/Documentation/driver-api/dma-buf.rst >>>>>>> b/Documentation/driver-api/dma-buf.rst >>>>>>> index a2133d69872c..e924c1e4f7a3 100644 >>>>>>> --- a/Documentation/driver-api/dma-buf.rst >>>>>>> +++ b/Documentation/driver-api/dma-buf.rst >>>>>>> @@ -257,3 +257,69 @@ fences in the kernel. This means: >>>>>>>      userspace is allowed to use userspace fencing or long running >>>>>>> compute >>>>>>>      workloads. This also means no implicit fencing for shared >>>>>>> buffers in these >>>>>>>      cases. >>>>>>> + >>>>>>> +Recoverable Hardware Page Faults Implications >>>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> + >>>>>>> +Modern hardware supports recoverable page faults, which has a >>>>>>> lot of >>>>>>> +implications for DMA fences. >>>>>>> + >>>>>>> +First, a pending page fault obviously holds up the work that's >>>>>>> running on the >>>>>>> +accelerator and a memory allocation is usually required to resolve >>>>>>> the fault. >>>>>>> +But memory allocations are not allowed to gate completion of DMA >>>>>>> fences, which >>>>>>> +means any workload using recoverable page faults cannot use DMA >>>>>>> fences for >>>>>>> +synchronization. Synchronization fences controlled by userspace >>>>>>> must be used >>>>>>> +instead. >>>>>>> + >>>>>>> +On GPUs this poses a problem, because current desktop compositor >>>>>>> protocols on >>>>>>> +Linus rely on DMA fences, which means without an entirely new >>>>>>> userspace stack >>>>>>> +built on top of userspace fences, they cannot benefit from >>>>>>> recoverable page >>>>>>> +faults. The exception is when page faults are only used as >>>>>>> migration hints and >>>>>>> +never to on-demand fill a memory request. For now this means >>>>>>> recoverable page >>>>>>> +faults on GPUs are limited to pure compute workloads. >>>>>>> + >>>>>>> +Furthermore GPUs usually have shared resources between the 3D >>>>>>> rendering and >>>>>>> +compute side, like compute units or command submission engines. If >>>>>>> both a 3D >>>>>>> +job with a DMA fence and a compute workload using recoverable page >>>>>>> faults are >>>>>>> +pending they could deadlock: >>>>>>> + >>>>>>> +- The 3D workload might need to wait for the compute job to finish >>>>>>> and release >>>>>>> +  hardware resources first. >>>>>>> + >>>>>>> +- The compute workload might be stuck in a page fault, because the >>>>>>> memory >>>>>>> +  allocation is waiting for the DMA fence of the 3D workload to >>>>>>> complete. >>>>>>> + >>>>>>> +There are a few ways to prevent this problem: >>>>>>> + >>>>>>> +- Compute workloads can always be preempted, even when a page >>>>>>> fault is pending >>>>>>> +  and not yet repaired. Not all hardware supports this. >>>>>>> + >>>>>>> +- DMA fence workloads and workloads which need page fault handling >>>>>>> have >>>>>>> +  independent hardware resources to guarantee forward progress. >>>>>>> This could be >>>>>>> +  achieved through e.g. through dedicated engines and minimal >>>>>>> compute unit >>>>>>> +  reservations for DMA fence workloads. >>>>>>> + >>>>>>> +- The reservation approach could be further refined by only >>>>>>> reserving the >>>>>>> +  hardware resources for DMA fence workloads when they are >>>>>>> in-flight. This must >>>>>>> +  cover the time from when the DMA fence is visible to other >>>>>>> threads up to >>>>>>> +  moment when fence is completed through dma_fence_signal(). >>>>>>> + >>>>>>> +- As a last resort, if the hardware provides no useful reservation >>>>>>> mechanics, >>>>>>> +  all workloads must be flushed from the GPU when switching >>>>>>> between jobs >>>>>>> +  requiring DMA fences or jobs requiring page fault handling: This >>>>>>> means all DMA >>>>>>> +  fences must complete before a compute job with page fault >>>>>>> handling can be >>>>>>> +  inserted into the scheduler queue. And vice versa, before a DMA >>>>>>> fence can be >>>>>>> +  made visible anywhere in the system, all compute workloads must >>>>>>> be preempted >>>>>>> +  to guarantee all pending GPU page faults are flushed. >>>>>> I thought of another possible workaround: >>>>>> >>>>>>     * Partition the memory. Servicing of page faults will use a >>>>>> separate >>>>>>       memory pool that can always be allocated from without >>>>>> waiting for >>>>>>       fences. This includes memory for page tables and memory for >>>>>>       migrating data to. You may steal memory from other processes >>>>>> that >>>>>>       can page fault, so no fence waiting is necessary. Being able to >>>>>>       steal memory at any time also means there are basically no >>>>>>       out-of-memory situations you need to worry about. Even page >>>>>> tables >>>>>>       (except the root page directory of each process) can be >>>>>> stolen in >>>>>>       the worst case. >>>>> I think 'overcommit' would be a nice way to describe this. But I'm not >>>>> sure how easy this is to implement in practice. You would basically >>>>> need >>>>> to create your own memory manager for this. >>>> Well you would need a completely separate pool for both device as well >>>> as system memory. >>>> >>>> E.g. on boot we say we steal X GB system memory only for HMM. >>> Why? The GPU driver doesn't need to allocate system memory for HMM. >>> Migrations to system memory are handled by the kernel's handle_mm_fault >>> and page allocator and swap logic. >> And that one depends on dma_fence completion because you can easily >> need to wait for an MMU notifier callback. > I see, the GFX MMU notifier for userpointers in amdgpu currently waits > for fences. For the KFD MMU notifier I am planning to fix this by > causing GPU page faults instead of preempting the queues. Can we limit > userptrs in amdgpu to engines that can page fault. Basically make it > illegal to attach userptr BOs to graphics CS BO lists, so they can only > be used in user mode command submissions, which can page fault. Then the > GFX MMU notifier could invalidate PTEs and would not have to wait for > fences. It's not only the MMU notifier, the TTM shrinker I'm adding needs to wait for dma_fences as well. And apart from that we can't limit userptrs since they are part of the UAPI and Vulkan/OpenGL. >> As Maarten wrote when you want to go down this route you need a >> complete separate memory management parallel to the one of the kernel. > Not really. I'm trying to make the GPU memory management more similar to > what the kernel does for system memory. > > I understood Maarten's comment as "I'm creating a new memory manager and > not using TTM any more". This is true. The idea is that this portion of > VRAM would be managed more like system memory. I don't think that will fly. We can have the backing store which TTM uses for allocation shared with HMM. But essentially both TTM allocations needs to be able to put pressure on HMM allocations as well as the other way around. Regards, Christian. > > Regards, >   Felix > > >> Regards, >> Christian. >> >>>   It doesn't depend on any fences, so >>> it cannot deadlock with any GPU driver-managed memory. The GPU driver >>> gets involved in the MMU notifier to invalidate device page tables. But >>> that also doesn't need to wait for any fences. >>> >>> And if the kernel runs out of pageable memory, you're in trouble anyway. >>> The OOM killer will step in, nothing new there. >>> >>> Regards, >>>    Felix >>> >>> >>>>> But from a design point of view, definitely a valid solution. >>>> I think the restriction above makes it pretty much unusable. >>>> >>>>> But this looks good, those solutions are definitely the valid >>>>> options we >>>>> can choose from. >>>> It's certainly worth noting, yes. And just to make sure that nobody >>>> has the idea to reserve only device memory. >>>> >>>> Christian. >>>> >>>>> ~Maarten >>>>> >>> _______________________________________________ >>> Linaro-mm-sig mailing list >>> Linaro-mm-sig@lists.linaro.org >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.linaro.org%2Fmailman%2Flistinfo%2Flinaro-mm-sig&data=04%7C01%7Cfelix.kuehling%40amd.com%7C4e4884be55d74c4dda1408d8c35fd0ab%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637474163592260552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gQj51eDK8OUWoQcbYliY639jOPleRjyLY3Q16nj2PL0%3D&reserved=0 >>>