From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D80BC433E0 for ; Tue, 9 Feb 2021 14:26:10 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9804A64E66 for ; Tue, 9 Feb 2021 14:26:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9804A64E66 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CCFFF6E061; Tue, 9 Feb 2021 14:26:07 +0000 (UTC) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2069.outbound.protection.outlook.com [40.107.244.69]) by gabe.freedesktop.org (Postfix) with ESMTPS id E3D2C6E07B for ; Tue, 9 Feb 2021 14:26:05 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iTmp17i4TX2FSm2RIztCXU+ktQlhfomWlsHPuY6Nkd64NCrkcnsWfLefCt+uXSVxJY/yutlpen61HAG58Cc/HaWtSfvgIXbxET99whGMaM1WTAbqcjhOAd+Xm16s5MTsyWsCZDXO6JT28hEQoEmXIGUGg8cuV7/e6K4kFfhqCx8NN4QRBz9r9+leJ3m1RPWCPBbaT8tglANG+phXQX9qAGvbGjKiHWwX4i/DYIEVQl18lnjnnG5F6gntE+1AgXOITxXB+6QvXkATbF2hsKZ+BpMrUCYZ76tsdPaKi9wfdHSAp3ukX+VFb/KmAaVjRqPDOTDDihFDQm0T8VNA9nQqWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pDvd/VnOcvi3KlTpZQflahw8ROalMpMj+jgJO559s3s=; b=FAI2nBq8LNauZB/8EtNQ/Pl2abf+4lE1Abr5nOSEXxlDFLOX07Rhyh63SAFCkJrUFEBdyOWATShUWWSbCv3lrp9cHNQZzQM1UUH/l4uLraVEMJlyCgNdDvdpkuUCyHSz14Lvqpm5MKH/zy1pH0TsCLBHqODWjQmSB4C/DFbFq0H0PGxFitz6E0jQXgtQocCEbpuRdSa3PSIMBi61YpnhmvTRegG3hZnmOGhTg4l+i1Mb/nJOVm11/wa5YwXEe7ESSMClSMeCDaLsJ2uYmensbMWmQp1RpbC5G01moAP/L1tk46dvoqQKAfau6LVKZ0UctAq8o8RovkJ4JcV9Bpn8UA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pDvd/VnOcvi3KlTpZQflahw8ROalMpMj+jgJO559s3s=; b=v6d+AYEg7AlDyJY9YqDSss8I0MYia1fFGcyPq/jmYOBUzSp/NBeT7xLKgEwriSUn5VZ+KaD110AhWvm5QFezXiWMZJ96SpEAXZRUfEmspmOaLeGWqbj9VCyCbWVR33CEbH0byoLTDP5ScmN4FxKqtGG+gIYcnXA/xkgnoaC8StQ= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4657.namprd12.prod.outlook.com (2603:10b6:207:1a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.23; Tue, 9 Feb 2021 14:26:00 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::ec8d:851e:525d:a6ab]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::ec8d:851e:525d:a6ab%8]) with mapi id 15.20.3784.022; Tue, 9 Feb 2021 14:26:00 +0000 Subject: Re: [Linaro-mm-sig] [PATCH] RFC: dma-fence: Document recoverable page fault implications To: Daniel Vetter References: <20210121194056.1734409-1-daniel.vetter@ffwll.ch> <6d373177-2645-1d67-9c14-dcad87c4f4d9@amd.com> <68740fcf-530e-b929-1c98-5810fc97ed23@linux.intel.com> <1e38efbc-ec52-e436-21e4-49a0d074b57b@amd.com> <18e7efbd-3d10-5ad1-49c9-7e26f0a27ef2@amd.com> <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> From: Felix Kuehling Message-ID: Date: Tue, 9 Feb 2021 09:25:58 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 In-Reply-To: Content-Language: en-US X-Originating-IP: [142.117.121.176] X-ClientProxiedBy: YTOPR0101CA0012.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::25) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.100] (142.117.121.176) by YTOPR0101CA0012.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17 via Frontend Transport; Tue, 9 Feb 2021 14:25:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 126342fd-2a2e-4e52-f9ff-08d8cd069f76 X-MS-TrafficTypeDiagnostic: BL0PR12MB4657: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2fwyosBrYQ+9XrH5RAkh3rpQ9wRK9Ac+lLdXOVN5auOxEJSQX7YSCgkYiRcWOpfOoVTMr+HXLd9OjpdGr4cr9KIOm5/LHy68CTMTLJ13hinNR9dOf5jounFpzsCpOtIExPBaUu+0TNrwwkl2M+V0aS/EYN2sFl7kkRhSGBoNhHzJzLN11keOXO06+mYfIx02UGJjWasOopyCYoFQFPEJ0CGojzbrpmWYecAylZPqT9P6QRsum1cHYy3zAYI/tTsOGMEZ7V5U8pu7i3ENCgo8GJB1kL6m4BVzGZr5IOh4ba+reAWjrUacIsTp2gS5ZhhMqDLTU+nCzOe1RToetRpC6GvWVsSUaA6RKQvg/Rez2y2fy2IG+K1WPkDjjAGXNE7+CPof2nNhmQoL35B8xBSyQfj5WE4EgF66U2bMyyT2bEj0bk3yfZ6DVeAATQrlBGk5UhPoIGEBEkHyRAOzEwPjefVDJV/g7TWVkBujVzsqZ9LWOkxsLIOjNal2jjhZU6014ybU84dlPgV3yMQ3P7kjpo3fzHgPXdtdQSjbDFvmNmIOzSE5c5k82f/OGXWGQ8ZBXAnxcIU5U9r+viRAzzE20iMPQkfwucUHMZbEsvQ9infbi6WRtWp44DTu3MU/8nw6CxYgoA1A4DUhvYuEyhepw+Y7VpVnn9lB/dwUPzf3WGXc88Q+evELFJbXJLlFCnOz X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(396003)(136003)(346002)(366004)(376002)(52116002)(4326008)(54906003)(31696002)(36756003)(6916009)(6486002)(30864003)(31686004)(8936002)(956004)(83380400001)(2906002)(44832011)(26005)(966005)(8676002)(86362001)(5660300002)(186003)(16526019)(2616005)(316002)(66476007)(478600001)(53546011)(16576012)(66556008)(66574015)(66946007)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?dFhiblBuWHdKVWdYQ1hQVlFsc1lyeXhpNDYyajNlRlBmQ0orVDhITzBRbGpD?= =?utf-8?B?YTBwNGF5eWV5V2ZDZ2tSUXZXdHAzeERNOFd5L0JoM2czOWZBMmJHanRpUXZo?= =?utf-8?B?WGF1Y3Ywa3lMZWdlUzk1Vm03OXZuM0NiQXlBa1JxVi9ZSklaMGVZdTdSQi8r?= =?utf-8?B?OHdlOXdmQ0hYU1BpM0srdkF5ZU5QSFRQMjFpYzkvVzFsRHhqdDlkS0YydlNP?= =?utf-8?B?bjBnZmF2UjBQaXNMbzF4VFBhY1lxY01Ea2dBRlJWTzNieXhLTXNHZGZ3LzRw?= =?utf-8?B?MWNZWU1LdVpJWlFla1FveUNJQmZlQ1NXVkppRVo3c1ZQMiswQzliTUVFRVpl?= =?utf-8?B?MFZJMUNDMDBsRnVoTmM1Y1dXRmQyOHNEZlErWGg3WFppSUZPbDZuMUNLQ1Ra?= =?utf-8?B?emJjT2E3bTVuRnhhRUxselZST1FoM1dlYWY2QWxtbUxWUTJ0UDhBQ0NETm1m?= =?utf-8?B?alU3SURZNk1pZk9WUm9CREhQVVd1Sk9CaE5hSStrZXZZNjNRQXRyMzRTMmp0?= =?utf-8?B?WUVSdlVrUFFhbzlCSCtuMzdvMHRRanpTMlMxd1c3T0Vqd1U0ZFNOVVEzVzZ1?= =?utf-8?B?d28wREFSY0tRd0ZTNzNXdkNWbTVUMmUvVDdOa3ZjdGlVa21BdkQxQ1BJc1py?= =?utf-8?B?SzltaUErNWZsKzVCVVdtTjBxaTI2ZDNrV25pSEpwSHFGdzNuQXdPa21QeWFU?= =?utf-8?B?bU8zamxTOWJ6OTd5MWdoTW1SVUNqTzN6bkI4bm1FbUlQdUpJaDhBODBnQWNw?= =?utf-8?B?WmplSGRQZzhOVVdHcWd6ZGlXYUVRdjFEd2s0RGdORTYrbHVEbUpmT1VOay9i?= =?utf-8?B?RU9yVzlrRno5OGZadVdrRHhBOE5yRVl5WVkxSXNhYmxzRy82WmgzT3NXaHZp?= =?utf-8?B?a041MEcrNmtuenN3enI5WXFCcWlNTTJLRjlIRENYRS83QXpOZ05iYnk5QSth?= =?utf-8?B?TXkxTUl0V2wxTjlRS2c2c3dMVGhpb2NIRUtDeFR5U0xIOVRVdzdzTTZwS0ZC?= =?utf-8?B?cGJOeC9HRm5WRlFlTkN2L0MxZlpqWHNzRVhOcks0c0JXaFJKM01zdkh3MDYy?= =?utf-8?B?L0RxWVlxcXYrOG1EMGg1SWE3NjY5ZC9UT21HMmhuM045NWdEanhRU3ZSbkNJ?= =?utf-8?B?bTVkRVYyTzBJMjFFMWQ2ZnV6UDZNek1MTHZKUUtwbDV5eWNXN3pEeHBYOFN6?= =?utf-8?B?NDFBZzEra1FzalR5Wk1nckt1c0wzTWlqcDJkbi9ROHEwTkFVa05kbTkvdlBK?= =?utf-8?B?QjJmc3BwNGtMZnZpS3NWYjIyMk5xdldwa0E5VllHdEZsblExdzZpemNLcXk3?= =?utf-8?B?T1VwbmozRm1pQlcrRDh4NWgxb0VZNEZpVVpjZ1BrV05oMUhzTERtRWFIbWtp?= =?utf-8?B?aEoyL0d4bXg4RVVsNnpSWjJaOU9KZHkzcnEzb1FvOG9tM2s3OVdBSVZwYmow?= =?utf-8?B?SDBsVVBDOS9idlFMYms4d1dtYllXQkY1NStKaDBtY2F1T3NJWEZucDh4RFYz?= =?utf-8?B?bnhJejE3TGE2bnh4eHgvb1MvL1orbm5xQzFQdGMxOFJVd0FzS3FmL21WNyt6?= =?utf-8?B?NlNOS0hJekhFMDhYbWQ4Q1dMRjZ1cHNSSFBUWFNQb3ZyOTBHbUFZUVFrRCtU?= =?utf-8?B?dnVPOFBBbDUzNFE0eW1XT1FOVC9hekRxNCtQZlBHQzB6V0d0VXo3dDhWRi9l?= =?utf-8?B?dlVkZ0hKOThSTWI2SGpGVlpQUzVSZy8yQndCUTNGOTlhRkVNTk1kUHVvaXI3?= =?utf-8?B?T0tPaGhuL2llbGd2UGpZMGVSMks2bkVPZFNudFJTcyt3UVpqeUpSRzJWSDRJ?= =?utf-8?B?dmJCdnJQZTJNdkJFMWpZUT09?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 126342fd-2a2e-4e52-f9ff-08d8cd069f76 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2021 14:26:00.3470 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: owZB781tZboWXXI6iHxusvy/Y7WuT5q6FUnpL6+CZ3/RPqZuxg6GULCIpRrrSKyZ6fq9yelyCz90KDTLEU3VXQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4657 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: DRI Development , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , Daniel Vetter , "Koenig, Christian" , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" QW0gMjAyMS0wMi0wOSB1bSA5OjA4IGEubS4gc2NocmllYiBEYW5pZWwgVmV0dGVyOgo+IE9uIFR1 ZSwgRmViIDksIDIwMjEgYXQgMTI6MTUgUE0gRmVsaXggS3VlaGxpbmcgPGZlbGl4Lmt1ZWhsaW5n QGFtZC5jb20+IHdyb3RlOgo+PiBBbSAyMDIxLTAyLTA5IHVtIDE6MzcgYS5tLiBzY2hyaWViIERh bmllbCBWZXR0ZXI6Cj4+PiBPbiBUdWUsIEZlYiA5LCAyMDIxIGF0IDQ6MTMgQU0gQmFzIE5pZXV3 ZW5odWl6ZW4KPj4+IDxiYXNAYmFzbmlldXdlbmh1aXplbi5ubD4gd3JvdGU6Cj4+Pj4gT24gVGh1 LCBKYW4gMjgsIDIwMjEgYXQgNDo0MCBQTSBGZWxpeCBLdWVobGluZyA8ZmVsaXgua3VlaGxpbmdA YW1kLmNvbT4gd3JvdGU6Cj4+Pj4+IEFtIDIwMjEtMDEtMjggdW0gMjozOSBhLm0uIHNjaHJpZWIg Q2hyaXN0aWFuIEvDtm5pZzoKPj4+Pj4+IEFtIDI3LjAxLjIxIHVtIDIzOjAwIHNjaHJpZWIgRmVs aXggS3VlaGxpbmc6Cj4+Pj4+Pj4gQW0gMjAyMS0wMS0yNyB1bSA3OjE2IGEubS4gc2NocmllYiBD aHJpc3RpYW4gS8O2bmlnOgo+Pj4+Pj4+PiBBbSAyNy4wMS4yMSB1bSAxMzoxMSBzY2hyaWViIE1h YXJ0ZW4gTGFua2hvcnN0Ogo+Pj4+Pj4+Pj4gT3AgMjctMDEtMjAyMSBvbSAwMToyMiBzY2hyZWVm IEZlbGl4IEt1ZWhsaW5nOgo+Pj4+Pj4+Pj4+IEFtIDIwMjEtMDEtMjEgdW0gMjo0MCBwLm0uIHNj aHJpZWIgRGFuaWVsIFZldHRlcjoKPj4+Pj4+Pj4+Pj4gUmVjZW50bHkgdGhlcmUgd2FzIGEgZmFp cmx5IGxvbmcgdGhyZWFkIGFib3V0IHJlY29yZWFibGUgaGFyZHdhcmUKPj4+Pj4+Pj4+Pj4gcGFn ZQo+Pj4+Pj4+Pj4+PiBmYXVsdHMsIGhvdyB0aGV5IGNhbiBkZWFkbG9jaywgYW5kIHdoYXQgdG8g ZG8gYWJvdXQgdGhhdC4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gV2hpbGUgdGhlIGRpc2N1c3Np b24gaXMgc3RpbGwgZnJlc2ggSSBmaWd1cmVkIGdvb2QgdGltZSB0byB0cnkgYW5kCj4+Pj4+Pj4+ Pj4+IGRvY3VtZW50IHRoZSBjb25jbHVzaW9ucyBhIGJpdC4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+ Pj4gUmVmZXJlbmNlczoKPj4+Pj4+Pj4+Pj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvZHJpLWRl dmVsLzIwMjEwMTA3MDMwMTI3LjIwMzkzLTEtRmVsaXguS3VlaGxpbmdAYW1kLmNvbS8KPj4+Pj4+ Pj4+Pj4KPj4+Pj4+Pj4+Pj4gQ2M6IE1hYXJ0ZW4gTGFua2hvcnN0IDxtYWFydGVuLmxhbmtob3Jz dEBsaW51eC5pbnRlbC5jb20+Cj4+Pj4+Pj4+Pj4+IENjOiBUaG9tYXMgSGVsbHN0csO2bSA8dGhv bWFzLmhlbGxzdHJvbUBpbnRlbC5jb20+Cj4+Pj4+Pj4+Pj4+IENjOiAiQ2hyaXN0aWFuIEvDtm5p ZyIgPGNocmlzdGlhbi5rb2VuaWdAYW1kLmNvbT4KPj4+Pj4+Pj4+Pj4gQ2M6IEplcm9tZSBHbGlz c2UgPGpnbGlzc2VAcmVkaGF0LmNvbT4KPj4+Pj4+Pj4+Pj4gQ2M6IEZlbGl4IEt1ZWhsaW5nIDxm ZWxpeC5rdWVobGluZ0BhbWQuY29tPgo+Pj4+Pj4+Pj4+PiBTaWduZWQtb2ZmLWJ5OiBEYW5pZWwg VmV0dGVyIDxkYW5pZWwudmV0dGVyQGludGVsLmNvbT4KPj4+Pj4+Pj4+Pj4gQ2M6IFN1bWl0IFNl bXdhbCA8c3VtaXQuc2Vtd2FsQGxpbmFyby5vcmc+Cj4+Pj4+Pj4+Pj4+IENjOiBsaW51eC1tZWRp YUB2Z2VyLmtlcm5lbC5vcmcKPj4+Pj4+Pj4+Pj4gQ2M6IGxpbmFyby1tbS1zaWdAbGlzdHMubGlu YXJvLm9yZwo+Pj4+Pj4+Pj4+PiAtLQo+Pj4+Pj4+Pj4+PiBJJ2xsIGJlIGF3YXkgbmV4dCB3ZWVr LCBidXQgZmlndXJlZCBJJ2xsIHR5cGUgdGhpcyB1cCBxdWlja2x5IGZvcgo+Pj4+Pj4+Pj4+PiBz b21lCj4+Pj4+Pj4+Pj4+IGNvbW1lbnRzIGFuZCB0byBjaGVjayB3aGV0aGVyIEkgZ290IHRoaXMg YWxsIHJvdWdobHkgcmlnaHQuCj4+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+IENyaXRpcXVlIHZlcnkg bXVjaCB3YW50ZWQgb24gdGhpcywgc28gdGhhdCB3ZSBjYW4gbWFrZSBzdXJlIGh3IHdoaWNoCj4+ Pj4+Pj4+Pj4+IGNhbid0IHByZWVtcHQgKHdpdGggcGFnZWZhdWx0cyBwZW5kaW5nKSBsaWtlIGdm eDEwIGhhcyBhIGNsZWFyCj4+Pj4+Pj4+Pj4+IHBhdGggdG8KPj4+Pj4+Pj4+Pj4gc3VwcG9ydCBw YWdlIGZhdWx0cyBpbiB1cHN0cmVhbS4gU28gYW55dGhpbmcgSSBtaXNzZWQsIGdvdCB3cm9uZyBv cgo+Pj4+Pj4+Pj4+PiBsaWtlIHRoYXQgd291bGQgYmUgZ29vZC4KPj4+Pj4+Pj4+Pj4gLURhbmll bAo+Pj4+Pj4+Pj4+PiAtLS0KPj4+Pj4+Pj4+Pj4gICAgRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBp L2RtYS1idWYucnN0IHwgNjYKPj4+Pj4+Pj4+Pj4gKysrKysrKysrKysrKysrKysrKysrKysrKysr Kwo+Pj4+Pj4+Pj4+PiAgICAxIGZpbGUgY2hhbmdlZCwgNjYgaW5zZXJ0aW9ucygrKQo+Pj4+Pj4+ Pj4+Pgo+Pj4+Pj4+Pj4+PiBkaWZmIC0tZ2l0IGEvRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2Rt YS1idWYucnN0Cj4+Pj4+Pj4+Pj4+IGIvRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2RtYS1idWYu cnN0Cj4+Pj4+Pj4+Pj4+IGluZGV4IGEyMTMzZDY5ODcyYy4uZTkyNGMxZTRmN2EzIDEwMDY0NAo+ Pj4+Pj4+Pj4+PiAtLS0gYS9Eb2N1bWVudGF0aW9uL2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QKPj4+ Pj4+Pj4+Pj4gKysrIGIvRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2RtYS1idWYucnN0Cj4+Pj4+ Pj4+Pj4+IEBAIC0yNTcsMyArMjU3LDY5IEBAIGZlbmNlcyBpbiB0aGUga2VybmVsLiBUaGlzIG1l YW5zOgo+Pj4+Pj4+Pj4+PiAgICAgIHVzZXJzcGFjZSBpcyBhbGxvd2VkIHRvIHVzZSB1c2Vyc3Bh Y2UgZmVuY2luZyBvciBsb25nIHJ1bm5pbmcKPj4+Pj4+Pj4+Pj4gY29tcHV0ZQo+Pj4+Pj4+Pj4+ PiAgICAgIHdvcmtsb2Fkcy4gVGhpcyBhbHNvIG1lYW5zIG5vIGltcGxpY2l0IGZlbmNpbmcgZm9y IHNoYXJlZAo+Pj4+Pj4+Pj4+PiBidWZmZXJzIGluIHRoZXNlCj4+Pj4+Pj4+Pj4+ICAgICAgY2Fz ZXMuCj4+Pj4+Pj4+Pj4+ICsKPj4+Pj4+Pj4+Pj4gK1JlY292ZXJhYmxlIEhhcmR3YXJlIFBhZ2Ug RmF1bHRzIEltcGxpY2F0aW9ucwo+Pj4+Pj4+Pj4+PiArfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+ fn5+fn5+fn5+fn5+fn5+fn5+fn5+Cj4+Pj4+Pj4+Pj4+ICsKPj4+Pj4+Pj4+Pj4gK01vZGVybiBo YXJkd2FyZSBzdXBwb3J0cyByZWNvdmVyYWJsZSBwYWdlIGZhdWx0cywgd2hpY2ggaGFzIGEKPj4+ Pj4+Pj4+Pj4gbG90IG9mCj4+Pj4+Pj4+Pj4+ICtpbXBsaWNhdGlvbnMgZm9yIERNQSBmZW5jZXMu Cj4+Pj4+Pj4+Pj4+ICsKPj4+Pj4+Pj4+Pj4gK0ZpcnN0LCBhIHBlbmRpbmcgcGFnZSBmYXVsdCBv YnZpb3VzbHkgaG9sZHMgdXAgdGhlIHdvcmsgdGhhdCdzCj4+Pj4+Pj4+Pj4+IHJ1bm5pbmcgb24g dGhlCj4+Pj4+Pj4+Pj4+ICthY2NlbGVyYXRvciBhbmQgYSBtZW1vcnkgYWxsb2NhdGlvbiBpcyB1 c3VhbGx5IHJlcXVpcmVkIHRvIHJlc29sdmUKPj4+Pj4+Pj4+Pj4gdGhlIGZhdWx0Lgo+Pj4+Pj4+ Pj4+PiArQnV0IG1lbW9yeSBhbGxvY2F0aW9ucyBhcmUgbm90IGFsbG93ZWQgdG8gZ2F0ZSBjb21w bGV0aW9uIG9mIERNQQo+Pj4+Pj4+Pj4+PiBmZW5jZXMsIHdoaWNoCj4+Pj4+Pj4+Pj4+ICttZWFu cyBhbnkgd29ya2xvYWQgdXNpbmcgcmVjb3ZlcmFibGUgcGFnZSBmYXVsdHMgY2Fubm90IHVzZSBE TUEKPj4+Pj4+Pj4+Pj4gZmVuY2VzIGZvcgo+Pj4+Pj4+Pj4+PiArc3luY2hyb25pemF0aW9uLiBT eW5jaHJvbml6YXRpb24gZmVuY2VzIGNvbnRyb2xsZWQgYnkgdXNlcnNwYWNlCj4+Pj4+Pj4+Pj4+ IG11c3QgYmUgdXNlZAo+Pj4+Pj4+Pj4+PiAraW5zdGVhZC4KPj4+Pj4+Pj4+Pj4gKwo+Pj4+Pj4+ Pj4+PiArT24gR1BVcyB0aGlzIHBvc2VzIGEgcHJvYmxlbSwgYmVjYXVzZSBjdXJyZW50IGRlc2t0 b3AgY29tcG9zaXRvcgo+Pj4+Pj4+Pj4+PiBwcm90b2NvbHMgb24KPj4+Pj4+Pj4+Pj4gK0xpbnVz IHJlbHkgb24gRE1BIGZlbmNlcywgd2hpY2ggbWVhbnMgd2l0aG91dCBhbiBlbnRpcmVseSBuZXcK Pj4+Pj4+Pj4+Pj4gdXNlcnNwYWNlIHN0YWNrCj4+Pj4+Pj4+Pj4+ICtidWlsdCBvbiB0b3Agb2Yg dXNlcnNwYWNlIGZlbmNlcywgdGhleSBjYW5ub3QgYmVuZWZpdCBmcm9tCj4+Pj4+Pj4+Pj4+IHJl Y292ZXJhYmxlIHBhZ2UKPj4+Pj4+Pj4+Pj4gK2ZhdWx0cy4gVGhlIGV4Y2VwdGlvbiBpcyB3aGVu IHBhZ2UgZmF1bHRzIGFyZSBvbmx5IHVzZWQgYXMKPj4+Pj4+Pj4+Pj4gbWlncmF0aW9uIGhpbnRz IGFuZAo+Pj4+Pj4+Pj4+PiArbmV2ZXIgdG8gb24tZGVtYW5kIGZpbGwgYSBtZW1vcnkgcmVxdWVz dC4gRm9yIG5vdyB0aGlzIG1lYW5zCj4+Pj4+Pj4+Pj4+IHJlY292ZXJhYmxlIHBhZ2UKPj4+Pj4+ Pj4+Pj4gK2ZhdWx0cyBvbiBHUFVzIGFyZSBsaW1pdGVkIHRvIHB1cmUgY29tcHV0ZSB3b3JrbG9h ZHMuCj4+Pj4+Pj4+Pj4+ICsKPj4+Pj4+Pj4+Pj4gK0Z1cnRoZXJtb3JlIEdQVXMgdXN1YWxseSBo YXZlIHNoYXJlZCByZXNvdXJjZXMgYmV0d2VlbiB0aGUgM0QKPj4+Pj4+Pj4+Pj4gcmVuZGVyaW5n IGFuZAo+Pj4+Pj4+Pj4+PiArY29tcHV0ZSBzaWRlLCBsaWtlIGNvbXB1dGUgdW5pdHMgb3IgY29t bWFuZCBzdWJtaXNzaW9uIGVuZ2luZXMuIElmCj4+Pj4+Pj4+Pj4+IGJvdGggYSAzRAo+Pj4+Pj4+ Pj4+PiAram9iIHdpdGggYSBETUEgZmVuY2UgYW5kIGEgY29tcHV0ZSB3b3JrbG9hZCB1c2luZyBy ZWNvdmVyYWJsZSBwYWdlCj4+Pj4+Pj4+Pj4+IGZhdWx0cyBhcmUKPj4+Pj4+Pj4+Pj4gK3BlbmRp bmcgdGhleSBjb3VsZCBkZWFkbG9jazoKPj4+Pj4+Pj4+Pj4gKwo+Pj4+Pj4+Pj4+PiArLSBUaGUg M0Qgd29ya2xvYWQgbWlnaHQgbmVlZCB0byB3YWl0IGZvciB0aGUgY29tcHV0ZSBqb2IgdG8gZmlu aXNoCj4+Pj4+Pj4+Pj4+IGFuZCByZWxlYXNlCj4+Pj4+Pj4+Pj4+ICsgIGhhcmR3YXJlIHJlc291 cmNlcyBmaXJzdC4KPj4+Pj4+Pj4+Pj4gKwo+Pj4+Pj4+Pj4+PiArLSBUaGUgY29tcHV0ZSB3b3Jr bG9hZCBtaWdodCBiZSBzdHVjayBpbiBhIHBhZ2UgZmF1bHQsIGJlY2F1c2UgdGhlCj4+Pj4+Pj4+ Pj4+IG1lbW9yeQo+Pj4+Pj4+Pj4+PiArICBhbGxvY2F0aW9uIGlzIHdhaXRpbmcgZm9yIHRoZSBE TUEgZmVuY2Ugb2YgdGhlIDNEIHdvcmtsb2FkIHRvCj4+Pj4+Pj4+Pj4+IGNvbXBsZXRlLgo+Pj4+ Pj4+Pj4+PiArCj4+Pj4+Pj4+Pj4+ICtUaGVyZSBhcmUgYSBmZXcgd2F5cyB0byBwcmV2ZW50IHRo aXMgcHJvYmxlbToKPj4+Pj4+Pj4+Pj4gKwo+Pj4+Pj4+Pj4+PiArLSBDb21wdXRlIHdvcmtsb2Fk cyBjYW4gYWx3YXlzIGJlIHByZWVtcHRlZCwgZXZlbiB3aGVuIGEgcGFnZQo+Pj4+Pj4+Pj4+PiBm YXVsdCBpcyBwZW5kaW5nCj4+Pj4+Pj4+Pj4+ICsgIGFuZCBub3QgeWV0IHJlcGFpcmVkLiBOb3Qg YWxsIGhhcmR3YXJlIHN1cHBvcnRzIHRoaXMuCj4+Pj4+Pj4+Pj4+ICsKPj4+Pj4+Pj4+Pj4gKy0g RE1BIGZlbmNlIHdvcmtsb2FkcyBhbmQgd29ya2xvYWRzIHdoaWNoIG5lZWQgcGFnZSBmYXVsdCBo YW5kbGluZwo+Pj4+Pj4+Pj4+PiBoYXZlCj4+Pj4+Pj4+Pj4+ICsgIGluZGVwZW5kZW50IGhhcmR3 YXJlIHJlc291cmNlcyB0byBndWFyYW50ZWUgZm9yd2FyZCBwcm9ncmVzcy4KPj4+Pj4+Pj4+Pj4g VGhpcyBjb3VsZCBiZQo+Pj4+Pj4+Pj4+PiArICBhY2hpZXZlZCB0aHJvdWdoIGUuZy4gdGhyb3Vn aCBkZWRpY2F0ZWQgZW5naW5lcyBhbmQgbWluaW1hbAo+Pj4+Pj4+Pj4+PiBjb21wdXRlIHVuaXQK Pj4+Pj4+Pj4+Pj4gKyAgcmVzZXJ2YXRpb25zIGZvciBETUEgZmVuY2Ugd29ya2xvYWRzLgo+Pj4+ Pj4+Pj4+PiArCj4+Pj4+Pj4+Pj4+ICstIFRoZSByZXNlcnZhdGlvbiBhcHByb2FjaCBjb3VsZCBi ZSBmdXJ0aGVyIHJlZmluZWQgYnkgb25seQo+Pj4+Pj4+Pj4+PiByZXNlcnZpbmcgdGhlCj4+Pj4+ Pj4+Pj4+ICsgIGhhcmR3YXJlIHJlc291cmNlcyBmb3IgRE1BIGZlbmNlIHdvcmtsb2FkcyB3aGVu IHRoZXkgYXJlCj4+Pj4+Pj4+Pj4+IGluLWZsaWdodC4gVGhpcyBtdXN0Cj4+Pj4+Pj4+Pj4+ICsg IGNvdmVyIHRoZSB0aW1lIGZyb20gd2hlbiB0aGUgRE1BIGZlbmNlIGlzIHZpc2libGUgdG8gb3Ro ZXIKPj4+Pj4+Pj4+Pj4gdGhyZWFkcyB1cCB0bwo+Pj4+Pj4+Pj4+PiArICBtb21lbnQgd2hlbiBm ZW5jZSBpcyBjb21wbGV0ZWQgdGhyb3VnaCBkbWFfZmVuY2Vfc2lnbmFsKCkuCj4+Pj4+Pj4+Pj4+ ICsKPj4+Pj4+Pj4+Pj4gKy0gQXMgYSBsYXN0IHJlc29ydCwgaWYgdGhlIGhhcmR3YXJlIHByb3Zp ZGVzIG5vIHVzZWZ1bCByZXNlcnZhdGlvbgo+Pj4+Pj4+Pj4+PiBtZWNoYW5pY3MsCj4+Pj4+Pj4+ Pj4+ICsgIGFsbCB3b3JrbG9hZHMgbXVzdCBiZSBmbHVzaGVkIGZyb20gdGhlIEdQVSB3aGVuIHN3 aXRjaGluZwo+Pj4+Pj4+Pj4+PiBiZXR3ZWVuIGpvYnMKPj4+Pj4+Pj4+Pj4gKyAgcmVxdWlyaW5n IERNQSBmZW5jZXMgb3Igam9icyByZXF1aXJpbmcgcGFnZSBmYXVsdCBoYW5kbGluZzogVGhpcwo+ Pj4+Pj4+Pj4+PiBtZWFucyBhbGwgRE1BCj4+Pj4+Pj4+Pj4+ICsgIGZlbmNlcyBtdXN0IGNvbXBs ZXRlIGJlZm9yZSBhIGNvbXB1dGUgam9iIHdpdGggcGFnZSBmYXVsdAo+Pj4+Pj4+Pj4+PiBoYW5k bGluZyBjYW4gYmUKPj4+Pj4+Pj4+Pj4gKyAgaW5zZXJ0ZWQgaW50byB0aGUgc2NoZWR1bGVyIHF1 ZXVlLiBBbmQgdmljZSB2ZXJzYSwgYmVmb3JlIGEgRE1BCj4+Pj4+Pj4+Pj4+IGZlbmNlIGNhbiBi ZQo+Pj4+Pj4+Pj4+PiArICBtYWRlIHZpc2libGUgYW55d2hlcmUgaW4gdGhlIHN5c3RlbSwgYWxs IGNvbXB1dGUgd29ya2xvYWRzIG11c3QKPj4+Pj4+Pj4+Pj4gYmUgcHJlZW1wdGVkCj4+Pj4+Pj4+ Pj4+ICsgIHRvIGd1YXJhbnRlZSBhbGwgcGVuZGluZyBHUFUgcGFnZSBmYXVsdHMgYXJlIGZsdXNo ZWQuCj4+Pj4+Pj4+Pj4gSSB0aG91Z2h0IG9mIGFub3RoZXIgcG9zc2libGUgd29ya2Fyb3VuZDoK Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+ICAgICAqIFBhcnRpdGlvbiB0aGUgbWVtb3J5LiBTZXJ2aWNp bmcgb2YgcGFnZSBmYXVsdHMgd2lsbCB1c2UgYQo+Pj4+Pj4+Pj4+IHNlcGFyYXRlCj4+Pj4+Pj4+ Pj4gICAgICAgbWVtb3J5IHBvb2wgdGhhdCBjYW4gYWx3YXlzIGJlIGFsbG9jYXRlZCBmcm9tIHdp dGhvdXQKPj4+Pj4+Pj4+PiB3YWl0aW5nIGZvcgo+Pj4+Pj4+Pj4+ICAgICAgIGZlbmNlcy4gVGhp cyBpbmNsdWRlcyBtZW1vcnkgZm9yIHBhZ2UgdGFibGVzIGFuZCBtZW1vcnkgZm9yCj4+Pj4+Pj4+ Pj4gICAgICAgbWlncmF0aW5nIGRhdGEgdG8uIFlvdSBtYXkgc3RlYWwgbWVtb3J5IGZyb20gb3Ro ZXIgcHJvY2Vzc2VzCj4+Pj4+Pj4+Pj4gdGhhdAo+Pj4+Pj4+Pj4+ICAgICAgIGNhbiBwYWdlIGZh dWx0LCBzbyBubyBmZW5jZSB3YWl0aW5nIGlzIG5lY2Vzc2FyeS4gQmVpbmcgYWJsZSB0bwo+Pj4+ Pj4+Pj4+ICAgICAgIHN0ZWFsIG1lbW9yeSBhdCBhbnkgdGltZSBhbHNvIG1lYW5zIHRoZXJlIGFy ZSBiYXNpY2FsbHkgbm8KPj4+Pj4+Pj4+PiAgICAgICBvdXQtb2YtbWVtb3J5IHNpdHVhdGlvbnMg eW91IG5lZWQgdG8gd29ycnkgYWJvdXQuIEV2ZW4gcGFnZQo+Pj4+Pj4+Pj4+IHRhYmxlcwo+Pj4+ Pj4+Pj4+ICAgICAgIChleGNlcHQgdGhlIHJvb3QgcGFnZSBkaXJlY3Rvcnkgb2YgZWFjaCBwcm9j ZXNzKSBjYW4gYmUKPj4+Pj4+Pj4+PiBzdG9sZW4gaW4KPj4+Pj4+Pj4+PiAgICAgICB0aGUgd29y c3QgY2FzZS4KPj4+Pj4+Pj4+IEkgdGhpbmsgJ292ZXJjb21taXQnIHdvdWxkIGJlIGEgbmljZSB3 YXkgdG8gZGVzY3JpYmUgdGhpcy4gQnV0IEknbSBub3QKPj4+Pj4+Pj4+IHN1cmUgaG93IGVhc3kg dGhpcyBpcyB0byBpbXBsZW1lbnQgaW4gcHJhY3RpY2UuIFlvdSB3b3VsZCBiYXNpY2FsbHkKPj4+ Pj4+Pj4+IG5lZWQKPj4+Pj4+Pj4+IHRvIGNyZWF0ZSB5b3VyIG93biBtZW1vcnkgbWFuYWdlciBm b3IgdGhpcy4KPj4+Pj4+Pj4gV2VsbCB5b3Ugd291bGQgbmVlZCBhIGNvbXBsZXRlbHkgc2VwYXJh dGUgcG9vbCBmb3IgYm90aCBkZXZpY2UgYXMgd2VsbAo+Pj4+Pj4+PiBhcyBzeXN0ZW0gbWVtb3J5 Lgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBFLmcuIG9uIGJvb3Qgd2Ugc2F5IHdlIHN0ZWFsIFggR0Igc3lz dGVtIG1lbW9yeSBvbmx5IGZvciBITU0uCj4+Pj4+Pj4gV2h5PyBUaGUgR1BVIGRyaXZlciBkb2Vz bid0IG5lZWQgdG8gYWxsb2NhdGUgc3lzdGVtIG1lbW9yeSBmb3IgSE1NLgo+Pj4+Pj4+IE1pZ3Jh dGlvbnMgdG8gc3lzdGVtIG1lbW9yeSBhcmUgaGFuZGxlZCBieSB0aGUga2VybmVsJ3MgaGFuZGxl X21tX2ZhdWx0Cj4+Pj4+Pj4gYW5kIHBhZ2UgYWxsb2NhdG9yIGFuZCBzd2FwIGxvZ2ljLgo+Pj4+ Pj4gQW5kIHRoYXQgb25lIGRlcGVuZHMgb24gZG1hX2ZlbmNlIGNvbXBsZXRpb24gYmVjYXVzZSB5 b3UgY2FuIGVhc2lseQo+Pj4+Pj4gbmVlZCB0byB3YWl0IGZvciBhbiBNTVUgbm90aWZpZXIgY2Fs bGJhY2suCj4+Pj4+IEkgc2VlLCB0aGUgR0ZYIE1NVSBub3RpZmllciBmb3IgdXNlcnBvaW50ZXJz IGluIGFtZGdwdSBjdXJyZW50bHkgd2FpdHMKPj4+Pj4gZm9yIGZlbmNlcy4gRm9yIHRoZSBLRkQg TU1VIG5vdGlmaWVyIEkgYW0gcGxhbm5pbmcgdG8gZml4IHRoaXMgYnkKPj4+Pj4gY2F1c2luZyBH UFUgcGFnZSBmYXVsdHMgaW5zdGVhZCBvZiBwcmVlbXB0aW5nIHRoZSBxdWV1ZXMuIENhbiB3ZSBs aW1pdAo+Pj4+PiB1c2VycHRycyBpbiBhbWRncHUgdG8gZW5naW5lcyB0aGF0IGNhbiBwYWdlIGZh dWx0LiBCYXNpY2FsbHkgbWFrZSBpdAo+Pj4+PiBpbGxlZ2FsIHRvIGF0dGFjaCB1c2VycHRyIEJP cyB0byBncmFwaGljcyBDUyBCTyBsaXN0cywgc28gdGhleSBjYW4gb25seQo+Pj4+PiBiZSB1c2Vk IGluIHVzZXIgbW9kZSBjb21tYW5kIHN1Ym1pc3Npb25zLCB3aGljaCBjYW4gcGFnZSBmYXVsdC4g VGhlbiB0aGUKPj4+Pj4gR0ZYIE1NVSBub3RpZmllciBjb3VsZCBpbnZhbGlkYXRlIFBURXMgYW5k IHdvdWxkIG5vdCBoYXZlIHRvIHdhaXQgZm9yCj4+Pj4+IGZlbmNlcy4KPj4+PiBzYWRseSBncmFw aGljcyArIHVzZXJwdHIgaXMgYWxyZWFkeSBleHBvc2VkIHZpYSBNZXNhLgo+Pj4gVGhpcyBpcyBu b3QgYWJvdXQgdXNlcnB0ciwgd2UgZmFrZSB1c2VycHRyIGVudGlyZWx5IGluIHNvZnR3YXJlLiBJ dCdzCj4+PiBhYm91dCBleHBvc2luZyByZWNvdmVyYWJsZSBncHUgcGFnZSBmYXVsdHMgKHdoaWNo IHdvdWxkIG1ha2UgdXNlcnB0cgo+Pj4gbWF5YmUgbW9yZSBlZmZpY2llbnQgc2luY2Ugd2UgY291 bGQgZG8gb24tZGVtYW5kIHBhZ2luZykuIHVzZXJwdHIKPj4+IGl0c2VsZiBpc24ndCBhIHByb2Js ZW0sIGJ1dCBpdCBpcyBwYXJ0IG9mIHRoZSByZWFzb25zIHdoeSB0aGlzIGlzCj4+PiB0cmlja3ku Cj4+Pgo+Pj4gQ2hyaXN0aWFuL0ZlbGl4LCBJIHRoaW5rIGZvciBrZXJuZWwgZm9sa3MgdGhpcyBp cyBjbGVhciBlbm91Z2ggdGhhdCBJCj4+PiBkb24ndCBuZWVkIHRvIGNsYXJpZnkgdGhpcyBpbiB0 aGUgdGV4dD8KPj4gWWVhaCwgaXQncyBjbGVhciB0byBtZS4gQW55d2F5LCB5b3VyIGxhdGVzdCB0 ZXh0IGRvZXNuJ3QgcmVmZXJlbmNlCj4+IHVzZXJwdHIgZGlyZWN0bHkgYW5kIGtlZXBzIHRoZSBk aXNjdXNzaW9uIGF0IGEgZmFpcmx5IGFic3RyYWN0IGxldmVsLiBTbwo+PiBJIHRoaW5rIGl0J3Mg ZmluZS4gSXQncyB0aGUgcHJhY3RpY2FsIGRldGFpbHMgb2YgdGhlIHByb3Bvc2VkCj4+IHdvcmth cm91bmRzIHdoZXJlIGl0IGZlZWwgbGlrZSB3YWxraW5nIHRocm91Z2ggYSBtaXJyb3IgY2FiaW5l dCwgYnVtcGluZwo+PiBpbnRvIHVuZXhwZWN0ZWQgb2JzdGFjbGVzIHdpdGggZXZlcnkgb3RoZXIg c3RlcC4KPiBPaCB5ZXMsIHRoaXMgaXMgdmVyeSBoaWdoLWxldmVsLiBUaGUgaW1wbGVtZW50YXRp b24gaXMgZ29pbmcgdG8gYmUKPiB2ZXJ5IHRyaWNrLCBubyBtYXR0ZXIgd2hpY2ggb25lIHdlJ3Jl IHBpY2tpbmcuIEFuZCB0YmggSSBleHBlY3QKPiBzdXJwcmlzZXMgYW5kIHRoaW5ncyB3ZSdsbCBs ZWFybi4gQnV0IEknbSBzdGlsbCBob3BpbmcgdGhhdCB0aGlzIGhpZ2gKPiBsZXZlbCBkb2MgcGF0 Y2ggd2lsbCBoZWxwIGEgbG90IHdpdGggYXZvaWRpbmcgdGhlIHdvcnN0IHByb2JsZW1zLgo+Cj4g T2YgY291cnNlIG9uY2Ugd2UgaGF2ZSBzb21lIG9mIHRoZXNlIGhhY2tzIGxhbmRlZCB3ZSBzaG91 bGQgbG9vayBhdCBpdAo+IGFnYWluIGFuZCBtYXliZSB1cGRhdGUgd2hlcmUgaXQncyB3cm9uZy91 bmNsZWFyLy4uLgo+Cj4gYnR3IHItYjogZnJvbSB5b3UgdG9vIG9uIHRoZSBwYXRjaD8KClllcy4K ClJldmlld2VkLWJ5OiBGZWxpeCBLdWVobGluZyA8RmVsaXguS3VlaGxpbmdAYW1kLmNvbT4KClRo YW5rcywKwqAgRmVsaXgKCgo+Cj4gQ2hlZXJzLCBEYW5pZWwKPgo+PiBSZWdhcmRzLAo+PiAgIEZl bGl4Cj4+Cj4+Cj4+PiAtRGFuaWVsCj4+Pgo+Pj4+Pj4gQXMgTWFhcnRlbiB3cm90ZSB3aGVuIHlv dSB3YW50IHRvIGdvIGRvd24gdGhpcyByb3V0ZSB5b3UgbmVlZCBhCj4+Pj4+PiBjb21wbGV0ZSBz ZXBhcmF0ZSBtZW1vcnkgbWFuYWdlbWVudCBwYXJhbGxlbCB0byB0aGUgb25lIG9mIHRoZSBrZXJu ZWwuCj4+Pj4+IE5vdCByZWFsbHkuIEknbSB0cnlpbmcgdG8gbWFrZSB0aGUgR1BVIG1lbW9yeSBt YW5hZ2VtZW50IG1vcmUgc2ltaWxhciB0bwo+Pj4+PiB3aGF0IHRoZSBrZXJuZWwgZG9lcyBmb3Ig c3lzdGVtIG1lbW9yeS4KPj4+Pj4KPj4+Pj4gSSB1bmRlcnN0b29kIE1hYXJ0ZW4ncyBjb21tZW50 IGFzICJJJ20gY3JlYXRpbmcgYSBuZXcgbWVtb3J5IG1hbmFnZXIgYW5kCj4+Pj4+IG5vdCB1c2lu ZyBUVE0gYW55IG1vcmUiLiBUaGlzIGlzIHRydWUuIFRoZSBpZGVhIGlzIHRoYXQgdGhpcyBwb3J0 aW9uIG9mCj4+Pj4+IFZSQU0gd291bGQgYmUgbWFuYWdlZCBtb3JlIGxpa2Ugc3lzdGVtIG1lbW9y eS4KPj4+Pj4KPj4+Pj4gUmVnYXJkcywKPj4+Pj4gICBGZWxpeAo+Pj4+Pgo+Pj4+Pgo+Pj4+Pj4g UmVnYXJkcywKPj4+Pj4+IENocmlzdGlhbi4KPj4+Pj4+Cj4+Pj4+Pj4gICBJdCBkb2Vzbid0IGRl cGVuZCBvbiBhbnkgZmVuY2VzLCBzbwo+Pj4+Pj4+IGl0IGNhbm5vdCBkZWFkbG9jayB3aXRoIGFu eSBHUFUgZHJpdmVyLW1hbmFnZWQgbWVtb3J5LiBUaGUgR1BVIGRyaXZlcgo+Pj4+Pj4+IGdldHMg aW52b2x2ZWQgaW4gdGhlIE1NVSBub3RpZmllciB0byBpbnZhbGlkYXRlIGRldmljZSBwYWdlIHRh Ymxlcy4gQnV0Cj4+Pj4+Pj4gdGhhdCBhbHNvIGRvZXNuJ3QgbmVlZCB0byB3YWl0IGZvciBhbnkg ZmVuY2VzLgo+Pj4+Pj4+Cj4+Pj4+Pj4gQW5kIGlmIHRoZSBrZXJuZWwgcnVucyBvdXQgb2YgcGFn ZWFibGUgbWVtb3J5LCB5b3UncmUgaW4gdHJvdWJsZSBhbnl3YXkuCj4+Pj4+Pj4gVGhlIE9PTSBr aWxsZXIgd2lsbCBzdGVwIGluLCBub3RoaW5nIG5ldyB0aGVyZS4KPj4+Pj4+Pgo+Pj4+Pj4+IFJl Z2FyZHMsCj4+Pj4+Pj4gICAgRmVsaXgKPj4+Pj4+Pgo+Pj4+Pj4+Cj4+Pj4+Pj4+PiBCdXQgZnJv bSBhIGRlc2lnbiBwb2ludCBvZiB2aWV3LCBkZWZpbml0ZWx5IGEgdmFsaWQgc29sdXRpb24uCj4+ Pj4+Pj4+IEkgdGhpbmsgdGhlIHJlc3RyaWN0aW9uIGFib3ZlIG1ha2VzIGl0IHByZXR0eSBtdWNo IHVudXNhYmxlLgo+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gQnV0IHRoaXMgbG9va3MgZ29vZCwgdGhvc2Ug c29sdXRpb25zIGFyZSBkZWZpbml0ZWx5IHRoZSB2YWxpZAo+Pj4+Pj4+Pj4gb3B0aW9ucyB3ZQo+ Pj4+Pj4+Pj4gY2FuIGNob29zZSBmcm9tLgo+Pj4+Pj4+PiBJdCdzIGNlcnRhaW5seSB3b3J0aCBu b3RpbmcsIHllcy4gQW5kIGp1c3QgdG8gbWFrZSBzdXJlIHRoYXQgbm9ib2R5Cj4+Pj4+Pj4+IGhh cyB0aGUgaWRlYSB0byByZXNlcnZlIG9ubHkgZGV2aWNlIG1lbW9yeS4KPj4+Pj4+Pj4KPj4+Pj4+ Pj4gQ2hyaXN0aWFuLgo+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gfk1hYXJ0ZW4KPj4+Pj4+Pj4+Cj4+Pj4+ Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPj4+Pj4+ PiBMaW5hcm8tbW0tc2lnIG1haWxpbmcgbGlzdAo+Pj4+Pj4+IExpbmFyby1tbS1zaWdAbGlzdHMu bGluYXJvLm9yZwo+Pj4+Pj4+IGh0dHBzOi8vbGlzdHMubGluYXJvLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2xpbmFyby1tbS1zaWcKPj4+Pj4+Pgo+Pj4+PiBfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXwo+Pj4+PiBkcmktZGV2ZWwgbWFpbGluZyBsaXN0Cj4+Pj4+ IGRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKPj4+Pj4gaHR0cHM6Ly9saXN0cy5mcmVl ZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwKPj4+Cj4KPgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBs aXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVz a3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.5 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF40BC433E0 for ; Tue, 9 Feb 2021 14:26:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71F6164E6C for ; Tue, 9 Feb 2021 14:26:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231520AbhBIO06 (ORCPT ); Tue, 9 Feb 2021 09:26:58 -0500 Received: from mail-mw2nam12on2074.outbound.protection.outlook.com ([40.107.244.74]:39008 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230269AbhBIO04 (ORCPT ); Tue, 9 Feb 2021 09:26:56 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iTmp17i4TX2FSm2RIztCXU+ktQlhfomWlsHPuY6Nkd64NCrkcnsWfLefCt+uXSVxJY/yutlpen61HAG58Cc/HaWtSfvgIXbxET99whGMaM1WTAbqcjhOAd+Xm16s5MTsyWsCZDXO6JT28hEQoEmXIGUGg8cuV7/e6K4kFfhqCx8NN4QRBz9r9+leJ3m1RPWCPBbaT8tglANG+phXQX9qAGvbGjKiHWwX4i/DYIEVQl18lnjnnG5F6gntE+1AgXOITxXB+6QvXkATbF2hsKZ+BpMrUCYZ76tsdPaKi9wfdHSAp3ukX+VFb/KmAaVjRqPDOTDDihFDQm0T8VNA9nQqWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pDvd/VnOcvi3KlTpZQflahw8ROalMpMj+jgJO559s3s=; b=FAI2nBq8LNauZB/8EtNQ/Pl2abf+4lE1Abr5nOSEXxlDFLOX07Rhyh63SAFCkJrUFEBdyOWATShUWWSbCv3lrp9cHNQZzQM1UUH/l4uLraVEMJlyCgNdDvdpkuUCyHSz14Lvqpm5MKH/zy1pH0TsCLBHqODWjQmSB4C/DFbFq0H0PGxFitz6E0jQXgtQocCEbpuRdSa3PSIMBi61YpnhmvTRegG3hZnmOGhTg4l+i1Mb/nJOVm11/wa5YwXEe7ESSMClSMeCDaLsJ2uYmensbMWmQp1RpbC5G01moAP/L1tk46dvoqQKAfau6LVKZ0UctAq8o8RovkJ4JcV9Bpn8UA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pDvd/VnOcvi3KlTpZQflahw8ROalMpMj+jgJO559s3s=; b=v6d+AYEg7AlDyJY9YqDSss8I0MYia1fFGcyPq/jmYOBUzSp/NBeT7xLKgEwriSUn5VZ+KaD110AhWvm5QFezXiWMZJ96SpEAXZRUfEmspmOaLeGWqbj9VCyCbWVR33CEbH0byoLTDP5ScmN4FxKqtGG+gIYcnXA/xkgnoaC8StQ= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4657.namprd12.prod.outlook.com (2603:10b6:207:1a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.23; Tue, 9 Feb 2021 14:26:00 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::ec8d:851e:525d:a6ab]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::ec8d:851e:525d:a6ab%8]) with mapi id 15.20.3784.022; Tue, 9 Feb 2021 14:26:00 +0000 Subject: Re: [Linaro-mm-sig] [PATCH] RFC: dma-fence: Document recoverable page fault implications To: Daniel Vetter Cc: Bas Nieuwenhuizen , "Koenig, Christian" , Maarten Lankhorst , DRI Development , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Daniel Vetter , Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , "open list:DMA BUFFER SHARING FRAMEWORK" References: <20210121194056.1734409-1-daniel.vetter@ffwll.ch> <6d373177-2645-1d67-9c14-dcad87c4f4d9@amd.com> <68740fcf-530e-b929-1c98-5810fc97ed23@linux.intel.com> <1e38efbc-ec52-e436-21e4-49a0d074b57b@amd.com> <18e7efbd-3d10-5ad1-49c9-7e26f0a27ef2@amd.com> <65b7a61c-b4b9-a210-5a37-0f69d01f667c@amd.com> From: Felix Kuehling Message-ID: Date: Tue, 9 Feb 2021 09:25:58 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: [142.117.121.176] X-ClientProxiedBy: YTOPR0101CA0012.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::25) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.100] (142.117.121.176) by YTOPR0101CA0012.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17 via Frontend Transport; Tue, 9 Feb 2021 14:25:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 126342fd-2a2e-4e52-f9ff-08d8cd069f76 X-MS-TrafficTypeDiagnostic: BL0PR12MB4657: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2fwyosBrYQ+9XrH5RAkh3rpQ9wRK9Ac+lLdXOVN5auOxEJSQX7YSCgkYiRcWOpfOoVTMr+HXLd9OjpdGr4cr9KIOm5/LHy68CTMTLJ13hinNR9dOf5jounFpzsCpOtIExPBaUu+0TNrwwkl2M+V0aS/EYN2sFl7kkRhSGBoNhHzJzLN11keOXO06+mYfIx02UGJjWasOopyCYoFQFPEJ0CGojzbrpmWYecAylZPqT9P6QRsum1cHYy3zAYI/tTsOGMEZ7V5U8pu7i3ENCgo8GJB1kL6m4BVzGZr5IOh4ba+reAWjrUacIsTp2gS5ZhhMqDLTU+nCzOe1RToetRpC6GvWVsSUaA6RKQvg/Rez2y2fy2IG+K1WPkDjjAGXNE7+CPof2nNhmQoL35B8xBSyQfj5WE4EgF66U2bMyyT2bEj0bk3yfZ6DVeAATQrlBGk5UhPoIGEBEkHyRAOzEwPjefVDJV/g7TWVkBujVzsqZ9LWOkxsLIOjNal2jjhZU6014ybU84dlPgV3yMQ3P7kjpo3fzHgPXdtdQSjbDFvmNmIOzSE5c5k82f/OGXWGQ8ZBXAnxcIU5U9r+viRAzzE20iMPQkfwucUHMZbEsvQ9infbi6WRtWp44DTu3MU/8nw6CxYgoA1A4DUhvYuEyhepw+Y7VpVnn9lB/dwUPzf3WGXc88Q+evELFJbXJLlFCnOz X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR12MB4948.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(396003)(136003)(346002)(366004)(376002)(52116002)(4326008)(54906003)(31696002)(36756003)(6916009)(6486002)(30864003)(31686004)(8936002)(956004)(83380400001)(2906002)(44832011)(26005)(966005)(8676002)(86362001)(5660300002)(186003)(16526019)(2616005)(316002)(66476007)(478600001)(53546011)(16576012)(66556008)(66574015)(66946007)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?dFhiblBuWHdKVWdYQ1hQVlFsc1lyeXhpNDYyajNlRlBmQ0orVDhITzBRbGpD?= =?utf-8?B?YTBwNGF5eWV5V2ZDZ2tSUXZXdHAzeERNOFd5L0JoM2czOWZBMmJHanRpUXZo?= =?utf-8?B?WGF1Y3Ywa3lMZWdlUzk1Vm03OXZuM0NiQXlBa1JxVi9ZSklaMGVZdTdSQi8r?= =?utf-8?B?OHdlOXdmQ0hYU1BpM0srdkF5ZU5QSFRQMjFpYzkvVzFsRHhqdDlkS0YydlNP?= =?utf-8?B?bjBnZmF2UjBQaXNMbzF4VFBhY1lxY01Ea2dBRlJWTzNieXhLTXNHZGZ3LzRw?= =?utf-8?B?MWNZWU1LdVpJWlFla1FveUNJQmZlQ1NXVkppRVo3c1ZQMiswQzliTUVFRVpl?= =?utf-8?B?MFZJMUNDMDBsRnVoTmM1Y1dXRmQyOHNEZlErWGg3WFppSUZPbDZuMUNLQ1Ra?= =?utf-8?B?emJjT2E3bTVuRnhhRUxselZST1FoM1dlYWY2QWxtbUxWUTJ0UDhBQ0NETm1m?= =?utf-8?B?alU3SURZNk1pZk9WUm9CREhQVVd1Sk9CaE5hSStrZXZZNjNRQXRyMzRTMmp0?= =?utf-8?B?WUVSdlVrUFFhbzlCSCtuMzdvMHRRanpTMlMxd1c3T0Vqd1U0ZFNOVVEzVzZ1?= =?utf-8?B?d28wREFSY0tRd0ZTNzNXdkNWbTVUMmUvVDdOa3ZjdGlVa21BdkQxQ1BJc1py?= =?utf-8?B?SzltaUErNWZsKzVCVVdtTjBxaTI2ZDNrV25pSEpwSHFGdzNuQXdPa21QeWFU?= =?utf-8?B?bU8zamxTOWJ6OTd5MWdoTW1SVUNqTzN6bkI4bm1FbUlQdUpJaDhBODBnQWNw?= =?utf-8?B?WmplSGRQZzhOVVdHcWd6ZGlXYUVRdjFEd2s0RGdORTYrbHVEbUpmT1VOay9i?= =?utf-8?B?RU9yVzlrRno5OGZadVdrRHhBOE5yRVl5WVkxSXNhYmxzRy82WmgzT3NXaHZp?= =?utf-8?B?a041MEcrNmtuenN3enI5WXFCcWlNTTJLRjlIRENYRS83QXpOZ05iYnk5QSth?= =?utf-8?B?TXkxTUl0V2wxTjlRS2c2c3dMVGhpb2NIRUtDeFR5U0xIOVRVdzdzTTZwS0ZC?= =?utf-8?B?cGJOeC9HRm5WRlFlTkN2L0MxZlpqWHNzRVhOcks0c0JXaFJKM01zdkh3MDYy?= =?utf-8?B?L0RxWVlxcXYrOG1EMGg1SWE3NjY5ZC9UT21HMmhuM045NWdEanhRU3ZSbkNJ?= =?utf-8?B?bTVkRVYyTzBJMjFFMWQ2ZnV6UDZNek1MTHZKUUtwbDV5eWNXN3pEeHBYOFN6?= =?utf-8?B?NDFBZzEra1FzalR5Wk1nckt1c0wzTWlqcDJkbi9ROHEwTkFVa05kbTkvdlBK?= =?utf-8?B?QjJmc3BwNGtMZnZpS3NWYjIyMk5xdldwa0E5VllHdEZsblExdzZpemNLcXk3?= =?utf-8?B?T1VwbmozRm1pQlcrRDh4NWgxb0VZNEZpVVpjZ1BrV05oMUhzTERtRWFIbWtp?= =?utf-8?B?aEoyL0d4bXg4RVVsNnpSWjJaOU9KZHkzcnEzb1FvOG9tM2s3OVdBSVZwYmow?= =?utf-8?B?SDBsVVBDOS9idlFMYms4d1dtYllXQkY1NStKaDBtY2F1T3NJWEZucDh4RFYz?= =?utf-8?B?bnhJejE3TGE2bnh4eHgvb1MvL1orbm5xQzFQdGMxOFJVd0FzS3FmL21WNyt6?= =?utf-8?B?NlNOS0hJekhFMDhYbWQ4Q1dMRjZ1cHNSSFBUWFNQb3ZyOTBHbUFZUVFrRCtU?= =?utf-8?B?dnVPOFBBbDUzNFE0eW1XT1FOVC9hekRxNCtQZlBHQzB6V0d0VXo3dDhWRi9l?= =?utf-8?B?dlVkZ0hKOThSTWI2SGpGVlpQUzVSZy8yQndCUTNGOTlhRkVNTk1kUHVvaXI3?= =?utf-8?B?T0tPaGhuL2llbGd2UGpZMGVSMks2bkVPZFNudFJTcyt3UVpqeUpSRzJWSDRJ?= =?utf-8?B?dmJCdnJQZTJNdkJFMWpZUT09?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 126342fd-2a2e-4e52-f9ff-08d8cd069f76 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2021 14:26:00.3470 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: owZB781tZboWXXI6iHxusvy/Y7WuT5q6FUnpL6+CZ3/RPqZuxg6GULCIpRrrSKyZ6fq9yelyCz90KDTLEU3VXQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4657 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Am 2021-02-09 um 9:08 a.m. schrieb Daniel Vetter: > On Tue, Feb 9, 2021 at 12:15 PM Felix Kuehling wrote: >> Am 2021-02-09 um 1:37 a.m. schrieb Daniel Vetter: >>> On Tue, Feb 9, 2021 at 4:13 AM Bas Nieuwenhuizen >>> wrote: >>>> On Thu, Jan 28, 2021 at 4:40 PM Felix Kuehling wrote: >>>>> Am 2021-01-28 um 2:39 a.m. schrieb Christian König: >>>>>> Am 27.01.21 um 23:00 schrieb Felix Kuehling: >>>>>>> Am 2021-01-27 um 7:16 a.m. schrieb Christian König: >>>>>>>> Am 27.01.21 um 13:11 schrieb Maarten Lankhorst: >>>>>>>>> Op 27-01-2021 om 01:22 schreef Felix Kuehling: >>>>>>>>>> Am 2021-01-21 um 2:40 p.m. schrieb Daniel Vetter: >>>>>>>>>>> Recently there was a fairly long thread about recoreable hardware >>>>>>>>>>> page >>>>>>>>>>> faults, how they can deadlock, and what to do about that. >>>>>>>>>>> >>>>>>>>>>> While the discussion is still fresh I figured good time to try and >>>>>>>>>>> document the conclusions a bit. >>>>>>>>>>> >>>>>>>>>>> References: >>>>>>>>>>> https://lore.kernel.org/dri-devel/20210107030127.20393-1-Felix.Kuehling@amd.com/ >>>>>>>>>>> >>>>>>>>>>> Cc: Maarten Lankhorst >>>>>>>>>>> Cc: Thomas Hellström >>>>>>>>>>> Cc: "Christian König" >>>>>>>>>>> Cc: Jerome Glisse >>>>>>>>>>> Cc: Felix Kuehling >>>>>>>>>>> Signed-off-by: Daniel Vetter >>>>>>>>>>> Cc: Sumit Semwal >>>>>>>>>>> Cc: linux-media@vger.kernel.org >>>>>>>>>>> Cc: linaro-mm-sig@lists.linaro.org >>>>>>>>>>> -- >>>>>>>>>>> I'll be away next week, but figured I'll type this up quickly for >>>>>>>>>>> some >>>>>>>>>>> comments and to check whether I got this all roughly right. >>>>>>>>>>> >>>>>>>>>>> Critique very much wanted on this, so that we can make sure hw which >>>>>>>>>>> can't preempt (with pagefaults pending) like gfx10 has a clear >>>>>>>>>>> path to >>>>>>>>>>> support page faults in upstream. So anything I missed, got wrong or >>>>>>>>>>> like that would be good. >>>>>>>>>>> -Daniel >>>>>>>>>>> --- >>>>>>>>>>> Documentation/driver-api/dma-buf.rst | 66 >>>>>>>>>>> ++++++++++++++++++++++++++++ >>>>>>>>>>> 1 file changed, 66 insertions(+) >>>>>>>>>>> >>>>>>>>>>> diff --git a/Documentation/driver-api/dma-buf.rst >>>>>>>>>>> b/Documentation/driver-api/dma-buf.rst >>>>>>>>>>> index a2133d69872c..e924c1e4f7a3 100644 >>>>>>>>>>> --- a/Documentation/driver-api/dma-buf.rst >>>>>>>>>>> +++ b/Documentation/driver-api/dma-buf.rst >>>>>>>>>>> @@ -257,3 +257,69 @@ fences in the kernel. This means: >>>>>>>>>>> userspace is allowed to use userspace fencing or long running >>>>>>>>>>> compute >>>>>>>>>>> workloads. This also means no implicit fencing for shared >>>>>>>>>>> buffers in these >>>>>>>>>>> cases. >>>>>>>>>>> + >>>>>>>>>>> +Recoverable Hardware Page Faults Implications >>>>>>>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> + >>>>>>>>>>> +Modern hardware supports recoverable page faults, which has a >>>>>>>>>>> lot of >>>>>>>>>>> +implications for DMA fences. >>>>>>>>>>> + >>>>>>>>>>> +First, a pending page fault obviously holds up the work that's >>>>>>>>>>> running on the >>>>>>>>>>> +accelerator and a memory allocation is usually required to resolve >>>>>>>>>>> the fault. >>>>>>>>>>> +But memory allocations are not allowed to gate completion of DMA >>>>>>>>>>> fences, which >>>>>>>>>>> +means any workload using recoverable page faults cannot use DMA >>>>>>>>>>> fences for >>>>>>>>>>> +synchronization. Synchronization fences controlled by userspace >>>>>>>>>>> must be used >>>>>>>>>>> +instead. >>>>>>>>>>> + >>>>>>>>>>> +On GPUs this poses a problem, because current desktop compositor >>>>>>>>>>> protocols on >>>>>>>>>>> +Linus rely on DMA fences, which means without an entirely new >>>>>>>>>>> userspace stack >>>>>>>>>>> +built on top of userspace fences, they cannot benefit from >>>>>>>>>>> recoverable page >>>>>>>>>>> +faults. The exception is when page faults are only used as >>>>>>>>>>> migration hints and >>>>>>>>>>> +never to on-demand fill a memory request. For now this means >>>>>>>>>>> recoverable page >>>>>>>>>>> +faults on GPUs are limited to pure compute workloads. >>>>>>>>>>> + >>>>>>>>>>> +Furthermore GPUs usually have shared resources between the 3D >>>>>>>>>>> rendering and >>>>>>>>>>> +compute side, like compute units or command submission engines. If >>>>>>>>>>> both a 3D >>>>>>>>>>> +job with a DMA fence and a compute workload using recoverable page >>>>>>>>>>> faults are >>>>>>>>>>> +pending they could deadlock: >>>>>>>>>>> + >>>>>>>>>>> +- The 3D workload might need to wait for the compute job to finish >>>>>>>>>>> and release >>>>>>>>>>> + hardware resources first. >>>>>>>>>>> + >>>>>>>>>>> +- The compute workload might be stuck in a page fault, because the >>>>>>>>>>> memory >>>>>>>>>>> + allocation is waiting for the DMA fence of the 3D workload to >>>>>>>>>>> complete. >>>>>>>>>>> + >>>>>>>>>>> +There are a few ways to prevent this problem: >>>>>>>>>>> + >>>>>>>>>>> +- Compute workloads can always be preempted, even when a page >>>>>>>>>>> fault is pending >>>>>>>>>>> + and not yet repaired. Not all hardware supports this. >>>>>>>>>>> + >>>>>>>>>>> +- DMA fence workloads and workloads which need page fault handling >>>>>>>>>>> have >>>>>>>>>>> + independent hardware resources to guarantee forward progress. >>>>>>>>>>> This could be >>>>>>>>>>> + achieved through e.g. through dedicated engines and minimal >>>>>>>>>>> compute unit >>>>>>>>>>> + reservations for DMA fence workloads. >>>>>>>>>>> + >>>>>>>>>>> +- The reservation approach could be further refined by only >>>>>>>>>>> reserving the >>>>>>>>>>> + hardware resources for DMA fence workloads when they are >>>>>>>>>>> in-flight. This must >>>>>>>>>>> + cover the time from when the DMA fence is visible to other >>>>>>>>>>> threads up to >>>>>>>>>>> + moment when fence is completed through dma_fence_signal(). >>>>>>>>>>> + >>>>>>>>>>> +- As a last resort, if the hardware provides no useful reservation >>>>>>>>>>> mechanics, >>>>>>>>>>> + all workloads must be flushed from the GPU when switching >>>>>>>>>>> between jobs >>>>>>>>>>> + requiring DMA fences or jobs requiring page fault handling: This >>>>>>>>>>> means all DMA >>>>>>>>>>> + fences must complete before a compute job with page fault >>>>>>>>>>> handling can be >>>>>>>>>>> + inserted into the scheduler queue. And vice versa, before a DMA >>>>>>>>>>> fence can be >>>>>>>>>>> + made visible anywhere in the system, all compute workloads must >>>>>>>>>>> be preempted >>>>>>>>>>> + to guarantee all pending GPU page faults are flushed. >>>>>>>>>> I thought of another possible workaround: >>>>>>>>>> >>>>>>>>>> * Partition the memory. Servicing of page faults will use a >>>>>>>>>> separate >>>>>>>>>> memory pool that can always be allocated from without >>>>>>>>>> waiting for >>>>>>>>>> fences. This includes memory for page tables and memory for >>>>>>>>>> migrating data to. You may steal memory from other processes >>>>>>>>>> that >>>>>>>>>> can page fault, so no fence waiting is necessary. Being able to >>>>>>>>>> steal memory at any time also means there are basically no >>>>>>>>>> out-of-memory situations you need to worry about. Even page >>>>>>>>>> tables >>>>>>>>>> (except the root page directory of each process) can be >>>>>>>>>> stolen in >>>>>>>>>> the worst case. >>>>>>>>> I think 'overcommit' would be a nice way to describe this. But I'm not >>>>>>>>> sure how easy this is to implement in practice. You would basically >>>>>>>>> need >>>>>>>>> to create your own memory manager for this. >>>>>>>> Well you would need a completely separate pool for both device as well >>>>>>>> as system memory. >>>>>>>> >>>>>>>> E.g. on boot we say we steal X GB system memory only for HMM. >>>>>>> Why? The GPU driver doesn't need to allocate system memory for HMM. >>>>>>> Migrations to system memory are handled by the kernel's handle_mm_fault >>>>>>> and page allocator and swap logic. >>>>>> And that one depends on dma_fence completion because you can easily >>>>>> need to wait for an MMU notifier callback. >>>>> I see, the GFX MMU notifier for userpointers in amdgpu currently waits >>>>> for fences. For the KFD MMU notifier I am planning to fix this by >>>>> causing GPU page faults instead of preempting the queues. Can we limit >>>>> userptrs in amdgpu to engines that can page fault. Basically make it >>>>> illegal to attach userptr BOs to graphics CS BO lists, so they can only >>>>> be used in user mode command submissions, which can page fault. Then the >>>>> GFX MMU notifier could invalidate PTEs and would not have to wait for >>>>> fences. >>>> sadly graphics + userptr is already exposed via Mesa. >>> This is not about userptr, we fake userptr entirely in software. It's >>> about exposing recoverable gpu page faults (which would make userptr >>> maybe more efficient since we could do on-demand paging). userptr >>> itself isn't a problem, but it is part of the reasons why this is >>> tricky. >>> >>> Christian/Felix, I think for kernel folks this is clear enough that I >>> don't need to clarify this in the text? >> Yeah, it's clear to me. Anyway, your latest text doesn't reference >> userptr directly and keeps the discussion at a fairly abstract level. So >> I think it's fine. It's the practical details of the proposed >> workarounds where it feel like walking through a mirror cabinet, bumping >> into unexpected obstacles with every other step. > Oh yes, this is very high-level. The implementation is going to be > very trick, no matter which one we're picking. And tbh I expect > surprises and things we'll learn. But I'm still hoping that this high > level doc patch will help a lot with avoiding the worst problems. > > Of course once we have some of these hacks landed we should look at it > again and maybe update where it's wrong/unclear/... > > btw r-b: from you too on the patch? Yes. Reviewed-by: Felix Kuehling Thanks,   Felix > > Cheers, Daniel > >> Regards, >> Felix >> >> >>> -Daniel >>> >>>>>> As Maarten wrote when you want to go down this route you need a >>>>>> complete separate memory management parallel to the one of the kernel. >>>>> Not really. I'm trying to make the GPU memory management more similar to >>>>> what the kernel does for system memory. >>>>> >>>>> I understood Maarten's comment as "I'm creating a new memory manager and >>>>> not using TTM any more". This is true. The idea is that this portion of >>>>> VRAM would be managed more like system memory. >>>>> >>>>> Regards, >>>>> Felix >>>>> >>>>> >>>>>> Regards, >>>>>> Christian. >>>>>> >>>>>>> It doesn't depend on any fences, so >>>>>>> it cannot deadlock with any GPU driver-managed memory. The GPU driver >>>>>>> gets involved in the MMU notifier to invalidate device page tables. But >>>>>>> that also doesn't need to wait for any fences. >>>>>>> >>>>>>> And if the kernel runs out of pageable memory, you're in trouble anyway. >>>>>>> The OOM killer will step in, nothing new there. >>>>>>> >>>>>>> Regards, >>>>>>> Felix >>>>>>> >>>>>>> >>>>>>>>> But from a design point of view, definitely a valid solution. >>>>>>>> I think the restriction above makes it pretty much unusable. >>>>>>>> >>>>>>>>> But this looks good, those solutions are definitely the valid >>>>>>>>> options we >>>>>>>>> can choose from. >>>>>>>> It's certainly worth noting, yes. And just to make sure that nobody >>>>>>>> has the idea to reserve only device memory. >>>>>>>> >>>>>>>> Christian. >>>>>>>> >>>>>>>>> ~Maarten >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Linaro-mm-sig mailing list >>>>>>> Linaro-mm-sig@lists.linaro.org >>>>>>> https://lists.linaro.org/mailman/listinfo/linaro-mm-sig >>>>>>> >>>>> _______________________________________________ >>>>> dri-devel mailing list >>>>> dri-devel@lists.freedesktop.org >>>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel >>> > >