From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAEBFC2D0E4 for ; Wed, 25 Nov 2020 03:21:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3F3B220782 for ; Wed, 25 Nov 2020 03:21:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=marvell.com header.i=@marvell.com header.b="Y1EFoFrw"; dkim=pass (1024-bit key) header.d=marvell.onmicrosoft.com header.i=@marvell.onmicrosoft.com header.b="M2ndBrzA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726602AbgKYDVt (ORCPT ); Tue, 24 Nov 2020 22:21:49 -0500 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174]:48148 "EHLO mx0b-0016f401.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726532AbgKYDVt (ORCPT ); Tue, 24 Nov 2020 22:21:49 -0500 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0AP3KBl7013934; Tue, 24 Nov 2020 19:20:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=pfpt0220; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=Y1EFoFrw0NDdL2aapeEA9O2VCCSSwM9zU6Nkg9SXQpuu9APKX2p/S9zSlDkzl4LDT6yZ HqQ08BvrUll2TGS5BmE8+JjpQOGegj+byzt7nhsKcR/HhQPK4xWDZeDytcsBdpmNeGT1 KYEIf4NCuwIC3C65kCMOdy7N9MYI6iLJo8JifC7uLuElAp0eDPO847DXXkvCE2/pH++k IJOSr3cMILC2UwUXdSOyb9OLDnwcw22x2llMJ/iPkHI4lzwONp9ge/tE+EzXs6SDSLU/ yUzkRDRH8X/5v+eSbn28dBXFjxv0LK1yAGr+CGPgD2k7fKqxi0PuQeSCtExr7kF/dB8N Gw== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0a-0016f401.pphosted.com with ESMTP id 34y14ucsam-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 24 Nov 2020 19:20:38 -0800 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 24 Nov 2020 19:20:36 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 24 Nov 2020 19:20:37 -0800 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.36.55) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Tue, 24 Nov 2020 19:20:36 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rghtfb/0DCZEhW4B2dHOQW4dlqugAa9ZMPZbsRdK4cKrIgE1rHxsRQ/1sRzZQUj2fiZ1YGIgGw7OMiE+zM1MsHuq3wIgRK+tvj1VcCdikYGZudkezEoxdFM5epAoHL0vijzh9a+xZBqJomQD3YPxGnrQrEhyJT+59jDbr3jmFucfqqFkh+Km0teG0THuUGkuB0sw3tql4YxojH8oCKzPH7YWfj9ZBYZqyTf0cezKPkmPXV/SA1U0AVuCTfkpf838iLBeKV9M4Fbl/4QMm7FKjOdyXb2uxm/L8p+KZKaIvnl7m7+W9AQ143Lyh4LWcaT+Ctk6CBRJfYXv4bTGsElHZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=MpG9DF3Ng7u3dOpYVlJ8ngKA9Cm2s9oHNabG4Wru+ikdCW8RXM/CwwidjFTIx4AKA4C2X9LIJsGqGgT+WYAf1J6GirE+7jeXBYzJMFcTxjtVUzOzsiHTWAE/iu9Q/XASgHCv5M1K/CyK3RJLWZWJ4/czigwW8ZqK+9LHxmYsHPJqJIDMaq0T6giaLzyNO/AfsoSDsLSHTjBRZ8P306ekWVHRN61dsLlu4ADZBUb0/wtGL9jq/PXKiXwOhecRcmNu6vhIm61GGvB9XOEzByGo9a+X6FqGY3sF/HGe0Wp0n+zmCbJJ/bl3A94y1Zqxjt4TGlxCNtsjExF3baPAvZfqTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=M2ndBrzAl3GOCUMh7yBqkALHQYDHmh9kqDC7Qiw5kB4vRj37VxHLop8MNiI92p065bBA7eaSSAsc0nrKhtzkw/bkJGJpm3um6C5DQ3BGk0D28Emb9G7qKIKq5JzwQxRAeuQX5u78Pkb2dXJ/Z73soyEJPRIiANX3ahTAQw+Vpko= Received: from MW2PR18MB2267.namprd18.prod.outlook.com (2603:10b6:907:3::11) by MWHPR18MB1440.namprd18.prod.outlook.com (2603:10b6:320:2c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Wed, 25 Nov 2020 03:20:35 +0000 Received: from MW2PR18MB2267.namprd18.prod.outlook.com ([fe80::e17f:37fb:4eb3:c703]) by MW2PR18MB2267.namprd18.prod.outlook.com ([fe80::e17f:37fb:4eb3:c703%4]) with mapi id 15.20.3564.039; Wed, 25 Nov 2020 03:20:34 +0000 From: Alex Belits To: "frederic@kernel.org" CC: Prasun Kapoor , "linux-api@vger.kernel.org" , "davem@davemloft.net" , "trix@redhat.com" , "mingo@kernel.org" , "linux-kernel@vger.kernel.org" , "rostedt@goodmis.org" , "peterx@redhat.com" , "tglx@linutronix.de" , "nitesh@redhat.com" , "linux-arch@vger.kernel.org" , "mtosatti@redhat.com" , "will@kernel.org" , "peterz@infradead.org" , "leon@sidebranch.com" , "linux-arm-kernel@lists.infradead.org" , "catalin.marinas@arm.com" , "pauld@redhat.com" , "netdev@vger.kernel.org" Subject: Re: [EXT] Re: [PATCH v5 9/9] task_isolation: kick_all_cpus_sync: don't kick isolated cpus Thread-Topic: [EXT] Re: [PATCH v5 9/9] task_isolation: kick_all_cpus_sync: don't kick isolated cpus Thread-Index: AQHWwcJHihlNClCxdEOODDjRi5wCaKnWTJeAgAAC64CAAAucAIAB1ToA Date: Wed, 25 Nov 2020 03:20:34 +0000 Message-ID: <45bb2800f2c966ed04ced8dfa85615c377fac4b2.camel@marvell.com> References: <8d887e59ca713726f4fcb25a316e1e932b02823e.camel@marvell.com> <3236b13f42679031960c5605be20664e90e75223.camel@marvell.com> <20201123222907.GC1751@lothringen> <20201123232106.GD1751@lothringen> In-Reply-To: <20201123232106.GD1751@lothringen> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=marvell.com; x-originating-ip: [173.228.7.197] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 55ef8eb2-533d-4ede-c770-08d890f112b5 x-ms-traffictypediagnostic: MWHPR18MB1440: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: B0byi0kQVG8Z+R5XExLn1lGCmWyT7IdrMdAAtstKnoT2oPnQKPv3tqPdXy2BzQP4YAZwrNuuOv8Rw39rZs+9pHRPUtibtZgJ3FYPt76/ay0HX4xYOqSFp/1/4ddbMw8eZNBpOlaaUDvIS8DDfTP8oUlpjbYlZDqrsKtb3pn3zE/uAVeQmflZepH5A21RFK2Wu/TuCIicag4MmJz9TiWzoKMx/vyPCtT3OPlcVs4FOGxpMZUuT2Zz6WYFvWeE9VBRnb5v4zARM991P124JaREMdy0NnFRoFO+BT5xMMrCY2rYpT/hW9LPA9T2pceV+VI8k8DmsRnByvz/ClOERXdfOg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR18MB2267.namprd18.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(136003)(376002)(366004)(39850400004)(396003)(6506007)(83380400001)(7416002)(4326008)(6486002)(6512007)(91956017)(36756003)(5660300002)(76116006)(66476007)(64756008)(66946007)(66556008)(66446008)(86362001)(186003)(26005)(478600001)(71200400001)(2616005)(316002)(54906003)(4001150100001)(2906002)(6916009)(8676002)(8936002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata: 4gvzcjKq4r5qQCylHcBxvSfgfiTFVF8LxUMCYWskBL1IYxGImcJRp1E+CkpEzn4utYa9c6rTFLmtF0YtDEp1LxsGBOIeZX7W3t+1csM6lOR+GuzJVw0uoNocReydcPX0OpcJ5z/GlTMihUHOM3T8RyHHu3p6xYTwoaZeoXMamBm8oHR3LfWAJEbDmVc9yqBVPX50uWZfixifHHz1N/ukdF5UmgPUVdP8Hid5C61SPV/UNh+NEFNxlrvmvs8wXudkEzgmNBtKAIjZL00tD+VISpYRAKFmbPRUPH18ufl20CYAvV1gO8ZpL52iUTKFkgWXSqfZlcmxYT9A3IXNKmVP3yY6o6BSWvAidc1UHYrYOXWPq6P6D6Bzyislk+tb6s8kSm6SPp/JqEI+hteTNP0TMbzNQZnkaUEPdludIXyKSgunLyCV3voyYZ2v9PALQOQjaWxW8m2aA57hdEvMZsrOtETRHYxPKYxQQwCUmKqbeJprpsror68lCD1QN9t0Eyh/X44RYtWIG4+1OIuvmj2fmvpx1soQ2E33viJr5OLRE2pT/pWBkmsbFK3/D5WNQod9lqQSVmXS0QUeaoTd7Us9Uo5xkbVTI2ubh28A1vg2CCzLDDGtNA+EdBQ27xRrW2GfDtLmYB41mSCdIGG9wGwW48wL2y/tj46VPJ0PF5CRptoS2vLyA0SjVFMO/ZTTw0VLYwIB6P0KeFv/ILVbpcgR0e0pc8kGfgcWTof7q0mW6rTK/dCqPEYVlYhVwItAKvuWjX6ZuPPYEW+DSNQXE/ABKjwA+NVTacI0g1EgeD5Bv/8Lkv0D1sv57tQgg6KzAavXUIu8NwKhz698B+aItbNizrZ1J8qFvpyRDfqnCBWlm/PxFlMzd1hVW2sWLRgDTSzP2BU4YS/0utqN0OOk1grC6g== Content-Type: text/plain; charset="utf-8" Content-ID: <6B9838A204A0934595C222765B6BC2D1@namprd18.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW2PR18MB2267.namprd18.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55ef8eb2-533d-4ede-c770-08d890f112b5 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Nov 2020 03:20:34.5307 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Vrk7gYZPmHhU5DNFtoSH5HrS+oyzfcJa7Ri2xtMDzdfTGOXATwsPWauLNOAzJqJXNiKq+O3LnJszUHcxSz8ujQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR18MB1440 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-11-24_11:2020-11-24,2020-11-24 signatures=0 Precedence: bulk List-ID: X-Mailing-List: linux-api@vger.kernel.org DQpPbiBUdWUsIDIwMjAtMTEtMjQgYXQgMDA6MjEgKzAxMDAsIEZyZWRlcmljIFdlaXNiZWNrZXIg d3JvdGU6DQo+IE9uIE1vbiwgTm92IDIzLCAyMDIwIGF0IDEwOjM5OjM0UE0gKzAwMDAsIEFsZXgg QmVsaXRzIHdyb3RlOg0KPiA+IA0KPiA+IFRoaXMgaXMgZGlmZmVyZW50IGZyb20gdGltZXJzLiBU aGUgb3JpZ2luYWwgZGVzaWduIHdhcyBiYXNlZCBvbiB0aGUNCj4gPiBpZGVhIHRoYXQgZXZlcnkg Q1BVIHNob3VsZCBiZSBhYmxlIHRvIGVudGVyIGtlcm5lbCBhdCBhbnkgdGltZSBhbmQNCj4gPiBy dW4NCj4gPiBrZXJuZWwgY29kZSB3aXRoIG5vIGFkZGl0aW9uYWwgcHJlcGFyYXRpb24uIFRoZW4g dGhlIG9ubHkgc29sdXRpb24NCj4gPiBpcw0KPiA+IHRvIGFsd2F5cyBkbyBmdWxsIGJyb2FkY2Fz dCBhbmQgcmVxdWlyZSBhbGwgQ1BVcyB0byBwcm9jZXNzIGl0Lg0KPiA+IA0KPiA+IFdoYXQgSSBh bSB0cnlpbmcgdG8gaW50cm9kdWNlIGlzIHRoZSBpZGVhIG9mIENQVSB0aGF0IGlzIG5vdCBsaWtl bHkNCj4gPiB0bw0KPiA+IHJ1biBrZXJuZWwgY29kZSBhbnkgc29vbiwgYW5kIGNhbiBhZmZvcmQg dG8gZ28gdGhyb3VnaCBhbg0KPiA+IGFkZGl0aW9uYWwNCj4gPiBzeW5jaHJvbml6YXRpb24gcHJv Y2VkdXJlIG9uIHRoZSBuZXh0IGVudHJ5IGludG8ga2VybmVsLiBUaGUNCj4gPiBzeW5jaHJvbml6 YXRpb24gaXMgbm90IHNraXBwZWQsIGl0IHNpbXBseSBoYXBwZW5zIGxhdGVyLCBlYXJseSBpbg0K PiA+IGtlcm5lbCBlbnRyeSBjb2RlLg0KPiANCj4gQWggSSBzZWUsIHRoaXMgaXMgb3JkZXJlZCB0 aGF0IHdheToNCj4gDQo+IGxsX2lzb2xfZmxhZ3MgPSBJU09MQVRFRA0KPiANCj4gICAgICAgICAg Q1BVIDAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENQVSAxDQo+ICAgICAtLS0tLS0t LS0tLS0tLS0tLS0gICAgICAgICAgICAgICAgICAgICAgIC0tLS0tLS0tLS0tLS0tLS0tDQo+ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLy8ga2VybmVsIGVudHJ5 DQo+ICAgICBkYXRhX3RvX3N5bmMgPSAxICAgICAgICAgICAgICAgICAgICAgICAgbGxfaXNvbF9m bGFncyA9IElTT0xBVEVEX0JST0tFTg0KPiAgICAgc21wX21iKCkgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHNtcF9tYigpDQo+ICAgICBpZiBsbF9pc29sX2ZsYWdzKENQVSAxKSA9PSBJ U09MQVRFRCAgICAgUkVBRCBkYXRhX3RvX3N5bmMNCj4gICAgICAgICAgc21wX2NhbGwoQ1BVIDEp DQo+IA0KDQpUaGUgY2hlY2sgZm9yIGxsX2lzb2xfZmxhZ3MoQ1BVIDEpIGlzIHJldmVyc2VkLCBh bmQgaXQncyBhIGJpdCBtb3JlDQpjb21wbGV4LiBJbiB0ZXJtcyBvZiBzY2VuYXJpb3MsIG9uIGVu dHJ5IGZyb20gaXNvbGF0aW9uIHRoZSBmb2xsb3dpbmcNCmNhbiBoYXBwZW46DQoNCjEuIEtlcm5l bCBlbnRyeSBoYXBwZW5zIHNpbXVsdGFuZW91c2x5IHdpdGggb3BlcmF0aW9uIHRoYXQgcmVxdWly ZXMNCnN5bmNocm9uaXphdGlvbiwga2VybmVsIGVudHJ5IHByb2Nlc3NpbmcgaGFwcGVucyBiZWZv cmUgdGhlIGNoZWNrIGZvcg0KaXNvbGF0aW9uIG9uIHRoZSBzZW5kZXIgc2lkZToNCg0KbGxfaXNv bF9mbGFncyhDUFUgMSkgPSBJU09MQVRFRA0KDQogICAgICAgICBDUFUgMCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgQ1BVIDENCiAgICAtLS0tLS0tLS0tLS0tLS0tLS0gICAgICAgICAg ICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0NCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgLy8ga2VybmVsIGVudHJ5DQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGlmIChsbF9pc29sX2ZsYWdzID09IElTT0xBVEVEKSB7DQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxsX2lzb2xf ZmxhZ3MgPSBJU09MQVRFRF9CUk9LRU4NCiAgICBkYXRhX3RvX3N5bmMgPSAxICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgc21wX21iKCkNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgLy8gZGF0YV90b19zeW5jIHVuZGV0ZXJtaW5lZA0KICAgIHNt cF9tYigpICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB9DQogICAgLy8gbGxfaXNvbF9m bGFncyhDUFUgMSkgdXBkYXRlZA0KICAgIGlmIGxsX2lzb2xfZmxhZ3MoQ1BVIDEpICE9IElTT0xB VEVEDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8vIGludGVy cnVwdHMgZW5hYmxlZA0KICAgICAgICAgc21wX2NhbGwoQ1BVIDEpICAgICAgICAgICAgICAgICAg ICAgICAgICAvLyBrZXJuZWwgZW50cnkgYWdhaW4NCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgaWYgKGxsX2lzb2xfZmxhZ3MgPT0gSVNPTEFURUQpDQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8v IG5vdGhpbmcgaGFwcGVucw0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAvLyBleHBsaWNpdCBvciBpbXBsaWVkIGJhcnJpZXJzDQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8vIGRhdGFfdG9fc3luYyB1cGRh dGVkDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8v IGtlcm5lbCBleGl0DQogICAgLy8gQ1BVIDAgYXNzdW1lcywgQ1BVIDEgd2lsbCBzZWUgICAgICAg IFJFQUQgZGF0YV90b19zeW5jDQogICAgLy8gZGF0YV90b19zeW5jID0gMSB3aGVuIGluIGtlcm5l bA0KDQoyLiBLZXJuZWwgZW50cnkgaGFwcGVucyBzaW11bHRhbmVvdXNseSB3aXRoIG9wZXJhdGlv biB0aGF0IHJlcXVpcmVzDQpzeW5jaHJvbml6YXRpb24sIGtlcm5lbCBlbnRyeSBwcm9jZXNzaW5n IGhhcHBlbnMgYWZ0ZXIgdGhlIGNoZWNrIGZvcg0KaXNvbGF0aW9uIG9uIHRoZSBzZW5kZXIgc2lk ZToNCg0KbGxfaXNvbF9mbGFncyhDUFUgMSkgPSBJU09MQVRFRA0KDQogICAgICAgICBDUFUgMCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgQ1BVIDENCiAgICAtLS0tLS0tLS0tLS0tLS0t LS0gICAgICAgICAgICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0NCiAgICBkYXRhX3RvX3N5 bmMgPSAxICAgICAgICAgICAgICAgICAgICAgICAgLy8ga2VybmVsIGVudHJ5DQogICAgc21wX21i KCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8vIGRhdGFfdG9fc3luYyB1bmRldGVy bWluZWQNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLy8gc2hv dWxkIG5vdCBhY2Nlc3MgZGF0YV90b19zeW5jIGhlcmUNCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgaWYgKGxsX2lzb2xfZmxhZ3MgPT0gSVNPTEFURUQpIHsgICAg ICANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxs X2lzb2xfZmxhZ3MgPSBJU09MQVRFRF9CUk9LRU4NCiAgICAvLyBsbF9pc29sX2ZsYWdzKENQVSAx KSB1bmRldGVybWluZWQgICAgICAgICAgIHNtcF9tYigpDQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvLyBkYXRhX3RvX3N5bmMgdXBkYXRlZA0KICAg IGlmIGxsX2lzb2xfZmxhZ3MoQ1BVIDEpICE9IElTT0xBVEVEICAgICB9DQogICAgICAgICAvLyBw b3NzaWJseSBub3RoaW5nIGhhcHBlbnMNCiAgICAvLyBDUFUgMCBhc3N1bWVzLCBDUFUgMSB3aWxs IHNlZSAgICAgICAgUkVBRCBkYXRhX3RvX3N5bmMNCiAgICAvLyBkYXRhX3RvX3N5bmMgPSAxIHdo ZW4gaW4ga2VybmVsDQoNCjMuIEtlcm5lbCBlbnRyeSBwcm9jZXNzaW5nIGNvbXBsZXRlZCBiZWZv cmUgdGhlIGNoZWNrIGZvciBpc29sYXRpb24gb24gdGhlIHNlbmRlcg0Kc2lkZToNCg0KbGxfaXNv bF9mbGFncyhDUFUgMSkgPSBJU09MQVRFRA0KDQogICAgICAgICBDUFUgMCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgQ1BVIDENCiAgICAtLS0tLS0tLS0tLS0tLS0tLS0gICAgICAgICAg ICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0NCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgLy8ga2VybmVsIGVudHJ5DQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGlmIChsbF9pc29sX2ZsYWdzID09IElTT0xBVEVEKSB7DQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxsX2lzb2xf ZmxhZ3MgPSBJU09MQVRFRF9CUk9LRU4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgc21wX21iKCkNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAvLyBpbnRlcnJ1cHRzIGFyZSBlbmFibGVkIGF0IHNvbWUNCiAgICBkYXRhX3RvX3N5bmMg PSAxICAgICAgICAgICAgICAgICAgICAgICAgLy8gcG9pbnQgaGVyZSwgZGF0YV90b19zeW5jIHZh bHVlDQogICAgc21wX21iKCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8vIGlzIHVu ZGV0ZXJtaW5lZCwgQ1BVIDAgbWFrZXMgbm8NCiAgICAvLyBsbF9pc29sX2ZsYWdzKENQVSAxKSB1 cGRhdGVkICAgICAgICAgLy8gYXNzdW1wdGlvbnMgYWJvdXQgaXQNCiAgICBpZiBsbF9pc29sX2Zs YWdzKENQVSAxKSAhPSBJU09MQVRFRCAgICAgLy8NCiAgICAgICAgICBzbXBfY2FsbChDUFUgMSkg ICAgICAgICAgICAgICAgICAgICAgICAgLy8ga2VybmVsIGVudHJ5IGFnYWluDQogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlmIChsbF9pc29sX2ZsYWdz ID09IElTT0xBVEVEKQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAvLyBub3RoaW5nIGhhcHBlbnMNCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgLy8gZXhwbGljaXQgb3IgaW1wbGllZCBiYXJyaWVy cw0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvLyBk YXRhX3RvX3N5bmMgdXBkYXRlZA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAvLyBrZXJuZWwgZXhpdA0KICAgIC8vIENQVSAwIGFzc3VtZXMsIENQVSAx IHdpbGwgc2VlDQogICAgLy8gZGF0YV90b19zeW5jID0gMSB3aGVuIGluIGtlcm5lbA0KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSRUFEIGRhdGFfdG9fc3luYw0K DQo0LiBLZXJuZWwgZW50cnkgcHJvY2Vzc2luZyBoYXBwZW5zIGFmdGVyIHRoZSBjaGVjayBmb3Ig aXNvbGF0aW9uIG9uIHRoZSBzZW5kZXINCnNpZGU6DQoNCmxsX2lzb2xfZmxhZ3MoQ1BVIDEpID0g SVNPTEFURUQNCg0KICAgICAgICAgQ1BVIDAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IENQVSAxDQogICAgLS0tLS0tLS0tLS0tLS0tLS0tICAgICAgICAgICAgICAgICAgICAgIC0tLS0t LS0tLS0tLS0tLS0tDQogICAgZGF0YV90b19zeW5jID0gMSAgICAgICAgICAgICAgICAgICAgICAg IC8vIHVzZXJzcGFjZSBvciBlYXJseSBrZXJuZWwgZW50cnkNCiAgICBzbXBfbWIoKQ0KICAgIGlm IGxsX2lzb2xfZmxhZ3MoQ1BVIDEpICE9IElTT0xBVEVEDQogICAgICAgICAgc21wX2NhbGwoQ1BV IDEpIC8vIHNraXBwZWQgICAgICAgIC8vIHVzZXJzcGFjZSBvciBlYXJseSBrZXJuZWwgZW50cnkN CiAgICAvLyBDUFUgMCBhc3N1bWVzLCBDUFUgMSB3aWxsIHNlZSAgICAgICAgLy8gY29udGludWVz IHVuZGlzdHVyYmVkDQogICAgLy8gZGF0YV90b19zeW5jID0gMSB3aGVuIGluIGtlcm5lbA0KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvLyBrZXJuZWwgZW50cnkN CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLy8gZGF0YV90b19z eW5jIHVuZGV0ZXJtaW5lZA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAvLyBzaG91bGQgbm90IGFjY2VzcyBkYXRhX3RvX3N5bmMgaGVyZQ0KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZiAobGxfaXNvbF9mbGFncyA9PSBJU09M QVRFRCkgew0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBsbF9pc29sX2ZsYWdzID0gSVNPTEFURURfQlJPS0VODQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNtcF9tYigpDQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8vIGRhdGFfdG9fc3luYyB1cGRhdGVkDQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH0NCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUkVBRCBkYXRhX3RvX3N5bmMNCg0KVGhp cyBhbHNvIGFwcGxpZXMgdG8gZXhpdCB0byB1c2Vyc3BhY2UgYWZ0ZXIgZW5hYmxpbmcgaXNvbGF0 aW9uIC0tIG9uY2UNCmxsX2lzb2xfZmxhZ3MgaXMgc2V0IHRvIElTT0xBVEVELCBzeW5jaHJvbml6 YXRpb24gd2lsbCBiZSBtaXNzZWQsIHNvDQpvbmUgZmluYWwgYmFycmllciBzaG91bGQgaGFwcGVu IGJlZm9yZSByZXR1cm5pbmcgdG8gdXNlcnNwYWNlIGFuZA0KZW5hYmxpbmcgaW50ZXJydXB0cyBp biB0aGUgcHJvY2Vzcy4gSW4gdGhpcyBjYXNlICJ1bmx1Y2t5IiB0aW1pbmcgd291bGQNCnJlc3Vs dCBpbiBzbXAgY2FsbCBpbnRlcnJ1cHRpbmcgdXNlcnNwYWNlIHRoYXQgaXMgYWxyZWFkeSBzdXBw b3NlZCB0bw0KYmUgaXNvbGF0ZWQsIGl0IHdpbGwgdHJpZ2dlciBub3JtYWwgaXNvbGF0aW9uIGJy ZWFraW5nIHByb2NlZHVyZSBidXQNCm90aGVyd2lzZSB3aWxsIGJlIGFuIHVucmVtYXJrYWJsZSBz eW5jaHJvbml6YXRpb24gY2FsbC4gT24gdGhlIG90aGVyDQpoYW5kLCBzeW5jaHJvbml6YXRpb24g dGhhdCB3YXMgc3VwcG9zZWQgdG8gaGFwcGVuIGFmdGVyIHNldHRpbmcNCmxsX2lzb2xfZmxhZ3Ms IHdpbGwgYmUgZGVsYXllZCB0byB0aGUgbmV4dCBrZXJuZWwgZW50cnksIHNvIHRoZXJlDQpzaG91 bGQgYmUgbm90aGluZyB0aGF0IG5lZWRzIHN5bmNocm9uaXphdGlvbiBiZXR3ZWVuIHRoZSBlbmQg b2YNCnRhc2tfaXNvbGF0aW9uX2V4aXRfdG9fdXNlcl9tb2RlKCkgYW5kIGVudGVyaW5nIHVzZXJz cGFjZSAoaW50ZXJydXB0cw0KYXJlIG9rLCB0aG91Z2gsIHRoZXkgd2lsbCB0cmlnZ2VyIGlzb2xh dGlvbiBicmVha2luZykuDQoNClRoaXMgaXMgd2h5IEkgaGF2ZSBwbGFjZWQgdGFza19pc29sYXRp b25fa2VybmVsX2VudGVyKCkNCmFuZCB0YXNrX2lzb2xhdGlvbl9leGl0X3RvX3VzZXJfbW9kZSgp IGluIGVudHJ5L2V4aXQgY29kZSwgYW5kIHdoZW4gSQ0KaGF2ZSBzZWVuIGFueSBhbWJpZ3VpdHkg b3IgaGFkIGFueSBkb3VidCBhbGxvd2VkIGR1cGxpY2F0ZSBjYWxscyB0bw0KdGFza19pc29sYXRp b25fa2VybmVsX2VudGVyKCkgb24gZW50cnkuIElmIEkgb3ZlcmRpZCBhbnkgb2YgdGhhdCwgSQ0K d291bGQgYXBwcmVjaWF0ZSBmaXhlcyBhbmQgY2xhcmlmaWNhdGlvbnMsIGhvd2V2ZXIgaXQgc2hv dWxkIGJlIHRha2VuDQppbnRvIGFjY291bnQgdGhhdCBmb3Igbm93IGRpZmZlcmVudCBhcmNoaXRl Y3R1cmVzIGFuZCBldmVuIGRyaXZlcnMgbWF5DQpjYWxsIGNvbW1vbiBhbmQgc3BlY2lmaWMgZnVu Y3Rpb25zIGluIGEgc2xpZ2h0bHkgZGlmZmVyZW50IG9yZGVyLg0KDQpTeW5jaHJvbml6YXRpb24g YWxzbyBhcHBsaWVzIHRvIHBvc3NpYmxlIGVmZmVjdHMgb24gcGlwZWxpbmUgKHdoZW4NCm9yaWdp bmF0aW5nIENQVSB3cml0ZXMgaW5zdHJ1Y3Rpb25zKS4NCg0KVGhlcmUgaXMgYSB2ZXJzaW9uIHVu ZGVyIGRldmVsb3BtZW50IHRoYXQgZGVsYXlzIFRMQiBmbHVzaGVzIGluIHRoaXMNCm1hbm5lci4g T24gYXJtNjQgdGhhdCByZXF1aXJlcyBhIHN3aXRjaCB0byBzb2Z0IFRMQiBmbHVzaGVzLCBhbmQg dGhhdCdzDQphbm90aGVyIHBvdGVudGlhbCBiYWxsIG9mIHdheC4gT24gYXJjaGl0ZWN0dXJlcyB0 aGF0IGFscmVhZHkgdXNlIHNvZnQNClRMQiBmbHVzaGVzLCB0aGlzIHdpbGwgYmUgdW5hdm9pZGFi bGUgYmVjYXVzZSBUTEIgZmx1c2ggYmVjb21lcyBhbm90aGVyDQpJUEksIGFuZCBJUElzIGJyZWFr IGlzb2xhdGlvbi4gTWF5YmUgaXQgd2lsbCBiZSBiZXR0ZXIgdG8gbWFrZSBhDQpzb21ld2hhdCBz bWFydGVyIFRMQiBoYW5kbGluZyBpbiBnZW5lcmFsLCBzbyBpdCB3aWxsIGJlIHBvc3NpYmxlIHRv DQphdm9pZCBib3RoZXJpbmcgdW5yZWxhdGVkIENQVXMuIEJ1dCB0aGVuLCBJIGd1ZXNzLCBoYXJk d2FyZSBtYXkgc3RpbGwNCm92ZXJkbyBpdC5UaGVuIGl0IHdpbGwgYmUgY2xvc2VyIHRvIHRoZSBz aXR1YXRpb24gd2l0aCB0aW1lcnMuIEZvciBub3csDQpJIHdhbnQgdG8gZmlyc3QgZG8gdGhlIG9i dmlvdXMgYW5kIGV4Y2x1ZGUgaXNvbGF0ZWQgdGFza3MgZnJvbSBUTEINCnVwZGF0ZXMgdW50aWwg dGhleSBhcmUgYmFjayBpbiBrZXJuZWwsIGFuZCBkbyBhIHNsb3cgZnVsbCBmbHVzaCBvbg0KaXNv bGF0aW9uIGJyZWFraW5nLg0KDQo+IFlvdSBzaG91bGQgZG9jdW1lbnQgdGhhdCwgaWU6IGV4cGxh aW4gd2h5IHdoYXQgeW91J3JlIGRvaW5nIGlzIHNhZmUuDQoNCkkgdHJpZWQgdG8gZG8gdGhhdCBp biBjb21tZW50cywgaG93ZXZlciB0aGlzIGlzIGNsZWFybHkgaW5zdWZmaWNpZW50DQpkZXNwaXRl IHRoZWlyIHZlcmJvc2l0eS4gV291bGQgaXQgbWFrZSBzZW5zZSB0byBjcmVhdGUgYSBzZXBhcmF0 ZQ0KZG9jdW1lbnRhdGlvbiB0ZXh0IGZpbGU/IEN1cnJlbnQgZG9jdW1lbnRhdGlvbiBzZWVtcyB0 byBiZSBsaWdodCBvbg0KZGV0YWlsZWQgc3BlY2lmaWNhdGlvbnMgZm9yIHRoaW5ncyB1bmRlciBo ZWF2eSBkZXZlbG9wbWVudCBleGNlcHQgZm9yDQpzb21ldGhpbmcgdmVyeSBzaWduaWZpY2FudCB0 aGF0IGFmZmVjdHMgbmVhcmx5IGV2ZXJ5b25lLg0KDQpXb3VsZCBpdCBtYWtlIHNlbnNlIHRvIGlu Y2x1ZGUgYWxsIHNjZW5hcmlvcyBsaWtlIHRoZSBhYm92ZSBwbHVzIGV4aXQNCnRvIHVzZXJzcGFj ZSwgYW5kIGV4aXN0aW5nIHNlcXVlbmNlcyBvZiBjYWxscyB0aGF0IHNob3VsZCBsZWFkIHRvDQpz eW5jaHJvbml6YXRpb24gY2FsbHMsIHRhc2tfaXNvbGF0aW9uX2tlcm5lbF9lbnRlcigpIG9yDQp0 YXNrX2lzb2xhdGlvbl9leGl0X3RvX3VzZXJfbW9kZSgpPw0KDQoNCj4gQWxzbyBCZXdhcmUgdGhv dWdoIHRoYXQgdGhlIGRhdGEgdG8gc3luYyBpbiBxdWVzdGlvbiBkb2Vzbid0IG5lZWQgdG8NCj4g YmUgdmlzaWJsZQ0KPiBpbiB0aGUgZW50cnkgY29kZSBiZWZvcmUgdGFza19pc29sYXRpb25fa2Vy bmVsX2VudGVyKCkuDQoNClJpZ2h0LiBBbmQgYWZ0ZXIgdGFza19pc29sYXRpb25fZXhpdF90b191 c2VyX21vZGUoKSBhcyB3ZWxsLiBUaGlzIGlzDQp3aHkgSSBoYWQgdG8gZGlnIHRocm91Z2ggZW50 cnkvZXhpdCBjb2RlLiBJIGNhbiBwcm9kdWNlIGEgc29tZXdoYXQNCnVzYWJsZSBjYWxsIHNlcXVl bmNlIGRpYWdyYW0uDQoNCklmIGJ5IGFueSBjaGFuY2UgSSBhbSB3cm9uZyB0aGVyZSBzb21ld2hl cmUsIEkgd2VsY29tZSBhbGwgcmVsZXZhbnQNCmNvbW1lbnRzIGFuZCBjb3JyZWN0aW9ucy4NCg0K QXQgdGhpcyBwb2ludCBJIGFtIG5vdCBzdXJlIG9ubHkgYWJvdXQgdGhpbmdzIHRoYXQgYXJlIGNh bGxlZCB3aGVuDQpDT05GSUdfVFJBQ0VfSVJRRkxBR1MgaXMgZW5hYmxlZCwgc2ltcGx5IGJlY2F1 c2UgSSBoYXZlIG5vdCB5ZXQgY2hlY2tlZA0Kd2hhdCB0aGV5IGRlcGVuZCBvbiwgYW5kIHZpcnR1 YWxpemF0aW9uLXNwZWNpZmljIGVudHJ5L2V4aXQgaXMgbm90DQplbnRpcmVseSBjbGVhciB0byBt ZS4gTWF5YmUgZm9yIGNvcnJlY3RuZXNzIHNha2UgSSBzaG91bGQgaGF2ZSBkZWNsYXJlZA0KdGFz ayBpc29sYXRpb24gaW5jb21wYXRpYmxlIHdpdGggdGhvc2UgdW50aWwgSSBlaXRoZXIga25vdyBm b3Igc3VyZSBvcg0KYWRkZWQgd29ya2luZyBzdXBwb3J0IGZvciB0aGVtLg0KDQo+ICBZb3UgbmVl ZCB0byBhdWRpdCBhbGwNCj4gdGhlIGNhbGxlcnMgb2Yga2lja19hbGxfY3B1c19zeW5jKCkuDQoN ClJpZ2h0IG5vdyBpdCdzIGp1c3QgdGhyZWUgcGxhY2VzLg0KDQpkb190dW5lX2NwdWNhY2hlKCkg ZG9lcyB0aGUgcmlnaHQgdGhpbmcsIGtlZXBzIENQVXMgZnJvbSBzZWVpbmcgb2xkDQp2YWx1ZXMg b2YgY2FjaGVwLT5jcHVfY2FjaGUgYXMgdGhleSBhcmUgYmVpbmcgZGVhbGxvY2F0ZWQuDQoNCnBv d2VycGMga3ZtX2FyY2hfZGVzdHJveV92bSgpIGNhcmVzIG9ubHkgYWJvdXQgQ1BVcyB3aXRoIFZD UFVzIG9uIHRoZW0sDQp3aGF0IGZvciBub3cgc2hvdWxkIG5vdCBiZSB1c2VkIHdpdGggaXNvbGF0 aW9uLg0KDQphcm02NCBmbHVzaF9pY2FjaGVfcmFuZ2UoKSBpcyB0aGUgbWFpbiByZWFzb24gd2h5 IHRoaXMgY291bGQgYmUNCnBvdGVudGlhbGx5IHByb2JsZW1hdGljLCBpdCBtYWtlcyBzdXJlIHRo YXQgb3RoZXIgQ1BVcyB3aWxsIG5vdCBydW4NCnN0YWxlIGluc3RydWN0aW9ucywgYW5kIGl0J3Mg Y2FsbGVkIGZyb20gYWxsIHBsYWNlcyB3aGVyZSBjb2RlIGlzDQptb2RpZmllZC4gSWYgaW4gb3Ro ZXIgc2l0dWF0aW9uIHNvbWUga2luZCBvZiBkZWxheWVkIHByb2Nlc3NpbmcgY291bGQNCmJlIHBv c3NpYmxlLCB0aGlzIG9uZSBoYXMgdG8gYmUgZG9uZSBpbiB0aGUgcmlnaHQgc2VxdWVuY2UuIEFu ZCB0aGF0J3MNCnRoZSByZWFzb24gZm9yIGluc3RyX3N5bmMoKSBhZnRlciBzbXBfbWIoKSBpbg0K dGFza19pc29sYXRpb25fa2VybmVsX2VudGVyKCkuIFNpbmNlIGZsdXNoX2ljYWNoZV9yYW5nZSgp IGNvdWxkIHNraXANCnRoaXMgQ1BVLCB3ZSBoYXZlIHRvIHN5bmNocm9uaXplIGl0IGJ5IG91cnNl bHZlcy4NCg0KVGhlcmUgaXMgYW4gaW1wb3J0YW50IGlzc3VlIG9mIG5vdCBoYXZpbmcgZWFybHkg a2VybmVsIGVudHJ5IC8gbGF0ZQ0Ka2VybmVsIGV4aXQgcmVseSBvbiBzb21ldGhpbmcgdGhhdCBt YXkgYmUgc3RhbGUgKHdpdGggYm90aCBkYXRhIGFuZA0KY29kZSkuIFdpdGggdGhlIGFib3ZlIG1l bnRpb25lZCBleGNlcHRpb25zIGZvciBub3csIGl0IGNhbiBiZQ0KZGVtb25zdHJhdGVkIHRoYXQg dGhpcyBpcyBjb3JyZWN0LiBJdCBzaG91bGQgYmUgdGFrZW4gaW50byBhY2NvdW50IHRoYXQNCmV2 ZW4gdGhvdWdoIHN0YXRpYyBrZXlzIGFyZSB1c2VkIGluIGVhcmx5IGtlcm5lbCBlbnRyeSBjb2Rl LCBzdGF0aWMNCmtleXMgc2V0dGluZyBkb2VzIG5vdCBzeW5jaHJvbml6ZSBmbHVzaGVzLCBhbmQg dGhlcmVmb3JlIGFscmVhZHkgc2hvdWxkDQpiZSBkb25lIGJlZm9yZSB1c2UuDQoNClRhc2sgaXNv bGF0aW9uIGluIHZpcnR1YWxpemF0aW9uIGd1ZXN0cyBjYW4gYmUgYSBwZXJmZWN0bHkgdmFsaWQg dGhpbmcNCnRvIHN1cHBvcnQgKGl0IGp1c3QgaGFzIHRvIGJlIHByb3BhZ2F0ZWQgdG8gdGhlIGhv c3QgaWYgcGVybWl0dGVkKSwNCmhvd2V2ZXIgdGhpcyBpcyBzb21ldGhpbmcgSSB3aWxsIHdhbnQg dG8gcmV2aXNpdCBpbiB0aGUgZnV0dXJlLiBGb3INCm5vdywgSSBhc3N1bWUgdGhhdCB2aXJ0dWFs aXphdGlvbi1yZWxhdGVkIGV2ZW50cyBhcmUgbm90IHN1cHBvc2VkIHRvDQpicmVhayBpc29sYXRp b24sIGhvd2V2ZXIgaXQgd291bGQgYmUgbmljZSB0byBiZSByZWFkeSBmb3IgaGFuZGxpbmcgdGhh dA0KcG9zc2liaWxpdHkuDQoNCi0tIA0KQWxleA0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE84CC56202 for ; Wed, 25 Nov 2020 03:22:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EE2C20782 for ; Wed, 25 Nov 2020 03:22:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="p/XYrzu5"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=marvell.com header.i=@marvell.com header.b="Y1EFoFrw"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=marvell.onmicrosoft.com header.i=@marvell.onmicrosoft.com header.b="M2ndBrzA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EE2C20782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=marvell.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Content-ID:In-Reply-To:References: Message-ID:Date:Subject:To:From:Reply-To:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UAnbaatqrSFBWe6R3wRrcLaPXCKSRNd7AtaFv1Fq+1s=; b=p/XYrzu5elm7dnXAKYMnJrIl9 4hXxgKErul7rJhyS/FxVzwrCrZfN2Zq3i52VvmpVRTgBiOTomv+lO/0MrMiocyTKWBXS6VzbSFAKg 9td/ds/1v5eh8w8Ib8zLn/G1qhjQTSbfstUAABi/YVASv71E/nmu7WG9h0ZuXPNSG/wmqYf0VuOtk 8lpIVzsovPtfOfugBrOvWoxLAz9DNQfkhAHh3wmrHgAk9/jcApJB/yywki729uPjysthLEpQlCFnI xO6lAXfzhiJOP48AcAR9RsHwbztfWp7hNsA0TEdS4dHi640DbFyaPX/E4Jv+Fig2uSsW0OESoP0ZA ZAZW+Q3Ow==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khlNI-00081o-5e; Wed, 25 Nov 2020 03:21:44 +0000 Received: from mx0a-0016f401.pphosted.com ([67.231.148.174] helo=mx0b-0016f401.pphosted.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khlND-00081L-PR for linux-arm-kernel@lists.infradead.org; Wed, 25 Nov 2020 03:21:41 +0000 Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0AP3KBl7013934; Tue, 24 Nov 2020 19:20:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=pfpt0220; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=Y1EFoFrw0NDdL2aapeEA9O2VCCSSwM9zU6Nkg9SXQpuu9APKX2p/S9zSlDkzl4LDT6yZ HqQ08BvrUll2TGS5BmE8+JjpQOGegj+byzt7nhsKcR/HhQPK4xWDZeDytcsBdpmNeGT1 KYEIf4NCuwIC3C65kCMOdy7N9MYI6iLJo8JifC7uLuElAp0eDPO847DXXkvCE2/pH++k IJOSr3cMILC2UwUXdSOyb9OLDnwcw22x2llMJ/iPkHI4lzwONp9ge/tE+EzXs6SDSLU/ yUzkRDRH8X/5v+eSbn28dBXFjxv0LK1yAGr+CGPgD2k7fKqxi0PuQeSCtExr7kF/dB8N Gw== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0a-0016f401.pphosted.com with ESMTP id 34y14ucsam-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 24 Nov 2020 19:20:38 -0800 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 24 Nov 2020 19:20:36 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 24 Nov 2020 19:20:37 -0800 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.36.55) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Tue, 24 Nov 2020 19:20:36 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rghtfb/0DCZEhW4B2dHOQW4dlqugAa9ZMPZbsRdK4cKrIgE1rHxsRQ/1sRzZQUj2fiZ1YGIgGw7OMiE+zM1MsHuq3wIgRK+tvj1VcCdikYGZudkezEoxdFM5epAoHL0vijzh9a+xZBqJomQD3YPxGnrQrEhyJT+59jDbr3jmFucfqqFkh+Km0teG0THuUGkuB0sw3tql4YxojH8oCKzPH7YWfj9ZBYZqyTf0cezKPkmPXV/SA1U0AVuCTfkpf838iLBeKV9M4Fbl/4QMm7FKjOdyXb2uxm/L8p+KZKaIvnl7m7+W9AQ143Lyh4LWcaT+Ctk6CBRJfYXv4bTGsElHZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=MpG9DF3Ng7u3dOpYVlJ8ngKA9Cm2s9oHNabG4Wru+ikdCW8RXM/CwwidjFTIx4AKA4C2X9LIJsGqGgT+WYAf1J6GirE+7jeXBYzJMFcTxjtVUzOzsiHTWAE/iu9Q/XASgHCv5M1K/CyK3RJLWZWJ4/czigwW8ZqK+9LHxmYsHPJqJIDMaq0T6giaLzyNO/AfsoSDsLSHTjBRZ8P306ekWVHRN61dsLlu4ADZBUb0/wtGL9jq/PXKiXwOhecRcmNu6vhIm61GGvB9XOEzByGo9a+X6FqGY3sF/HGe0Wp0n+zmCbJJ/bl3A94y1Zqxjt4TGlxCNtsjExF3baPAvZfqTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JmlDDgFD5V/E4vxIzz4pM6tRCjfleqwaAKG2KAcKtYM=; b=M2ndBrzAl3GOCUMh7yBqkALHQYDHmh9kqDC7Qiw5kB4vRj37VxHLop8MNiI92p065bBA7eaSSAsc0nrKhtzkw/bkJGJpm3um6C5DQ3BGk0D28Emb9G7qKIKq5JzwQxRAeuQX5u78Pkb2dXJ/Z73soyEJPRIiANX3ahTAQw+Vpko= Received: from MW2PR18MB2267.namprd18.prod.outlook.com (2603:10b6:907:3::11) by MWHPR18MB1440.namprd18.prod.outlook.com (2603:10b6:320:2c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Wed, 25 Nov 2020 03:20:35 +0000 Received: from MW2PR18MB2267.namprd18.prod.outlook.com ([fe80::e17f:37fb:4eb3:c703]) by MW2PR18MB2267.namprd18.prod.outlook.com ([fe80::e17f:37fb:4eb3:c703%4]) with mapi id 15.20.3564.039; Wed, 25 Nov 2020 03:20:34 +0000 From: Alex Belits To: "frederic@kernel.org" Subject: Re: [EXT] Re: [PATCH v5 9/9] task_isolation: kick_all_cpus_sync: don't kick isolated cpus Thread-Topic: [EXT] Re: [PATCH v5 9/9] task_isolation: kick_all_cpus_sync: don't kick isolated cpus Thread-Index: AQHWwcJHihlNClCxdEOODDjRi5wCaKnWTJeAgAAC64CAAAucAIAB1ToA Date: Wed, 25 Nov 2020 03:20:34 +0000 Message-ID: <45bb2800f2c966ed04ced8dfa85615c377fac4b2.camel@marvell.com> References: <8d887e59ca713726f4fcb25a316e1e932b02823e.camel@marvell.com> <3236b13f42679031960c5605be20664e90e75223.camel@marvell.com> <20201123222907.GC1751@lothringen> <20201123232106.GD1751@lothringen> In-Reply-To: <20201123232106.GD1751@lothringen> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=marvell.com; x-originating-ip: [173.228.7.197] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 55ef8eb2-533d-4ede-c770-08d890f112b5 x-ms-traffictypediagnostic: MWHPR18MB1440: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: B0byi0kQVG8Z+R5XExLn1lGCmWyT7IdrMdAAtstKnoT2oPnQKPv3tqPdXy2BzQP4YAZwrNuuOv8Rw39rZs+9pHRPUtibtZgJ3FYPt76/ay0HX4xYOqSFp/1/4ddbMw8eZNBpOlaaUDvIS8DDfTP8oUlpjbYlZDqrsKtb3pn3zE/uAVeQmflZepH5A21RFK2Wu/TuCIicag4MmJz9TiWzoKMx/vyPCtT3OPlcVs4FOGxpMZUuT2Zz6WYFvWeE9VBRnb5v4zARM991P124JaREMdy0NnFRoFO+BT5xMMrCY2rYpT/hW9LPA9T2pceV+VI8k8DmsRnByvz/ClOERXdfOg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MW2PR18MB2267.namprd18.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(376002)(366004)(39850400004)(396003)(6506007)(83380400001)(7416002)(4326008)(6486002)(6512007)(91956017)(36756003)(5660300002)(76116006)(66476007)(64756008)(66946007)(66556008)(66446008)(86362001)(186003)(26005)(478600001)(71200400001)(2616005)(316002)(54906003)(4001150100001)(2906002)(6916009)(8676002)(8936002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 4gvzcjKq4r5qQCylHcBxvSfgfiTFVF8LxUMCYWskBL1IYxGImcJRp1E+CkpEzn4utYa9c6rTFLmtF0YtDEp1LxsGBOIeZX7W3t+1csM6lOR+GuzJVw0uoNocReydcPX0OpcJ5z/GlTMihUHOM3T8RyHHu3p6xYTwoaZeoXMamBm8oHR3LfWAJEbDmVc9yqBVPX50uWZfixifHHz1N/ukdF5UmgPUVdP8Hid5C61SPV/UNh+NEFNxlrvmvs8wXudkEzgmNBtKAIjZL00tD+VISpYRAKFmbPRUPH18ufl20CYAvV1gO8ZpL52iUTKFkgWXSqfZlcmxYT9A3IXNKmVP3yY6o6BSWvAidc1UHYrYOXWPq6P6D6Bzyislk+tb6s8kSm6SPp/JqEI+hteTNP0TMbzNQZnkaUEPdludIXyKSgunLyCV3voyYZ2v9PALQOQjaWxW8m2aA57hdEvMZsrOtETRHYxPKYxQQwCUmKqbeJprpsror68lCD1QN9t0Eyh/X44RYtWIG4+1OIuvmj2fmvpx1soQ2E33viJr5OLRE2pT/pWBkmsbFK3/D5WNQod9lqQSVmXS0QUeaoTd7Us9Uo5xkbVTI2ubh28A1vg2CCzLDDGtNA+EdBQ27xRrW2GfDtLmYB41mSCdIGG9wGwW48wL2y/tj46VPJ0PF5CRptoS2vLyA0SjVFMO/ZTTw0VLYwIB6P0KeFv/ILVbpcgR0e0pc8kGfgcWTof7q0mW6rTK/dCqPEYVlYhVwItAKvuWjX6ZuPPYEW+DSNQXE/ABKjwA+NVTacI0g1EgeD5Bv/8Lkv0D1sv57tQgg6KzAavXUIu8NwKhz698B+aItbNizrZ1J8qFvpyRDfqnCBWlm/PxFlMzd1hVW2sWLRgDTSzP2BU4YS/0utqN0OOk1grC6g== Content-ID: <6B9838A204A0934595C222765B6BC2D1@namprd18.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW2PR18MB2267.namprd18.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55ef8eb2-533d-4ede-c770-08d890f112b5 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Nov 2020 03:20:34.5307 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Vrk7gYZPmHhU5DNFtoSH5HrS+oyzfcJa7Ri2xtMDzdfTGOXATwsPWauLNOAzJqJXNiKq+O3LnJszUHcxSz8ujQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR18MB1440 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-24_11:2020-11-24, 2020-11-24 signatures=0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_222140_760646_9D9A5416 X-CRM114-Status: GOOD ( 26.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "linux-arch@vger.kernel.org" , "nitesh@redhat.com" , "pauld@redhat.com" , "catalin.marinas@arm.com" , "peterz@infradead.org" , Prasun Kapoor , "mtosatti@redhat.com" , "linux-kernel@vger.kernel.org" , "rostedt@goodmis.org" , "davem@davemloft.net" , "leon@sidebranch.com" , "netdev@vger.kernel.org" , "peterx@redhat.com" , "trix@redhat.com" , "tglx@linutronix.de" , "linux-api@vger.kernel.org" , "will@kernel.org" , "mingo@kernel.org" , "linux-arm-kernel@lists.infradead.org" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 2020-11-24 at 00:21 +0100, Frederic Weisbecker wrote: > On Mon, Nov 23, 2020 at 10:39:34PM +0000, Alex Belits wrote: > > > > This is different from timers. The original design was based on the > > idea that every CPU should be able to enter kernel at any time and > > run > > kernel code with no additional preparation. Then the only solution > > is > > to always do full broadcast and require all CPUs to process it. > > > > What I am trying to introduce is the idea of CPU that is not likely > > to > > run kernel code any soon, and can afford to go through an > > additional > > synchronization procedure on the next entry into kernel. The > > synchronization is not skipped, it simply happens later, early in > > kernel entry code. > > Ah I see, this is ordered that way: > > ll_isol_flags = ISOLATED > > CPU 0 CPU 1 > ------------------ ----------------- > // kernel entry > data_to_sync = 1 ll_isol_flags = ISOLATED_BROKEN > smp_mb() smp_mb() > if ll_isol_flags(CPU 1) == ISOLATED READ data_to_sync > smp_call(CPU 1) > The check for ll_isol_flags(CPU 1) is reversed, and it's a bit more complex. In terms of scenarios, on entry from isolation the following can happen: 1. Kernel entry happens simultaneously with operation that requires synchronization, kernel entry processing happens before the check for isolation on the sender side: ll_isol_flags(CPU 1) = ISOLATED CPU 0 CPU 1 ------------------ ----------------- // kernel entry if (ll_isol_flags == ISOLATED) { ll_isol_flags = ISOLATED_BROKEN data_to_sync = 1 smp_mb() // data_to_sync undetermined smp_mb() } // ll_isol_flags(CPU 1) updated if ll_isol_flags(CPU 1) != ISOLATED // interrupts enabled smp_call(CPU 1) // kernel entry again if (ll_isol_flags == ISOLATED) // nothing happens // explicit or implied barriers // data_to_sync updated // kernel exit // CPU 0 assumes, CPU 1 will see READ data_to_sync // data_to_sync = 1 when in kernel 2. Kernel entry happens simultaneously with operation that requires synchronization, kernel entry processing happens after the check for isolation on the sender side: ll_isol_flags(CPU 1) = ISOLATED CPU 0 CPU 1 ------------------ ----------------- data_to_sync = 1 // kernel entry smp_mb() // data_to_sync undetermined // should not access data_to_sync here if (ll_isol_flags == ISOLATED) { ll_isol_flags = ISOLATED_BROKEN // ll_isol_flags(CPU 1) undetermined smp_mb() // data_to_sync updated if ll_isol_flags(CPU 1) != ISOLATED } // possibly nothing happens // CPU 0 assumes, CPU 1 will see READ data_to_sync // data_to_sync = 1 when in kernel 3. Kernel entry processing completed before the check for isolation on the sender side: ll_isol_flags(CPU 1) = ISOLATED CPU 0 CPU 1 ------------------ ----------------- // kernel entry if (ll_isol_flags == ISOLATED) { ll_isol_flags = ISOLATED_BROKEN smp_mb() } // interrupts are enabled at some data_to_sync = 1 // point here, data_to_sync value smp_mb() // is undetermined, CPU 0 makes no // ll_isol_flags(CPU 1) updated // assumptions about it if ll_isol_flags(CPU 1) != ISOLATED // smp_call(CPU 1) // kernel entry again if (ll_isol_flags == ISOLATED) // nothing happens // explicit or implied barriers // data_to_sync updated // kernel exit // CPU 0 assumes, CPU 1 will see // data_to_sync = 1 when in kernel READ data_to_sync 4. Kernel entry processing happens after the check for isolation on the sender side: ll_isol_flags(CPU 1) = ISOLATED CPU 0 CPU 1 ------------------ ----------------- data_to_sync = 1 // userspace or early kernel entry smp_mb() if ll_isol_flags(CPU 1) != ISOLATED smp_call(CPU 1) // skipped // userspace or early kernel entry // CPU 0 assumes, CPU 1 will see // continues undisturbed // data_to_sync = 1 when in kernel // kernel entry // data_to_sync undetermined // should not access data_to_sync here if (ll_isol_flags == ISOLATED) { ll_isol_flags = ISOLATED_BROKEN smp_mb() // data_to_sync updated } READ data_to_sync This also applies to exit to userspace after enabling isolation -- once ll_isol_flags is set to ISOLATED, synchronization will be missed, so one final barrier should happen before returning to userspace and enabling interrupts in the process. In this case "unlucky" timing would result in smp call interrupting userspace that is already supposed to be isolated, it will trigger normal isolation breaking procedure but otherwise will be an unremarkable synchronization call. On the other hand, synchronization that was supposed to happen after setting ll_isol_flags, will be delayed to the next kernel entry, so there should be nothing that needs synchronization between the end of task_isolation_exit_to_user_mode() and entering userspace (interrupts are ok, though, they will trigger isolation breaking). This is why I have placed task_isolation_kernel_enter() and task_isolation_exit_to_user_mode() in entry/exit code, and when I have seen any ambiguity or had any doubt allowed duplicate calls to task_isolation_kernel_enter() on entry. If I overdid any of that, I would appreciate fixes and clarifications, however it should be taken into account that for now different architectures and even drivers may call common and specific functions in a slightly different order. Synchronization also applies to possible effects on pipeline (when originating CPU writes instructions). There is a version under development that delays TLB flushes in this manner. On arm64 that requires a switch to soft TLB flushes, and that's another potential ball of wax. On architectures that already use soft TLB flushes, this will be unavoidable because TLB flush becomes another IPI, and IPIs break isolation. Maybe it will be better to make a somewhat smarter TLB handling in general, so it will be possible to avoid bothering unrelated CPUs. But then, I guess, hardware may still overdo it.Then it will be closer to the situation with timers. For now, I want to first do the obvious and exclude isolated tasks from TLB updates until they are back in kernel, and do a slow full flush on isolation breaking. > You should document that, ie: explain why what you're doing is safe. I tried to do that in comments, however this is clearly insufficient despite their verbosity. Would it make sense to create a separate documentation text file? Current documentation seems to be light on detailed specifications for things under heavy development except for something very significant that affects nearly everyone. Would it make sense to include all scenarios like the above plus exit to userspace, and existing sequences of calls that should lead to synchronization calls, task_isolation_kernel_enter() or task_isolation_exit_to_user_mode()? > Also Beware though that the data to sync in question doesn't need to > be visible > in the entry code before task_isolation_kernel_enter(). Right. And after task_isolation_exit_to_user_mode() as well. This is why I had to dig through entry/exit code. I can produce a somewhat usable call sequence diagram. If by any chance I am wrong there somewhere, I welcome all relevant comments and corrections. At this point I am not sure only about things that are called when CONFIG_TRACE_IRQFLAGS is enabled, simply because I have not yet checked what they depend on, and virtualization-specific entry/exit is not entirely clear to me. Maybe for correctness sake I should have declared task isolation incompatible with those until I either know for sure or added working support for them. > You need to audit all > the callers of kick_all_cpus_sync(). Right now it's just three places. do_tune_cpucache() does the right thing, keeps CPUs from seeing old values of cachep->cpu_cache as they are being deallocated. powerpc kvm_arch_destroy_vm() cares only about CPUs with VCPUs on them, what for now should not be used with isolation. arm64 flush_icache_range() is the main reason why this could be potentially problematic, it makes sure that other CPUs will not run stale instructions, and it's called from all places where code is modified. If in other situation some kind of delayed processing could be possible, this one has to be done in the right sequence. And that's the reason for instr_sync() after smp_mb() in task_isolation_kernel_enter(). Since flush_icache_range() could skip this CPU, we have to synchronize it by ourselves. There is an important issue of not having early kernel entry / late kernel exit rely on something that may be stale (with both data and code). With the above mentioned exceptions for now, it can be demonstrated that this is correct. It should be taken into account that even though static keys are used in early kernel entry code, static keys setting does not synchronize flushes, and therefore already should be done before use. Task isolation in virtualization guests can be a perfectly valid thing to support (it just has to be propagated to the host if permitted), however this is something I will want to revisit in the future. For now, I assume that virtualization-related events are not supposed to break isolation, however it would be nice to be ready for handling that possibility. -- Alex _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel