From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C885C433F5 for ; Mon, 29 Nov 2021 16:33:24 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-401-jlkdaxS-NGGZQIua2QJsqg-1; Mon, 29 Nov 2021 11:33:20 -0500 X-MC-Unique: jlkdaxS-NGGZQIua2QJsqg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E8B451934101; Mon, 29 Nov 2021 16:33:00 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 98BC25D6B1; Mon, 29 Nov 2021 16:32:58 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id CB2211809C89; Mon, 29 Nov 2021 16:32:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 1AT7Zs4v032764 for ; Mon, 29 Nov 2021 02:35:54 -0500 Received: by smtp.corp.redhat.com (Postfix) id 2B0342166B3F; Mon, 29 Nov 2021 07:35:54 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2387F2166B2D for ; Mon, 29 Nov 2021 07:35:51 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30EA01066681 for ; Mon, 29 Nov 2021 07:35:51 +0000 (UTC) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-587-XT177KO3Oh6OmhhKw8vqkQ-1; Mon, 29 Nov 2021 02:35:46 -0500 X-MC-Unique: XT177KO3Oh6OmhhKw8vqkQ-1 Received: from dggpemm500024.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4J2cZN300Kz91GK; Mon, 29 Nov 2021 15:35:08 +0800 (CST) Received: from dggpemm500023.china.huawei.com (7.185.36.83) by dggpemm500024.china.huawei.com (7.185.36.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 29 Nov 2021 15:35:39 +0800 Received: from dggpemm500023.china.huawei.com ([7.185.36.83]) by dggpemm500023.china.huawei.com ([7.185.36.83]) with mapi id 15.01.2308.020; Mon, 29 Nov 2021 15:35:39 +0800 From: "zhaozixuan (C)" To: Paul Moore Subject: Re: [PATCH] audit: accelerate audit rule filter Thread-Topic: [PATCH] audit: accelerate audit rule filter Thread-Index: AQHX4UnhpG+k05BPVEqO6SQrGWmNIKwaIv3w Date: Mon, 29 Nov 2021 07:35:39 +0000 Message-ID: <4aac209c744848a38bb2003d601083e4@huawei.com> References: <20211123075001.3676-1-zhaozixuan2@huawei.com> In-Reply-To: Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.174.176.92] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-MIME-Autoconverted: from base64 to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id 1AT7Zs4v032764 X-loop: linux-audit@redhat.com X-Mailman-Approved-At: Mon, 29 Nov 2021 11:32:54 -0500 Cc: "linux-audit@redhat.com" , "linux-kernel@vger.kernel.org" , "eparis@redhat.com" X-BeenThere: linux-audit@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Linux Audit Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=linux-audit-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit >On Tue, Nov 23, 2021 at 2:50 AM Zixuan Zhao wrote: >> We used lat_syscall of lmbench3 to test the performance impact of this >> patch. We changed the number of rules and run lat_syscall with 1000 >> repetitions at each test. Syscalls measured by lat_syscall are not >> monitored by rules. >> >> Before this optimization: >> >> null read write stat fstat open >> 0 rules 1.87ms 2.74ms 2.56ms 26.31ms 4.13ms 69.66ms >> 10 rules 2.15ms 3.13ms 3.32ms 26.99ms 4.16ms 74.70ms >> 20 rules 2.45ms 3.97ms 3.82ms 27.05ms 4.60ms 76.35ms >> 30 rules 2.64ms 4.52ms 3.95ms 30.30ms 4.94ms 78.94ms >> 40 rules 2.83ms 4.97ms 4.23ms 32.16ms 5.40ms 81.88ms >> 50 rules 3.00ms 5.30ms 4.84ms 33.49ms 5.79ms 83.20ms >> 100 rules 4.24ms 9.75ms 7.42ms 37.68ms 6.55ms 93.70ms >> 160 rules 5.50ms 16.89ms 12.18ms 51.53ms 17.45ms 155.40ms >> >> After this optimization: >> >> null read write stat fstat open >> 0 rules 1.81ms 2.84ms 2.42ms 27.70ms 4.15ms 69.10ms >> 10 rules 1.97ms 2.83ms 2.69ms 27.70ms 4.15ms 69.30ms >> 20 rules 1.72ms 2.91ms 2.41ms 26.49ms 3.91ms 71.19ms >> 30 rules 1.85ms 2.94ms 2.48ms 26.27ms 3.97ms 71.43ms >> 40 rules 1.88ms 2.94ms 2.78ms 26.85ms 4.08ms 69.79ms >> 50 rules 1.86ms 3.17ms 3.08ms 26.25ms 4.03ms 72.32ms >> 100 rules 1.84ms 3.00ms 2.81ms 26.25ms 3.98ms 70.25ms >> 160 rules 1.92ms 3.32ms 3.06ms 26.81ms 4.57ms 71.41ms >> >> As the result shown above, the syscall latencies increase as the >> number of rules increases, while with the patch the latencies remain stable. >> This could help when a user adds many audit rules for purposes (such >> as attack tracing or process behavior recording) but suffers from low >> performance. > >I have general concerns about trading memory and complexity for performance gains, but beyond that the numbers you posted above don't yet make sense to me. Thanks for your reply. The memory cost of this patch is less than 4KB (1820 bytes on x64 and 3640 bytes on compatible x86_64) which is trivial in many cases. Besides, syscalls are called frequently on a system so a small optimization could bring a good income. >Why are the latency increases due to rule count not similar across the different syscalls? For example, I would think that if the increase in syscall latency was >directly attributed to the audit rule processing then the increase on the "open" syscall should be similar to that of the "null" syscall. In other phrasing, if we >can process 160 rules in ~4ms in the "null" case, why does it take us ~86ms in the "open" case? As to the test result, we did some investigations and concluded two reasons: 1. The chosen rule sets were not very suitable. Though they were not hit by syscalls being measured, some of them were hit by other processes, which reduced the system performance and affected the test result; 2. The routine of lat_syscall is much more complicated than we thought. It called many other syscalls during the test, which may cause the result not to be linear. Due to the reasons above, we did another test. We modified audit rule sets and made sure they wouldn't be hit at runtime. Then, we added ktime_get_real_ts64 to auditsc.c to record the time of executing __audit_syscall_exit. We ran "stat" syscall 10000 times for each rule set and recorded the time interval. The result is shown below: Before this optimization: rule set time 0 rules 3843.96ns 1 rules 13119.08ns 10 rules 14003.13ns 20 rules 15420.18ns 30 rules 17284.84ns 40 rules 19010.67ns 50 rules 21112.63ns 100 rules 25815.02ns 130 rules 29447.09ns After this optimization: rule set time 0 rules 3597.78ns 1 rules 13498.73ns 10 rules 13122.57ns 20 rules 12874.88ns 30 rules 14351.99ns 40 rules 14181.07ns 50 rules 13806.45ns 100 rules 13890.85ns 130 rules 14441.45ns As the result showed, the interval is linearly increased before optimization while the interval remains stable after optimization. Note that audit skips some operations if there are no rules, so there is a gap between 0 rule and 1 rule set. -- Linux-audit mailing list Linux-audit@redhat.com https://listman.redhat.com/mailman/listinfo/linux-audit From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 417DEC433F5 for ; Mon, 29 Nov 2021 07:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240097AbhK2HlL (ORCPT ); Mon, 29 Nov 2021 02:41:11 -0500 Received: from szxga02-in.huawei.com ([45.249.212.188]:16313 "EHLO szxga02-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240827AbhK2HjJ (ORCPT ); Mon, 29 Nov 2021 02:39:09 -0500 Received: from dggpemm500024.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4J2cZN300Kz91GK; Mon, 29 Nov 2021 15:35:08 +0800 (CST) Received: from dggpemm500023.china.huawei.com (7.185.36.83) by dggpemm500024.china.huawei.com (7.185.36.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 29 Nov 2021 15:35:39 +0800 Received: from dggpemm500023.china.huawei.com ([7.185.36.83]) by dggpemm500023.china.huawei.com ([7.185.36.83]) with mapi id 15.01.2308.020; Mon, 29 Nov 2021 15:35:39 +0800 From: "zhaozixuan (C)" To: Paul Moore CC: "eparis@redhat.com" , "linux-audit@redhat.com" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] audit: accelerate audit rule filter Thread-Topic: [PATCH] audit: accelerate audit rule filter Thread-Index: AQHX4UnhpG+k05BPVEqO6SQrGWmNIKwaIv3w Date: Mon, 29 Nov 2021 07:35:39 +0000 Message-ID: <4aac209c744848a38bb2003d601083e4@huawei.com> References: <20211123075001.3676-1-zhaozixuan2@huawei.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.174.176.92] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Pk9uIFR1ZSwgTm92IDIzLCAyMDIxIGF0IDI6NTAgQU0gWml4dWFuIFpoYW8gPHpoYW96aXh1YW4y QGh1YXdlaS5jb20+IHdyb3RlOg0KPj4gV2UgdXNlZCBsYXRfc3lzY2FsbCBvZiBsbWJlbmNoMyB0 byB0ZXN0IHRoZSBwZXJmb3JtYW5jZSBpbXBhY3Qgb2YgdGhpcyAgDQo+PiBwYXRjaC4gV2UgY2hh bmdlZCB0aGUgbnVtYmVyIG9mIHJ1bGVzIGFuZCBydW4gbGF0X3N5c2NhbGwgd2l0aCAxMDAwICAN Cj4+IHJlcGV0aXRpb25zIGF0IGVhY2ggdGVzdC4gU3lzY2FsbHMgbWVhc3VyZWQgYnkgbGF0X3N5 c2NhbGwgYXJlIG5vdCAgDQo+PiBtb25pdG9yZWQgYnkgcnVsZXMuDQo+Pg0KPj4gQmVmb3JlIHRo aXMgb3B0aW1pemF0aW9uOg0KPj4NCj4+ICAgICAgICAgICAgICBudWxsICAgICByZWFkICAgIHdy aXRlICAgICBzdGF0ICAgIGZzdGF0ICAgICAgb3Blbg0KPj4gICAwIHJ1bGVzICAxLjg3bXMgICAy Ljc0bXMgICAyLjU2bXMgICAyNi4zMW1zICA0LjEzbXMgICA2OS42Nm1zDQo+PiAgMTAgcnVsZXMg IDIuMTVtcyAgIDMuMTNtcyAgIDMuMzJtcyAgIDI2Ljk5bXMgIDQuMTZtcyAgIDc0LjcwbXMNCj4+ ICAyMCBydWxlcyAgMi40NW1zICAgMy45N21zICAgMy44Mm1zICAgMjcuMDVtcyAgNC42MG1zICAg NzYuMzVtcw0KPj4gIDMwIHJ1bGVzICAyLjY0bXMgICA0LjUybXMgICAzLjk1bXMgICAzMC4zMG1z ICA0Ljk0bXMgICA3OC45NG1zDQo+PiAgNDAgcnVsZXMgIDIuODNtcyAgIDQuOTdtcyAgIDQuMjNt cyAgIDMyLjE2bXMgIDUuNDBtcyAgIDgxLjg4bXMNCj4+ICA1MCBydWxlcyAgMy4wMG1zICAgNS4z MG1zICAgNC44NG1zICAgMzMuNDltcyAgNS43OW1zICAgODMuMjBtcw0KPj4gMTAwIHJ1bGVzICA0 LjI0bXMgICA5Ljc1bXMgICA3LjQybXMgICAzNy42OG1zICA2LjU1bXMgICA5My43MG1zDQo+PiAx NjAgcnVsZXMgIDUuNTBtcyAgIDE2Ljg5bXMgIDEyLjE4bXMgIDUxLjUzbXMgIDE3LjQ1bXMgIDE1 NS40MG1zDQo+Pg0KPj4gQWZ0ZXIgdGhpcyBvcHRpbWl6YXRpb246DQo+Pg0KPj4gICAgICAgICAg ICAgIG51bGwgICAgIHJlYWQgICAgd3JpdGUgICAgIHN0YXQgICAgZnN0YXQgICAgICBvcGVuDQo+ PiAgIDAgcnVsZXMgIDEuODFtcyAgIDIuODRtcyAgIDIuNDJtcyAgMjcuNzBtcyAgIDQuMTVtcyAg IDY5LjEwbXMNCj4+ICAxMCBydWxlcyAgMS45N21zICAgMi44M21zICAgMi42OW1zICAyNy43MG1z ICAgNC4xNW1zICAgNjkuMzBtcw0KPj4gIDIwIHJ1bGVzICAxLjcybXMgICAyLjkxbXMgICAyLjQx bXMgIDI2LjQ5bXMgICAzLjkxbXMgICA3MS4xOW1zDQo+PiAgMzAgcnVsZXMgIDEuODVtcyAgIDIu OTRtcyAgIDIuNDhtcyAgMjYuMjdtcyAgIDMuOTdtcyAgIDcxLjQzbXMNCj4+ICA0MCBydWxlcyAg MS44OG1zICAgMi45NG1zICAgMi43OG1zICAyNi44NW1zICAgNC4wOG1zICAgNjkuNzltcw0KPj4g IDUwIHJ1bGVzICAxLjg2bXMgICAzLjE3bXMgICAzLjA4bXMgIDI2LjI1bXMgICA0LjAzbXMgICA3 Mi4zMm1zDQo+PiAxMDAgcnVsZXMgIDEuODRtcyAgIDMuMDBtcyAgIDIuODFtcyAgMjYuMjVtcyAg IDMuOThtcyAgIDcwLjI1bXMNCj4+IDE2MCBydWxlcyAgMS45Mm1zICAgMy4zMm1zICAgMy4wNm1z ICAyNi44MW1zICAgNC41N21zICAgNzEuNDFtcw0KPj4NCj4+IEFzIHRoZSByZXN1bHQgc2hvd24g YWJvdmUsIHRoZSBzeXNjYWxsIGxhdGVuY2llcyBpbmNyZWFzZSBhcyAgdGhlIA0KPj4gbnVtYmVy ICBvZiBydWxlcyBpbmNyZWFzZXMsIHdoaWxlIHdpdGggdGhlIHBhdGNoIHRoZSBsYXRlbmNpZXMg cmVtYWluIHN0YWJsZS4NCj4+ICBUaGlzIGNvdWxkIGhlbHAgd2hlbiBhIHVzZXIgYWRkcyBtYW55 IGF1ZGl0IHJ1bGVzIGZvciBwdXJwb3NlcyAoc3VjaCANCj4+IGFzICBhdHRhY2sgdHJhY2luZyBv ciBwcm9jZXNzIGJlaGF2aW9yIHJlY29yZGluZykgYnV0IHN1ZmZlcnMgZnJvbSBsb3cgIA0KPj4g cGVyZm9ybWFuY2UuDQo+DQo+SSBoYXZlIGdlbmVyYWwgY29uY2VybnMgYWJvdXQgdHJhZGluZyBt ZW1vcnkgYW5kIGNvbXBsZXhpdHkgZm9yIHBlcmZvcm1hbmNlIGdhaW5zLCBidXQgYmV5b25kIHRo YXQgdGhlIG51bWJlcnMgeW91IHBvc3RlZCBhYm92ZSBkb24ndCB5ZXQgbWFrZSBzZW5zZSB0byBt ZS4NCg0KVGhhbmtzIGZvciB5b3VyIHJlcGx5Lg0KDQpUaGUgbWVtb3J5IGNvc3Qgb2YgdGhpcyBw YXRjaCBpcyBsZXNzIHRoYW4gNEtCICgxODIwIGJ5dGVzIG9uIHg2NCBhbmQNCiAzNjQwIGJ5dGVz IG9uIGNvbXBhdGlibGUgeDg2XzY0KSB3aGljaCBpcyB0cml2aWFsIGluIG1hbnkgY2FzZXMuDQog QmVzaWRlcywgc3lzY2FsbHMgYXJlIGNhbGxlZCBmcmVxdWVudGx5IG9uIGEgc3lzdGVtIHNvIGEg c21hbGwNCiBvcHRpbWl6YXRpb24gY291bGQgYnJpbmcgYSBnb29kIGluY29tZS4NCg0KPldoeSBh cmUgdGhlIGxhdGVuY3kgaW5jcmVhc2VzIGR1ZSB0byBydWxlIGNvdW50IG5vdCBzaW1pbGFyIGFj cm9zcyB0aGUgZGlmZmVyZW50IHN5c2NhbGxzPyBGb3IgZXhhbXBsZSwgSSB3b3VsZCB0aGluayB0 aGF0IGlmIHRoZSBpbmNyZWFzZSBpbiBzeXNjYWxsIGxhdGVuY3kgd2FzID5kaXJlY3RseSBhdHRy aWJ1dGVkIHRvIHRoZSBhdWRpdCBydWxlIHByb2Nlc3NpbmcgdGhlbiB0aGUgaW5jcmVhc2Ugb24g dGhlICJvcGVuIiBzeXNjYWxsIHNob3VsZCBiZSBzaW1pbGFyIHRvIHRoYXQgb2YgdGhlICJudWxs IiBzeXNjYWxsLiAgSW4gb3RoZXIgcGhyYXNpbmcsIGlmIHdlID5jYW4gcHJvY2VzcyAxNjAgcnVs ZXMgaW4gfjRtcyBpbiB0aGUgIm51bGwiIGNhc2UsIHdoeSBkb2VzIGl0IHRha2UgdXMgfjg2bXMg aW4gdGhlICJvcGVuIiBjYXNlPw0KDQpBcyB0byB0aGUgdGVzdCByZXN1bHQsIHdlIGRpZCBzb21l IGludmVzdGlnYXRpb25zIGFuZCBjb25jbHVkZWQgdHdvDQogcmVhc29uczoNCjEuIFRoZSBjaG9z ZW4gcnVsZSBzZXRzIHdlcmUgbm90IHZlcnkgc3VpdGFibGUuIFRob3VnaCB0aGV5IHdlcmUgbm90 IGhpdA0KIGJ5IHN5c2NhbGxzIGJlaW5nIG1lYXN1cmVkLCBzb21lIG9mIHRoZW0gd2VyZSBoaXQg Ynkgb3RoZXIgcHJvY2Vzc2VzLA0KIHdoaWNoIHJlZHVjZWQgdGhlIHN5c3RlbSBwZXJmb3JtYW5j ZSBhbmQgYWZmZWN0ZWQgdGhlIHRlc3QgcmVzdWx0Ow0KMi4gVGhlIHJvdXRpbmUgb2YgbGF0X3N5 c2NhbGwgaXMgbXVjaCBtb3JlIGNvbXBsaWNhdGVkIHRoYW4gd2UgdGhvdWdodC4gSXQNCiBjYWxs ZWQgbWFueSBvdGhlciBzeXNjYWxscyBkdXJpbmcgdGhlIHRlc3QsIHdoaWNoIG1heSBjYXVzZSB0 aGUgcmVzdWx0DQogbm90IHRvIGJlIGxpbmVhci4NCg0KRHVlIHRvIHRoZSByZWFzb25zIGFib3Zl LCB3ZSBkaWQgYW5vdGhlciB0ZXN0LiBXZSBtb2RpZmllZCBhdWRpdCBydWxlIHNldHMNCiBhbmQg bWFkZSBzdXJlIHRoZXkgd291bGRuJ3QgYmUgaGl0IGF0IHJ1bnRpbWUuIFRoZW4sIHdlIGFkZGVk DQoga3RpbWVfZ2V0X3JlYWxfdHM2NCB0byBhdWRpdHNjLmMgdG8gcmVjb3JkIHRoZSB0aW1lIG9m IGV4ZWN1dGluZw0KIF9fYXVkaXRfc3lzY2FsbF9leGl0LiBXZSByYW4gInN0YXQiIHN5c2NhbGwg MTAwMDAgdGltZXMgZm9yIGVhY2ggcnVsZSBzZXQNCiBhbmQgcmVjb3JkZWQgdGhlIHRpbWUgaW50 ZXJ2YWwuIFRoZSByZXN1bHQgaXMgc2hvd24gYmVsb3c6DQoNCkJlZm9yZSB0aGlzIG9wdGltaXph dGlvbjoNCg0KcnVsZSBzZXQgICAgICAgICAgdGltZQ0KICAwIHJ1bGVzICAgICAzODQzLjk2bnMN CiAgMSBydWxlcyAgICAxMzExOS4wOG5zDQogMTAgcnVsZXMgICAgMTQwMDMuMTNucw0KIDIwIHJ1 bGVzICAgIDE1NDIwLjE4bnMNCiAzMCBydWxlcyAgICAxNzI4NC44NG5zDQogNDAgcnVsZXMgICAg MTkwMTAuNjducw0KIDUwIHJ1bGVzICAgIDIxMTEyLjYzbnMNCjEwMCBydWxlcyAgICAyNTgxNS4w Mm5zDQoxMzAgcnVsZXMgICAgMjk0NDcuMDlucw0KDQpBZnRlciB0aGlzIG9wdGltaXphdGlvbjoN Cg0KIHJ1bGUgc2V0ICAgICAgICAgIHRpbWUNCiAgMCBydWxlcyAgICAgMzU5Ny43OG5zDQogIDEg cnVsZXMgICAgMTM0OTguNzNucw0KIDEwIHJ1bGVzICAgIDEzMTIyLjU3bnMNCiAyMCBydWxlcyAg ICAxMjg3NC44OG5zDQogMzAgcnVsZXMgICAgMTQzNTEuOTlucw0KIDQwIHJ1bGVzICAgIDE0MTgx LjA3bnMNCiA1MCBydWxlcyAgICAxMzgwNi40NW5zDQoxMDAgcnVsZXMgICAgMTM4OTAuODVucw0K MTMwIHJ1bGVzICAgIDE0NDQxLjQ1bnMNCg0KQXMgdGhlIHJlc3VsdCBzaG93ZWQsIHRoZSBpbnRl cnZhbCBpcyBsaW5lYXJseSBpbmNyZWFzZWQgYmVmb3JlDQogb3B0aW1pemF0aW9uIHdoaWxlIHRo ZSBpbnRlcnZhbCByZW1haW5zIHN0YWJsZSBhZnRlciBvcHRpbWl6YXRpb24uIE5vdGUgDQogdGhh dCBhdWRpdCBza2lwcyBzb21lIG9wZXJhdGlvbnMgaWYgdGhlcmUgYXJlIG5vIHJ1bGVzLCBzbyB0 aGVyZSBpcyBhIGdhcA0KIGJldHdlZW4gMCBydWxlIGFuZCAxIHJ1bGUgc2V0Lg0K