From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28363C8 for ; Sun, 3 Dec 2023 09:22:02 -0800 (PST) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VxgQUtK_1701624118; Received: from 30.27.123.200(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgQUtK_1701624118) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 01:22:00 +0800 Message-ID: <649a3bc4-58bb-1dc8-85fb-a56e47b3d5c9@linux.alibaba.com> Date: Mon, 4 Dec 2023 01:21:58 +0800 Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: Weird EROFS data corruption To: Juhyung Park Cc: Gao Xiang , linux-erofs@lists.ozlabs.org, linux-f2fs-devel@lists.sourceforge.net, linux-crypto@vger.kernel.org, Yann Collet References: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> From: Gao Xiang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2023/12/4 01:01, Juhyung Park wrote: > Hi Gao, > > On Mon, Dec 4, 2023 at 1:52 AM Gao Xiang wrote: >> >> Hi Juhyung, >> >> On 2023/12/4 00:22, Juhyung Park wrote: >>> (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a >>> while ago, which may mean that this is not specific to EROFS: >>> https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+1EEjK+cw@mail.gmail.com/ >>> ) >>> >>> Hi. >>> >>> I'm encountering a very weird EROFS data corruption. >>> >>> I noticed when I build an EROFS image for AOSP development, the device >>> would randomly not boot from a certain build. >>> After inspecting the log, I noticed that a file got corrupted. >> >> Is it observed on your laptop (i7-1185G7), yes? or some other arm64 >> device? > > Yes, only on my laptop. The arm64 device seems fine. > The reason that it would not boot was that the host machine (my > laptop) was repacking the EROFS image wrongfully. > > The workflow is something like this: > Server-built EROFS AOSP image -> Image copied to laptop -> Laptop > mounts the EROFS image -> Copies the entire content to a scratch > directory (CORRUPT!) -> Changes some files -> mkfs.erofs > > So the device is not responsible for the corruption, the laptop is. Ok. > >> >>> >>> After adding a hash check during the build flow, I noticed that EROFS >>> would randomly read data wrong. >>> >>> I now have a reliable method of reproducing the issue, but here's the >>> funny/weird part: it's only happening on my laptop (i7-1185G7). This >>> is not happening with my 128 cores buildfarm machine (Threadripper >>> 3990X).> >>> I first suspected a hardware issue, but: >>> a. The laptop had its motherboard replaced recently (due to a failing >>> physical Type-C port). >>> b. The laptop passes memory test (memtest86). >>> c. This happens on all kernel versions from v5.4 to the latest v6.6 >>> including my personal custom builds and Canonical's official Ubuntu >>> kernels. >>> d. This happens on different host SSDs and file-system combinations. >>> e. This only happens on LZ4. LZ4HC doesn't trigger the issue. >>> f. This only happens when mounting the image natively by the kernel. >>> Using fuse with erofsfuse is fine. >> >> I think it's a weird issue with inplace decompression because you said >> it depends on the hardware. In addition, with your dataset sadly I >> cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). > > As I feared. Bummer :( > >> >> What is the difference between these two machines? just different CPU or >> they have some other difference like different compliers? > > I fully and exclusively control both devices, and the setup is almost the same. > Same Ubuntu version, kernel/compiler version. > > But as I said, on my laptop, the issue happens on kernels that someone > else (Canonical) built, so I don't think it matters. The only thing I could say is that the kernel side has optimized inplace decompression compared to fuse so that it will reuse the same buffer for decompression but with a safe margin (according to the current lz4 decompression implementation). It shouldn't behave different just due to different CPUs. Let me find more clues later, also maybe we should introduce a way for users to turn off this if needed. Thanks, Gao Xiang From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4F9C5C4167B for ; Sun, 3 Dec 2023 17:22:20 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Sjts66njFz3cST for ; Mon, 4 Dec 2023 04:22:18 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.alibaba.com (client-ip=115.124.30.100; helo=out30-100.freemail.mail.aliyun.com; envelope-from=hsiangkao@linux.alibaba.com; receiver=lists.ozlabs.org) Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Sjtry3qtVz2ykC for ; Mon, 4 Dec 2023 04:22:07 +1100 (AEDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VxgQUtK_1701624118; Received: from 30.27.123.200(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgQUtK_1701624118) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 01:22:00 +0800 Message-ID: <649a3bc4-58bb-1dc8-85fb-a56e47b3d5c9@linux.alibaba.com> Date: Mon, 4 Dec 2023 01:21:58 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: Weird EROFS data corruption To: Juhyung Park References: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> From: Gao Xiang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: linux-erofs@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development of Linux EROFS file system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yann Collet , linux-erofs@lists.ozlabs.org, linux-crypto@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Errors-To: linux-erofs-bounces+linux-erofs=archiver.kernel.org@lists.ozlabs.org Sender: "Linux-erofs" On 2023/12/4 01:01, Juhyung Park wrote: > Hi Gao, > > On Mon, Dec 4, 2023 at 1:52 AM Gao Xiang wrote: >> >> Hi Juhyung, >> >> On 2023/12/4 00:22, Juhyung Park wrote: >>> (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a >>> while ago, which may mean that this is not specific to EROFS: >>> https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+1EEjK+cw@mail.gmail.com/ >>> ) >>> >>> Hi. >>> >>> I'm encountering a very weird EROFS data corruption. >>> >>> I noticed when I build an EROFS image for AOSP development, the device >>> would randomly not boot from a certain build. >>> After inspecting the log, I noticed that a file got corrupted. >> >> Is it observed on your laptop (i7-1185G7), yes? or some other arm64 >> device? > > Yes, only on my laptop. The arm64 device seems fine. > The reason that it would not boot was that the host machine (my > laptop) was repacking the EROFS image wrongfully. > > The workflow is something like this: > Server-built EROFS AOSP image -> Image copied to laptop -> Laptop > mounts the EROFS image -> Copies the entire content to a scratch > directory (CORRUPT!) -> Changes some files -> mkfs.erofs > > So the device is not responsible for the corruption, the laptop is. Ok. > >> >>> >>> After adding a hash check during the build flow, I noticed that EROFS >>> would randomly read data wrong. >>> >>> I now have a reliable method of reproducing the issue, but here's the >>> funny/weird part: it's only happening on my laptop (i7-1185G7). This >>> is not happening with my 128 cores buildfarm machine (Threadripper >>> 3990X).> >>> I first suspected a hardware issue, but: >>> a. The laptop had its motherboard replaced recently (due to a failing >>> physical Type-C port). >>> b. The laptop passes memory test (memtest86). >>> c. This happens on all kernel versions from v5.4 to the latest v6.6 >>> including my personal custom builds and Canonical's official Ubuntu >>> kernels. >>> d. This happens on different host SSDs and file-system combinations. >>> e. This only happens on LZ4. LZ4HC doesn't trigger the issue. >>> f. This only happens when mounting the image natively by the kernel. >>> Using fuse with erofsfuse is fine. >> >> I think it's a weird issue with inplace decompression because you said >> it depends on the hardware. In addition, with your dataset sadly I >> cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). > > As I feared. Bummer :( > >> >> What is the difference between these two machines? just different CPU or >> they have some other difference like different compliers? > > I fully and exclusively control both devices, and the setup is almost the same. > Same Ubuntu version, kernel/compiler version. > > But as I said, on my laptop, the issue happens on kernels that someone > else (Canonical) built, so I don't think it matters. The only thing I could say is that the kernel side has optimized inplace decompression compared to fuse so that it will reuse the same buffer for decompression but with a safe margin (according to the current lz4 decompression implementation). It shouldn't behave different just due to different CPUs. Let me find more clues later, also maybe we should introduce a way for users to turn off this if needed. Thanks, Gao Xiang From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 96D2CC4167B for ; Sun, 3 Dec 2023 17:22:40 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1r9qAv-0005OU-II; Sun, 03 Dec 2023 17:22:37 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r9qAt-0005OO-Ol for linux-f2fs-devel@lists.sourceforge.net; Sun, 03 Dec 2023 17:22:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: From:References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=3KdTh7qu2wyEVz7mbvzfcC6qo4vY1PLBcIu+HyBe8cE=; b=HJg9qMbS5wKq1K2dRcsQjwlbDG wCvHsUFfqxZOSYaB1uuy55ournnFD800eJWtg6JnzQJM/ZqOgW5mp8hOX0lPPPEgqhh0OxO2fEMQx E4bOsfW6SlwazlvDR4COnwKz608Cq1a4NE9GhnqBSdVJVBS5cI6nBL9VHTa5u4GsVn+k=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3KdTh7qu2wyEVz7mbvzfcC6qo4vY1PLBcIu+HyBe8cE=; b=TdsTl8bmVBtToI38KJM/k3Cyav G66pGxQMiS3+OAB0ZJ1JweSKx7XjEbBqh7VTdfu51vcUTEpwJaPbHuGatb1FCngyL1aYJGD7aHRbx H8HMlmL9MGCi55L9zXp9oOJlaTLUQ6oeJ/l1TprBSJC50sknWr64WLb+MFNaB3sQMvnM=; Received: from out30-133.freemail.mail.aliyun.com ([115.124.30.133]) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1r9qAo-00033v-HQ for linux-f2fs-devel@lists.sourceforge.net; Sun, 03 Dec 2023 17:22:36 +0000 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R191e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=ay29a033018045170; MF=hsiangkao@linux.alibaba.com; NM=1; PH=DS; RN=6; SR=0; TI=SMTPD_---0VxgQUtK_1701624118; Received: from 30.27.123.200(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgQUtK_1701624118) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 01:22:00 +0800 Message-ID: <649a3bc4-58bb-1dc8-85fb-a56e47b3d5c9@linux.alibaba.com> Date: Mon, 4 Dec 2023 01:21:58 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 To: Juhyung Park References: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> From: Gao Xiang In-Reply-To: X-Headers-End: 1r9qAo-00033v-HQ Subject: Re: [f2fs-dev] Weird EROFS data corruption X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yann Collet , Gao Xiang , linux-erofs@lists.ozlabs.org, linux-crypto@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net CgpPbiAyMDIzLzEyLzQgMDE6MDEsIEp1aHl1bmcgUGFyayB3cm90ZToKPiBIaSBHYW8sCj4gCj4g T24gTW9uLCBEZWMgNCwgMjAyMyBhdCAxOjUy4oCvQU0gR2FvIFhpYW5nIDxoc2lhbmdrYW9AbGlu dXguYWxpYmFiYS5jb20+IHdyb3RlOgo+Pgo+PiBIaSBKdWh5dW5nLAo+Pgo+PiBPbiAyMDIzLzEy LzQgMDA6MjIsIEp1aHl1bmcgUGFyayB3cm90ZToKPj4+IChDYydpbmcgZjJmcyBhbmQgY3J5cHRv IGFzIEkndmUgbm90aWNlZCBzb21ldGhpbmcgc2ltaWxhciB3aXRoIGYyZnMgYQo+Pj4gd2hpbGUg YWdvLCB3aGljaCBtYXkgbWVhbiB0aGF0IHRoaXMgaXMgbm90IHNwZWNpZmljIHRvIEVST0ZTOgo+ Pj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvYWxsL0NBRDE0K2YybkJadExmTEM2Q3dOamdDT3VS UlJqd3p0dHAzRDNpSzRPZisxRUVqSytjd0BtYWlsLmdtYWlsLmNvbS8KPj4+ICkKPj4+Cj4+PiBI aS4KPj4+Cj4+PiBJJ20gZW5jb3VudGVyaW5nIGEgdmVyeSB3ZWlyZCBFUk9GUyBkYXRhIGNvcnJ1 cHRpb24uCj4+Pgo+Pj4gSSBub3RpY2VkIHdoZW4gSSBidWlsZCBhbiBFUk9GUyBpbWFnZSBmb3Ig QU9TUCBkZXZlbG9wbWVudCwgdGhlIGRldmljZQo+Pj4gd291bGQgcmFuZG9tbHkgbm90IGJvb3Qg ZnJvbSBhIGNlcnRhaW4gYnVpbGQuCj4+PiBBZnRlciBpbnNwZWN0aW5nIHRoZSBsb2csIEkgbm90 aWNlZCB0aGF0IGEgZmlsZSBnb3QgY29ycnVwdGVkLgo+Pgo+PiBJcyBpdCBvYnNlcnZlZCBvbiB5 b3VyIGxhcHRvcCAoaTctMTE4NUc3KSwgeWVzPyBvciBzb21lIG90aGVyIGFybTY0Cj4+IGRldmlj ZT8KPiAKPiBZZXMsIG9ubHkgb24gbXkgbGFwdG9wLiBUaGUgYXJtNjQgZGV2aWNlIHNlZW1zIGZp bmUuCj4gVGhlIHJlYXNvbiB0aGF0IGl0IHdvdWxkIG5vdCBib290IHdhcyB0aGF0IHRoZSBob3N0 IG1hY2hpbmUgKG15Cj4gbGFwdG9wKSB3YXMgcmVwYWNraW5nIHRoZSBFUk9GUyBpbWFnZSB3cm9u Z2Z1bGx5Lgo+IAo+IFRoZSB3b3JrZmxvdyBpcyBzb21ldGhpbmcgbGlrZSB0aGlzOgo+IFNlcnZl ci1idWlsdCBFUk9GUyBBT1NQIGltYWdlIC0+IEltYWdlIGNvcGllZCB0byBsYXB0b3AgLT4gTGFw dG9wCj4gbW91bnRzIHRoZSBFUk9GUyBpbWFnZSAtPiBDb3BpZXMgdGhlIGVudGlyZSBjb250ZW50 IHRvIGEgc2NyYXRjaAo+IGRpcmVjdG9yeSAoQ09SUlVQVCEpIC0+IENoYW5nZXMgc29tZSBmaWxl cyAtPiBta2ZzLmVyb2ZzCj4gCj4gU28gdGhlIGRldmljZSBpcyBub3QgcmVzcG9uc2libGUgZm9y IHRoZSBjb3JydXB0aW9uLCB0aGUgbGFwdG9wIGlzLgoKT2suCgo+IAo+Pgo+Pj4KPj4+IEFmdGVy IGFkZGluZyBhIGhhc2ggY2hlY2sgZHVyaW5nIHRoZSBidWlsZCBmbG93LCBJIG5vdGljZWQgdGhh dCBFUk9GUwo+Pj4gd291bGQgcmFuZG9tbHkgcmVhZCBkYXRhIHdyb25nLgo+Pj4KPj4+IEkgbm93 IGhhdmUgYSByZWxpYWJsZSBtZXRob2Qgb2YgcmVwcm9kdWNpbmcgdGhlIGlzc3VlLCBidXQgaGVy ZSdzIHRoZQo+Pj4gZnVubnkvd2VpcmQgcGFydDogaXQncyBvbmx5IGhhcHBlbmluZyBvbiBteSBs YXB0b3AgKGk3LTExODVHNykuIFRoaXMKPj4+IGlzIG5vdCBoYXBwZW5pbmcgd2l0aCBteSAxMjgg Y29yZXMgYnVpbGRmYXJtIG1hY2hpbmUgKFRocmVhZHJpcHBlcgo+Pj4gMzk5MFgpLj4KPj4+IEkg Zmlyc3Qgc3VzcGVjdGVkIGEgaGFyZHdhcmUgaXNzdWUsIGJ1dDoKPj4+IGEuIFRoZSBsYXB0b3Ag aGFkIGl0cyBtb3RoZXJib2FyZCByZXBsYWNlZCByZWNlbnRseSAoZHVlIHRvIGEgZmFpbGluZwo+ Pj4gcGh5c2ljYWwgVHlwZS1DIHBvcnQpLgo+Pj4gYi4gVGhlIGxhcHRvcCBwYXNzZXMgbWVtb3J5 IHRlc3QgKG1lbXRlc3Q4NikuCj4+PiBjLiBUaGlzIGhhcHBlbnMgb24gYWxsIGtlcm5lbCB2ZXJz aW9ucyBmcm9tIHY1LjQgdG8gdGhlIGxhdGVzdCB2Ni42Cj4+PiBpbmNsdWRpbmcgbXkgcGVyc29u YWwgY3VzdG9tIGJ1aWxkcyBhbmQgQ2Fub25pY2FsJ3Mgb2ZmaWNpYWwgVWJ1bnR1Cj4+PiBrZXJu ZWxzLgo+Pj4gZC4gVGhpcyBoYXBwZW5zIG9uIGRpZmZlcmVudCBob3N0IFNTRHMgYW5kIGZpbGUt c3lzdGVtIGNvbWJpbmF0aW9ucy4KPj4+IGUuIFRoaXMgb25seSBoYXBwZW5zIG9uIExaNC4gTFo0 SEMgZG9lc24ndCB0cmlnZ2VyIHRoZSBpc3N1ZS4KPj4+IGYuIFRoaXMgb25seSBoYXBwZW5zIHdo ZW4gbW91bnRpbmcgdGhlIGltYWdlIG5hdGl2ZWx5IGJ5IHRoZSBrZXJuZWwuCj4+PiBVc2luZyBm dXNlIHdpdGggZXJvZnNmdXNlIGlzIGZpbmUuCj4+Cj4+IEkgdGhpbmsgaXQncyBhIHdlaXJkIGlz c3VlIHdpdGggaW5wbGFjZSBkZWNvbXByZXNzaW9uIGJlY2F1c2UgeW91IHNhaWQKPj4gaXQgZGVw ZW5kcyBvbiB0aGUgaGFyZHdhcmUuICBJbiBhZGRpdGlvbiwgd2l0aCB5b3VyIGRhdGFzZXQgc2Fk bHkgSQo+PiBjYW5ub3QgcmVwcm9kdWNlIG9uIG15IGxvY2FsIHNlcnZlciAoWGVvbihSKSBDUFUg RTUtMjY4MiB2NCkuCj4gCj4gQXMgSSBmZWFyZWQuIEJ1bW1lciA6KAo+IAo+Pgo+PiBXaGF0IGlz IHRoZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlc2UgdHdvIG1hY2hpbmVzPyBqdXN0IGRpZmZlcmVu dCBDUFUgb3IKPj4gdGhleSBoYXZlIHNvbWUgb3RoZXIgZGlmZmVyZW5jZSBsaWtlIGRpZmZlcmVu dCBjb21wbGllcnM/Cj4gCj4gSSBmdWxseSBhbmQgZXhjbHVzaXZlbHkgY29udHJvbCBib3RoIGRl dmljZXMsIGFuZCB0aGUgc2V0dXAgaXMgYWxtb3N0IHRoZSBzYW1lLgo+IFNhbWUgVWJ1bnR1IHZl cnNpb24sIGtlcm5lbC9jb21waWxlciB2ZXJzaW9uLgo+IAo+IEJ1dCBhcyBJIHNhaWQsIG9uIG15 IGxhcHRvcCwgdGhlIGlzc3VlIGhhcHBlbnMgb24ga2VybmVscyB0aGF0IHNvbWVvbmUKPiBlbHNl IChDYW5vbmljYWwpIGJ1aWx0LCBzbyBJIGRvbid0IHRoaW5rIGl0IG1hdHRlcnMuCgpUaGUgb25s eSB0aGluZyBJIGNvdWxkIHNheSBpcyB0aGF0IHRoZSBrZXJuZWwgc2lkZSBoYXMgb3B0aW1pemVk CmlucGxhY2UgZGVjb21wcmVzc2lvbiBjb21wYXJlZCB0byBmdXNlIHNvIHRoYXQgaXQgd2lsbCBy ZXVzZSB0aGUKc2FtZSBidWZmZXIgZm9yIGRlY29tcHJlc3Npb24gYnV0IHdpdGggYSBzYWZlIG1h cmdpbiAoYWNjb3JkaW5nIHRvCnRoZSBjdXJyZW50IGx6NCBkZWNvbXByZXNzaW9uIGltcGxlbWVu dGF0aW9uKS4gIEl0IHNob3VsZG4ndCBiZWhhdmUKZGlmZmVyZW50IGp1c3QgZHVlIHRvIGRpZmZl cmVudCBDUFVzLiAgTGV0IG1lIGZpbmQgbW9yZSBjbHVlcwpsYXRlciwgYWxzbyBtYXliZSB3ZSBz aG91bGQgaW50cm9kdWNlIGEgd2F5IGZvciB1c2VycyB0byB0dXJuIG9mZgp0aGlzIGlmIG5lZWRl ZC4KClRoYW5rcywKR2FvIFhpYW5nCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18KTGludXgtZjJmcy1kZXZlbCBtYWlsaW5nIGxpc3QKTGludXgtZjJmcy1k ZXZlbEBsaXN0cy5zb3VyY2Vmb3JnZS5uZXQKaHR0cHM6Ly9saXN0cy5zb3VyY2Vmb3JnZS5uZXQv bGlzdHMvbGlzdGluZm8vbGludXgtZjJmcy1kZXZlbAo=