From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2231C4167B for ; Fri, 3 Nov 2023 16:22:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234339AbjKCQV7 (ORCPT ); Fri, 3 Nov 2023 12:21:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230121AbjKCQV7 (ORCPT ); Fri, 3 Nov 2023 12:21:59 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E374CA for ; Fri, 3 Nov 2023 09:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699028472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cNgudPCc13D4/tjUOv7XPhTpnXTcLS65Xsn8KNXaBsc=; b=FEY8/S/ezQ2gqLle70SneHmHCM4mdELWBIvqij3LrsiHm///3ED9MqYeT8KJd7ufwALQQn ekjJiEmycd3nUzuMV5WmkQ9767YRyv9LYTx4ArsIaToU+ASaRQVapp+N45/LGKTcbvd5MB jzvV11ozhwviKvQaVstwM0ZhnmMTqLo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-362-qTo86_hcP5C2SOhjH-WgDg-1; Fri, 03 Nov 2023 12:21:07 -0400 X-MC-Unique: qTo86_hcP5C2SOhjH-WgDg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 77B522932483; Fri, 3 Nov 2023 16:21:05 +0000 (UTC) Received: from fedora (unknown [10.72.120.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A91BC2026D4C; Fri, 3 Nov 2023 16:20:56 +0000 (UTC) Date: Sat, 4 Nov 2023 00:20:51 +0800 From: Ming Lei To: Ed Tsai =?utf-8?B?KOiUoeWul+i7kik=?= Cc: "matthias.bgg@gmail.com" , "angelogioacchino.delregno@collabora.com" , "axboe@kernel.dk" , Will Shiu =?utf-8?B?KOioseaBreeRnCk=?= , Peter Wang =?utf-8?B?KOeOi+S/oeWPiyk=?= , "linux-block@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Alice Chao =?utf-8?B?KOi2meePruWdhyk=?= , "linux-mediatek@lists.infradead.org" , wsd_upstream , Casper Li =?utf-8?B?KOadjuS4reamrik=?= , Chun-Hung Wu =?utf-8?B?KOW3q+mnv+Wujyk=?= , Powen Kao =?utf-8?B?KOmrmOS8r+aWhyk=?= , Naomi Chu =?utf-8?B?KOacseipoOeUsCk=?= , "linux-arm-kernel@lists.infradead.org" , Stanley Chu =?utf-8?B?KOacseWOn+mZnik=?= , ming.lei@redhat.com Subject: Re: [PATCH 1/1] block: Check the queue limit before bio submitting Message-ID: References: <20231025092255.27930-1-ed.tsai@mediatek.com> <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Wed, Nov 01, 2023 at 02:23:26AM +0000, Ed Tsai (蔡宗軒) wrote: > On Wed, 2023-10-25 at 17:22 +0800, ed.tsai@mediatek.com wrote: > > From: Ed Tsai > > > > Referring to commit 07173c3ec276 ("block: enable multipage bvecs"), > > each bio_vec now holds more than one page, potentially exceeding > > 1MB in size and causing alignment issues with the queue limit. > > > > In a sequential read/write scenario, the file system maximizes the > > bio's capacity before submitting. However, misalignment with the > > queue limit can result in the bio being split into smaller I/O > > operations. > > > > For instance, assuming the maximum I/O size is set to 512KB and the > > memory is highly fragmented, resulting in each bio containing only > > one 2-pages bio_vec (i.e., bi_size = 1028KB). This would cause the > > bio to be split into two 512KB portions and one 4KB portion. As a > > result, the originally expected continuous large I/O operations are > > interspersed with many small I/O operations. > > > > To address this issue, this patch adds a check for the max_sectors > > before submitting the bio. This allows the upper layers to > > proactively > > detect and handle alignment issues. > > > > I performed the Antutu V10 Storage Test on a UFS 4.0 device, which > > resulted in a significant improvement in the Sequential test: > > > > Sequential Read (average of 5 rounds): > > Original: 3033.7 MB/sec > > Patched: 3520.9 MB/sec > > > > Sequential Write (average of 5 rounds): > > Original: 2225.4 MB/sec > > Patched: 2800.3 MB/sec > > > > Signed-off-by: Ed Tsai > > --- > > block/bio.c | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/block/bio.c b/block/bio.c > > index 816d412c06e9..a4a1f775b9ea 100644 > > --- a/block/bio.c > > +++ b/block/bio.c > > @@ -1227,6 +1227,7 @@ static int __bio_iov_iter_get_pages(struct bio > > *bio, struct iov_iter *iter) > > iov_iter_extraction_t extraction_flags = 0; > > unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt; > > unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; > > + struct queue_limits *lim = &bdev_get_queue(bio->bi_bdev)- > > >limits; > > struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; > > struct page **pages = (struct page **)bv; > > ssize_t size, left; > > @@ -1275,6 +1276,11 @@ static int __bio_iov_iter_get_pages(struct bio > > *bio, struct iov_iter *iter) > > struct page *page = pages[i]; > > > > len = min_t(size_t, PAGE_SIZE - offset, left); > > + if (bio->bi_iter.bi_size + len > > > + lim->max_sectors << SECTOR_SHIFT) { > > + ret = left; > > + break; > > + } > > if (bio_op(bio) == REQ_OP_ZONE_APPEND) { > > ret = bio_iov_add_zone_append_page(bio, page, > > len, > > offset); > > -- > > 2.18.0 > > > > > Hi Jens, > > Just to clarify any potential confusion, I would like to provide > further details based on the assumed scenario mentioned above. > > When the upper layer continuously sends 1028KB full-sized bios for > sequential reads, the Block Layer sees the following sequence: > submit bio: size = 1028KB, start LBA = n > submit bio: size = 1028KB, start LBA = n + 1028KB > submit bio: size = 1028KB, start LBA = n + 2056KB > ... > > However, due to the queue limit restricting the I/O size to a maximum > of 512KB, the Block Layer splits into the following sequence: > submit bio: size = 512KB, start LBA = n > submit bio: size = 512KB, start LBA = n + 512KB > submit bio: size = 4KB, start LBA = n + 1024KB > submit bio: size = 512KB, start LBA = n + 1028KB > submit bio: size = 512KB, start LBA = n + 1540KB > submit bio: size = 4KB, start LBA = n + 2052KB > submit bio: size = 512KB, start LBA = n + 2056KB > submit bio: size = 512KB, start LBA = n + 2568KB > submit bio: size = 4KB, start LBA = n + 3080KB > ... > > The original expectation was for the storage to receive large, > contiguous requests. However, due to non-alignment, many small I/O > requests are generated. This problem is easily visible because the > user pages passed in are often allocated by the buddy system as order 0 > pages during page faults, resulting in highly non-contiguous memory. If order 0 page is added to bio, the multipage bvec becomes nop basically(256bvec holds 256 pages), then how can it make a difference for you? > > As observed in the Antutu Sequential Read test below, it is similar to > the description above where the splitting caused by the queue limit > leaves small requests sandwiched in between: > > block_bio_queue: 8,32 R 86925864 + 2144 [Thread-51] > block_split: 8,32 R 86925864 / 86926888 [Thread-51] > block_split: 8,32 R 86926888 / 86927912 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86925864 + 1024 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86926888 + 1024 [Thread-51] > block_bio_queue: 8,32 R 86928008 + 2144 [Thread-51] > block_split: 8,32 R 86928008 / 86929032 [Thread-51] > block_split: 8,32 R 86929032 / 86930056 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86928008 + 1024 [Thread-51] > block_rq_issue: 8,32 R 49152 () 86927912 + 96 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86929032 + 1024 [Thread-51] > block_bio_queue: 8,32 R 86930152 + 2112 [Thread-51] > block_split: 8,32 R 86930152 / 86931176 [Thread-51] > block_split: 8,32 R 86931176 / 86932200 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86930152 + 1024 [Thread-51] > block_rq_issue: 8,32 R 49152 () 86930056 + 96 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86931176 + 1024 [Thread-51] > block_bio_queue: 8,32 R 86932264 + 2096 [Thread-51] > block_split: 8,32 R 86932264 / 86933288 [Thread-51] > block_split: 8,32 R 86933288 / 86934312 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86932264 + 1024 [Thread-51] > block_rq_issue: 8,32 R 32768 () 86932200 + 64 [Thread-51] > block_rq_issue: 8,32 R 524288 () 86933288 + 1024 [Thread-51] > > I simply prevents non-aligned situations in bio_iov_iter_get_pages. But there is still 4KB IO left if you limit max bio size is 512KB, then how does this 4KB IO finally go in case of 1028KB IO? > Besides making the upper layer application aware of the queue limit, I > would appreciate any other directions or suggestions you may have. The problem is related with IO size from application. If you send unaligned IO, you can't avoid the last IO with small size, no matter if block layer bio split is involved or not. Your patch just lets __bio_iov_iter_get_pages split the bio, and you still have 4KB left finally when application submits 1028KB, right? Then I don't understand why your patch improves sequential IO performance. Thanks, Ming From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 374EBC4167D for ; Fri, 3 Nov 2023 16:21:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sOLR7QBd4gtCpF4NxpC/IrI1yFzYTBkdrGkyrUCJlrQ=; b=E5KNt+czkSDkXC LyhwhTzXmlW66eGbhu3lWj7O2KEF4zZbdl99FNeftnRTCFxijnGYr5FfnhFFEThFaXdKJMGxQCx+/ jtW9h2xEmbTxtpplOMPZdQpXce1pP2mgyID+u21GxDnMZfslwdtHuvwg9hDVvfKBqJ4/10FfkwEWb vlXpYff4suYYGobAxxKEvxJJ2fao0ot0wsOrjNqwPZ/ONA/gPuNFiPvWzRKdXfJmReECocbVDZz3y dueQQymwfDnv4g/rF3uITbsjVn9IvqwL2vJWsZiohcXTyD2ng43FivmUL36iQFG6WfLFJzX9hDbVg PtTMRXLORT3Ob2xQDIkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qywv9-00BmSE-2P; Fri, 03 Nov 2023 16:21:19 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qywv6-00BmRH-1i for linux-arm-kernel@lists.infradead.org; Fri, 03 Nov 2023 16:21:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699028472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cNgudPCc13D4/tjUOv7XPhTpnXTcLS65Xsn8KNXaBsc=; b=FEY8/S/ezQ2gqLle70SneHmHCM4mdELWBIvqij3LrsiHm///3ED9MqYeT8KJd7ufwALQQn ekjJiEmycd3nUzuMV5WmkQ9767YRyv9LYTx4ArsIaToU+ASaRQVapp+N45/LGKTcbvd5MB jzvV11ozhwviKvQaVstwM0ZhnmMTqLo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-362-qTo86_hcP5C2SOhjH-WgDg-1; Fri, 03 Nov 2023 12:21:07 -0400 X-MC-Unique: qTo86_hcP5C2SOhjH-WgDg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 77B522932483; Fri, 3 Nov 2023 16:21:05 +0000 (UTC) Received: from fedora (unknown [10.72.120.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A91BC2026D4C; Fri, 3 Nov 2023 16:20:56 +0000 (UTC) Date: Sat, 4 Nov 2023 00:20:51 +0800 From: Ming Lei To: Ed Tsai =?utf-8?B?KOiUoeWul+i7kik=?= Cc: "matthias.bgg@gmail.com" , "angelogioacchino.delregno@collabora.com" , "axboe@kernel.dk" , Will Shiu =?utf-8?B?KOioseaBreeRnCk=?= , Peter Wang =?utf-8?B?KOeOi+S/oeWPiyk=?= , "linux-block@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Alice Chao =?utf-8?B?KOi2meePruWdhyk=?= , "linux-mediatek@lists.infradead.org" , wsd_upstream , Casper Li =?utf-8?B?KOadjuS4reamrik=?= , Chun-Hung Wu =?utf-8?B?KOW3q+mnv+Wujyk=?= , Powen Kao =?utf-8?B?KOmrmOS8r+aWhyk=?= , Naomi Chu =?utf-8?B?KOacseipoOeUsCk=?= , "linux-arm-kernel@lists.infradead.org" , Stanley Chu =?utf-8?B?KOacseWOn+mZnik=?= , ming.lei@redhat.com Subject: Re: [PATCH 1/1] block: Check the queue limit before bio submitting Message-ID: References: <20231025092255.27930-1-ed.tsai@mediatek.com> <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231103_092116_655952_B152EB07 X-CRM114-Status: GOOD ( 36.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBOb3YgMDEsIDIwMjMgYXQgMDI6MjM6MjZBTSArMDAwMCwgRWQgVHNhaSAo6JSh5a6X 6LuSKSB3cm90ZToKPiBPbiBXZWQsIDIwMjMtMTAtMjUgYXQgMTc6MjIgKzA4MDAsIGVkLnRzYWlA bWVkaWF0ZWsuY29tIHdyb3RlOgo+ID4gRnJvbTogRWQgVHNhaSA8ZWQudHNhaUBtZWRpYXRlay5j b20+Cj4gPiAKPiA+IFJlZmVycmluZyB0byBjb21taXQgMDcxNzNjM2VjMjc2ICgiYmxvY2s6IGVu YWJsZSBtdWx0aXBhZ2UgYnZlY3MiKSwKPiA+IGVhY2ggYmlvX3ZlYyBub3cgaG9sZHMgbW9yZSB0 aGFuIG9uZSBwYWdlLCBwb3RlbnRpYWxseSBleGNlZWRpbmcKPiA+IDFNQiBpbiBzaXplIGFuZCBj YXVzaW5nIGFsaWdubWVudCBpc3N1ZXMgd2l0aCB0aGUgcXVldWUgbGltaXQuCj4gPiAKPiA+IElu IGEgc2VxdWVudGlhbCByZWFkL3dyaXRlIHNjZW5hcmlvLCB0aGUgZmlsZSBzeXN0ZW0gbWF4aW1p emVzIHRoZQo+ID4gYmlvJ3MgY2FwYWNpdHkgYmVmb3JlIHN1Ym1pdHRpbmcuIEhvd2V2ZXIsIG1p c2FsaWdubWVudCB3aXRoIHRoZQo+ID4gcXVldWUgbGltaXQgY2FuIHJlc3VsdCBpbiB0aGUgYmlv IGJlaW5nIHNwbGl0IGludG8gc21hbGxlciBJL08KPiA+IG9wZXJhdGlvbnMuCj4gPiAKPiA+IEZv ciBpbnN0YW5jZSwgYXNzdW1pbmcgdGhlIG1heGltdW0gSS9PIHNpemUgaXMgc2V0IHRvIDUxMktC IGFuZCB0aGUKPiA+IG1lbW9yeSBpcyBoaWdobHkgZnJhZ21lbnRlZCwgcmVzdWx0aW5nIGluIGVh Y2ggYmlvIGNvbnRhaW5pbmcgb25seQo+ID4gb25lIDItcGFnZXMgYmlvX3ZlYyAoaS5lLiwgYmlf c2l6ZSA9IDEwMjhLQikuIFRoaXMgd291bGQgY2F1c2UgdGhlCj4gPiBiaW8gdG8gYmUgc3BsaXQg aW50byB0d28gNTEyS0IgcG9ydGlvbnMgYW5kIG9uZSA0S0IgcG9ydGlvbi4gQXMgYQo+ID4gcmVz dWx0LCB0aGUgb3JpZ2luYWxseSBleHBlY3RlZCBjb250aW51b3VzIGxhcmdlIEkvTyBvcGVyYXRp b25zIGFyZQo+ID4gaW50ZXJzcGVyc2VkIHdpdGggbWFueSBzbWFsbCBJL08gb3BlcmF0aW9ucy4K PiA+IAo+ID4gVG8gYWRkcmVzcyB0aGlzIGlzc3VlLCB0aGlzIHBhdGNoIGFkZHMgYSBjaGVjayBm b3IgdGhlIG1heF9zZWN0b3JzCj4gPiBiZWZvcmUgc3VibWl0dGluZyB0aGUgYmlvLiBUaGlzIGFs bG93cyB0aGUgdXBwZXIgbGF5ZXJzIHRvCj4gPiBwcm9hY3RpdmVseQo+ID4gZGV0ZWN0IGFuZCBo YW5kbGUgYWxpZ25tZW50IGlzc3Vlcy4KPiA+IAo+ID4gSSBwZXJmb3JtZWQgdGhlIEFudHV0dSBW MTAgU3RvcmFnZSBUZXN0IG9uIGEgVUZTIDQuMCBkZXZpY2UsIHdoaWNoCj4gPiByZXN1bHRlZCBp biBhIHNpZ25pZmljYW50IGltcHJvdmVtZW50IGluIHRoZSBTZXF1ZW50aWFsIHRlc3Q6Cj4gPiAK PiA+IFNlcXVlbnRpYWwgUmVhZCAoYXZlcmFnZSBvZiA1IHJvdW5kcyk6Cj4gPiBPcmlnaW5hbDog MzAzMy43IE1CL3NlYwo+ID4gUGF0Y2hlZDogMzUyMC45IE1CL3NlYwo+ID4gCj4gPiBTZXF1ZW50 aWFsIFdyaXRlIChhdmVyYWdlIG9mIDUgcm91bmRzKToKPiA+IE9yaWdpbmFsOiAyMjI1LjQgTUIv c2VjCj4gPiBQYXRjaGVkOiAyODAwLjMgTUIvc2VjCj4gPiAKPiA+IFNpZ25lZC1vZmYtYnk6IEVk IFRzYWkgPGVkLnRzYWlAbWVkaWF0ZWsuY29tPgo+ID4gLS0tCj4gPiAgYmxvY2svYmlvLmMgfCA2 ICsrKysrKwo+ID4gIDEgZmlsZSBjaGFuZ2VkLCA2IGluc2VydGlvbnMoKykKPiA+IAo+ID4gZGlm ZiAtLWdpdCBhL2Jsb2NrL2Jpby5jIGIvYmxvY2svYmlvLmMKPiA+IGluZGV4IDgxNmQ0MTJjMDZl OS4uYTRhMWY3NzViOWVhIDEwMDY0NAo+ID4gLS0tIGEvYmxvY2svYmlvLmMKPiA+ICsrKyBiL2Js b2NrL2Jpby5jCj4gPiBAQCAtMTIyNyw2ICsxMjI3LDcgQEAgc3RhdGljIGludCBfX2Jpb19pb3Zf aXRlcl9nZXRfcGFnZXMoc3RydWN0IGJpbwo+ID4gKmJpbywgc3RydWN0IGlvdl9pdGVyICppdGVy KQo+ID4gIAlpb3ZfaXRlcl9leHRyYWN0aW9uX3QgZXh0cmFjdGlvbl9mbGFncyA9IDA7Cj4gPiAg CXVuc2lnbmVkIHNob3J0IG5yX3BhZ2VzID0gYmlvLT5iaV9tYXhfdmVjcyAtIGJpby0+YmlfdmNu dDsKPiA+ICAJdW5zaWduZWQgc2hvcnQgZW50cmllc19sZWZ0ID0gYmlvLT5iaV9tYXhfdmVjcyAt IGJpby0+YmlfdmNudDsKPiA+ICsJc3RydWN0IHF1ZXVlX2xpbWl0cyAqbGltID0gJmJkZXZfZ2V0 X3F1ZXVlKGJpby0+YmlfYmRldiktCj4gPiA+bGltaXRzOwo+ID4gIAlzdHJ1Y3QgYmlvX3ZlYyAq YnYgPSBiaW8tPmJpX2lvX3ZlYyArIGJpby0+YmlfdmNudDsKPiA+ICAJc3RydWN0IHBhZ2UgKipw YWdlcyA9IChzdHJ1Y3QgcGFnZSAqKilidjsKPiA+ICAJc3NpemVfdCBzaXplLCBsZWZ0Owo+ID4g QEAgLTEyNzUsNiArMTI3NiwxMSBAQCBzdGF0aWMgaW50IF9fYmlvX2lvdl9pdGVyX2dldF9wYWdl cyhzdHJ1Y3QgYmlvCj4gPiAqYmlvLCBzdHJ1Y3QgaW92X2l0ZXIgKml0ZXIpCj4gPiAgCQlzdHJ1 Y3QgcGFnZSAqcGFnZSA9IHBhZ2VzW2ldOwo+ID4gIAo+ID4gIAkJbGVuID0gbWluX3Qoc2l6ZV90 LCBQQUdFX1NJWkUgLSBvZmZzZXQsIGxlZnQpOwo+ID4gKwkJaWYgKGJpby0+YmlfaXRlci5iaV9z aXplICsgbGVuID4KPiA+ICsJCSAgICBsaW0tPm1heF9zZWN0b3JzIDw8IFNFQ1RPUl9TSElGVCkg ewo+ID4gKwkJCXJldCA9IGxlZnQ7Cj4gPiArCQkJYnJlYWs7Cj4gPiArCQl9Cj4gPiAgCQlpZiAo YmlvX29wKGJpbykgPT0gUkVRX09QX1pPTkVfQVBQRU5EKSB7Cj4gPiAgCQkJcmV0ID0gYmlvX2lv dl9hZGRfem9uZV9hcHBlbmRfcGFnZShiaW8sIHBhZ2UsCj4gPiBsZW4sCj4gPiAgCQkJCQlvZmZz ZXQpOwo+ID4gLS0gCj4gPiAyLjE4LjAKPiA+IAo+IAo+IAo+IEhpIEplbnMsCj4gCj4gSnVzdCB0 byBjbGFyaWZ5IGFueSBwb3RlbnRpYWwgY29uZnVzaW9uLCBJIHdvdWxkIGxpa2UgdG8gcHJvdmlk ZQo+IGZ1cnRoZXIgZGV0YWlscyBiYXNlZCBvbiB0aGUgYXNzdW1lZCBzY2VuYXJpbyBtZW50aW9u ZWQgYWJvdmUuCj4gCj4gV2hlbiB0aGUgdXBwZXIgbGF5ZXIgY29udGludW91c2x5IHNlbmRzIDEw MjhLQiBmdWxsLXNpemVkIGJpb3MgZm9yCj4gc2VxdWVudGlhbCByZWFkcywgdGhlIEJsb2NrIExh eWVyIHNlZXMgdGhlIGZvbGxvd2luZyBzZXF1ZW5jZToKPiAJc3VibWl0IGJpbzogc2l6ZSA9IDEw MjhLQiwgc3RhcnQgTEJBID0gbgo+IAlzdWJtaXQgYmlvOiBzaXplID0gMTAyOEtCLCBzdGFydCBM QkEgPSBuICsgMTAyOEtCIAo+IAlzdWJtaXQgYmlvOiBzaXplID0gMTAyOEtCLCBzdGFydCBMQkEg PSBuICsgMjA1NktCCj4gCS4uLgo+IAo+IEhvd2V2ZXIsIGR1ZSB0byB0aGUgcXVldWUgbGltaXQg cmVzdHJpY3RpbmcgdGhlIEkvTyBzaXplIHRvIGEgbWF4aW11bQo+IG9mIDUxMktCLCB0aGUgQmxv Y2sgTGF5ZXIgc3BsaXRzIGludG8gdGhlIGZvbGxvd2luZyBzZXF1ZW5jZToKPiAJc3VibWl0IGJp bzogc2l6ZSA9IDUxMktCLCBzdGFydCBMQkEgPSBuCj4gCXN1Ym1pdCBiaW86IHNpemUgPSA1MTJL Qiwgc3RhcnQgTEJBID0gbiArICA1MTJLQgo+IAlzdWJtaXQgYmlvOiBzaXplID0gICA0S0IsIHN0 YXJ0IExCQSA9IG4gKyAxMDI0S0IJCj4gCXN1Ym1pdCBiaW86IHNpemUgPSA1MTJLQiwgc3RhcnQg TEJBID0gbiArIDEwMjhLQgo+IAlzdWJtaXQgYmlvOiBzaXplID0gNTEyS0IsIHN0YXJ0IExCQSA9 IG4gKyAxNTQwS0IKPiAJc3VibWl0IGJpbzogc2l6ZSA9ICAgNEtCLCBzdGFydCBMQkEgPSBuICsg MjA1MktCCj4gCXN1Ym1pdCBiaW86IHNpemUgPSA1MTJLQiwgc3RhcnQgTEJBID0gbiArIDIwNTZL Qgo+IAlzdWJtaXQgYmlvOiBzaXplID0gNTEyS0IsIHN0YXJ0IExCQSA9IG4gKyAyNTY4S0IKPiAJ c3VibWl0IGJpbzogc2l6ZSA9ICAgNEtCLCBzdGFydCBMQkEgPSBuICsgMzA4MEtCCj4gCS4uLgo+ IAkKPiBUaGUgb3JpZ2luYWwgZXhwZWN0YXRpb24gd2FzIGZvciB0aGUgc3RvcmFnZSB0byByZWNl aXZlIGxhcmdlLAo+IGNvbnRpZ3VvdXMgcmVxdWVzdHMuIEhvd2V2ZXIsIGR1ZSB0byBub24tYWxp Z25tZW50LCBtYW55IHNtYWxsIEkvTwo+IHJlcXVlc3RzIGFyZSBnZW5lcmF0ZWQuIFRoaXMgcHJv YmxlbSBpcyBlYXNpbHkgdmlzaWJsZSBiZWNhdXNlIHRoZQo+IHVzZXIgcGFnZXMgcGFzc2VkIGlu IGFyZSBvZnRlbiBhbGxvY2F0ZWQgYnkgdGhlIGJ1ZGR5IHN5c3RlbSBhcyBvcmRlciAwCj4gcGFn ZXMgZHVyaW5nIHBhZ2UgZmF1bHRzLCByZXN1bHRpbmcgaW4gaGlnaGx5IG5vbi1jb250aWd1b3Vz IG1lbW9yeS4KCklmIG9yZGVyIDAgcGFnZSBpcyBhZGRlZCB0byBiaW8sIHRoZSBtdWx0aXBhZ2Ug YnZlYyBiZWNvbWVzIG5vcApiYXNpY2FsbHkoMjU2YnZlYyBob2xkcyAyNTYgcGFnZXMpLCB0aGVu IGhvdyBjYW4gaXQgbWFrZSBhIGRpZmZlcmVuY2UKZm9yIHlvdT8KCj4gCj4gQXMgb2JzZXJ2ZWQg aW4gdGhlIEFudHV0dSBTZXF1ZW50aWFsIFJlYWQgdGVzdCBiZWxvdywgaXQgaXMgc2ltaWxhciB0 bwo+IHRoZSBkZXNjcmlwdGlvbiBhYm92ZSB3aGVyZSB0aGUgc3BsaXR0aW5nIGNhdXNlZCBieSB0 aGUgcXVldWUgbGltaXQKPiBsZWF2ZXMgc21hbGwgcmVxdWVzdHMgc2FuZHdpY2hlZCBpbiBiZXR3 ZWVuOgo+IAo+IGJsb2NrX2Jpb19xdWV1ZTogOCwzMiBSIDg2OTI1ODY0ICsgMjE0NCBbVGhyZWFk LTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MjU4NjQgLyA4NjkyNjg4OCBbVGhyZWFkLTUx XQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MjY4ODggLyA4NjkyNzkxMiBbVGhyZWFkLTUxXQo+ IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTI1ODY0ICsgMTAyNCBbVGhyZWFk LTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTI2ODg4ICsgMTAyNCBb VGhyZWFkLTUxXQo+IGJsb2NrX2Jpb19xdWV1ZTogOCwzMiBSIDg2OTI4MDA4ICsgMjE0NCBbVGhy ZWFkLTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MjgwMDggLyA4NjkyOTAzMiBbVGhyZWFk LTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MjkwMzIgLyA4NjkzMDA1NiBbVGhyZWFkLTUx XQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTI4MDA4ICsgMTAyNCBbVGhy ZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNDkxNTIgKCkgODY5Mjc5MTIgKyA5NiBb VGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTI5MDMyICsg MTAyNCBbVGhyZWFkLTUxXQo+IGJsb2NrX2Jpb19xdWV1ZTogOCwzMiBSIDg2OTMwMTUyICsgMjEx MiBbVGhyZWFkLTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MzAxNTIgLyA4NjkzMTE3NiBb VGhyZWFkLTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MzExNzYgLyA4NjkzMjIwMCBbVGhy ZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTMwMTUyICsgMTAy NCBbVGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNDkxNTIgKCkgODY5MzAwNTYg KyA5NiBbVGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTMx MTc2ICsgMTAyNCBbVGhyZWFkLTUxXQo+IGJsb2NrX2Jpb19xdWV1ZTogOCwzMiBSIDg2OTMyMjY0 ICsgMjA5NiBbVGhyZWFkLTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MzIyNjQgLyA4Njkz MzI4OCBbVGhyZWFkLTUxXQo+IGJsb2NrX3NwbGl0OiA4LDMyIFIgODY5MzMyODggLyA4NjkzNDMx MiBbVGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgpIDg2OTMyMjY0 ICsgMTAyNCBbVGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgMzI3NjggKCkgODY5 MzIyMDAgKyA2NCBbVGhyZWFkLTUxXQo+IGJsb2NrX3JxX2lzc3VlOiA4LDMyIFIgNTI0Mjg4ICgp IDg2OTMzMjg4ICsgMTAyNCBbVGhyZWFkLTUxXQo+IAo+IEkgc2ltcGx5IHByZXZlbnRzIG5vbi1h bGlnbmVkIHNpdHVhdGlvbnMgaW4gYmlvX2lvdl9pdGVyX2dldF9wYWdlcy4KCkJ1dCB0aGVyZSBp cyBzdGlsbCA0S0IgSU8gbGVmdCBpZiB5b3UgbGltaXQgbWF4IGJpbyBzaXplIGlzIDUxMktCLAp0 aGVuIGhvdyBkb2VzIHRoaXMgNEtCIElPIGZpbmFsbHkgZ28gaW4gY2FzZSBvZiAxMDI4S0IgSU8/ Cgo+IEJlc2lkZXMgbWFraW5nIHRoZSB1cHBlciBsYXllciBhcHBsaWNhdGlvbiBhd2FyZSBvZiB0 aGUgcXVldWUgbGltaXQsIEkKPiB3b3VsZCBhcHByZWNpYXRlIGFueSBvdGhlciBkaXJlY3Rpb25z IG9yIHN1Z2dlc3Rpb25zIHlvdSBtYXkgaGF2ZS4KClRoZSBwcm9ibGVtIGlzIHJlbGF0ZWQgd2l0 aCBJTyBzaXplIGZyb20gYXBwbGljYXRpb24uCgpJZiB5b3Ugc2VuZCB1bmFsaWduZWQgSU8sIHlv dSBjYW4ndCBhdm9pZCB0aGUgbGFzdCBJTyB3aXRoIHNtYWxsIHNpemUsIG5vCm1hdHRlciBpZiBi bG9jayBsYXllciBiaW8gc3BsaXQgaXMgaW52b2x2ZWQgb3Igbm90LiBZb3VyIHBhdGNoIGp1c3Qg bGV0cwpfX2Jpb19pb3ZfaXRlcl9nZXRfcGFnZXMgc3BsaXQgdGhlIGJpbywgYW5kIHlvdSBzdGls bCBoYXZlIDRLQiBsZWZ0CmZpbmFsbHkgd2hlbiBhcHBsaWNhdGlvbiBzdWJtaXRzIDEwMjhLQiwg cmlnaHQ/CgpUaGVuIEkgZG9uJ3QgdW5kZXJzdGFuZCB3aHkgeW91ciBwYXRjaCBpbXByb3ZlcyBz ZXF1ZW50aWFsIElPCnBlcmZvcm1hbmNlLgoKVGhhbmtzLApNaW5nCgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5n IGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5p bmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=