From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9180E6126 for ; Tue, 7 Nov 2023 03:49:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IuYIcNrL" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3C5ED47 for ; Mon, 6 Nov 2023 19:49:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699328960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y9FCMlk2IkrIEzQB8L9xTTGDJXhqSvTWTL0IDXr95cA=; b=IuYIcNrLlUZSo/Asb4z+ApP+JT0LaOjzuZVe7oY/91WqLfu9qKDowRltkUHG/76AYIho0s oI1dzII8i+4JmREzAXx5U+zJliG0hgndj4C+f3PpLLSL41T9NpDrVu0gFvWIQ3NPqyYOnY XEh/wRBhmK1zGSMUsicz4zEKe7fRD+A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-112-H3pkEELXNwqDhxmZ4mCm1w-1; Mon, 06 Nov 2023 22:48:58 -0500 X-MC-Unique: H3pkEELXNwqDhxmZ4mCm1w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B7C1F185A781; Tue, 7 Nov 2023 03:48:57 +0000 (UTC) Received: from fedora (unknown [10.72.120.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 18CD240C6EB9; Tue, 7 Nov 2023 03:48:48 +0000 (UTC) Date: Tue, 7 Nov 2023 11:48:44 +0800 From: Ming Lei To: Ed Tsai =?utf-8?B?KOiUoeWul+i7kik=?= Cc: Will Shiu =?utf-8?B?KOioseaBreeRnCk=?= , "linux-kernel@vger.kernel.org" , "linux-mediatek@lists.infradead.org" , Peter Wang =?utf-8?B?KOeOi+S/oeWPiyk=?= , Alice Chao =?utf-8?B?KOi2meePruWdhyk=?= , "linux-block@vger.kernel.org" , wsd_upstream , "axboe@kernel.dk" , Casper Li =?utf-8?B?KOadjuS4reamrik=?= , Chun-Hung Wu =?utf-8?B?KOW3q+mnv+Wujyk=?= , Powen Kao =?utf-8?B?KOmrmOS8r+aWhyk=?= , Naomi Chu =?utf-8?B?KOacseipoOeUsCk=?= , "linux-arm-kernel@lists.infradead.org" , Stanley Chu =?utf-8?B?KOacseWOn+mZnik=?= , "matthias.bgg@gmail.com" , "angelogioacchino.delregno@collabora.com" , ming.lei@redhat.com Subject: Re: [PATCH 1/1] block: Check the queue limit before bio submitting Message-ID: References: <20231025092255.27930-1-ed.tsai@mediatek.com> <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> <2bc847a83849973b7658145f2efdda86cc47e3d5.camel@mediatek.com> <5ecedad658bf28abf9bbeeb70dcac09b4b404cf5.camel@mediatek.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 On Tue, Nov 07, 2023 at 02:53:20AM +0000, Ed Tsai (蔡宗軒) wrote: > On Mon, 2023-11-06 at 19:54 +0800, Ming Lei wrote: > > On Mon, Nov 06, 2023 at 12:53:31PM +0800, Ming Lei wrote: > > > On Mon, Nov 06, 2023 at 01:40:12AM +0000, Ed Tsai (蔡宗軒) wrote: > > > > On Mon, 2023-11-06 at 09:33 +0800, Ed Tsai wrote: > > > > > On Sat, 2023-11-04 at 11:43 +0800, Ming Lei wrote: > > > > > > ... > > > > > > > Sorry for missing out on my dd command. Here it is: > > > > dd if=/data/test_file of=/dev/null bs=64m count=1 iflag=direct > > > > > > OK, thanks for the sharing. > > > > > > I understand the issue now, but not sure if it is one good idea to > > check > > > queue limit in __bio_iov_iter_get_pages(): > > > > > > 1) bio->bi_bdev may not be set > > > > > > 2) what matters is actually bio's alignment, and bio size still can > > > be big enough > > > > > > So I cooked one patch, and it should address your issue: > > > > The following one fixes several bugs, and is verified to be capable > > of > > making big & aligned bios, feel free to run your test against this > > one: > > > > block/bio.c | 28 +++++++++++++++++++++++++++- > > 1 file changed, 27 insertions(+), 1 deletion(-) > > > > diff --git a/block/bio.c b/block/bio.c > > index 816d412c06e9..80b36ce57510 100644 > > --- a/block/bio.c > > +++ b/block/bio.c > > @@ -1211,6 +1211,7 @@ static int bio_iov_add_zone_append_page(struct > > bio *bio, struct page *page, > > } > > > > #define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / > > sizeof(struct page *)) > > +#define BIO_CHUNK_SIZE(256U << 10) > > > > /** > > * __bio_iov_iter_get_pages - pin user or kernel pages and add them > > to a bio > > @@ -1266,6 +1267,31 @@ static int __bio_iov_iter_get_pages(struct bio > > *bio, struct iov_iter *iter) > > size -= trim; > > } > > > > +/* > > + * Try to make bio aligned with 128KB if it isn't the last one, so > > + * we can avoid small bio in case of big chunk sequential IO because > > + * of bio split and multipage bvec. > > + * > > + * If nothing is added to this bio, simply allow unaligned since we > > + * have chance to add more bytes > > + */ > > +if (iov_iter_count(iter) && bio->bi_iter.bi_size) { > > +unsigned int aligned_size = (bio->bi_iter.bi_size + size) & > > +~(BIO_CHUNK_SIZE - 1); > > + > > +if (aligned_size <= bio->bi_iter.bi_size) { > > +/* stop to add page if this bio can't keep aligned */ > > +if (!(bio->bi_iter.bi_size & (BIO_CHUNK_SIZE - 1))) { > > +ret = left = size; > > +goto revert; > > +} > > +} else { > > +aligned_size -= bio->bi_iter.bi_size; > > +iov_iter_revert(iter, size - aligned_size); > > +size = aligned_size; > > +} > > +} > > + > > if (unlikely(!size)) { > > ret = -EFAULT; > > goto out; > > @@ -1285,7 +1311,7 @@ static int __bio_iov_iter_get_pages(struct bio > > *bio, struct iov_iter *iter) > > > > offset = 0; > > } > > - > > +revert: > > iov_iter_revert(iter, left); > > out: > > while (i < nr_pages) > > -- > > 2.41.0 > > > > > > > > Thanks, > > Ming > > > > The latest patch you provided with 256 alignment does help alleviate > the severity of fragmentation. However, the actual aligned size may > vary depending on the device. Using a fixed and universal size of 128 > or 256KB only provides partial relief from fragmentation. > > I performed a dd direct I/O read of 64MB with your patch, and although > most of the bios were aligned, there were still cases of missalignment > to the device limit (e.g., 512MB for my device), as shown below: 512MB is really big, and actually you have reached 3520MB in READ by limiting max bio size to 1MB in your original patch. Just be curious what is the data if you change to align with max sectors against my last patch? which can try to maximize & align bio. > > dd [000] ..... 392.976830: block_bio_queue: 254,52 R 2997760 + 3584 > dd [000] ..... 392.979940: block_bio_queue: 254,52 R 3001344 + 3584 > dd [000] ..... 392.983235: block_bio_queue: 254,52 R 3004928 + 3584 > dd [000] ..... 392.986468: block_bio_queue: 254,52 R 3008512 + 3584 Yeah, I thought that 128KB should be fine for usual hardware, but looks not good enough. > > Comparing the results of the Antutu Sequential test to the previous > data, it is indeed an improvement, but still a slight difference with > limiting the bio size to max_sectors: > > Sequential Read (average of 5 rounds): > Original: 3033.7 MB/sec > Limited to max_sectors: 3520.9 MB/sec > Aligned 256KB: 3471.5 MB/sec > > Sequential Write (average of 5 rounds): > Original: 2225.4 MB/sec > Limited to max_sectors: 2800.3 MB/sec > Aligned 256KB: 2618.1 MB/sec Thanks for sharing the data. > > What if we limit the bio size only for those who have set the > max_sectors? I think it may be doable, but need more smart approach for avoiding extra cost of iov_iter_revert(), and one way is to add bio_shrink() (or bio_revert()) to run the alignment just once. I will think further and write a new patch if it is doable. Thanks, Ming From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 512D2C4332F for ; Tue, 7 Nov 2023 03:49:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Bw++3rp9yS1QYqVzROQ8AjIcpH5atvcCjfSjmIBnrik=; b=JS4wsdjBXXPiqy 3sLfqdl86ZR7HYKiQZHSTpb0bbbUTmDhwgYHoGMR5I3KxOUpzr9n5RQSxVDHvAb9p8qtk9m3QV7C4 t93oR7aQo1hX/f5MOfcsyhJNdoyRveRDbpnIhNfJ77b91Pu1MTQSKQy196hHrewL9HBtAxvOzw38Y k33Xacco4/TFSk3Ieck84yV71rBe2+0sftsCewnI1AktVdWy5wzeOiaIBl/QzRwQOv7nPC7Hp15CF rYXJfaAjQa6fpNFQq8616YEUTyCkY5ib4dIL9U2qAki3J1tHngXEAAw1O6NvLr0uPd/yvXVa0QGcx 3feajcpQN6Vzt574jEQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r0D5e-000SA7-19; Tue, 07 Nov 2023 03:49:22 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r0D5c-000S8y-0Y for linux-arm-kernel@lists.infradead.org; Tue, 07 Nov 2023 03:49:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699328959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y9FCMlk2IkrIEzQB8L9xTTGDJXhqSvTWTL0IDXr95cA=; b=gsDj7asPHqmJGXdY8x13q725pxOG0XZSwrwsdtpN7WYIAilC5HEnW1/xkPVBaIbk9MZpWn DoKBaB/cucwVj2HGAZVIDACtpzDax5KVY9+0KUDEGOexnp7d41RdCvBUUmrzD9Vh0JlNgh 0MedcRvsQP8OJT/o3tjLv1DBthxU9Yw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-112-H3pkEELXNwqDhxmZ4mCm1w-1; Mon, 06 Nov 2023 22:48:58 -0500 X-MC-Unique: H3pkEELXNwqDhxmZ4mCm1w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B7C1F185A781; Tue, 7 Nov 2023 03:48:57 +0000 (UTC) Received: from fedora (unknown [10.72.120.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 18CD240C6EB9; Tue, 7 Nov 2023 03:48:48 +0000 (UTC) Date: Tue, 7 Nov 2023 11:48:44 +0800 From: Ming Lei To: Ed Tsai =?utf-8?B?KOiUoeWul+i7kik=?= Cc: Will Shiu =?utf-8?B?KOioseaBreeRnCk=?= , "linux-kernel@vger.kernel.org" , "linux-mediatek@lists.infradead.org" , Peter Wang =?utf-8?B?KOeOi+S/oeWPiyk=?= , Alice Chao =?utf-8?B?KOi2meePruWdhyk=?= , "linux-block@vger.kernel.org" , wsd_upstream , "axboe@kernel.dk" , Casper Li =?utf-8?B?KOadjuS4reamrik=?= , Chun-Hung Wu =?utf-8?B?KOW3q+mnv+Wujyk=?= , Powen Kao =?utf-8?B?KOmrmOS8r+aWhyk=?= , Naomi Chu =?utf-8?B?KOacseipoOeUsCk=?= , "linux-arm-kernel@lists.infradead.org" , Stanley Chu =?utf-8?B?KOacseWOn+mZnik=?= , "matthias.bgg@gmail.com" , "angelogioacchino.delregno@collabora.com" , ming.lei@redhat.com Subject: Re: [PATCH 1/1] block: Check the queue limit before bio submitting Message-ID: References: <20231025092255.27930-1-ed.tsai@mediatek.com> <64db8f5406571c2f89b70f852eb411320201abe6.camel@mediatek.com> <2bc847a83849973b7658145f2efdda86cc47e3d5.camel@mediatek.com> <5ecedad658bf28abf9bbeeb70dcac09b4b404cf5.camel@mediatek.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231106_194920_301341_40AD6752 X-CRM114-Status: GOOD ( 47.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBOb3YgMDcsIDIwMjMgYXQgMDI6NTM6MjBBTSArMDAwMCwgRWQgVHNhaSAo6JSh5a6X 6LuSKSB3cm90ZToKPiBPbiBNb24sIDIwMjMtMTEtMDYgYXQgMTk6NTQgKzA4MDAsIE1pbmcgTGVp IHdyb3RlOgo+ID4gIE9uIE1vbiwgTm92IDA2LCAyMDIzIGF0IDEyOjUzOjMxUE0gKzA4MDAsIE1p bmcgTGVpIHdyb3RlOgo+ID4gPiBPbiBNb24sIE5vdiAwNiwgMjAyMyBhdCAwMTo0MDoxMkFNICsw MDAwLCBFZCBUc2FpICjolKHlrpfou5IpIHdyb3RlOgo+ID4gPiA+IE9uIE1vbiwgMjAyMy0xMS0w NiBhdCAwOTozMyArMDgwMCwgRWQgVHNhaSB3cm90ZToKPiA+ID4gPiA+IE9uIFNhdCwgMjAyMy0x MS0wNCBhdCAxMTo0MyArMDgwMCwgTWluZyBMZWkgd3JvdGU6Cj4gPiA+IAo+ID4gPiAuLi4KPiA+ ID4gCj4gPiA+ID4gU29ycnkgZm9yIG1pc3Npbmcgb3V0IG9uIG15IGRkIGNvbW1hbmQuIEhlcmUg aXQgaXM6Cj4gPiA+ID4gZGQgaWY9L2RhdGEvdGVzdF9maWxlIG9mPS9kZXYvbnVsbCBicz02NG0g Y291bnQ9MSBpZmxhZz1kaXJlY3QKPiA+ID4gCj4gPiA+IE9LLCB0aGFua3MgZm9yIHRoZSBzaGFy aW5nLgo+ID4gPiAKPiA+ID4gSSB1bmRlcnN0YW5kIHRoZSBpc3N1ZSBub3csIGJ1dCBub3Qgc3Vy ZSBpZiBpdCBpcyBvbmUgZ29vZCBpZGVhIHRvCj4gPiBjaGVjawo+ID4gPiBxdWV1ZSBsaW1pdCBp biBfX2Jpb19pb3ZfaXRlcl9nZXRfcGFnZXMoKToKPiA+ID4gCj4gPiA+IDEpIGJpby0+YmlfYmRl diBtYXkgbm90IGJlIHNldAo+ID4gPiAKPiA+ID4gMikgd2hhdCBtYXR0ZXJzIGlzIGFjdHVhbGx5 IGJpbydzIGFsaWdubWVudCwgYW5kIGJpbyBzaXplIHN0aWxsIGNhbgo+ID4gPiBiZSBiaWcgZW5v dWdoCj4gPiA+IAo+ID4gPiBTbyBJIGNvb2tlZCBvbmUgcGF0Y2gsIGFuZCBpdCBzaG91bGQgYWRk cmVzcyB5b3VyIGlzc3VlOgo+ID4gCj4gPiBUaGUgZm9sbG93aW5nIG9uZSBmaXhlcyBzZXZlcmFs IGJ1Z3MsIGFuZCBpcyB2ZXJpZmllZCB0byBiZSBjYXBhYmxlCj4gPiBvZgo+ID4gbWFraW5nIGJp ZyAmIGFsaWduZWQgYmlvcywgZmVlbCBmcmVlIHRvIHJ1biB5b3VyIHRlc3QgYWdhaW5zdCB0aGlz Cj4gPiBvbmU6Cj4gPiAKPiA+ICBibG9jay9iaW8uYyB8IDI4ICsrKysrKysrKysrKysrKysrKysr KysrKysrKy0KPiA+ICAxIGZpbGUgY2hhbmdlZCwgMjcgaW5zZXJ0aW9ucygrKSwgMSBkZWxldGlv bigtKQo+ID4gCj4gPiBkaWZmIC0tZ2l0IGEvYmxvY2svYmlvLmMgYi9ibG9jay9iaW8uYwo+ID4g aW5kZXggODE2ZDQxMmMwNmU5Li44MGIzNmNlNTc1MTAgMTAwNjQ0Cj4gPiAtLS0gYS9ibG9jay9i aW8uYwo+ID4gKysrIGIvYmxvY2svYmlvLmMKPiA+IEBAIC0xMjExLDYgKzEyMTEsNyBAQCBzdGF0 aWMgaW50IGJpb19pb3ZfYWRkX3pvbmVfYXBwZW5kX3BhZ2Uoc3RydWN0Cj4gPiBiaW8gKmJpbywg c3RydWN0IHBhZ2UgKnBhZ2UsCj4gPiAgfQo+ID4gIAo+ID4gICNkZWZpbmUgUEFHRV9QVFJTX1BF Ul9CVkVDICAgICAoc2l6ZW9mKHN0cnVjdCBiaW9fdmVjKSAvCj4gPiBzaXplb2Yoc3RydWN0IHBh Z2UgKikpCj4gPiArI2RlZmluZSBCSU9fQ0hVTktfU0laRSgyNTZVIDw8IDEwKQo+ID4gIAo+ID4g IC8qKgo+ID4gICAqIF9fYmlvX2lvdl9pdGVyX2dldF9wYWdlcyAtIHBpbiB1c2VyIG9yIGtlcm5l bCBwYWdlcyBhbmQgYWRkIHRoZW0KPiA+IHRvIGEgYmlvCj4gPiBAQCAtMTI2Niw2ICsxMjY3LDMx IEBAIHN0YXRpYyBpbnQgX19iaW9faW92X2l0ZXJfZ2V0X3BhZ2VzKHN0cnVjdCBiaW8KPiA+ICpi aW8sIHN0cnVjdCBpb3ZfaXRlciAqaXRlcikKPiA+ICBzaXplIC09IHRyaW07Cj4gPiAgfQo+ID4g IAo+ID4gKy8qCj4gPiArICogVHJ5IHRvIG1ha2UgYmlvIGFsaWduZWQgd2l0aCAxMjhLQiBpZiBp dCBpc24ndCB0aGUgbGFzdCBvbmUsIHNvCj4gPiArICogd2UgY2FuIGF2b2lkIHNtYWxsIGJpbyBp biBjYXNlIG9mIGJpZyBjaHVuayBzZXF1ZW50aWFsIElPIGJlY2F1c2UKPiA+ICsgKiBvZiBiaW8g c3BsaXQgYW5kIG11bHRpcGFnZSBidmVjLgo+ID4gKyAqCj4gPiArICogSWYgbm90aGluZyBpcyBh ZGRlZCB0byB0aGlzIGJpbywgc2ltcGx5IGFsbG93IHVuYWxpZ25lZCBzaW5jZSB3ZQo+ID4gKyAq IGhhdmUgY2hhbmNlIHRvIGFkZCBtb3JlIGJ5dGVzCj4gPiArICovCj4gPiAraWYgKGlvdl9pdGVy X2NvdW50KGl0ZXIpICYmIGJpby0+YmlfaXRlci5iaV9zaXplKSB7Cj4gPiArdW5zaWduZWQgaW50 IGFsaWduZWRfc2l6ZSA9IChiaW8tPmJpX2l0ZXIuYmlfc2l6ZSArIHNpemUpICYKPiA+ICt+KEJJ T19DSFVOS19TSVpFIC0gMSk7Cj4gPiArCj4gPiAraWYgKGFsaWduZWRfc2l6ZSA8PSBiaW8tPmJp X2l0ZXIuYmlfc2l6ZSkgewo+ID4gKy8qIHN0b3AgdG8gYWRkIHBhZ2UgaWYgdGhpcyBiaW8gY2Fu J3Qga2VlcCBhbGlnbmVkICovCj4gPiAraWYgKCEoYmlvLT5iaV9pdGVyLmJpX3NpemUgJiAoQklP X0NIVU5LX1NJWkUgLSAxKSkpIHsKPiA+ICtyZXQgPSBsZWZ0ID0gc2l6ZTsKPiA+ICtnb3RvIHJl dmVydDsKPiA+ICt9Cj4gPiArfSBlbHNlIHsKPiA+ICthbGlnbmVkX3NpemUgLT0gYmlvLT5iaV9p dGVyLmJpX3NpemU7Cj4gPiAraW92X2l0ZXJfcmV2ZXJ0KGl0ZXIsIHNpemUgLSBhbGlnbmVkX3Np emUpOwo+ID4gK3NpemUgPSBhbGlnbmVkX3NpemU7Cj4gPiArfQo+ID4gK30KPiA+ICsKPiA+ICBp ZiAodW5saWtlbHkoIXNpemUpKSB7Cj4gPiAgcmV0ID0gLUVGQVVMVDsKPiA+ICBnb3RvIG91dDsK PiA+IEBAIC0xMjg1LDcgKzEzMTEsNyBAQCBzdGF0aWMgaW50IF9fYmlvX2lvdl9pdGVyX2dldF9w YWdlcyhzdHJ1Y3QgYmlvCj4gPiAqYmlvLCBzdHJ1Y3QgaW92X2l0ZXIgKml0ZXIpCj4gPiAgCj4g PiAgb2Zmc2V0ID0gMDsKPiA+ICB9Cj4gPiAtCj4gPiArcmV2ZXJ0Ogo+ID4gIGlvdl9pdGVyX3Jl dmVydChpdGVyLCBsZWZ0KTsKPiA+ICBvdXQ6Cj4gPiAgd2hpbGUgKGkgPCBucl9wYWdlcykKPiA+ IC0tIAo+ID4gMi40MS4wCj4gPiAKPiA+IAo+ID4gCj4gPiBUaGFua3MsIAo+ID4gTWluZwo+ID4g Cj4gCj4gVGhlIGxhdGVzdCBwYXRjaCB5b3UgcHJvdmlkZWQgd2l0aCAyNTYgYWxpZ25tZW50IGRv ZXMgaGVscCBhbGxldmlhdGUKPiB0aGUgc2V2ZXJpdHkgb2YgZnJhZ21lbnRhdGlvbi4gSG93ZXZl ciwgdGhlIGFjdHVhbCBhbGlnbmVkIHNpemUgbWF5Cj4gdmFyeSBkZXBlbmRpbmcgb24gdGhlIGRl dmljZS4gVXNpbmcgYSBmaXhlZCBhbmQgdW5pdmVyc2FsIHNpemUgb2YgMTI4Cj4gb3IgMjU2S0Ig b25seSBwcm92aWRlcyBwYXJ0aWFsIHJlbGllZiBmcm9tIGZyYWdtZW50YXRpb24uCj4gCj4gSSBw ZXJmb3JtZWQgYSBkZCBkaXJlY3QgSS9PIHJlYWQgb2YgNjRNQiB3aXRoIHlvdXIgcGF0Y2gsIGFu ZCBhbHRob3VnaAo+IG1vc3Qgb2YgdGhlIGJpb3Mgd2VyZSBhbGlnbmVkLCB0aGVyZSB3ZXJlIHN0 aWxsIGNhc2VzIG9mIG1pc3NhbGlnbm1lbnQKPiB0byB0aGUgZGV2aWNlIGxpbWl0IChlLmcuLCA1 MTJNQiBmb3IgbXkgZGV2aWNlKSwgYXMgc2hvd24gYmVsb3c6Cgo1MTJNQiBpcyByZWFsbHkgYmln LCBhbmQgYWN0dWFsbHkgeW91IGhhdmUgcmVhY2hlZCAzNTIwTUIgaW4gUkVBRCBieQpsaW1pdGlu ZyBtYXggYmlvIHNpemUgdG8gMU1CIGluIHlvdXIgb3JpZ2luYWwgcGF0Y2guCgpKdXN0IGJlIGN1 cmlvdXMgd2hhdCBpcyB0aGUgZGF0YSBpZiB5b3UgY2hhbmdlIHRvIGFsaWduIHdpdGggbWF4IHNl Y3RvcnMKYWdhaW5zdCBteSBsYXN0IHBhdGNoPyB3aGljaCBjYW4gdHJ5IHRvIG1heGltaXplICYg YWxpZ24gYmlvLgoKPiAKPiBkZCBbMDAwXSAuLi4uLiAzOTIuOTc2ODMwOiBibG9ja19iaW9fcXVl dWU6IDI1NCw1MiBSIDI5OTc3NjAgKyAzNTg0Cj4gZGQgWzAwMF0gLi4uLi4gMzkyLjk3OTk0MDog YmxvY2tfYmlvX3F1ZXVlOiAyNTQsNTIgUiAzMDAxMzQ0ICsgMzU4NAo+IGRkIFswMDBdIC4uLi4u IDM5Mi45ODMyMzU6IGJsb2NrX2Jpb19xdWV1ZTogMjU0LDUyIFIgMzAwNDkyOCArIDM1ODQKPiBk ZCBbMDAwXSAuLi4uLiAzOTIuOTg2NDY4OiBibG9ja19iaW9fcXVldWU6IDI1NCw1MiBSIDMwMDg1 MTIgKyAzNTg0CgpZZWFoLCBJIHRob3VnaHQgdGhhdCAxMjhLQiBzaG91bGQgYmUgZmluZSBmb3Ig dXN1YWwgaGFyZHdhcmUsIGJ1dApsb29rcyBub3QgZ29vZCBlbm91Z2guCgo+IAo+IENvbXBhcmlu ZyB0aGUgcmVzdWx0cyBvZiB0aGUgQW50dXR1IFNlcXVlbnRpYWwgdGVzdCB0byB0aGUgcHJldmlv dXMKPiBkYXRhLCBpdCBpcyBpbmRlZWQgYW4gaW1wcm92ZW1lbnQsIGJ1dCBzdGlsbCBhIHNsaWdo dCBkaWZmZXJlbmNlIHdpdGgKPiBsaW1pdGluZyB0aGUgYmlvIHNpemUgdG8gbWF4X3NlY3RvcnM6 Cj4gCj4gU2VxdWVudGlhbCBSZWFkIChhdmVyYWdlIG9mIDUgcm91bmRzKToKPiBPcmlnaW5hbDog MzAzMy43IE1CL3NlYwo+IExpbWl0ZWQgdG8gbWF4X3NlY3RvcnM6IDM1MjAuOSBNQi9zZWMKPiBB bGlnbmVkIDI1NktCOiAzNDcxLjUgTUIvc2VjCj4gCj4gU2VxdWVudGlhbCBXcml0ZSAoYXZlcmFn ZSBvZiA1IHJvdW5kcyk6Cj4gT3JpZ2luYWw6IDIyMjUuNCBNQi9zZWMKPiBMaW1pdGVkIHRvIG1h eF9zZWN0b3JzOiAyODAwLjMgTUIvc2VjCj4gQWxpZ25lZCAyNTZLQjogMjYxOC4xIE1CL3NlYwoK VGhhbmtzIGZvciBzaGFyaW5nIHRoZSBkYXRhLgoKPiAKPiBXaGF0IGlmIHdlIGxpbWl0IHRoZSBi aW8gc2l6ZSBvbmx5IGZvciB0aG9zZSB3aG8gaGF2ZSBzZXQgdGhlCj4gbWF4X3NlY3RvcnM/CgpJ IHRoaW5rIGl0IG1heSBiZSBkb2FibGUsIGJ1dCBuZWVkIG1vcmUgc21hcnQgYXBwcm9hY2ggZm9y IGF2b2lkaW5nCmV4dHJhIGNvc3Qgb2YgaW92X2l0ZXJfcmV2ZXJ0KCksIGFuZCBvbmUgd2F5IGlz IHRvIGFkZCBiaW9fc2hyaW5rKCkKKG9yIGJpb19yZXZlcnQoKSkgdG8gcnVuIHRoZSBhbGlnbm1l bnQganVzdCBvbmNlLgoKSSB3aWxsIHRoaW5rIGZ1cnRoZXIgYW5kIHdyaXRlIGEgbmV3IHBhdGNo IGlmIGl0IGlzIGRvYWJsZS4KCgoKVGhhbmtzLApNaW5nCgoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QK bGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRl YWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=