From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 581B5C43603 for ; Wed, 18 Dec 2019 18:38:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1B48E227BF for ; Wed, 18 Dec 2019 18:38:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="O9Qwor/0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726939AbfLRSiC (ORCPT ); Wed, 18 Dec 2019 13:38:02 -0500 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:20596 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726698AbfLRSiC (ORCPT ); Wed, 18 Dec 2019 13:38:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576694282; x=1608230282; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=X60GVNIN6w+57tWVEw55R0ONYNcF3stArJhhbGYl2Dg=; b=O9Qwor/0weoBYe2fxK+hPtWw7zzZ94dQkdI0MjONoZHW4DRiYeUO2eb0 RwG/dgqzlLZoO4hwqtGdu/a52hzzuM/S0kzhXbr/0cgKDBsRbC2OACsIh lB9qUBBahtSyxvuzuBz5lQf+jABONecoGer7TCRuitbq3qGynIync9aWT Y=; IronPort-SDR: E5P+HmBPSKhnZ1Uqeqxwy175m071GVvW3x4FgpkSmjihLCcfZLx1ZVOLf2IL/JzqKgOYTJusGT WVAXWsznea0w== X-IronPort-AV: E=Sophos;i="5.69,330,1571702400"; d="scan'208";a="9082785" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 18 Dec 2019 18:38:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com (Postfix) with ESMTPS id 900582822B8; Wed, 18 Dec 2019 18:37:58 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Wed, 18 Dec 2019 18:37:57 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.109) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 18 Dec 2019 18:37:52 +0000 From: SeongJae Park To: , , , CC: SeongJae Park , , , , , , Subject: [PATCH v13 3/5] xen/blkback: Squeeze page pools if a memory pressure is detected Date: Wed, 18 Dec 2019 19:37:16 +0100 Message-ID: <20191218183718.31719-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218183718.31719-1-sjpark@amazon.com> References: <20191218183718.31719-1-sjpark@amazon.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.43.160.109] X-ClientProxiedBy: EX13D10UWB003.ant.amazon.com (10.43.161.106) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: SeongJae Park Each `blkif` has a free pages pool for the grant mapping. The size of the pool starts from zero and is increased on demand while processing the I/O requests. If current I/O requests handling is finished or 100 milliseconds has passed since last I/O requests handling, it checks and shrinks the pool to not exceed the size limit, `max_buffer_pages`. Therefore, host administrators can cause memory pressure in blkback by attaching a large number of block devices and inducing I/O. Such problematic situations can be avoided by limiting the maximum number of devices that can be attached, but finding the optimal limit is not so easy. Improper set of the limit can results in memory pressure or a resource underutilization. This commit avoids such problematic situations by squeezing the pools (returns every free page in the pool to the system) for a while (users can set this duration via a module parameter) if memory pressure is detected. Discussions =========== The `blkback`'s original shrinking mechanism returns only pages in the pool which are not currently be used by `blkback` to the system. In other words, the pages that are not mapped with granted pages. Because this commit is changing only the shrink limit but still uses the same freeing mechanism it does not touch pages which are currently mapping grants. Once memory pressure is detected, this commit keeps the squeezing limit for a user-specified time duration. The duration should be neither too long nor too short. If it is too long, the squeezing incurring overhead can reduce the I/O performance. If it is too short, `blkback` will not free enough pages to reduce the memory pressure. This commit sets the value as `10 milliseconds` by default because it is a short time in terms of I/O while it is a long time in terms of memory operations. Also, as the original shrinking mechanism works for at least every 100 milliseconds, this could be a somewhat reasonable choice. I also tested other durations (refer to the below section for more details) and confirmed that 10 milliseconds is the one that works best with the test. That said, the proper duration depends on actual configurations and workloads. That's why this commit allows users to set the duration as a module parameter. Memory Pressure Test ==================== To show how this commit fixes the memory pressure situation well, I configured a test environment on a xen-running virtualization system. On the `blkfront` running guest instances, I attach a large number of network-backed volume devices and induce I/O to those. Meanwhile, I measure the number of pages that swapped in (pswpin) and out (pswpout) on the `blkback` running guest. The test ran twice, once for the `blkback` before this commit and once for that after this commit. As shown below, this commit has dramatically reduced the memory pressure: pswpin pswpout before 76,672 185,799 after 867 3,967 Optimal Aggressive Shrinking Duration ------------------------------------- To find a best squeezing duration, I repeated the test with three different durations (1ms, 10ms, and 100ms). The results are as below: duration pswpin pswpout 1 707 5,095 10 867 3,967 100 362 3,348 As expected, the memory pressure decreases as the duration increases, but the reduction become slow from the `10ms`. Based on this results, I chose the default duration as 10ms. Performance Overhead Test ========================= This commit could incur I/O performance degradation under severe memory pressure because the squeezing will require more page allocations per I/O. To show the overhead, I artificially made a worst-case squeezing situation and measured the I/O performance of a `blkfront` running guest. For the artificial squeezing, I set the `blkback.max_buffer_pages` using the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. In this test, I set the value to `1024` and `0`. The `1024` is the default value. Setting the value as `0` is same to a situation doing the squeezing always (worst-case). If the underlying block device is slow enough, the squeezing overhead could be hidden. For the reason, I use a fast block device, namely the rbd[1]: # xl block-attach guest phy:/dev/ram0 xvdb w For the I/O performance measurement, I run a simple `dd` command 5 times directly to the device as below and collect the 'MB/s' results. $ for i in {1..5}; do dd if=/dev/zero of=/dev/xvdb \ bs=4k count=$((256*512)); sync; done The results are as below. 'max_pgs' represents the value of the `blkback.max_buffer_pages` parameter. max_pgs Min Max Median Avg Stddev 0 417 423 420 419.4 2.5099801 1024 414 425 416 417.8 4.4384682 No difference proven at 95.0% confidence In short, even worst case squeezing on ramdisk based fast block device makes no visible performance degradation. Please note that this is just a very simple and minimal test. On systems using super-fast block devices and a special I/O workload, the results might be different. If you have any doubt, test on your machine with your workload to find the optimal squeezing duration for you. [1] https://www.kernel.org/doc/html/latest/admin-guide/blockdev/ramdisk.html Signed-off-by: SeongJae Park --- .../ABI/testing/sysfs-driver-xen-blkback | 10 ++++++++ drivers/block/xen-blkback/blkback.c | 7 ++++-- drivers/block/xen-blkback/common.h | 1 + drivers/block/xen-blkback/xenbus.c | 23 ++++++++++++++++++- 4 files changed, 38 insertions(+), 3 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback index 4e7babb3ba1f..f01224231f3f 100644 --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback @@ -25,3 +25,13 @@ Description: allocated without being in use. The time is in seconds, 0 means indefinitely long. The default is 60 seconds. + +What: /sys/module/xen_blkback/parameters/buffer_squeeze_duration_ms +Date: December 2019 +KernelVersion: 5.5 +Contact: SeongJae Park +Description: + When memory pressure is reported to blkback this option + controls the duration in milliseconds that blkback will not + cache any page not backed by a grant mapping. + The default is 10ms. diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index fd1e19f1a49f..79f677aeb5cc 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -656,8 +656,11 @@ int xen_blkif_schedule(void *arg) ring->next_lru = jiffies + msecs_to_jiffies(LRU_INTERVAL); } - /* Shrink if we have more than xen_blkif_max_buffer_pages */ - shrink_free_pagepool(ring, xen_blkif_max_buffer_pages); + /* Shrink the free pages pool if it is too large. */ + if (time_before(jiffies, blkif->buffer_squeeze_end)) + shrink_free_pagepool(ring, 0); + else + shrink_free_pagepool(ring, xen_blkif_max_buffer_pages); if (log_stats && time_after(jiffies, ring->st_print)) print_stats(ring); diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h index 1d3002d773f7..536c84f61fed 100644 --- a/drivers/block/xen-blkback/common.h +++ b/drivers/block/xen-blkback/common.h @@ -319,6 +319,7 @@ struct xen_blkif { /* All rings for this device. */ struct xen_blkif_ring *rings; unsigned int nr_rings; + unsigned long buffer_squeeze_end; }; struct seg_buf { diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index b90dbcd99c03..24172c180f5f 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -824,6 +824,26 @@ static void frontend_changed(struct xenbus_device *dev, } +/* Once a memory pressure is detected, squeeze free page pools for a while. */ +static unsigned int buffer_squeeze_duration_ms = 10; +module_param_named(buffer_squeeze_duration_ms, + buffer_squeeze_duration_ms, int, 0644); +MODULE_PARM_DESC(buffer_squeeze_duration_ms, +"Duration in ms to squeeze pages buffer when a memory pressure is detected"); + +/* + * Callback received when the memory pressure is detected. + */ +static void reclaim_memory(struct xenbus_device *dev) +{ + struct backend_info *be = dev_get_drvdata(&dev->dev); + + if (!be) + return; + be->blkif->buffer_squeeze_end = jiffies + + msecs_to_jiffies(buffer_squeeze_duration_ms); +} + /* ** Connection ** */ @@ -1115,7 +1135,8 @@ static struct xenbus_driver xen_blkbk_driver = { .ids = xen_blkbk_ids, .probe = xen_blkbk_probe, .remove = xen_blkbk_remove, - .otherend_changed = frontend_changed + .otherend_changed = frontend_changed, + .reclaim_memory = reclaim_memory, }; int xen_blkif_xenbus_init(void) -- 2.17.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIM_ADSP_ALL,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96C0FC43603 for ; Wed, 18 Dec 2019 18:38:17 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5EF81227BF for ; Wed, 18 Dec 2019 18:38:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="O9Qwor/0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5EF81227BF Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iheCx-0001Qs-EV; Wed, 18 Dec 2019 18:38:03 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iheCw-0001QW-GA for xen-devel@lists.xenproject.org; Wed, 18 Dec 2019 18:38:02 +0000 X-Inumbo-ID: 84d9be06-21c5-11ea-a914-bc764e2007e4 Received: from smtp-fw-4101.amazon.com (unknown [72.21.198.25]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 84d9be06-21c5-11ea-a914-bc764e2007e4; Wed, 18 Dec 2019 18:38:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576694282; x=1608230282; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=X60GVNIN6w+57tWVEw55R0ONYNcF3stArJhhbGYl2Dg=; b=O9Qwor/0weoBYe2fxK+hPtWw7zzZ94dQkdI0MjONoZHW4DRiYeUO2eb0 RwG/dgqzlLZoO4hwqtGdu/a52hzzuM/S0kzhXbr/0cgKDBsRbC2OACsIh lB9qUBBahtSyxvuzuBz5lQf+jABONecoGer7TCRuitbq3qGynIync9aWT Y=; IronPort-SDR: E5P+HmBPSKhnZ1Uqeqxwy175m071GVvW3x4FgpkSmjihLCcfZLx1ZVOLf2IL/JzqKgOYTJusGT WVAXWsznea0w== X-IronPort-AV: E=Sophos;i="5.69,330,1571702400"; d="scan'208";a="9082785" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 18 Dec 2019 18:38:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com (Postfix) with ESMTPS id 900582822B8; Wed, 18 Dec 2019 18:37:58 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Wed, 18 Dec 2019 18:37:57 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.109) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 18 Dec 2019 18:37:52 +0000 From: SeongJae Park To: , , , Date: Wed, 18 Dec 2019 19:37:16 +0100 Message-ID: <20191218183718.31719-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191218183718.31719-1-sjpark@amazon.com> References: <20191218183718.31719-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.109] X-ClientProxiedBy: EX13D10UWB003.ant.amazon.com (10.43.161.106) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Precedence: Bulk Subject: [Xen-devel] [PATCH v13 3/5] xen/blkback: Squeeze page pools if a memory pressure is detected X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, sjpark@amazon.com, pdurrant@amazon.com, SeongJae Park , linux-kernel@vger.kernel.org, sj38.park@gmail.com, xen-devel@lists.xenproject.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" RnJvbTogU2VvbmdKYWUgUGFyayA8c2pwYXJrQGFtYXpvbi5kZT4KCkVhY2ggYGJsa2lmYCBoYXMg YSBmcmVlIHBhZ2VzIHBvb2wgZm9yIHRoZSBncmFudCBtYXBwaW5nLiAgVGhlIHNpemUgb2YKdGhl IHBvb2wgc3RhcnRzIGZyb20gemVybyBhbmQgaXMgaW5jcmVhc2VkIG9uIGRlbWFuZCB3aGlsZSBw cm9jZXNzaW5nCnRoZSBJL08gcmVxdWVzdHMuICBJZiBjdXJyZW50IEkvTyByZXF1ZXN0cyBoYW5k bGluZyBpcyBmaW5pc2hlZCBvciAxMDAKbWlsbGlzZWNvbmRzIGhhcyBwYXNzZWQgc2luY2UgbGFz dCBJL08gcmVxdWVzdHMgaGFuZGxpbmcsIGl0IGNoZWNrcyBhbmQKc2hyaW5rcyB0aGUgcG9vbCB0 byBub3QgZXhjZWVkIHRoZSBzaXplIGxpbWl0LCBgbWF4X2J1ZmZlcl9wYWdlc2AuCgpUaGVyZWZv cmUsIGhvc3QgYWRtaW5pc3RyYXRvcnMgY2FuIGNhdXNlIG1lbW9yeSBwcmVzc3VyZSBpbiBibGti YWNrIGJ5CmF0dGFjaGluZyBhIGxhcmdlIG51bWJlciBvZiBibG9jayBkZXZpY2VzIGFuZCBpbmR1 Y2luZyBJL08uICBTdWNoCnByb2JsZW1hdGljIHNpdHVhdGlvbnMgY2FuIGJlIGF2b2lkZWQgYnkg bGltaXRpbmcgdGhlIG1heGltdW0gbnVtYmVyIG9mCmRldmljZXMgdGhhdCBjYW4gYmUgYXR0YWNo ZWQsIGJ1dCBmaW5kaW5nIHRoZSBvcHRpbWFsIGxpbWl0IGlzIG5vdCBzbwplYXN5LiAgSW1wcm9w ZXIgc2V0IG9mIHRoZSBsaW1pdCBjYW4gcmVzdWx0cyBpbiBtZW1vcnkgcHJlc3N1cmUgb3IgYQpy ZXNvdXJjZSB1bmRlcnV0aWxpemF0aW9uLiAgVGhpcyBjb21taXQgYXZvaWRzIHN1Y2ggcHJvYmxl bWF0aWMKc2l0dWF0aW9ucyBieSBzcXVlZXppbmcgdGhlIHBvb2xzIChyZXR1cm5zIGV2ZXJ5IGZy ZWUgcGFnZSBpbiB0aGUgcG9vbAp0byB0aGUgc3lzdGVtKSBmb3IgYSB3aGlsZSAodXNlcnMgY2Fu IHNldCB0aGlzIGR1cmF0aW9uIHZpYSBhIG1vZHVsZQpwYXJhbWV0ZXIpIGlmIG1lbW9yeSBwcmVz c3VyZSBpcyBkZXRlY3RlZC4KCkRpc2N1c3Npb25zCj09PT09PT09PT09CgpUaGUgYGJsa2JhY2tg J3Mgb3JpZ2luYWwgc2hyaW5raW5nIG1lY2hhbmlzbSByZXR1cm5zIG9ubHkgcGFnZXMgaW4gdGhl CnBvb2wgd2hpY2ggYXJlIG5vdCBjdXJyZW50bHkgYmUgdXNlZCBieSBgYmxrYmFja2AgdG8gdGhl IHN5c3RlbS4gIEluCm90aGVyIHdvcmRzLCB0aGUgcGFnZXMgdGhhdCBhcmUgbm90IG1hcHBlZCB3 aXRoIGdyYW50ZWQgcGFnZXMuICBCZWNhdXNlCnRoaXMgY29tbWl0IGlzIGNoYW5naW5nIG9ubHkg dGhlIHNocmluayBsaW1pdCBidXQgc3RpbGwgdXNlcyB0aGUgc2FtZQpmcmVlaW5nIG1lY2hhbmlz bSBpdCBkb2VzIG5vdCB0b3VjaCBwYWdlcyB3aGljaCBhcmUgY3VycmVudGx5IG1hcHBpbmcKZ3Jh bnRzLgoKT25jZSBtZW1vcnkgcHJlc3N1cmUgaXMgZGV0ZWN0ZWQsIHRoaXMgY29tbWl0IGtlZXBz IHRoZSBzcXVlZXppbmcgbGltaXQKZm9yIGEgdXNlci1zcGVjaWZpZWQgdGltZSBkdXJhdGlvbi4g IFRoZSBkdXJhdGlvbiBzaG91bGQgYmUgbmVpdGhlciB0b28KbG9uZyBub3IgdG9vIHNob3J0LiAg SWYgaXQgaXMgdG9vIGxvbmcsIHRoZSBzcXVlZXppbmcgaW5jdXJyaW5nIG92ZXJoZWFkCmNhbiBy ZWR1Y2UgdGhlIEkvTyBwZXJmb3JtYW5jZS4gIElmIGl0IGlzIHRvbyBzaG9ydCwgYGJsa2JhY2tg IHdpbGwgbm90CmZyZWUgZW5vdWdoIHBhZ2VzIHRvIHJlZHVjZSB0aGUgbWVtb3J5IHByZXNzdXJl LiAgVGhpcyBjb21taXQgc2V0cyB0aGUKdmFsdWUgYXMgYDEwIG1pbGxpc2Vjb25kc2AgYnkgZGVm YXVsdCBiZWNhdXNlIGl0IGlzIGEgc2hvcnQgdGltZSBpbgp0ZXJtcyBvZiBJL08gd2hpbGUgaXQg aXMgYSBsb25nIHRpbWUgaW4gdGVybXMgb2YgbWVtb3J5IG9wZXJhdGlvbnMuCkFsc28sIGFzIHRo ZSBvcmlnaW5hbCBzaHJpbmtpbmcgbWVjaGFuaXNtIHdvcmtzIGZvciBhdCBsZWFzdCBldmVyeSAx MDAKbWlsbGlzZWNvbmRzLCB0aGlzIGNvdWxkIGJlIGEgc29tZXdoYXQgcmVhc29uYWJsZSBjaG9p Y2UuICBJIGFsc28gdGVzdGVkCm90aGVyIGR1cmF0aW9ucyAocmVmZXIgdG8gdGhlIGJlbG93IHNl Y3Rpb24gZm9yIG1vcmUgZGV0YWlscykgYW5kCmNvbmZpcm1lZCB0aGF0IDEwIG1pbGxpc2Vjb25k cyBpcyB0aGUgb25lIHRoYXQgd29ya3MgYmVzdCB3aXRoIHRoZSB0ZXN0LgpUaGF0IHNhaWQsIHRo ZSBwcm9wZXIgZHVyYXRpb24gZGVwZW5kcyBvbiBhY3R1YWwgY29uZmlndXJhdGlvbnMgYW5kCndv cmtsb2Fkcy4gIFRoYXQncyB3aHkgdGhpcyBjb21taXQgYWxsb3dzIHVzZXJzIHRvIHNldCB0aGUg ZHVyYXRpb24gYXMgYQptb2R1bGUgcGFyYW1ldGVyLgoKTWVtb3J5IFByZXNzdXJlIFRlc3QKPT09 PT09PT09PT09PT09PT09PT0KClRvIHNob3cgaG93IHRoaXMgY29tbWl0IGZpeGVzIHRoZSBtZW1v cnkgcHJlc3N1cmUgc2l0dWF0aW9uIHdlbGwsIEkKY29uZmlndXJlZCBhIHRlc3QgZW52aXJvbm1l bnQgb24gYSB4ZW4tcnVubmluZyB2aXJ0dWFsaXphdGlvbiBzeXN0ZW0uCk9uIHRoZSBgYmxrZnJv bnRgIHJ1bm5pbmcgZ3Vlc3QgaW5zdGFuY2VzLCBJIGF0dGFjaCBhIGxhcmdlIG51bWJlciBvZgpu ZXR3b3JrLWJhY2tlZCB2b2x1bWUgZGV2aWNlcyBhbmQgaW5kdWNlIEkvTyB0byB0aG9zZS4gIE1l YW53aGlsZSwgSQptZWFzdXJlIHRoZSBudW1iZXIgb2YgcGFnZXMgdGhhdCBzd2FwcGVkIGluIChw c3dwaW4pIGFuZCBvdXQgKHBzd3BvdXQpCm9uIHRoZSBgYmxrYmFja2AgcnVubmluZyBndWVzdC4g IFRoZSB0ZXN0IHJhbiB0d2ljZSwgb25jZSBmb3IgdGhlCmBibGtiYWNrYCBiZWZvcmUgdGhpcyBj b21taXQgYW5kIG9uY2UgZm9yIHRoYXQgYWZ0ZXIgdGhpcyBjb21taXQuICBBcwpzaG93biBiZWxv dywgdGhpcyBjb21taXQgaGFzIGRyYW1hdGljYWxseSByZWR1Y2VkIHRoZSBtZW1vcnkgcHJlc3N1 cmU6CgogICAgICAgICAgICAgICAgcHN3cGluICBwc3dwb3V0CiAgICBiZWZvcmUgICAgICA3Niw2 NzIgIDE4NSw3OTkKICAgIGFmdGVyICAgICAgICAgIDg2NyAgICAzLDk2NwoKT3B0aW1hbCBBZ2dy ZXNzaXZlIFNocmlua2luZyBEdXJhdGlvbgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tCgpUbyBmaW5kIGEgYmVzdCBzcXVlZXppbmcgZHVyYXRpb24sIEkgcmVwZWF0ZWQgdGhl IHRlc3Qgd2l0aCB0aHJlZQpkaWZmZXJlbnQgZHVyYXRpb25zICgxbXMsIDEwbXMsIGFuZCAxMDBt cykuICBUaGUgcmVzdWx0cyBhcmUgYXMgYmVsb3c6CgogICAgZHVyYXRpb24gICAgcHN3cGluICBw c3dwb3V0CiAgICAxICAgICAgICAgICA3MDcgICAgIDUsMDk1CiAgICAxMCAgICAgICAgICA4Njcg ICAgIDMsOTY3CiAgICAxMDAgICAgICAgICAzNjIgICAgIDMsMzQ4CgpBcyBleHBlY3RlZCwgdGhl IG1lbW9yeSBwcmVzc3VyZSBkZWNyZWFzZXMgYXMgdGhlIGR1cmF0aW9uIGluY3JlYXNlcywKYnV0 IHRoZSByZWR1Y3Rpb24gYmVjb21lIHNsb3cgZnJvbSB0aGUgYDEwbXNgLiAgQmFzZWQgb24gdGhp cyByZXN1bHRzLCBJCmNob3NlIHRoZSBkZWZhdWx0IGR1cmF0aW9uIGFzIDEwbXMuCgpQZXJmb3Jt YW5jZSBPdmVyaGVhZCBUZXN0Cj09PT09PT09PT09PT09PT09PT09PT09PT0KClRoaXMgY29tbWl0 IGNvdWxkIGluY3VyIEkvTyBwZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiB1bmRlciBzZXZlcmUgbWVt b3J5CnByZXNzdXJlIGJlY2F1c2UgdGhlIHNxdWVlemluZyB3aWxsIHJlcXVpcmUgbW9yZSBwYWdl IGFsbG9jYXRpb25zIHBlcgpJL08uICBUbyBzaG93IHRoZSBvdmVyaGVhZCwgSSBhcnRpZmljaWFs bHkgbWFkZSBhIHdvcnN0LWNhc2Ugc3F1ZWV6aW5nCnNpdHVhdGlvbiBhbmQgbWVhc3VyZWQgdGhl IEkvTyBwZXJmb3JtYW5jZSBvZiBhIGBibGtmcm9udGAgcnVubmluZwpndWVzdC4KCkZvciB0aGUg YXJ0aWZpY2lhbCBzcXVlZXppbmcsIEkgc2V0IHRoZSBgYmxrYmFjay5tYXhfYnVmZmVyX3BhZ2Vz YCB1c2luZwp0aGUgYC9zeXMvbW9kdWxlL3hlbl9ibGtiYWNrL3BhcmFtZXRlcnMvbWF4X2J1ZmZl cl9wYWdlc2AgZmlsZS4gIEluIHRoaXMKdGVzdCwgSSBzZXQgdGhlIHZhbHVlIHRvIGAxMDI0YCBh bmQgYDBgLiAgVGhlIGAxMDI0YCBpcyB0aGUgZGVmYXVsdAp2YWx1ZS4gIFNldHRpbmcgdGhlIHZh bHVlIGFzIGAwYCBpcyBzYW1lIHRvIGEgc2l0dWF0aW9uIGRvaW5nIHRoZQpzcXVlZXppbmcgYWx3 YXlzICh3b3JzdC1jYXNlKS4KCklmIHRoZSB1bmRlcmx5aW5nIGJsb2NrIGRldmljZSBpcyBzbG93 IGVub3VnaCwgdGhlIHNxdWVlemluZyBvdmVyaGVhZApjb3VsZCBiZSBoaWRkZW4uICBGb3IgdGhl IHJlYXNvbiwgSSB1c2UgYSBmYXN0IGJsb2NrIGRldmljZSwgbmFtZWx5IHRoZQpyYmRbMV06Cgog ICAgIyB4bCBibG9jay1hdHRhY2ggZ3Vlc3QgcGh5Oi9kZXYvcmFtMCB4dmRiIHcKCkZvciB0aGUg SS9PIHBlcmZvcm1hbmNlIG1lYXN1cmVtZW50LCBJIHJ1biBhIHNpbXBsZSBgZGRgIGNvbW1hbmQg NSB0aW1lcwpkaXJlY3RseSB0byB0aGUgZGV2aWNlIGFzIGJlbG93IGFuZCBjb2xsZWN0IHRoZSAn TUIvcycgcmVzdWx0cy4KCiAgICAkIGZvciBpIGluIHsxLi41fTsgZG8gZGQgaWY9L2Rldi96ZXJv IG9mPS9kZXYveHZkYiBcCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYnM9NGsgY291bnQ9 JCgoMjU2KjUxMikpOyBzeW5jOyBkb25lCgpUaGUgcmVzdWx0cyBhcmUgYXMgYmVsb3cuICAnbWF4 X3BncycgcmVwcmVzZW50cyB0aGUgdmFsdWUgb2YgdGhlCmBibGtiYWNrLm1heF9idWZmZXJfcGFn ZXNgIHBhcmFtZXRlci4KCiAgICBtYXhfcGdzICAgTWluICAgICAgIE1heCAgICAgICBNZWRpYW4g ICAgIEF2ZyAgICBTdGRkZXYKICAgIDAgICAgICAgICA0MTcgICAgICAgNDIzICAgICAgIDQyMCAg ICAgICAgNDE5LjQgIDIuNTA5OTgwMQogICAgMTAyNCAgICAgIDQxNCAgICAgICA0MjUgICAgICAg NDE2ICAgICAgICA0MTcuOCAgNC40Mzg0NjgyCiAgICBObyBkaWZmZXJlbmNlIHByb3ZlbiBhdCA5 NS4wJSBjb25maWRlbmNlCgpJbiBzaG9ydCwgZXZlbiB3b3JzdCBjYXNlIHNxdWVlemluZyBvbiBy YW1kaXNrIGJhc2VkIGZhc3QgYmxvY2sgZGV2aWNlCm1ha2VzIG5vIHZpc2libGUgcGVyZm9ybWFu Y2UgZGVncmFkYXRpb24uICBQbGVhc2Ugbm90ZSB0aGF0IHRoaXMgaXMganVzdAphIHZlcnkgc2lt cGxlIGFuZCBtaW5pbWFsIHRlc3QuICBPbiBzeXN0ZW1zIHVzaW5nIHN1cGVyLWZhc3QgYmxvY2sK ZGV2aWNlcyBhbmQgYSBzcGVjaWFsIEkvTyB3b3JrbG9hZCwgdGhlIHJlc3VsdHMgbWlnaHQgYmUg ZGlmZmVyZW50LiAgSWYKeW91IGhhdmUgYW55IGRvdWJ0LCB0ZXN0IG9uIHlvdXIgbWFjaGluZSB3 aXRoIHlvdXIgd29ya2xvYWQgdG8gZmluZCB0aGUKb3B0aW1hbCBzcXVlZXppbmcgZHVyYXRpb24g Zm9yIHlvdS4KClsxXSBodHRwczovL3d3dy5rZXJuZWwub3JnL2RvYy9odG1sL2xhdGVzdC9hZG1p bi1ndWlkZS9ibG9ja2Rldi9yYW1kaXNrLmh0bWwKClNpZ25lZC1vZmYtYnk6IFNlb25nSmFlIFBh cmsgPHNqcGFya0BhbWF6b24uZGU+Ci0tLQogLi4uL0FCSS90ZXN0aW5nL3N5c2ZzLWRyaXZlci14 ZW4tYmxrYmFjayAgICAgIHwgMTAgKysrKysrKysKIGRyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2sv YmxrYmFjay5jICAgICAgICAgICB8ICA3ICsrKystLQogZHJpdmVycy9ibG9jay94ZW4tYmxrYmFj ay9jb21tb24uaCAgICAgICAgICAgIHwgIDEgKwogZHJpdmVycy9ibG9jay94ZW4tYmxrYmFjay94 ZW5idXMuYyAgICAgICAgICAgIHwgMjMgKysrKysrKysrKysrKysrKysrLQogNCBmaWxlcyBjaGFu Z2VkLCAzOCBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL0RvY3Vt ZW50YXRpb24vQUJJL3Rlc3Rpbmcvc3lzZnMtZHJpdmVyLXhlbi1ibGtiYWNrIGIvRG9jdW1lbnRh dGlvbi9BQkkvdGVzdGluZy9zeXNmcy1kcml2ZXIteGVuLWJsa2JhY2sKaW5kZXggNGU3YmFiYjNi YTFmLi5mMDEyMjQyMzFmM2YgMTAwNjQ0Ci0tLSBhL0RvY3VtZW50YXRpb24vQUJJL3Rlc3Rpbmcv c3lzZnMtZHJpdmVyLXhlbi1ibGtiYWNrCisrKyBiL0RvY3VtZW50YXRpb24vQUJJL3Rlc3Rpbmcv c3lzZnMtZHJpdmVyLXhlbi1ibGtiYWNrCkBAIC0yNSwzICsyNSwxMyBAQCBEZXNjcmlwdGlvbjoK ICAgICAgICAgICAgICAgICBhbGxvY2F0ZWQgd2l0aG91dCBiZWluZyBpbiB1c2UuIFRoZSB0aW1l IGlzIGluCiAgICAgICAgICAgICAgICAgc2Vjb25kcywgMCBtZWFucyBpbmRlZmluaXRlbHkgbG9u Zy4KICAgICAgICAgICAgICAgICBUaGUgZGVmYXVsdCBpcyA2MCBzZWNvbmRzLgorCitXaGF0OiAg ICAgICAgICAgL3N5cy9tb2R1bGUveGVuX2Jsa2JhY2svcGFyYW1ldGVycy9idWZmZXJfc3F1ZWV6 ZV9kdXJhdGlvbl9tcworRGF0ZTogICAgICAgICAgIERlY2VtYmVyIDIwMTkKK0tlcm5lbFZlcnNp b246ICA1LjUKK0NvbnRhY3Q6ICAgICAgICBTZW9uZ0phZSBQYXJrIDxzanBhcmtAYW1hem9uLmRl PgorRGVzY3JpcHRpb246CisgICAgICAgICAgICAgICAgV2hlbiBtZW1vcnkgcHJlc3N1cmUgaXMg cmVwb3J0ZWQgdG8gYmxrYmFjayB0aGlzIG9wdGlvbgorICAgICAgICAgICAgICAgIGNvbnRyb2xz IHRoZSBkdXJhdGlvbiBpbiBtaWxsaXNlY29uZHMgdGhhdCBibGtiYWNrIHdpbGwgbm90CisgICAg ICAgICAgICAgICAgY2FjaGUgYW55IHBhZ2Ugbm90IGJhY2tlZCBieSBhIGdyYW50IG1hcHBpbmcu CisgICAgICAgICAgICAgICAgVGhlIGRlZmF1bHQgaXMgMTBtcy4KZGlmZiAtLWdpdCBhL2RyaXZl cnMvYmxvY2sveGVuLWJsa2JhY2svYmxrYmFjay5jIGIvZHJpdmVycy9ibG9jay94ZW4tYmxrYmFj ay9ibGtiYWNrLmMKaW5kZXggZmQxZTE5ZjFhNDlmLi43OWY2NzdhZWI1Y2MgMTAwNjQ0Ci0tLSBh L2RyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2svYmxrYmFjay5jCisrKyBiL2RyaXZlcnMvYmxvY2sv eGVuLWJsa2JhY2svYmxrYmFjay5jCkBAIC02NTYsOCArNjU2LDExIEBAIGludCB4ZW5fYmxraWZf c2NoZWR1bGUodm9pZCAqYXJnKQogCQkJcmluZy0+bmV4dF9scnUgPSBqaWZmaWVzICsgbXNlY3Nf dG9famlmZmllcyhMUlVfSU5URVJWQUwpOwogCQl9CiAKLQkJLyogU2hyaW5rIGlmIHdlIGhhdmUg bW9yZSB0aGFuIHhlbl9ibGtpZl9tYXhfYnVmZmVyX3BhZ2VzICovCi0JCXNocmlua19mcmVlX3Bh Z2Vwb29sKHJpbmcsIHhlbl9ibGtpZl9tYXhfYnVmZmVyX3BhZ2VzKTsKKwkJLyogU2hyaW5rIHRo ZSBmcmVlIHBhZ2VzIHBvb2wgaWYgaXQgaXMgdG9vIGxhcmdlLiAqLworCQlpZiAodGltZV9iZWZv cmUoamlmZmllcywgYmxraWYtPmJ1ZmZlcl9zcXVlZXplX2VuZCkpCisJCQlzaHJpbmtfZnJlZV9w YWdlcG9vbChyaW5nLCAwKTsKKwkJZWxzZQorCQkJc2hyaW5rX2ZyZWVfcGFnZXBvb2wocmluZywg eGVuX2Jsa2lmX21heF9idWZmZXJfcGFnZXMpOwogCiAJCWlmIChsb2dfc3RhdHMgJiYgdGltZV9h ZnRlcihqaWZmaWVzLCByaW5nLT5zdF9wcmludCkpCiAJCQlwcmludF9zdGF0cyhyaW5nKTsKZGlm ZiAtLWdpdCBhL2RyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2svY29tbW9uLmggYi9kcml2ZXJzL2Js b2NrL3hlbi1ibGtiYWNrL2NvbW1vbi5oCmluZGV4IDFkMzAwMmQ3NzNmNy4uNTM2Yzg0ZjYxZmVk IDEwMDY0NAotLS0gYS9kcml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL2NvbW1vbi5oCisrKyBiL2Ry aXZlcnMvYmxvY2sveGVuLWJsa2JhY2svY29tbW9uLmgKQEAgLTMxOSw2ICszMTksNyBAQCBzdHJ1 Y3QgeGVuX2Jsa2lmIHsKIAkvKiBBbGwgcmluZ3MgZm9yIHRoaXMgZGV2aWNlLiAqLwogCXN0cnVj dCB4ZW5fYmxraWZfcmluZwkqcmluZ3M7CiAJdW5zaWduZWQgaW50CQlucl9yaW5nczsKKwl1bnNp Z25lZCBsb25nCQlidWZmZXJfc3F1ZWV6ZV9lbmQ7CiB9OwogCiBzdHJ1Y3Qgc2VnX2J1ZiB7CmRp ZmYgLS1naXQgYS9kcml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL3hlbmJ1cy5jIGIvZHJpdmVycy9i bG9jay94ZW4tYmxrYmFjay94ZW5idXMuYwppbmRleCBiOTBkYmNkOTljMDMuLjI0MTcyYzE4MGY1 ZiAxMDA2NDQKLS0tIGEvZHJpdmVycy9ibG9jay94ZW4tYmxrYmFjay94ZW5idXMuYworKysgYi9k cml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL3hlbmJ1cy5jCkBAIC04MjQsNiArODI0LDI2IEBAIHN0 YXRpYyB2b2lkIGZyb250ZW5kX2NoYW5nZWQoc3RydWN0IHhlbmJ1c19kZXZpY2UgKmRldiwKIH0K IAogCisvKiBPbmNlIGEgbWVtb3J5IHByZXNzdXJlIGlzIGRldGVjdGVkLCBzcXVlZXplIGZyZWUg cGFnZSBwb29scyBmb3IgYSB3aGlsZS4gKi8KK3N0YXRpYyB1bnNpZ25lZCBpbnQgYnVmZmVyX3Nx dWVlemVfZHVyYXRpb25fbXMgPSAxMDsKK21vZHVsZV9wYXJhbV9uYW1lZChidWZmZXJfc3F1ZWV6 ZV9kdXJhdGlvbl9tcywKKwkJYnVmZmVyX3NxdWVlemVfZHVyYXRpb25fbXMsIGludCwgMDY0NCk7 CitNT0RVTEVfUEFSTV9ERVNDKGJ1ZmZlcl9zcXVlZXplX2R1cmF0aW9uX21zLAorIkR1cmF0aW9u IGluIG1zIHRvIHNxdWVlemUgcGFnZXMgYnVmZmVyIHdoZW4gYSBtZW1vcnkgcHJlc3N1cmUgaXMg ZGV0ZWN0ZWQiKTsKKworLyoKKyAqIENhbGxiYWNrIHJlY2VpdmVkIHdoZW4gdGhlIG1lbW9yeSBw cmVzc3VyZSBpcyBkZXRlY3RlZC4KKyAqLworc3RhdGljIHZvaWQgcmVjbGFpbV9tZW1vcnkoc3Ry dWN0IHhlbmJ1c19kZXZpY2UgKmRldikKK3sKKwlzdHJ1Y3QgYmFja2VuZF9pbmZvICpiZSA9IGRl dl9nZXRfZHJ2ZGF0YSgmZGV2LT5kZXYpOworCisJaWYgKCFiZSkKKwkJcmV0dXJuOworCWJlLT5i bGtpZi0+YnVmZmVyX3NxdWVlemVfZW5kID0gamlmZmllcyArCisJCW1zZWNzX3RvX2ppZmZpZXMo YnVmZmVyX3NxdWVlemVfZHVyYXRpb25fbXMpOworfQorCiAvKiAqKiBDb25uZWN0aW9uICoqICov CiAKIApAQCAtMTExNSw3ICsxMTM1LDggQEAgc3RhdGljIHN0cnVjdCB4ZW5idXNfZHJpdmVyIHhl bl9ibGtia19kcml2ZXIgPSB7CiAJLmlkcyAgPSB4ZW5fYmxrYmtfaWRzLAogCS5wcm9iZSA9IHhl bl9ibGtia19wcm9iZSwKIAkucmVtb3ZlID0geGVuX2Jsa2JrX3JlbW92ZSwKLQkub3RoZXJlbmRf Y2hhbmdlZCA9IGZyb250ZW5kX2NoYW5nZWQKKwkub3RoZXJlbmRfY2hhbmdlZCA9IGZyb250ZW5k X2NoYW5nZWQsCisJLnJlY2xhaW1fbWVtb3J5ID0gcmVjbGFpbV9tZW1vcnksCiB9OwogCiBpbnQg eGVuX2Jsa2lmX3hlbmJ1c19pbml0KHZvaWQpCi0tIAoyLjE3LjEKCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhl bi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3Jn L21haWxtYW4vbGlzdGluZm8veGVuLWRldmVs