From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63786C43603 for ; Tue, 17 Dec 2019 16:09:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2390121582 for ; Tue, 17 Dec 2019 16:09:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="XnfCSKi6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728102AbfLQQJK (ORCPT ); Tue, 17 Dec 2019 11:09:10 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:41981 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726858AbfLQQJK (ORCPT ); Tue, 17 Dec 2019 11:09:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576598949; x=1608134949; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0ZYoJMeZeJgi97Oa401PwKhbr/wNKMQ/89p6u0X86e8=; b=XnfCSKi6WLhc76hoDAjw0Lnevd4vyOfoF34xteNqdYy9QUCng0PBnkVF uMpUyx4ClS8urSl+GGnPF4F3L5wVSSw/nDLMcsglQ3Ma7ErTeDbv42Lx5 RnO9ZRjmptuHxTEYu3f1pXe0kas+wpTN78V3ivu2EkQBkP2BAcsZErzCZ c=; IronPort-SDR: cIaB9NeD8xqTERM+CXt0x4VPSx5N/Y9oITTlAlbJ8WmvAjTuBLhCRUT9f4nvqRyET01I8zl4Qv V8FY2lapr4Dw== X-IronPort-AV: E=Sophos;i="5.69,326,1571702400"; d="scan'208";a="14050305" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 17 Dec 2019 16:08:56 +0000 Received: from EX13MTAUEA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com (Postfix) with ESMTPS id 37C98C0713; Tue, 17 Dec 2019 16:08:55 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA001.ant.amazon.com (10.43.61.243) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 17 Dec 2019 16:08:54 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.179) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 17 Dec 2019 16:08:49 +0000 From: SeongJae Park To: , , , CC: SeongJae Park , , , , , , Subject: [PATCH v11 3/6] xen/blkback: Squeeze page pools if a memory pressure is detected Date: Tue, 17 Dec 2019 17:07:45 +0100 Message-ID: <20191217160748.693-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191217160748.693-1-sjpark@amazon.com> References: <20191217160748.693-1-sjpark@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.43.161.179] X-ClientProxiedBy: EX13d09UWC002.ant.amazon.com (10.43.162.102) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: SeongJae Park Each `blkif` has a free pages pool for the grant mapping. The size of the pool starts from zero and is increased on demand while processing the I/O requests. If current I/O requests handling is finished or 100 milliseconds has passed since last I/O requests handling, it checks and shrinks the pool to not exceed the size limit, `max_buffer_pages`. Therefore, host administrators can cause memory pressure in blkback by attaching a large number of block devices and inducing I/O. Such problematic situations can be avoided by limiting the maximum number of devices that can be attached, but finding the optimal limit is not so easy. Improper set of the limit can results in memory pressure or a resource underutilization. This commit avoids such problematic situations by squeezing the pools (returns every free page in the pool to the system) for a while (users can set this duration via a module parameter) if memory pressure is detected. Discussions =========== The `blkback`'s original shrinking mechanism returns only pages in the pool which are not currently be used by `blkback` to the system. In other words, the pages that are not mapped with granted pages. Because this commit is changing only the shrink limit but still uses the same freeing mechanism it does not touch pages which are currently mapping grants. Once memory pressure is detected, this commit keeps the squeezing limit for a user-specified time duration. The duration should be neither too long nor too short. If it is too long, the squeezing incurring overhead can reduce the I/O performance. If it is too short, `blkback` will not free enough pages to reduce the memory pressure. This commit sets the value as `10 milliseconds` by default because it is a short time in terms of I/O while it is a long time in terms of memory operations. Also, as the original shrinking mechanism works for at least every 100 milliseconds, this could be a somewhat reasonable choice. I also tested other durations (refer to the below section for more details) and confirmed that 10 milliseconds is the one that works best with the test. That said, the proper duration depends on actual configurations and workloads. That's why this commit allows users to set the duration as a module parameter. Memory Pressure Test ==================== To show how this commit fixes the memory pressure situation well, I configured a test environment on a xen-running virtualization system. On the `blkfront` running guest instances, I attach a large number of network-backed volume devices and induce I/O to those. Meanwhile, I measure the number of pages that swapped in (pswpin) and out (pswpout) on the `blkback` running guest. The test ran twice, once for the `blkback` before this commit and once for that after this commit. As shown below, this commit has dramatically reduced the memory pressure: pswpin pswpout before 76,672 185,799 after 212 3,325 Optimal Aggressive Shrinking Duration ------------------------------------- To find a best squeezing duration, I repeated the test with three different durations (1ms, 10ms, and 100ms). The results are as below: duration pswpin pswpout 1 852 6,424 10 212 3,325 100 203 3,340 As expected, the memory pressure has decreased as the duration is increased, but the reduction stopped from the `10ms`. Based on this results, I chose the default duration as 10ms. Performance Overhead Test ========================= This commit could incur I/O performance degradation under severe memory pressure because the squeezing will require more page allocations per I/O. To show the overhead, I artificially made a worst-case squeezing situation and measured the I/O performance of a `blkfront` running guest. For the artificial squeezing, I set the `blkback.max_buffer_pages` using the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. In this test, I set the value to `1024` and `0`. The `1024` is the default value. Setting the value as `0` is same to a situation doing the squeezing always (worst-case). If the underlying block device is slow enough, the squeezing overhead could be hidden. For the reason, I use a fast block device, namely the rbd[1]: # xl block-attach guest phy:/dev/ram0 xvdb w For the I/O performance measurement, I run a simple `dd` command 5 times directly to the device as below and collect the 'MB/s' results. $ for i in {1..5}; do dd if=/dev/zero of=/dev/xvdb \ bs=4k count=$((256*512)); sync; done The results are as below. 'max_pgs' represents the value of the `blkback.max_buffer_pages` parameter. max_pgs Min Max Median Avg Stddev 0 417 423 420 419.4 2.5099801 1024 414 425 416 417.8 4.4384682 No difference proven at 95.0% confidence In short, even worst case squeezing on ramdisk based fast block device makes no visible performance degradation. Please note that this is just a very simple and minimal test. On systems using super-fast block devices and a special I/O workload, the results might be different. If you have any doubt, test on your machine with your workload to find the optimal squeezing duration for you. [1] https://www.kernel.org/doc/html/latest/admin-guide/blockdev/ramdisk.html Reviewed-by: Roger Pau Monné Signed-off-by: SeongJae Park --- .../ABI/testing/sysfs-driver-xen-blkback | 10 +++++++++ drivers/block/xen-blkback/blkback.c | 7 +++++-- drivers/block/xen-blkback/common.h | 1 + drivers/block/xen-blkback/xenbus.c | 21 ++++++++++++++++++- 4 files changed, 36 insertions(+), 3 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback index 4e7babb3ba1f..f01224231f3f 100644 --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback @@ -25,3 +25,13 @@ Description: allocated without being in use. The time is in seconds, 0 means indefinitely long. The default is 60 seconds. + +What: /sys/module/xen_blkback/parameters/buffer_squeeze_duration_ms +Date: December 2019 +KernelVersion: 5.5 +Contact: SeongJae Park +Description: + When memory pressure is reported to blkback this option + controls the duration in milliseconds that blkback will not + cache any page not backed by a grant mapping. + The default is 10ms. diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index fd1e19f1a49f..79f677aeb5cc 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -656,8 +656,11 @@ int xen_blkif_schedule(void *arg) ring->next_lru = jiffies + msecs_to_jiffies(LRU_INTERVAL); } - /* Shrink if we have more than xen_blkif_max_buffer_pages */ - shrink_free_pagepool(ring, xen_blkif_max_buffer_pages); + /* Shrink the free pages pool if it is too large. */ + if (time_before(jiffies, blkif->buffer_squeeze_end)) + shrink_free_pagepool(ring, 0); + else + shrink_free_pagepool(ring, xen_blkif_max_buffer_pages); if (log_stats && time_after(jiffies, ring->st_print)) print_stats(ring); diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h index 1d3002d773f7..536c84f61fed 100644 --- a/drivers/block/xen-blkback/common.h +++ b/drivers/block/xen-blkback/common.h @@ -319,6 +319,7 @@ struct xen_blkif { /* All rings for this device. */ struct xen_blkif_ring *rings; unsigned int nr_rings; + unsigned long buffer_squeeze_end; }; struct seg_buf { diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index b90dbcd99c03..4f6ea4feca79 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -824,6 +824,24 @@ static void frontend_changed(struct xenbus_device *dev, } +/* Once a memory pressure is detected, squeeze free page pools for a while. */ +static unsigned int buffer_squeeze_duration_ms = 10; +module_param_named(buffer_squeeze_duration_ms, + buffer_squeeze_duration_ms, int, 0644); +MODULE_PARM_DESC(buffer_squeeze_duration_ms, +"Duration in ms to squeeze pages buffer when a memory pressure is detected"); + +/* + * Callback received when the memory pressure is detected. + */ +static void reclaim_memory(struct xenbus_device *dev) +{ + struct backend_info *be = dev_get_drvdata(&dev->dev); + + be->blkif->buffer_squeeze_end = jiffies + + msecs_to_jiffies(buffer_squeeze_duration_ms); +} + /* ** Connection ** */ @@ -1115,7 +1133,8 @@ static struct xenbus_driver xen_blkbk_driver = { .ids = xen_blkbk_ids, .probe = xen_blkbk_probe, .remove = xen_blkbk_remove, - .otherend_changed = frontend_changed + .otherend_changed = frontend_changed, + .reclaim_memory = reclaim_memory, }; int xen_blkif_xenbus_init(void) -- 2.17.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=DKIM_ADSP_ALL,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 987B6C2D0CD for ; Tue, 17 Dec 2019 16:09:21 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6222F24679 for ; Tue, 17 Dec 2019 16:09:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="XnfCSKi6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6222F24679 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihFPL-0006Ej-Ae; Tue, 17 Dec 2019 16:09:11 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihFPJ-0006EB-MD for xen-devel@lists.xenproject.org; Tue, 17 Dec 2019 16:09:09 +0000 X-Inumbo-ID: 8e6dff06-20e7-11ea-88e7-bc764e2007e4 Received: from smtp-fw-9102.amazon.com (unknown [207.171.184.29]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8e6dff06-20e7-11ea-88e7-bc764e2007e4; Tue, 17 Dec 2019 16:09:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576598949; x=1608134949; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0ZYoJMeZeJgi97Oa401PwKhbr/wNKMQ/89p6u0X86e8=; b=XnfCSKi6WLhc76hoDAjw0Lnevd4vyOfoF34xteNqdYy9QUCng0PBnkVF uMpUyx4ClS8urSl+GGnPF4F3L5wVSSw/nDLMcsglQ3Ma7ErTeDbv42Lx5 RnO9ZRjmptuHxTEYu3f1pXe0kas+wpTN78V3ivu2EkQBkP2BAcsZErzCZ c=; IronPort-SDR: cIaB9NeD8xqTERM+CXt0x4VPSx5N/Y9oITTlAlbJ8WmvAjTuBLhCRUT9f4nvqRyET01I8zl4Qv V8FY2lapr4Dw== X-IronPort-AV: E=Sophos;i="5.69,326,1571702400"; d="scan'208";a="14050305" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 17 Dec 2019 16:08:56 +0000 Received: from EX13MTAUEA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com (Postfix) with ESMTPS id 37C98C0713; Tue, 17 Dec 2019 16:08:55 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA001.ant.amazon.com (10.43.61.243) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 17 Dec 2019 16:08:54 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.161.179) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 17 Dec 2019 16:08:49 +0000 From: SeongJae Park To: , , , Date: Tue, 17 Dec 2019 17:07:45 +0100 Message-ID: <20191217160748.693-4-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191217160748.693-1-sjpark@amazon.com> References: <20191217160748.693-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.179] X-ClientProxiedBy: EX13d09UWC002.ant.amazon.com (10.43.162.102) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Precedence: Bulk Subject: [Xen-devel] [PATCH v11 3/6] xen/blkback: Squeeze page pools if a memory pressure is detected X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, sjpark@amazon.com, pdurrant@amazon.com, SeongJae Park , linux-kernel@vger.kernel.org, sj38.park@gmail.com, xen-devel@lists.xenproject.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" RnJvbTogU2VvbmdKYWUgUGFyayA8c2pwYXJrQGFtYXpvbi5kZT4KCkVhY2ggYGJsa2lmYCBoYXMg YSBmcmVlIHBhZ2VzIHBvb2wgZm9yIHRoZSBncmFudCBtYXBwaW5nLiAgVGhlIHNpemUgb2YKdGhl IHBvb2wgc3RhcnRzIGZyb20gemVybyBhbmQgaXMgaW5jcmVhc2VkIG9uIGRlbWFuZCB3aGlsZSBw cm9jZXNzaW5nCnRoZSBJL08gcmVxdWVzdHMuICBJZiBjdXJyZW50IEkvTyByZXF1ZXN0cyBoYW5k bGluZyBpcyBmaW5pc2hlZCBvciAxMDAKbWlsbGlzZWNvbmRzIGhhcyBwYXNzZWQgc2luY2UgbGFz dCBJL08gcmVxdWVzdHMgaGFuZGxpbmcsIGl0IGNoZWNrcyBhbmQKc2hyaW5rcyB0aGUgcG9vbCB0 byBub3QgZXhjZWVkIHRoZSBzaXplIGxpbWl0LCBgbWF4X2J1ZmZlcl9wYWdlc2AuCgpUaGVyZWZv cmUsIGhvc3QgYWRtaW5pc3RyYXRvcnMgY2FuIGNhdXNlIG1lbW9yeSBwcmVzc3VyZSBpbiBibGti YWNrIGJ5CmF0dGFjaGluZyBhIGxhcmdlIG51bWJlciBvZiBibG9jayBkZXZpY2VzIGFuZCBpbmR1 Y2luZyBJL08uICBTdWNoCnByb2JsZW1hdGljIHNpdHVhdGlvbnMgY2FuIGJlIGF2b2lkZWQgYnkg bGltaXRpbmcgdGhlIG1heGltdW0gbnVtYmVyIG9mCmRldmljZXMgdGhhdCBjYW4gYmUgYXR0YWNo ZWQsIGJ1dCBmaW5kaW5nIHRoZSBvcHRpbWFsIGxpbWl0IGlzIG5vdCBzbwplYXN5LiAgSW1wcm9w ZXIgc2V0IG9mIHRoZSBsaW1pdCBjYW4gcmVzdWx0cyBpbiBtZW1vcnkgcHJlc3N1cmUgb3IgYQpy ZXNvdXJjZSB1bmRlcnV0aWxpemF0aW9uLiAgVGhpcyBjb21taXQgYXZvaWRzIHN1Y2ggcHJvYmxl bWF0aWMKc2l0dWF0aW9ucyBieSBzcXVlZXppbmcgdGhlIHBvb2xzIChyZXR1cm5zIGV2ZXJ5IGZy ZWUgcGFnZSBpbiB0aGUgcG9vbAp0byB0aGUgc3lzdGVtKSBmb3IgYSB3aGlsZSAodXNlcnMgY2Fu IHNldCB0aGlzIGR1cmF0aW9uIHZpYSBhIG1vZHVsZQpwYXJhbWV0ZXIpIGlmIG1lbW9yeSBwcmVz c3VyZSBpcyBkZXRlY3RlZC4KCkRpc2N1c3Npb25zCj09PT09PT09PT09CgpUaGUgYGJsa2JhY2tg J3Mgb3JpZ2luYWwgc2hyaW5raW5nIG1lY2hhbmlzbSByZXR1cm5zIG9ubHkgcGFnZXMgaW4gdGhl CnBvb2wgd2hpY2ggYXJlIG5vdCBjdXJyZW50bHkgYmUgdXNlZCBieSBgYmxrYmFja2AgdG8gdGhl IHN5c3RlbS4gIEluCm90aGVyIHdvcmRzLCB0aGUgcGFnZXMgdGhhdCBhcmUgbm90IG1hcHBlZCB3 aXRoIGdyYW50ZWQgcGFnZXMuICBCZWNhdXNlCnRoaXMgY29tbWl0IGlzIGNoYW5naW5nIG9ubHkg dGhlIHNocmluayBsaW1pdCBidXQgc3RpbGwgdXNlcyB0aGUgc2FtZQpmcmVlaW5nIG1lY2hhbmlz bSBpdCBkb2VzIG5vdCB0b3VjaCBwYWdlcyB3aGljaCBhcmUgY3VycmVudGx5IG1hcHBpbmcKZ3Jh bnRzLgoKT25jZSBtZW1vcnkgcHJlc3N1cmUgaXMgZGV0ZWN0ZWQsIHRoaXMgY29tbWl0IGtlZXBz IHRoZSBzcXVlZXppbmcgbGltaXQKZm9yIGEgdXNlci1zcGVjaWZpZWQgdGltZSBkdXJhdGlvbi4g IFRoZSBkdXJhdGlvbiBzaG91bGQgYmUgbmVpdGhlciB0b28KbG9uZyBub3IgdG9vIHNob3J0LiAg SWYgaXQgaXMgdG9vIGxvbmcsIHRoZSBzcXVlZXppbmcgaW5jdXJyaW5nIG92ZXJoZWFkCmNhbiBy ZWR1Y2UgdGhlIEkvTyBwZXJmb3JtYW5jZS4gIElmIGl0IGlzIHRvbyBzaG9ydCwgYGJsa2JhY2tg IHdpbGwgbm90CmZyZWUgZW5vdWdoIHBhZ2VzIHRvIHJlZHVjZSB0aGUgbWVtb3J5IHByZXNzdXJl LiAgVGhpcyBjb21taXQgc2V0cyB0aGUKdmFsdWUgYXMgYDEwIG1pbGxpc2Vjb25kc2AgYnkgZGVm YXVsdCBiZWNhdXNlIGl0IGlzIGEgc2hvcnQgdGltZSBpbgp0ZXJtcyBvZiBJL08gd2hpbGUgaXQg aXMgYSBsb25nIHRpbWUgaW4gdGVybXMgb2YgbWVtb3J5IG9wZXJhdGlvbnMuCkFsc28sIGFzIHRo ZSBvcmlnaW5hbCBzaHJpbmtpbmcgbWVjaGFuaXNtIHdvcmtzIGZvciBhdCBsZWFzdCBldmVyeSAx MDAKbWlsbGlzZWNvbmRzLCB0aGlzIGNvdWxkIGJlIGEgc29tZXdoYXQgcmVhc29uYWJsZSBjaG9p Y2UuICBJIGFsc28gdGVzdGVkCm90aGVyIGR1cmF0aW9ucyAocmVmZXIgdG8gdGhlIGJlbG93IHNl Y3Rpb24gZm9yIG1vcmUgZGV0YWlscykgYW5kCmNvbmZpcm1lZCB0aGF0IDEwIG1pbGxpc2Vjb25k cyBpcyB0aGUgb25lIHRoYXQgd29ya3MgYmVzdCB3aXRoIHRoZSB0ZXN0LgpUaGF0IHNhaWQsIHRo ZSBwcm9wZXIgZHVyYXRpb24gZGVwZW5kcyBvbiBhY3R1YWwgY29uZmlndXJhdGlvbnMgYW5kCndv cmtsb2Fkcy4gIFRoYXQncyB3aHkgdGhpcyBjb21taXQgYWxsb3dzIHVzZXJzIHRvIHNldCB0aGUg ZHVyYXRpb24gYXMgYQptb2R1bGUgcGFyYW1ldGVyLgoKTWVtb3J5IFByZXNzdXJlIFRlc3QKPT09 PT09PT09PT09PT09PT09PT0KClRvIHNob3cgaG93IHRoaXMgY29tbWl0IGZpeGVzIHRoZSBtZW1v cnkgcHJlc3N1cmUgc2l0dWF0aW9uIHdlbGwsIEkKY29uZmlndXJlZCBhIHRlc3QgZW52aXJvbm1l bnQgb24gYSB4ZW4tcnVubmluZyB2aXJ0dWFsaXphdGlvbiBzeXN0ZW0uCk9uIHRoZSBgYmxrZnJv bnRgIHJ1bm5pbmcgZ3Vlc3QgaW5zdGFuY2VzLCBJIGF0dGFjaCBhIGxhcmdlIG51bWJlciBvZgpu ZXR3b3JrLWJhY2tlZCB2b2x1bWUgZGV2aWNlcyBhbmQgaW5kdWNlIEkvTyB0byB0aG9zZS4gIE1l YW53aGlsZSwgSQptZWFzdXJlIHRoZSBudW1iZXIgb2YgcGFnZXMgdGhhdCBzd2FwcGVkIGluIChw c3dwaW4pIGFuZCBvdXQgKHBzd3BvdXQpCm9uIHRoZSBgYmxrYmFja2AgcnVubmluZyBndWVzdC4g IFRoZSB0ZXN0IHJhbiB0d2ljZSwgb25jZSBmb3IgdGhlCmBibGtiYWNrYCBiZWZvcmUgdGhpcyBj b21taXQgYW5kIG9uY2UgZm9yIHRoYXQgYWZ0ZXIgdGhpcyBjb21taXQuICBBcwpzaG93biBiZWxv dywgdGhpcyBjb21taXQgaGFzIGRyYW1hdGljYWxseSByZWR1Y2VkIHRoZSBtZW1vcnkgcHJlc3N1 cmU6CgogICAgICAgICAgICAgICAgcHN3cGluICBwc3dwb3V0CiAgICBiZWZvcmUgICAgICA3Niw2 NzIgIDE4NSw3OTkKICAgIGFmdGVyICAgICAgICAgIDIxMiAgICAzLDMyNQoKT3B0aW1hbCBBZ2dy ZXNzaXZlIFNocmlua2luZyBEdXJhdGlvbgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tCgpUbyBmaW5kIGEgYmVzdCBzcXVlZXppbmcgZHVyYXRpb24sIEkgcmVwZWF0ZWQgdGhl IHRlc3Qgd2l0aCB0aHJlZQpkaWZmZXJlbnQgZHVyYXRpb25zICgxbXMsIDEwbXMsIGFuZCAxMDBt cykuICBUaGUgcmVzdWx0cyBhcmUgYXMgYmVsb3c6CgogICAgZHVyYXRpb24gICAgcHN3cGluICBw c3dwb3V0CiAgICAxICAgICAgICAgICA4NTIgICAgIDYsNDI0CiAgICAxMCAgICAgICAgICAyMTIg ICAgIDMsMzI1CiAgICAxMDAgICAgICAgICAyMDMgICAgIDMsMzQwCgpBcyBleHBlY3RlZCwgdGhl IG1lbW9yeSBwcmVzc3VyZSBoYXMgZGVjcmVhc2VkIGFzIHRoZSBkdXJhdGlvbiBpcwppbmNyZWFz ZWQsIGJ1dCB0aGUgcmVkdWN0aW9uIHN0b3BwZWQgZnJvbSB0aGUgYDEwbXNgLiAgQmFzZWQgb24g dGhpcwpyZXN1bHRzLCBJIGNob3NlIHRoZSBkZWZhdWx0IGR1cmF0aW9uIGFzIDEwbXMuCgpQZXJm b3JtYW5jZSBPdmVyaGVhZCBUZXN0Cj09PT09PT09PT09PT09PT09PT09PT09PT0KClRoaXMgY29t bWl0IGNvdWxkIGluY3VyIEkvTyBwZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiB1bmRlciBzZXZlcmUg bWVtb3J5CnByZXNzdXJlIGJlY2F1c2UgdGhlIHNxdWVlemluZyB3aWxsIHJlcXVpcmUgbW9yZSBw YWdlIGFsbG9jYXRpb25zIHBlcgpJL08uICBUbyBzaG93IHRoZSBvdmVyaGVhZCwgSSBhcnRpZmlj aWFsbHkgbWFkZSBhIHdvcnN0LWNhc2Ugc3F1ZWV6aW5nCnNpdHVhdGlvbiBhbmQgbWVhc3VyZWQg dGhlIEkvTyBwZXJmb3JtYW5jZSBvZiBhIGBibGtmcm9udGAgcnVubmluZwpndWVzdC4KCkZvciB0 aGUgYXJ0aWZpY2lhbCBzcXVlZXppbmcsIEkgc2V0IHRoZSBgYmxrYmFjay5tYXhfYnVmZmVyX3Bh Z2VzYCB1c2luZwp0aGUgYC9zeXMvbW9kdWxlL3hlbl9ibGtiYWNrL3BhcmFtZXRlcnMvbWF4X2J1 ZmZlcl9wYWdlc2AgZmlsZS4gIEluIHRoaXMKdGVzdCwgSSBzZXQgdGhlIHZhbHVlIHRvIGAxMDI0 YCBhbmQgYDBgLiAgVGhlIGAxMDI0YCBpcyB0aGUgZGVmYXVsdAp2YWx1ZS4gIFNldHRpbmcgdGhl IHZhbHVlIGFzIGAwYCBpcyBzYW1lIHRvIGEgc2l0dWF0aW9uIGRvaW5nIHRoZQpzcXVlZXppbmcg YWx3YXlzICh3b3JzdC1jYXNlKS4KCklmIHRoZSB1bmRlcmx5aW5nIGJsb2NrIGRldmljZSBpcyBz bG93IGVub3VnaCwgdGhlIHNxdWVlemluZyBvdmVyaGVhZApjb3VsZCBiZSBoaWRkZW4uICBGb3Ig dGhlIHJlYXNvbiwgSSB1c2UgYSBmYXN0IGJsb2NrIGRldmljZSwgbmFtZWx5IHRoZQpyYmRbMV06 CgogICAgIyB4bCBibG9jay1hdHRhY2ggZ3Vlc3QgcGh5Oi9kZXYvcmFtMCB4dmRiIHcKCkZvciB0 aGUgSS9PIHBlcmZvcm1hbmNlIG1lYXN1cmVtZW50LCBJIHJ1biBhIHNpbXBsZSBgZGRgIGNvbW1h bmQgNSB0aW1lcwpkaXJlY3RseSB0byB0aGUgZGV2aWNlIGFzIGJlbG93IGFuZCBjb2xsZWN0IHRo ZSAnTUIvcycgcmVzdWx0cy4KCiAgICAkIGZvciBpIGluIHsxLi41fTsgZG8gZGQgaWY9L2Rldi96 ZXJvIG9mPS9kZXYveHZkYiBcCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYnM9NGsgY291 bnQ9JCgoMjU2KjUxMikpOyBzeW5jOyBkb25lCgpUaGUgcmVzdWx0cyBhcmUgYXMgYmVsb3cuICAn bWF4X3BncycgcmVwcmVzZW50cyB0aGUgdmFsdWUgb2YgdGhlCmBibGtiYWNrLm1heF9idWZmZXJf cGFnZXNgIHBhcmFtZXRlci4KCiAgICBtYXhfcGdzICAgTWluICAgICAgIE1heCAgICAgICBNZWRp YW4gICAgIEF2ZyAgICBTdGRkZXYKICAgIDAgICAgICAgICA0MTcgICAgICAgNDIzICAgICAgIDQy MCAgICAgICAgNDE5LjQgIDIuNTA5OTgwMQogICAgMTAyNCAgICAgIDQxNCAgICAgICA0MjUgICAg ICAgNDE2ICAgICAgICA0MTcuOCAgNC40Mzg0NjgyCiAgICBObyBkaWZmZXJlbmNlIHByb3ZlbiBh dCA5NS4wJSBjb25maWRlbmNlCgpJbiBzaG9ydCwgZXZlbiB3b3JzdCBjYXNlIHNxdWVlemluZyBv biByYW1kaXNrIGJhc2VkIGZhc3QgYmxvY2sgZGV2aWNlCm1ha2VzIG5vIHZpc2libGUgcGVyZm9y bWFuY2UgZGVncmFkYXRpb24uICBQbGVhc2Ugbm90ZSB0aGF0IHRoaXMgaXMganVzdAphIHZlcnkg c2ltcGxlIGFuZCBtaW5pbWFsIHRlc3QuICBPbiBzeXN0ZW1zIHVzaW5nIHN1cGVyLWZhc3QgYmxv Y2sKZGV2aWNlcyBhbmQgYSBzcGVjaWFsIEkvTyB3b3JrbG9hZCwgdGhlIHJlc3VsdHMgbWlnaHQg YmUgZGlmZmVyZW50LiAgSWYKeW91IGhhdmUgYW55IGRvdWJ0LCB0ZXN0IG9uIHlvdXIgbWFjaGlu ZSB3aXRoIHlvdXIgd29ya2xvYWQgdG8gZmluZCB0aGUKb3B0aW1hbCBzcXVlZXppbmcgZHVyYXRp b24gZm9yIHlvdS4KClsxXSBodHRwczovL3d3dy5rZXJuZWwub3JnL2RvYy9odG1sL2xhdGVzdC9h ZG1pbi1ndWlkZS9ibG9ja2Rldi9yYW1kaXNrLmh0bWwKClJldmlld2VkLWJ5OiBSb2dlciBQYXUg TW9ubsOpIDxyb2dlci5wYXVAY2l0cml4LmNvbT4KU2lnbmVkLW9mZi1ieTogU2VvbmdKYWUgUGFy ayA8c2pwYXJrQGFtYXpvbi5kZT4KLS0tCiAuLi4vQUJJL3Rlc3Rpbmcvc3lzZnMtZHJpdmVyLXhl bi1ibGtiYWNrICAgICAgfCAxMCArKysrKysrKysKIGRyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2sv YmxrYmFjay5jICAgICAgICAgICB8ICA3ICsrKysrLS0KIGRyaXZlcnMvYmxvY2sveGVuLWJsa2Jh Y2svY29tbW9uLmggICAgICAgICAgICB8ICAxICsKIGRyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2sv eGVuYnVzLmMgICAgICAgICAgICB8IDIxICsrKysrKysrKysrKysrKysrKy0KIDQgZmlsZXMgY2hh bmdlZCwgMzYgaW5zZXJ0aW9ucygrKSwgMyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9Eb2N1 bWVudGF0aW9uL0FCSS90ZXN0aW5nL3N5c2ZzLWRyaXZlci14ZW4tYmxrYmFjayBiL0RvY3VtZW50 YXRpb24vQUJJL3Rlc3Rpbmcvc3lzZnMtZHJpdmVyLXhlbi1ibGtiYWNrCmluZGV4IDRlN2JhYmIz YmExZi4uZjAxMjI0MjMxZjNmIDEwMDY0NAotLS0gYS9Eb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5n L3N5c2ZzLWRyaXZlci14ZW4tYmxrYmFjaworKysgYi9Eb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5n L3N5c2ZzLWRyaXZlci14ZW4tYmxrYmFjawpAQCAtMjUsMyArMjUsMTMgQEAgRGVzY3JpcHRpb246 CiAgICAgICAgICAgICAgICAgYWxsb2NhdGVkIHdpdGhvdXQgYmVpbmcgaW4gdXNlLiBUaGUgdGlt ZSBpcyBpbgogICAgICAgICAgICAgICAgIHNlY29uZHMsIDAgbWVhbnMgaW5kZWZpbml0ZWx5IGxv bmcuCiAgICAgICAgICAgICAgICAgVGhlIGRlZmF1bHQgaXMgNjAgc2Vjb25kcy4KKworV2hhdDog ICAgICAgICAgIC9zeXMvbW9kdWxlL3hlbl9ibGtiYWNrL3BhcmFtZXRlcnMvYnVmZmVyX3NxdWVl emVfZHVyYXRpb25fbXMKK0RhdGU6ICAgICAgICAgICBEZWNlbWJlciAyMDE5CitLZXJuZWxWZXJz aW9uOiAgNS41CitDb250YWN0OiAgICAgICAgU2VvbmdKYWUgUGFyayA8c2pwYXJrQGFtYXpvbi5k ZT4KK0Rlc2NyaXB0aW9uOgorICAgICAgICAgICAgICAgIFdoZW4gbWVtb3J5IHByZXNzdXJlIGlz IHJlcG9ydGVkIHRvIGJsa2JhY2sgdGhpcyBvcHRpb24KKyAgICAgICAgICAgICAgICBjb250cm9s cyB0aGUgZHVyYXRpb24gaW4gbWlsbGlzZWNvbmRzIHRoYXQgYmxrYmFjayB3aWxsIG5vdAorICAg ICAgICAgICAgICAgIGNhY2hlIGFueSBwYWdlIG5vdCBiYWNrZWQgYnkgYSBncmFudCBtYXBwaW5n LgorICAgICAgICAgICAgICAgIFRoZSBkZWZhdWx0IGlzIDEwbXMuCmRpZmYgLS1naXQgYS9kcml2 ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL2Jsa2JhY2suYyBiL2RyaXZlcnMvYmxvY2sveGVuLWJsa2Jh Y2svYmxrYmFjay5jCmluZGV4IGZkMWUxOWYxYTQ5Zi4uNzlmNjc3YWViNWNjIDEwMDY0NAotLS0g YS9kcml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL2Jsa2JhY2suYworKysgYi9kcml2ZXJzL2Jsb2Nr L3hlbi1ibGtiYWNrL2Jsa2JhY2suYwpAQCAtNjU2LDggKzY1NiwxMSBAQCBpbnQgeGVuX2Jsa2lm X3NjaGVkdWxlKHZvaWQgKmFyZykKIAkJCXJpbmctPm5leHRfbHJ1ID0gamlmZmllcyArIG1zZWNz X3RvX2ppZmZpZXMoTFJVX0lOVEVSVkFMKTsKIAkJfQogCi0JCS8qIFNocmluayBpZiB3ZSBoYXZl IG1vcmUgdGhhbiB4ZW5fYmxraWZfbWF4X2J1ZmZlcl9wYWdlcyAqLwotCQlzaHJpbmtfZnJlZV9w YWdlcG9vbChyaW5nLCB4ZW5fYmxraWZfbWF4X2J1ZmZlcl9wYWdlcyk7CisJCS8qIFNocmluayB0 aGUgZnJlZSBwYWdlcyBwb29sIGlmIGl0IGlzIHRvbyBsYXJnZS4gKi8KKwkJaWYgKHRpbWVfYmVm b3JlKGppZmZpZXMsIGJsa2lmLT5idWZmZXJfc3F1ZWV6ZV9lbmQpKQorCQkJc2hyaW5rX2ZyZWVf cGFnZXBvb2wocmluZywgMCk7CisJCWVsc2UKKwkJCXNocmlua19mcmVlX3BhZ2Vwb29sKHJpbmcs IHhlbl9ibGtpZl9tYXhfYnVmZmVyX3BhZ2VzKTsKIAogCQlpZiAobG9nX3N0YXRzICYmIHRpbWVf YWZ0ZXIoamlmZmllcywgcmluZy0+c3RfcHJpbnQpKQogCQkJcHJpbnRfc3RhdHMocmluZyk7CmRp ZmYgLS1naXQgYS9kcml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL2NvbW1vbi5oIGIvZHJpdmVycy9i bG9jay94ZW4tYmxrYmFjay9jb21tb24uaAppbmRleCAxZDMwMDJkNzczZjcuLjUzNmM4NGY2MWZl ZCAxMDA2NDQKLS0tIGEvZHJpdmVycy9ibG9jay94ZW4tYmxrYmFjay9jb21tb24uaAorKysgYi9k cml2ZXJzL2Jsb2NrL3hlbi1ibGtiYWNrL2NvbW1vbi5oCkBAIC0zMTksNiArMzE5LDcgQEAgc3Ry dWN0IHhlbl9ibGtpZiB7CiAJLyogQWxsIHJpbmdzIGZvciB0aGlzIGRldmljZS4gKi8KIAlzdHJ1 Y3QgeGVuX2Jsa2lmX3JpbmcJKnJpbmdzOwogCXVuc2lnbmVkIGludAkJbnJfcmluZ3M7CisJdW5z aWduZWQgbG9uZwkJYnVmZmVyX3NxdWVlemVfZW5kOwogfTsKIAogc3RydWN0IHNlZ19idWYgewpk aWZmIC0tZ2l0IGEvZHJpdmVycy9ibG9jay94ZW4tYmxrYmFjay94ZW5idXMuYyBiL2RyaXZlcnMv YmxvY2sveGVuLWJsa2JhY2sveGVuYnVzLmMKaW5kZXggYjkwZGJjZDk5YzAzLi40ZjZlYTRmZWNh NzkgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvYmxvY2sveGVuLWJsa2JhY2sveGVuYnVzLmMKKysrIGIv ZHJpdmVycy9ibG9jay94ZW4tYmxrYmFjay94ZW5idXMuYwpAQCAtODI0LDYgKzgyNCwyNCBAQCBz dGF0aWMgdm9pZCBmcm9udGVuZF9jaGFuZ2VkKHN0cnVjdCB4ZW5idXNfZGV2aWNlICpkZXYsCiB9 CiAKIAorLyogT25jZSBhIG1lbW9yeSBwcmVzc3VyZSBpcyBkZXRlY3RlZCwgc3F1ZWV6ZSBmcmVl IHBhZ2UgcG9vbHMgZm9yIGEgd2hpbGUuICovCitzdGF0aWMgdW5zaWduZWQgaW50IGJ1ZmZlcl9z cXVlZXplX2R1cmF0aW9uX21zID0gMTA7Cittb2R1bGVfcGFyYW1fbmFtZWQoYnVmZmVyX3NxdWVl emVfZHVyYXRpb25fbXMsCisJCWJ1ZmZlcl9zcXVlZXplX2R1cmF0aW9uX21zLCBpbnQsIDA2NDQp OworTU9EVUxFX1BBUk1fREVTQyhidWZmZXJfc3F1ZWV6ZV9kdXJhdGlvbl9tcywKKyJEdXJhdGlv biBpbiBtcyB0byBzcXVlZXplIHBhZ2VzIGJ1ZmZlciB3aGVuIGEgbWVtb3J5IHByZXNzdXJlIGlz IGRldGVjdGVkIik7CisKKy8qCisgKiBDYWxsYmFjayByZWNlaXZlZCB3aGVuIHRoZSBtZW1vcnkg cHJlc3N1cmUgaXMgZGV0ZWN0ZWQuCisgKi8KK3N0YXRpYyB2b2lkIHJlY2xhaW1fbWVtb3J5KHN0 cnVjdCB4ZW5idXNfZGV2aWNlICpkZXYpCit7CisJc3RydWN0IGJhY2tlbmRfaW5mbyAqYmUgPSBk ZXZfZ2V0X2RydmRhdGEoJmRldi0+ZGV2KTsKKworCWJlLT5ibGtpZi0+YnVmZmVyX3NxdWVlemVf ZW5kID0gamlmZmllcyArCisJCW1zZWNzX3RvX2ppZmZpZXMoYnVmZmVyX3NxdWVlemVfZHVyYXRp b25fbXMpOworfQorCiAvKiAqKiBDb25uZWN0aW9uICoqICovCiAKIApAQCAtMTExNSw3ICsxMTMz LDggQEAgc3RhdGljIHN0cnVjdCB4ZW5idXNfZHJpdmVyIHhlbl9ibGtia19kcml2ZXIgPSB7CiAJ LmlkcyAgPSB4ZW5fYmxrYmtfaWRzLAogCS5wcm9iZSA9IHhlbl9ibGtia19wcm9iZSwKIAkucmVt b3ZlID0geGVuX2Jsa2JrX3JlbW92ZSwKLQkub3RoZXJlbmRfY2hhbmdlZCA9IGZyb250ZW5kX2No YW5nZWQKKwkub3RoZXJlbmRfY2hhbmdlZCA9IGZyb250ZW5kX2NoYW5nZWQsCisJLnJlY2xhaW1f bWVtb3J5ID0gcmVjbGFpbV9tZW1vcnksCiB9OwogCiBpbnQgeGVuX2Jsa2lmX3hlbmJ1c19pbml0 KHZvaWQpCi0tIAoyLjE3LjEKCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW5wcm9q ZWN0Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlzdGluZm8veGVu LWRldmVs