From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 80BA1C76195 for ; Mon, 27 Mar 2023 20:13:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679948024; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=0F4BJ+rDji4kQYj2D9nqdfUUnZJ1CPs4eA1iUd2H4zQ=; b=Vmva6TNxVmz7NCzZSuyg0sz/l+T80VQThLDmvTF8qZCj40siZ7hn4tyu0f4FsG0gab7lUW oWIyNv++8WcR6K6Zmx3mqh0v2/nJNr9m8pvNeh6CbhfBKgj19EpCKEQxPi0j3PapV6liaK m2Iciv4UWblfgr7tkBeuXm/UNwrFmC4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-441-2K6YMqwcNcKFBTgleG1V7A-1; Mon, 27 Mar 2023 16:13:40 -0400 X-MC-Unique: 2K6YMqwcNcKFBTgleG1V7A-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 632A9185A791; Mon, 27 Mar 2023 20:13:37 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4DB3F440BC; Mon, 27 Mar 2023 20:13:37 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 8A6DF1946A6F; Mon, 27 Mar 2023 20:13:27 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 4CE5719465A4 for ; Mon, 27 Mar 2023 20:13:26 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 431782166B2A; Mon, 27 Mar 2023 20:13:26 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3B1362166B29 for ; Mon, 27 Mar 2023 20:13:21 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F16E118E0047 for ; Mon, 27 Mar 2023 20:13:20 +0000 (UTC) Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-656-GaOc5spjPg2clnKb03cH_g-2; Mon, 27 Mar 2023 16:13:19 -0400 X-MC-Unique: GaOc5spjPg2clnKb03cH_g-2 Received: by mail-qt1-f169.google.com with SMTP id p2so4725617qtw.13 for ; Mon, 27 Mar 2023 13:13:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679947999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KZEmP2Lma0r81Bos8w4U89OEq/V+OweIm/gue2zysiY=; b=wLx1eoGTIgSb7LNvfZMHLI1vk08DvpcY0SZ+Oc4iMR92LS9RMjQOTDWiIH6ifq73Es c/pG2vPgucZWsf8HtntxHU1YoHeok0nPj4jk8rlY6kPMRNKlDF//OyvrQawdlcmwoxwT daaHjix/IOT8kPpvMrFVeeXdeyrWIMnj8mXE6jJjnB6hPratTxQffrrY+dfXHVdE/6fM BJxRkz1pl2yMeKHIp/DLHrcgnFTS06jwtWa1/OVY10NFdnqn8O/eKOFGTfBldEsaOfxV Fzja2wBcY1hCVKBEAsu8/kWWmjE25KCxudVDfzYs82YegPiJmxJ7+XCk8nxcJPPsmSEE PZKg== X-Gm-Message-State: AO0yUKX1OcIo/+N60hR/LsigGixD8tIGsMifE8QgIduUNg1877viEmcF 4kWfiA0hEUA7hFIxikS5dcSdGMGgkIUNe/lzy35dDxSPEAeJRYBDzOUHG5AZHp0V8tOArarC6ps RxB/n5mAIK+ZoWhviV2PiRX/W/xoHlQIm1FXqofC4M+SIapqisDZU+1FQnRiJKqBAY8tIu0awbq I= X-Google-Smtp-Source: AK7set/4u5wtNp1+7i337s6/269TkCLiJgXwRPcmr5VJh9wfIB7rZepSb8dvmBloMyznyDlN/1yP2Q== X-Received: by 2002:a05:622a:1109:b0:3e3:82c4:db44 with SMTP id e9-20020a05622a110900b003e382c4db44mr23802990qty.52.1679947998246; Mon, 27 Mar 2023 13:13:18 -0700 (PDT) Received: from localhost (pool-68-160-166-30.bstnma.fios.verizon.net. [68.160.166.30]) by smtp.gmail.com with ESMTPSA id u23-20020a37ab17000000b0071f0d0aaef7sm13735581qke.80.2023.03.27.13.13.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Mar 2023 13:13:17 -0700 (PDT) From: Mike Snitzer To: dm-devel@redhat.com Date: Mon, 27 Mar 2023 16:11:30 -0400 Message-Id: <20230327201143.51026-8-snitzer@kernel.org> In-Reply-To: <20230327201143.51026-1-snitzer@kernel.org> References: <20230327201143.51026-1-snitzer@kernel.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Subject: [dm-devel] [dm-6.4 PATCH v3 07/20] dm bufio: improve concurrent IO performance X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: axboe@kernel.dk, ebiggers@kernel.org, keescook@chromium.org, heinzm@redhat.com, Mike Snitzer , nhuck@google.com, linux-block@vger.kernel.org, ejt@redhat.com, mpatocka@redhat.com, luomeng12@huawei.com Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 RnJvbTogSm9lIFRob3JuYmVyIDxlanRAcmVkaGF0LmNvbT4KCldoZW4gbXVsdGlwbGUgdGhyZWFk cyBwZXJmb3JtIElPIHRvIGEgdGhpbiBkZXZpY2UsIHRoZSB1bmRlcmx5aW5nCmRtX2J1ZmlvIG9i amVjdCBjYW4gYmVjb21lIGEgYm90dGxlbmVjazsgc2xvd2luZyBkb3duIGFjY2VzcyB0byBidHJl ZQpub2RlcyB0aGF0IHN0b3JlIHRoZSB0aGluIG1ldGFkYXRhLiBQcmlvciB0byB0aGlzIGNvbW1p dCwgZWFjaCBidWZpbwppbnN0YW5jZSBoYWQgYSBzaW5nbGUgbXV0ZXggdGhhdCB3YXMgdGFrZW4g Zm9yIGV2ZXJ5IGJ1ZmlvIG9wZXJhdGlvbi4KClRoaXMgY29tbWl0IGNvbmNlbnRyYXRlcyBvbiBp bXByb3ZpbmcgdGhlIGNvbW1vbiBjYXNlIHdoZXJlOiBhIHVzZXIgb2YKZG1fYnVmaW8gd2lzaGVz IHRvIGFjY2VzcywgYnV0IG5vdCBtb2RpZnksIGEgYnVmZmVyIHdoaWNoIGlzIGFscmVhZHkKd2l0 aGluIHRoZSBkbV9idWZpbyBjYWNoZS4KCkltcGxlbWVudGF0aW9uOjoKCiAgVGhlIGNvZGUgaGFz IGJlZW4gcmVmYWN0b3JlZDsgcHVsbGluZyBvdXQgYW4gJ2xydScgYWJzdHJhY3Rpb24gYW5kIGEK ICAnYnVmZmVyIGNhY2hlJyBhYnN0cmFjdGlvbiAoc2VlIDIgcHJldmlvdXMgY29tbWl0cykuIFRo aXMgY29tbWl0CiAgdXBkYXRlcyBoaWdoZXIgbGV2ZWwgYnVmaW8gY29kZSAodGhhdCBwZXJmb3Jt cyBhbGxvY2F0aW9uIG9mIGJ1ZmZlcnMsCiAgSU8gYW5kIGV2aWN0aW9uL2NhY2hlIHNpemluZykg dG8gbGV2ZXJhZ2UgYm90aCBhYnN0cmFjdGlvbnMuIEl0IGFsc28KICBkZWFscyB3aXRoIHRoZSBk ZWxpY2F0ZSBsb2NraW5nIHJlcXVpcmVtZW50cyBvZiBib3RoIGFic3RyYWN0aW9ucyB0bwogIHBy b3ZpZGUgZmluZXIgZ3JhaW5lZCBsb2NraW5nLiBUaGUgcmVzdWx0IGlzIHNpZ25pZmljYW50bHkg YmV0dGVyCiAgY29uY3VycmVudCBJTyBwZXJmb3JtYW5jZS4KCiAgQmVmb3JlIHRoaXMgY29tbWl0 LCBidWZpbyBoYXMgYSBnbG9iYWwgbHJ1IGxpc3QgaXQgdXNlZCB0byBldmljdCB0aGUKICBvbGRl c3QsIGNsZWFuIGJ1ZmZlcnMgZnJvbSBfYWxsXyBjbGllbnRzLiBXaXRoIHRoZSBuZXcgbG9ja2lu ZyB3ZQogIGRvbuKAmXQgd2FudCBkaWZmZXJlbnQgd2F5cyB0byBhY2Nlc3MgdGhlIHNhbWUgYnVm ZmVyLCBzbyBpbnN0ZWFkCiAgZG9fZ2xvYmFsX2NsZWFudXAoKSBsb29wcyBhcm91bmQgdGhlIGNs aWVudHMgYXNraW5nIHRoZW0gdG8gZnJlZQogIGJ1ZmZlcnMgb2xkZXIgdGhhbiBhIGNlcnRhaW4g dGltZS4KCiAgVGhpcyBjb21taXQgYWxzbyBjb252ZXJ0cyBtYW55IG9sZCBCVUdfT05zIHRvIFdB Uk5fT05fT05DRSwgc2VlIHRoZQogIGxydV9ldmljdCBhbmQgY2FjaGVfZXZpY3QgY29kZSBpbiBw YXJ0aWN1bGFyLiAgVGhleSB3aWxsIHJldHVybgogIEVSX0RPTlRfRVZJQ1QgaWYgYSBnaXZlbiBi dWZmZXIgc29tZWhvdyBtZWV0cyB0aGUgaW52YXJpYW50cyB0aGF0CiAgc2hvdWxkIF9uZXZlcl8g aGFwcGVuLiBbQXNpZGUgZnJvbSByZXZpc2luZyB0aGlzIGNvbW1pdCdzIGhlYWRlciBhbmQKICBm aXhpbmcgY29kaW5nIHN0eWxlIGFuZCB3aGl0ZXNwYWNlIG5pdHM6IHRoaXMgc3dpdGNoaW5nIHRv CiAgV0FSTl9PTl9PTkNFIGlzIE1pa2UgU25pdHplcidzIGxvbmUgY29udHJpYnV0aW9uIHRvIHRo aXMgY29tbWl0XQoKVGVzdGluZzo6CgogIFNvbWUgb2YgdGhlIGxvdyBsZXZlbCBmdW5jdGlvbnMg aGF2ZSBiZWVuIHVuaXQgdGVzdGVkIHVzaW5nIGRtLXVuaXQ6CiAgICBodHRwczovL2dpdGh1Yi5j b20vanRob3JuYmVyL2RtLXVuaXQvYmxvYi9tYWluL3NyYy90ZXN0cy9idWZpby5ycwoKICBIaWdo ZXIgbGV2ZWwgY29uY3VycmVuY3kgYW5kIElPIGlzIHRlc3RlZCB2aWEgYSB0ZXN0IG9ubHkgdGFy Z2V0CiAgZm91bmQgaGVyZToKICAgIGh0dHBzOi8vZ2l0aHViLmNvbS9qdGhvcm5iZXIvbGludXgv YmxvYi8yMDIzLTAzLTI0LXRoaW4tY29uY3VycmVuY3ktOS9kcml2ZXJzL21kL2RtLWJ1ZmlvLXRl c3QuYwoKICBUaGUgYXNzb2NpYXRlZCB1c2VybGFuZCBzaWRlIG9mIHRoZXNlIHRlc3RzIGlzIGhl cmU6CiAgICBodHRwczovL2dpdGh1Yi5jb20vanRob3JuYmVyL2RtdGVzdC1weXRob24vYmxvYi9t YWluL3NyYy9kbXRlc3QvYnVmaW8vYnVmaW9fdGVzdHMucHkKCiAgSW4gYWRkaXRpb24gdGhlIGZ1 bGwgZG10ZXN0IHN1aXRlIG9mIHRlc3RzIChkbS10aGluLCBkbS1jYWNoZSwgZXRjKQogIGhhcyBi ZWVuIHJ1biAofjQ1MCB0ZXN0cykuCgpQZXJmb3JtYW5jZTo6CgogIE1vc3QgYnVmaW8gb3BlcmF0 aW9ucyBoYXZlIHVuY2hhbmdlZCBwZXJmb3JtYW5jZS4gQnV0IGlmIG11bHRpcGxlCiAgdGhyZWFk cyBhcmUgYXR0ZW1wdGluZyB0byBnZXQgYnVmZmVycyBjb25jdXJyZW50bHksIGFuZCB0aGVzZQog IGJ1ZmZlcnMgYXJlIGFscmVhZHkgaW4gdGhlIGNhY2hlIHRoZW4gdGhlcmUncyBhIGJpZyBzcGVl ZCB1cC4gRWcsCiAgb25lIHRlc3QgaGFzIDE2ICdob3RzcG90JyB0aHJlYWRzIHNpbXVsYXRpbmcg YnRyZWUgbG9va3VwcyB3aGlsZQogIGFub3RoZXIgdGhyZWFkIGRpcnRpZXMgdGhlIHdob2xlIGRl dmljZS4gSW4gdGhpcyBjYXNlIHRoZSBob3RzcG90CiAgdGhyZWFkcyBhY3F1aXJlIHRoZSBidWZm ZXJzIGFib3V0IDI1IHRpbWVzIGZhc3Rlci4KClNpZ25lZC1vZmYtYnk6IEpvZSBUaG9ybmJlciA8 ZWp0QHJlZGhhdC5jb20+ClNpZ25lZC1vZmYtYnk6IE1pa2UgU25pdHplciA8c25pdHplckBrZXJu ZWwub3JnPgotLS0KIGRyaXZlcnMvbWQvZG0tYnVmaW8uYyB8IDk1OSArKysrKysrKysrKysrKysr KysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0KIDEgZmlsZSBjaGFuZ2VkLCA0ODcgaW5zZXJ0aW9u cygrKSwgNDcyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2RyaXZlcnMvbWQvZG0tYnVmaW8u YyBiL2RyaXZlcnMvbWQvZG0tYnVmaW8uYwppbmRleCA5ZGFmZjliNzdjZWUuLjFlMDAwZWM3M2Jk NiAxMDA2NDQKLS0tIGEvZHJpdmVycy9tZC9kbS1idWZpby5jCisrKyBiL2RyaXZlcnMvbWQvZG0t YnVmaW8uYwpAQCAtMzIxLDM3ICszMjEsNDIgQEAgZW51bSBkYXRhX21vZGUgewogfTsKIAogc3Ry dWN0IGRtX2J1ZmZlciB7CisJLyogcHJvdGVjdGVkIGJ5IHRoZSBsb2NrcyBpbiBkbV9idWZmZXJf Y2FjaGUgKi8KIAlzdHJ1Y3QgcmJfbm9kZSBub2RlOwotCXN0cnVjdCBsaXN0X2hlYWQgbHJ1X2xp c3Q7Ci0Jc3RydWN0IGxpc3RfaGVhZCBnbG9iYWxfbGlzdDsKIAorCS8qIGltbXV0YWJsZSwgc28g ZG9uJ3QgbmVlZCBwcm90ZWN0aW5nICovCiAJc2VjdG9yX3QgYmxvY2s7CiAJdm9pZCAqZGF0YTsK IAl1bnNpZ25lZCBjaGFyIGRhdGFfbW9kZTsJCS8qIERBVEFfTU9ERV8qICovCiAKLQl1bnNpZ25l ZCBpbnQgYWNjZXNzZWQ7Ci0JdW5zaWduZWQgaW50IF9faG9sZF9jb3VudDsKKwkvKgorCSAqIFRo ZXNlIHR3byBmaWVsZHMgYXJlIHVzZWQgaW4gaXNvbGF0aW9uLCBzbyBkbyBub3QgbmVlZAorCSAq IGEgc3Vycm91bmRpbmcgbG9jay4KKwkgKi8KKwlhdG9taWNfdCBob2xkX2NvdW50OwogCXVuc2ln bmVkIGxvbmcgbGFzdF9hY2Nlc3NlZDsKIAorCS8qCisJICogRXZlcnl0aGluZyBlbHNlIGlzIHBy b3RlY3RlZCBieSB0aGUgbXV0ZXggaW4KKwkgKiBkbV9idWZpb19jbGllbnQKKwkgKi8KKwl1bnNp Z25lZCBsb25nIHN0YXRlOworCXN0cnVjdCBscnVfZW50cnkgbHJ1OwogCXVuc2lnbmVkIGNoYXIg bGlzdF9tb2RlOwkJLyogTElTVF8qICovCiAJYmxrX3N0YXR1c190IHJlYWRfZXJyb3I7CiAJYmxr X3N0YXR1c190IHdyaXRlX2Vycm9yOwotCXVuc2lnbmVkIGxvbmcgc3RhdGU7CiAJdW5zaWduZWQg aW50IGRpcnR5X3N0YXJ0OwogCXVuc2lnbmVkIGludCBkaXJ0eV9lbmQ7CiAJdW5zaWduZWQgaW50 IHdyaXRlX3N0YXJ0OwogCXVuc2lnbmVkIGludCB3cml0ZV9lbmQ7Ci0Jc3RydWN0IGRtX2J1Zmlv X2NsaWVudCAqYzsKIAlzdHJ1Y3QgbGlzdF9oZWFkIHdyaXRlX2xpc3Q7CisJc3RydWN0IGRtX2J1 ZmlvX2NsaWVudCAqYzsKIAl2b2lkICgqZW5kX2lvKShzdHJ1Y3QgZG1fYnVmZmVyICpiLCBibGtf c3RhdHVzX3QgYnMpOwogI2lmZGVmIENPTkZJR19ETV9ERUJVR19CTE9DS19TVEFDS19UUkFDSU5H CiAjZGVmaW5lIE1BWF9TVEFDSyAxMAogCXVuc2lnbmVkIGludCBzdGFja19sZW47CiAJdW5zaWdu ZWQgbG9uZyBzdGFja19lbnRyaWVzW01BWF9TVEFDS107CiAjZW5kaWYKLQkvKiBUZW1wIG1lbWJl cnMgdG8gYWxsb3cgZG1fYnVmZmVyX2NhY2hlIGNvZGUgdG8gY29tcGlsZSAqLwotCWF0b21pY190 IGhvbGRfY291bnQ7Ci0Jc3RydWN0IGxydV9lbnRyeSBscnU7CiB9OwogCiAvKi0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKi8KQEAg LTgxNCw3ICs4MTksNyBAQCBzdGF0aWMgdm9pZCBjYWNoZV9yZW1vdmVfcmFuZ2Uoc3RydWN0IGRt X2J1ZmZlcl9jYWNoZSAqYmMsCiAKIC8qCiAgKiBMaW5raW5nIG9mIGJ1ZmZlcnM6Ci0gKglBbGwg YnVmZmVycyBhcmUgbGlua2VkIHRvIGJ1ZmZlcl90cmVlIHdpdGggdGhlaXIgbm9kZSBmaWVsZC4K KyAqCUFsbCBidWZmZXJzIGFyZSBsaW5rZWQgdG8gYnVmZmVyX2NhY2hlIHdpdGggdGhlaXIgbm9k ZSBmaWVsZC4KICAqCiAgKglDbGVhbiBidWZmZXJzIHRoYXQgYXJlIG5vdCBiZWluZyB3cml0dGVu IChCX1dSSVRJTkcgbm90IHNldCkKICAqCWFyZSBsaW5rZWQgdG8gbHJ1W0xJU1RfQ0xFQU5dIHdp dGggdGhlaXIgbHJ1X2xpc3QgZmllbGQuCkBAIC04MzIsOSArODM3LDYgQEAgc3RydWN0IGRtX2J1 ZmlvX2NsaWVudCB7CiAJc3BpbmxvY2tfdCBzcGlubG9jazsKIAlib29sIG5vX3NsZWVwOwogCi0J c3RydWN0IGxpc3RfaGVhZCBscnVbTElTVF9TSVpFXTsKLQl1bnNpZ25lZCBsb25nIG5fYnVmZmVy c1tMSVNUX1NJWkVdOwotCiAJc3RydWN0IGJsb2NrX2RldmljZSAqYmRldjsKIAl1bnNpZ25lZCBp bnQgYmxvY2tfc2l6ZTsKIAlzOCBzZWN0b3JzX3Blcl9ibG9ja19iaXRzOwpAQCAtODQ5LDcgKzg1 MSw3IEBAIHN0cnVjdCBkbV9idWZpb19jbGllbnQgewogCiAJdW5zaWduZWQgaW50IG1pbmltdW1f YnVmZmVyczsKIAotCXN0cnVjdCByYl9yb290IGJ1ZmZlcl90cmVlOworCXN0cnVjdCBkbV9idWZm ZXJfY2FjaGUgY2FjaGU7CiAJd2FpdF9xdWV1ZV9oZWFkX3QgZnJlZV9idWZmZXJfd2FpdDsKIAog CXNlY3Rvcl90IHN0YXJ0OwpAQCAtODYxLDYgKzg2MywxMSBAQCBzdHJ1Y3QgZG1fYnVmaW9fY2xp ZW50IHsKIAlzdHJ1Y3Qgc2hyaW5rZXIgc2hyaW5rZXI7CiAJc3RydWN0IHdvcmtfc3RydWN0IHNo cmlua193b3JrOwogCWF0b21pY19sb25nX3QgbmVlZF9zaHJpbms7CisKKwkvKgorCSAqIFVzZWQg YnkgZ2xvYmFsX2NsZWFudXAgdG8gc29ydCB0aGUgY2xpZW50cyBsaXN0LgorCSAqLworCXVuc2ln bmVkIGxvbmcgb2xkZXN0X2J1ZmZlcjsKIH07CiAKIHN0YXRpYyBERUZJTkVfU1RBVElDX0tFWV9G QUxTRShub19zbGVlcF9lbmFibGVkKTsKQEAgLTg3NywxNCArODg0LDYgQEAgc3RhdGljIHZvaWQg ZG1fYnVmaW9fbG9jayhzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjKQogCQltdXRleF9sb2NrX25l c3RlZCgmYy0+bG9jaywgZG1fYnVmaW9faW5fcmVxdWVzdCgpKTsKIH0KIAotc3RhdGljIGludCBk bV9idWZpb190cnlsb2NrKHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMpCi17Ci0JaWYgKHN0YXRp Y19icmFuY2hfdW5saWtlbHkoJm5vX3NsZWVwX2VuYWJsZWQpICYmIGMtPm5vX3NsZWVwKQotCQly ZXR1cm4gc3Bpbl90cnlsb2NrX2JoKCZjLT5zcGlubG9jayk7Ci0JZWxzZQotCQlyZXR1cm4gbXV0 ZXhfdHJ5bG9jaygmYy0+bG9jayk7Ci19Ci0KIHN0YXRpYyB2b2lkIGRtX2J1ZmlvX3VubG9jayhz dHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjKQogewogCWlmIChzdGF0aWNfYnJhbmNoX3VubGlrZWx5 KCZub19zbGVlcF9lbmFibGVkKSAmJiBjLT5ub19zbGVlcCkKQEAgLTkxMywxMCArOTEyLDYgQEAg c3RhdGljIHVuc2lnbmVkIGxvbmcgZG1fYnVmaW9fY2FjaGVfc2l6ZV9sYXRjaDsKIAogc3RhdGlj IERFRklORV9TUElOTE9DSyhnbG9iYWxfc3BpbmxvY2spOwogCi1zdGF0aWMgTElTVF9IRUFEKGds b2JhbF9xdWV1ZSk7Ci0KLXN0YXRpYyB1bnNpZ25lZCBsb25nIGdsb2JhbF9udW07Ci0KIC8qCiAg KiBCdWZmZXJzIGFyZSBmcmVlZCBhZnRlciB0aGlzIHRpbWVvdXQKICAqLwpAQCAtOTU4LDc4ICs5 NTMsNiBAQCBzdGF0aWMgdm9pZCBidWZmZXJfcmVjb3JkX3N0YWNrKHN0cnVjdCBkbV9idWZmZXIg KmIpCiB9CiAjZW5kaWYKIAotLyoKLSAqLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQotICogQSByZWQvYmxhY2sgdHJlZSBhY3Rz IGFzIGFuIGluZGV4IGZvciBhbGwgdGhlIGJ1ZmZlcnMuCi0gKi0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KLSAqLwotc3RhdGlj IHN0cnVjdCBkbV9idWZmZXIgKl9fZmluZChzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjLCBzZWN0 b3JfdCBibG9jaykKLXsKLQlzdHJ1Y3QgcmJfbm9kZSAqbiA9IGMtPmJ1ZmZlcl90cmVlLnJiX25v ZGU7Ci0Jc3RydWN0IGRtX2J1ZmZlciAqYjsKLQotCXdoaWxlIChuKSB7Ci0JCWIgPSBjb250YWlu ZXJfb2Yobiwgc3RydWN0IGRtX2J1ZmZlciwgbm9kZSk7Ci0KLQkJaWYgKGItPmJsb2NrID09IGJs b2NrKQotCQkJcmV0dXJuIGI7Ci0KLQkJbiA9IGJsb2NrIDwgYi0+YmxvY2sgPyBuLT5yYl9sZWZ0 IDogbi0+cmJfcmlnaHQ7Ci0JfQotCi0JcmV0dXJuIE5VTEw7Ci19Ci0KLXN0YXRpYyBzdHJ1Y3Qg ZG1fYnVmZmVyICpfX2ZpbmRfbmV4dF9vbGQoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywgc2Vj dG9yX3QgYmxvY2spCi17Ci0Jc3RydWN0IHJiX25vZGUgKm4gPSBjLT5idWZmZXJfdHJlZS5yYl9u b2RlOwotCXN0cnVjdCBkbV9idWZmZXIgKmI7Ci0Jc3RydWN0IGRtX2J1ZmZlciAqYmVzdCA9IE5V TEw7Ci0KLQl3aGlsZSAobikgewotCQliID0gY29udGFpbmVyX29mKG4sIHN0cnVjdCBkbV9idWZm ZXIsIG5vZGUpOwotCi0JCWlmIChiLT5ibG9jayA9PSBibG9jaykKLQkJCXJldHVybiBiOwotCi0J CWlmIChibG9jayA8PSBiLT5ibG9jaykgewotCQkJbiA9IG4tPnJiX2xlZnQ7Ci0JCQliZXN0ID0g YjsKLQkJfSBlbHNlIHsKLQkJCW4gPSBuLT5yYl9yaWdodDsKLQkJfQotCX0KLQotCXJldHVybiBi ZXN0OwotfQotCi1zdGF0aWMgdm9pZCBfX2luc2VydChzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpj LCBzdHJ1Y3QgZG1fYnVmZmVyICpiKQotewotCXN0cnVjdCByYl9ub2RlICoqbmV3ID0gJmMtPmJ1 ZmZlcl90cmVlLnJiX25vZGUsICpwYXJlbnQgPSBOVUxMOwotCXN0cnVjdCBkbV9idWZmZXIgKmZv dW5kOwotCi0Jd2hpbGUgKCpuZXcpIHsKLQkJZm91bmQgPSBjb250YWluZXJfb2YoKm5ldywgc3Ry dWN0IGRtX2J1ZmZlciwgbm9kZSk7Ci0KLQkJaWYgKGZvdW5kLT5ibG9jayA9PSBiLT5ibG9jaykg ewotCQkJQlVHX09OKGZvdW5kICE9IGIpOwotCQkJcmV0dXJuOwotCQl9Ci0KLQkJcGFyZW50ID0g Km5ldzsKLQkJbmV3ID0gYi0+YmxvY2sgPCBmb3VuZC0+YmxvY2sgPwotCQkJJmZvdW5kLT5ub2Rl LnJiX2xlZnQgOiAmZm91bmQtPm5vZGUucmJfcmlnaHQ7Ci0JfQotCi0JcmJfbGlua19ub2RlKCZi LT5ub2RlLCBwYXJlbnQsIG5ldyk7Ci0JcmJfaW5zZXJ0X2NvbG9yKCZiLT5ub2RlLCAmYy0+YnVm ZmVyX3RyZWUpOwotfQotCi1zdGF0aWMgdm9pZCBfX3JlbW92ZShzdHJ1Y3QgZG1fYnVmaW9fY2xp ZW50ICpjLCBzdHJ1Y3QgZG1fYnVmZmVyICpiKQotewotCXJiX2VyYXNlKCZiLT5ub2RlLCAmYy0+ YnVmZmVyX3RyZWUpOwotfQotCiAvKi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0qLwogCiBzdGF0aWMgdm9pZCBhZGp1c3RfdG90 YWxfYWxsb2NhdGVkKHN0cnVjdCBkbV9idWZmZXIgKmIsIGJvb2wgdW5saW5rKQpAQCAtMTA1Nywx NiArOTgwLDkgQEAgc3RhdGljIHZvaWQgYWRqdXN0X3RvdGFsX2FsbG9jYXRlZChzdHJ1Y3QgZG1f YnVmZmVyICpiLCBib29sIHVubGluaykKIAlpZiAoZG1fYnVmaW9fY3VycmVudF9hbGxvY2F0ZWQg PiBkbV9idWZpb19wZWFrX2FsbG9jYXRlZCkKIAkJZG1fYnVmaW9fcGVha19hbGxvY2F0ZWQgPSBk bV9idWZpb19jdXJyZW50X2FsbG9jYXRlZDsKIAotCWItPmFjY2Vzc2VkID0gMTsKLQogCWlmICgh dW5saW5rKSB7Ci0JCWxpc3RfYWRkKCZiLT5nbG9iYWxfbGlzdCwgJmdsb2JhbF9xdWV1ZSk7Ci0J CWdsb2JhbF9udW0rKzsKIAkJaWYgKGRtX2J1ZmlvX2N1cnJlbnRfYWxsb2NhdGVkID4gZG1fYnVm aW9fY2FjaGVfc2l6ZSkKIAkJCXF1ZXVlX3dvcmsoZG1fYnVmaW9fd3EsICZkbV9idWZpb19yZXBs YWNlbWVudF93b3JrKTsKLQl9IGVsc2UgewotCQlsaXN0X2RlbCgmYi0+Z2xvYmFsX2xpc3QpOwot CQlnbG9iYWxfbnVtLS07CiAJfQogCiAJc3Bpbl91bmxvY2soJmdsb2JhbF9zcGlubG9jayk7CkBA IC0xMTk2LDYgKzExMTIsNyBAQCBzdGF0aWMgc3RydWN0IGRtX2J1ZmZlciAqYWxsb2NfYnVmZmVy KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMsIGdmcF90IGdmcF9tYXNrKQogCQlrbWVtX2NhY2hl X2ZyZWUoYy0+c2xhYl9idWZmZXIsIGIpOwogCQlyZXR1cm4gTlVMTDsKIAl9CisJYWRqdXN0X3Rv dGFsX2FsbG9jYXRlZChiLCBmYWxzZSk7CiAKICNpZmRlZiBDT05GSUdfRE1fREVCVUdfQkxPQ0tf U1RBQ0tfVFJBQ0lORwogCWItPnN0YWNrX2xlbiA9IDA7CkBAIC0xMjEwLDYxICsxMTI3LDExIEBA IHN0YXRpYyB2b2lkIGZyZWVfYnVmZmVyKHN0cnVjdCBkbV9idWZmZXIgKmIpCiB7CiAJc3RydWN0 IGRtX2J1ZmlvX2NsaWVudCAqYyA9IGItPmM7CiAKKwlhZGp1c3RfdG90YWxfYWxsb2NhdGVkKGIs IHRydWUpOwogCWZyZWVfYnVmZmVyX2RhdGEoYywgYi0+ZGF0YSwgYi0+ZGF0YV9tb2RlKTsKIAlr bWVtX2NhY2hlX2ZyZWUoYy0+c2xhYl9idWZmZXIsIGIpOwogfQogCi0vKgotICogTGluayBidWZm ZXIgdG8gdGhlIGJ1ZmZlciB0cmVlIGFuZCBjbGVhbiBvciBkaXJ0eSBxdWV1ZS4KLSAqLwotc3Rh dGljIHZvaWQgX19saW5rX2J1ZmZlcihzdHJ1Y3QgZG1fYnVmZmVyICpiLCBzZWN0b3JfdCBibG9j aywgaW50IGRpcnR5KQotewotCXN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMgPSBiLT5jOwotCi0J Yy0+bl9idWZmZXJzW2RpcnR5XSsrOwotCWItPmJsb2NrID0gYmxvY2s7Ci0JYi0+bGlzdF9tb2Rl ID0gZGlydHk7Ci0JbGlzdF9hZGQoJmItPmxydV9saXN0LCAmYy0+bHJ1W2RpcnR5XSk7Ci0JX19p bnNlcnQoYi0+YywgYik7Ci0JYi0+bGFzdF9hY2Nlc3NlZCA9IGppZmZpZXM7Ci0KLQlhZGp1c3Rf dG90YWxfYWxsb2NhdGVkKGIsIGZhbHNlKTsKLX0KLQotLyoKLSAqIFVubGluayBidWZmZXIgZnJv bSB0aGUgYnVmZmVyIHRyZWUgYW5kIGRpcnR5IG9yIGNsZWFuIHF1ZXVlLgotICovCi1zdGF0aWMg dm9pZCBfX3VubGlua19idWZmZXIoc3RydWN0IGRtX2J1ZmZlciAqYikKLXsKLQlzdHJ1Y3QgZG1f YnVmaW9fY2xpZW50ICpjID0gYi0+YzsKLQotCUJVR19PTighYy0+bl9idWZmZXJzW2ItPmxpc3Rf bW9kZV0pOwotCi0JYy0+bl9idWZmZXJzW2ItPmxpc3RfbW9kZV0tLTsKLQlfX3JlbW92ZShiLT5j LCBiKTsKLQlsaXN0X2RlbCgmYi0+bHJ1X2xpc3QpOwotCi0JYWRqdXN0X3RvdGFsX2FsbG9jYXRl ZChiLCB0cnVlKTsKLX0KLQotLyoKLSAqIFBsYWNlIHRoZSBidWZmZXIgdG8gdGhlIGhlYWQgb2Yg ZGlydHkgb3IgY2xlYW4gTFJVIHF1ZXVlLgotICovCi1zdGF0aWMgdm9pZCBfX3JlbGlua19scnUo c3RydWN0IGRtX2J1ZmZlciAqYiwgaW50IGRpcnR5KQotewotCXN0cnVjdCBkbV9idWZpb19jbGll bnQgKmMgPSBiLT5jOwotCi0JYi0+YWNjZXNzZWQgPSAxOwotCi0JQlVHX09OKCFjLT5uX2J1ZmZl cnNbYi0+bGlzdF9tb2RlXSk7Ci0KLQljLT5uX2J1ZmZlcnNbYi0+bGlzdF9tb2RlXS0tOwotCWMt Pm5fYnVmZmVyc1tkaXJ0eV0rKzsKLQliLT5saXN0X21vZGUgPSBkaXJ0eTsKLQlsaXN0X21vdmUo JmItPmxydV9saXN0LCAmYy0+bHJ1W2RpcnR5XSk7Ci0JYi0+bGFzdF9hY2Nlc3NlZCA9IGppZmZp ZXM7Ci19Ci0KIC8qCiAgKi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiAgKiBTdWJtaXQgSS9PIG9uIHRoZSBi dWZmZXIuCkBAIC0xNTA0LDcgKzEzNzEsNyBAQCBzdGF0aWMgdm9pZCBfX2ZsdXNoX3dyaXRlX2xp c3Qoc3RydWN0IGxpc3RfaGVhZCAqd3JpdGVfbGlzdCkKICAqLwogc3RhdGljIHZvaWQgX19tYWtl X2J1ZmZlcl9jbGVhbihzdHJ1Y3QgZG1fYnVmZmVyICpiKQogewotCUJVR19PTihiLT5fX2hvbGRf Y291bnQpOworCUJVR19PTihhdG9taWNfcmVhZCgmYi0+aG9sZF9jb3VudCkpOwogCiAJLyogc21w X2xvYWRfYWNxdWlyZSgpIHBhaXJzIHdpdGggcmVhZF9lbmRpbygpJ3Mgc21wX21iX19iZWZvcmVf YXRvbWljKCkgKi8KIAlpZiAoIXNtcF9sb2FkX2FjcXVpcmUoJmItPnN0YXRlKSkJLyogZmFzdCBj YXNlICovCkBAIC0xNTE1LDYgKzEzODIsMzYgQEAgc3RhdGljIHZvaWQgX19tYWtlX2J1ZmZlcl9j bGVhbihzdHJ1Y3QgZG1fYnVmZmVyICpiKQogCXdhaXRfb25fYml0X2lvKCZiLT5zdGF0ZSwgQl9X UklUSU5HLCBUQVNLX1VOSU5URVJSVVBUSUJMRSk7CiB9CiAKK3N0YXRpYyBlbnVtIGV2aWN0X3Jl c3VsdCBpc19jbGVhbihzdHJ1Y3QgZG1fYnVmZmVyICpiLCB2b2lkICpjb250ZXh0KQoreworCXN0 cnVjdCBkbV9idWZpb19jbGllbnQgKmMgPSBjb250ZXh0OworCisJLyogVGhlc2Ugc2hvdWxkIG5l dmVyIGhhcHBlbiAqLworCWlmIChXQVJOX09OX09OQ0UodGVzdF9iaXQoQl9XUklUSU5HLCAmYi0+ c3RhdGUpKSkKKwkJcmV0dXJuIEVSX0RPTlRfRVZJQ1Q7CisJaWYgKFdBUk5fT05fT05DRSh0ZXN0 X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUpKSkKKwkJcmV0dXJuIEVSX0RPTlRfRVZJQ1Q7CisJaWYg KFdBUk5fT05fT05DRShiLT5saXN0X21vZGUgIT0gTElTVF9DTEVBTikpCisJCXJldHVybiBFUl9E T05UX0VWSUNUOworCisJaWYgKHN0YXRpY19icmFuY2hfdW5saWtlbHkoJm5vX3NsZWVwX2VuYWJs ZWQpICYmIGMtPm5vX3NsZWVwICYmCisJICAgIHVubGlrZWx5KHRlc3RfYml0KEJfUkVBRElORywg JmItPnN0YXRlKSkpCisJCXJldHVybiBFUl9ET05UX0VWSUNUOworCisJcmV0dXJuIEVSX0VWSUNU OworfQorCitzdGF0aWMgZW51bSBldmljdF9yZXN1bHQgaXNfZGlydHkoc3RydWN0IGRtX2J1ZmZl ciAqYiwgdm9pZCAqY29udGV4dCkKK3sKKwkvKiBUaGVzZSBzaG91bGQgbmV2ZXIgaGFwcGVuICov CisJaWYgKFdBUk5fT05fT05DRSh0ZXN0X2JpdChCX1JFQURJTkcsICZiLT5zdGF0ZSkpKQorCQly ZXR1cm4gRVJfRE9OVF9FVklDVDsKKwlpZiAoV0FSTl9PTl9PTkNFKGItPmxpc3RfbW9kZSAhPSBM SVNUX0RJUlRZKSkKKwkJcmV0dXJuIEVSX0RPTlRfRVZJQ1Q7CisKKwlyZXR1cm4gRVJfRVZJQ1Q7 Cit9CisKIC8qCiAgKiBGaW5kIHNvbWUgYnVmZmVyIHRoYXQgaXMgbm90IGhlbGQgYnkgYW55Ym9k eSwgY2xlYW4gaXQsIHVubGluayBpdCBhbmQKICAqIHJldHVybiBpdC4KQEAgLTE1MjMsMzQgKzE0 MjAsMjAgQEAgc3RhdGljIHN0cnVjdCBkbV9idWZmZXIgKl9fZ2V0X3VuY2xhaW1lZF9idWZmZXIo c3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYykKIHsKIAlzdHJ1Y3QgZG1fYnVmZmVyICpiOwogCi0J bGlzdF9mb3JfZWFjaF9lbnRyeV9yZXZlcnNlKGIsICZjLT5scnVbTElTVF9DTEVBTl0sIGxydV9s aXN0KSB7Ci0JCUJVR19PTih0ZXN0X2JpdChCX1dSSVRJTkcsICZiLT5zdGF0ZSkpOwotCQlCVUdf T04odGVzdF9iaXQoQl9ESVJUWSwgJmItPnN0YXRlKSk7Ci0KLQkJaWYgKHN0YXRpY19icmFuY2hf dW5saWtlbHkoJm5vX3NsZWVwX2VuYWJsZWQpICYmIGMtPm5vX3NsZWVwICYmCi0JCSAgICB1bmxp a2VseSh0ZXN0X2JpdF9hY3F1aXJlKEJfUkVBRElORywgJmItPnN0YXRlKSkpCi0JCQljb250aW51 ZTsKLQotCQlpZiAoIWItPl9faG9sZF9jb3VudCkgewotCQkJX19tYWtlX2J1ZmZlcl9jbGVhbihi KTsKLQkJCV9fdW5saW5rX2J1ZmZlcihiKTsKLQkJCXJldHVybiBiOwotCQl9Ci0JCWNvbmRfcmVz Y2hlZCgpOworCWIgPSBjYWNoZV9ldmljdCgmYy0+Y2FjaGUsIExJU1RfQ0xFQU4sIGlzX2NsZWFu LCBjKTsKKwlpZiAoYikgeworCQkvKiB0aGlzIGFsc28gd2FpdHMgZm9yIHBlbmRpbmcgcmVhZHMg Ki8KKwkJX19tYWtlX2J1ZmZlcl9jbGVhbihiKTsKKwkJcmV0dXJuIGI7CiAJfQogCiAJaWYgKHN0 YXRpY19icmFuY2hfdW5saWtlbHkoJm5vX3NsZWVwX2VuYWJsZWQpICYmIGMtPm5vX3NsZWVwKQog CQlyZXR1cm4gTlVMTDsKIAotCWxpc3RfZm9yX2VhY2hfZW50cnlfcmV2ZXJzZShiLCAmYy0+bHJ1 W0xJU1RfRElSVFldLCBscnVfbGlzdCkgewotCQlCVUdfT04odGVzdF9iaXQoQl9SRUFESU5HLCAm Yi0+c3RhdGUpKTsKLQotCQlpZiAoIWItPl9faG9sZF9jb3VudCkgewotCQkJX19tYWtlX2J1ZmZl cl9jbGVhbihiKTsKLQkJCV9fdW5saW5rX2J1ZmZlcihiKTsKLQkJCXJldHVybiBiOwotCQl9Ci0J CWNvbmRfcmVzY2hlZCgpOworCWIgPSBjYWNoZV9ldmljdCgmYy0+Y2FjaGUsIExJU1RfRElSVFks IGlzX2RpcnR5LCBOVUxMKTsKKwlpZiAoYikgeworCQlfX21ha2VfYnVmZmVyX2NsZWFuKGIpOwor CQlyZXR1cm4gYjsKIAl9CiAKIAlyZXR1cm4gTlVMTDsKQEAgLTE1NzEsNyArMTQ1NCwxMiBAQCBz dGF0aWMgdm9pZCBfX3dhaXRfZm9yX2ZyZWVfYnVmZmVyKHN0cnVjdCBkbV9idWZpb19jbGllbnQg KmMpCiAJc2V0X2N1cnJlbnRfc3RhdGUoVEFTS19VTklOVEVSUlVQVElCTEUpOwogCWRtX2J1Zmlv X3VubG9jayhjKTsKIAotCWlvX3NjaGVkdWxlKCk7CisJLyoKKwkgKiBJdCdzIHBvc3NpYmxlIHRv IG1pc3MgYSB3YWtlIHVwIGV2ZW50IHNpbmNlIHdlIGRvbid0IGFsd2F5cworCSAqIGhvbGQgYy0+ bG9jayB3aGVuIHdha2VfdXAgaXMgY2FsbGVkLiAgU28gd2UgaGF2ZSBhIHRpbWVvdXQgaGVyZSwK KwkgKiBqdXN0IGluIGNhc2UuCisJICovCisJaW9fc2NoZWR1bGVfdGltZW91dCg1ICogSFopOwog CiAJcmVtb3ZlX3dhaXRfcXVldWUoJmMtPmZyZWVfYnVmZmVyX3dhaXQsICZ3YWl0KTsKIApAQCAt MTYyOSw5ICsxNTE3LDggQEAgc3RhdGljIHN0cnVjdCBkbV9idWZmZXIgKl9fYWxsb2NfYnVmZmVy X3dhaXRfbm9fY2FsbGJhY2soc3RydWN0IGRtX2J1ZmlvX2NsaWVudAogCQl9CiAKIAkJaWYgKCFs aXN0X2VtcHR5KCZjLT5yZXNlcnZlZF9idWZmZXJzKSkgewotCQkJYiA9IGxpc3RfZW50cnkoYy0+ cmVzZXJ2ZWRfYnVmZmVycy5uZXh0LAotCQkJCSAgICAgICBzdHJ1Y3QgZG1fYnVmZmVyLCBscnVf bGlzdCk7Ci0JCQlsaXN0X2RlbCgmYi0+bHJ1X2xpc3QpOworCQkJYiA9IGxpc3RfdG9fYnVmZmVy KGMtPnJlc2VydmVkX2J1ZmZlcnMubmV4dCk7CisJCQlsaXN0X2RlbCgmYi0+bHJ1Lmxpc3QpOwog CQkJYy0+bmVlZF9yZXNlcnZlZF9idWZmZXJzKys7CiAKIAkJCXJldHVybiBiOwpAQCAtMTY2NSwz NiArMTU1Miw1NiBAQCBzdGF0aWMgdm9pZCBfX2ZyZWVfYnVmZmVyX3dha2Uoc3RydWN0IGRtX2J1 ZmZlciAqYikKIHsKIAlzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjID0gYi0+YzsKIAorCWItPmJs b2NrID0gLTE7CiAJaWYgKCFjLT5uZWVkX3Jlc2VydmVkX2J1ZmZlcnMpCiAJCWZyZWVfYnVmZmVy KGIpOwogCWVsc2UgewotCQlsaXN0X2FkZCgmYi0+bHJ1X2xpc3QsICZjLT5yZXNlcnZlZF9idWZm ZXJzKTsKKwkJbGlzdF9hZGQoJmItPmxydS5saXN0LCAmYy0+cmVzZXJ2ZWRfYnVmZmVycyk7CiAJ CWMtPm5lZWRfcmVzZXJ2ZWRfYnVmZmVycy0tOwogCX0KIAogCXdha2VfdXAoJmMtPmZyZWVfYnVm ZmVyX3dhaXQpOwogfQogCitzdGF0aWMgZW51bSBldmljdF9yZXN1bHQgY2xlYW5lZChzdHJ1Y3Qg ZG1fYnVmZmVyICpiLCB2b2lkICpjb250ZXh0KQoreworCWlmIChXQVJOX09OX09OQ0UodGVzdF9i aXQoQl9SRUFESU5HLCAmYi0+c3RhdGUpKSkKKwkJcmV0dXJuIEVSX0RPTlRfRVZJQ1Q7IC8qIHNo b3VsZCBuZXZlciBoYXBwZW4gKi8KKworCWlmICh0ZXN0X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUp IHx8IHRlc3RfYml0KEJfV1JJVElORywgJmItPnN0YXRlKSkKKwkJcmV0dXJuIEVSX0RPTlRfRVZJ Q1Q7CisJZWxzZQorCQlyZXR1cm4gRVJfRVZJQ1Q7Cit9CisKK3N0YXRpYyB2b2lkIF9fbW92ZV9j bGVhbl9idWZmZXJzKHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMpCit7CisJY2FjaGVfbWFya19t YW55KCZjLT5jYWNoZSwgTElTVF9ESVJUWSwgTElTVF9DTEVBTiwgY2xlYW5lZCwgTlVMTCk7Cit9 CisKK3N0cnVjdCB3cml0ZV9jb250ZXh0IHsKKwlpbnQgbm9fd2FpdDsKKwlzdHJ1Y3QgbGlzdF9o ZWFkICp3cml0ZV9saXN0OworfTsKKworc3RhdGljIGVudW0gaXRfYWN0aW9uIHdyaXRlX29uZShz dHJ1Y3QgZG1fYnVmZmVyICpiLCB2b2lkICpjb250ZXh0KQoreworCXN0cnVjdCB3cml0ZV9jb250 ZXh0ICp3YyA9IGNvbnRleHQ7CisKKwlpZiAod2MtPm5vX3dhaXQgJiYgdGVzdF9iaXQoQl9XUklU SU5HLCAmYi0+c3RhdGUpKQorCQlyZXR1cm4gSVRfQ09NUExFVEU7CisKKwlfX3dyaXRlX2RpcnR5 X2J1ZmZlcihiLCB3Yy0+d3JpdGVfbGlzdCk7CisJcmV0dXJuIElUX05FWFQ7Cit9CisKIHN0YXRp YyB2b2lkIF9fd3JpdGVfZGlydHlfYnVmZmVyc19hc3luYyhzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50 ICpjLCBpbnQgbm9fd2FpdCwKIAkJCQkJc3RydWN0IGxpc3RfaGVhZCAqd3JpdGVfbGlzdCkKIHsK LQlzdHJ1Y3QgZG1fYnVmZmVyICpiLCAqdG1wOworCXN0cnVjdCB3cml0ZV9jb250ZXh0IHdjID0g ey5ub193YWl0ID0gbm9fd2FpdCwgLndyaXRlX2xpc3QgPSB3cml0ZV9saXN0fTsKIAotCWxpc3Rf Zm9yX2VhY2hfZW50cnlfc2FmZV9yZXZlcnNlKGIsIHRtcCwgJmMtPmxydVtMSVNUX0RJUlRZXSwg bHJ1X2xpc3QpIHsKLQkJQlVHX09OKHRlc3RfYml0KEJfUkVBRElORywgJmItPnN0YXRlKSk7Ci0K LQkJaWYgKCF0ZXN0X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUpICYmCi0JCSAgICAhdGVzdF9iaXQo Ql9XUklUSU5HLCAmYi0+c3RhdGUpKSB7Ci0JCQlfX3JlbGlua19scnUoYiwgTElTVF9DTEVBTik7 Ci0JCQljb250aW51ZTsKLQkJfQotCi0JCWlmIChub193YWl0ICYmIHRlc3RfYml0KEJfV1JJVElO RywgJmItPnN0YXRlKSkKLQkJCXJldHVybjsKLQotCQlfX3dyaXRlX2RpcnR5X2J1ZmZlcihiLCB3 cml0ZV9saXN0KTsKLQkJY29uZF9yZXNjaGVkKCk7Ci0JfQorCV9fbW92ZV9jbGVhbl9idWZmZXJz KGMpOworCWNhY2hlX2l0ZXJhdGUoJmMtPmNhY2hlLCBMSVNUX0RJUlRZLCB3cml0ZV9vbmUsICZ3 Yyk7CiB9CiAKIC8qCkBAIC0xNzA1LDcgKzE2MTIsOCBAQCBzdGF0aWMgdm9pZCBfX3dyaXRlX2Rp cnR5X2J1ZmZlcnNfYXN5bmMoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywgaW50IG5vX3dhaXQs CiBzdGF0aWMgdm9pZCBfX2NoZWNrX3dhdGVybWFyayhzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpj LAogCQkJICAgICAgc3RydWN0IGxpc3RfaGVhZCAqd3JpdGVfbGlzdCkKIHsKLQlpZiAoYy0+bl9i dWZmZXJzW0xJU1RfRElSVFldID4gYy0+bl9idWZmZXJzW0xJU1RfQ0xFQU5dICogRE1fQlVGSU9f V1JJVEVCQUNLX1JBVElPKQorCWlmIChjYWNoZV9jb3VudCgmYy0+Y2FjaGUsIExJU1RfRElSVFkp ID4KKwkgICAgY2FjaGVfY291bnQoJmMtPmNhY2hlLCBMSVNUX0NMRUFOKSAqIERNX0JVRklPX1dS SVRFQkFDS19SQVRJTykKIAkJX193cml0ZV9kaXJ0eV9idWZmZXJzX2FzeW5jKGMsIDEsIHdyaXRl X2xpc3QpOwogfQogCkBAIC0xNzE1LDYgKzE2MjMsMjEgQEAgc3RhdGljIHZvaWQgX19jaGVja193 YXRlcm1hcmsoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywKICAqLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KICAqLwogCitzdGF0 aWMgdm9pZCBjYWNoZV9wdXRfYW5kX3dha2Uoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywgc3Ry dWN0IGRtX2J1ZmZlciAqYikKK3sKKwkvKgorCSAqIFJlbHlpbmcgb24gd2FpdHF1ZXVlX2FjdGl2 ZSgpIGlzIHJhY2V5LCBidXQgd2Ugc2xlZXAKKwkgKiB3aXRoIHNjaGVkdWxlX3RpbWVvdXQgYW55 d2F5LgorCSAqLworCWlmIChjYWNoZV9wdXQoJmMtPmNhY2hlLCBiKSAmJgorCSAgICB1bmxpa2Vs eSh3YWl0cXVldWVfYWN0aXZlKCZjLT5mcmVlX2J1ZmZlcl93YWl0KSkpCisJCXdha2VfdXAoJmMt PmZyZWVfYnVmZmVyX3dhaXQpOworfQorCisvKgorICogVGhpcyBhc3N1bWVzIHlvdSBoYXZlIGFs cmVhZHkgY2hlY2tlZCB0aGUgY2FjaGUgdG8gc2VlIGlmIHRoZSBidWZmZXIKKyAqIGlzIGFscmVh ZHkgcHJlc2VudCAoaXQgd2lsbCByZWNoZWNrIGFmdGVyIGRyb3BwaW5nIHRoZSBsb2NrIGZvciBh bGxvY2F0aW9uKS4KKyAqLwogc3RhdGljIHN0cnVjdCBkbV9idWZmZXIgKl9fYnVmaW9fbmV3KHN0 cnVjdCBkbV9idWZpb19jbGllbnQgKmMsIHNlY3Rvcl90IGJsb2NrLAogCQkJCSAgICAgZW51bSBu ZXdfZmxhZyBuZiwgaW50ICpuZWVkX3N1Ym1pdCwKIAkJCQkgICAgIHN0cnVjdCBsaXN0X2hlYWQg KndyaXRlX2xpc3QpCkBAIC0xNzIzLDExICsxNjQ2LDggQEAgc3RhdGljIHN0cnVjdCBkbV9idWZm ZXIgKl9fYnVmaW9fbmV3KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMsIHNlY3Rvcl90IGJsb2Nr LAogCiAJKm5lZWRfc3VibWl0ID0gMDsKIAotCWIgPSBfX2ZpbmQoYywgYmxvY2spOwotCWlmIChi KQotCQlnb3RvIGZvdW5kX2J1ZmZlcjsKLQotCWlmIChuZiA9PSBORl9HRVQpCisJLyogVGhpcyBj YW4ndCBiZSBjYWxsZWQgd2l0aCBORl9HRVQgKi8KKwlpZiAoV0FSTl9PTl9PTkNFKG5mID09IE5G X0dFVCkpCiAJCXJldHVybiBOVUxMOwogCiAJbmV3X2IgPSBfX2FsbG9jX2J1ZmZlcl93YWl0KGMs IG5mKTsKQEAgLTE3MzgsNyArMTY1OCw3IEBAIHN0YXRpYyBzdHJ1Y3QgZG1fYnVmZmVyICpfX2J1 ZmlvX25ldyhzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjLCBzZWN0b3JfdCBibG9jaywKIAkgKiBX ZSd2ZSBoYWQgYSBwZXJpb2Qgd2hlcmUgdGhlIG11dGV4IHdhcyB1bmxvY2tlZCwgc28gbmVlZCB0 bwogCSAqIHJlY2hlY2sgdGhlIGJ1ZmZlciB0cmVlLgogCSAqLwotCWIgPSBfX2ZpbmQoYywgYmxv Y2spOworCWIgPSBjYWNoZV9nZXQoJmMtPmNhY2hlLCBibG9jayk7CiAJaWYgKGIpIHsKIAkJX19m cmVlX2J1ZmZlcl93YWtlKG5ld19iKTsKIAkJZ290byBmb3VuZF9idWZmZXI7CkBAIC0xNzQ3LDI0 ICsxNjY3LDM1IEBAIHN0YXRpYyBzdHJ1Y3QgZG1fYnVmZmVyICpfX2J1ZmlvX25ldyhzdHJ1Y3Qg ZG1fYnVmaW9fY2xpZW50ICpjLCBzZWN0b3JfdCBibG9jaywKIAlfX2NoZWNrX3dhdGVybWFyayhj LCB3cml0ZV9saXN0KTsKIAogCWIgPSBuZXdfYjsKLQliLT5fX2hvbGRfY291bnQgPSAxOworCWF0 b21pY19zZXQoJmItPmhvbGRfY291bnQsIDEpOworCVdSSVRFX09OQ0UoYi0+bGFzdF9hY2Nlc3Nl ZCwgamlmZmllcyk7CisJYi0+YmxvY2sgPSBibG9jazsKIAliLT5yZWFkX2Vycm9yID0gMDsKIAli LT53cml0ZV9lcnJvciA9IDA7Ci0JX19saW5rX2J1ZmZlcihiLCBibG9jaywgTElTVF9DTEVBTik7 CisJYi0+bGlzdF9tb2RlID0gTElTVF9DTEVBTjsKIAotCWlmIChuZiA9PSBORl9GUkVTSCkgewor CWlmIChuZiA9PSBORl9GUkVTSCkKIAkJYi0+c3RhdGUgPSAwOwotCQlyZXR1cm4gYjsKKwllbHNl IHsKKwkJYi0+c3RhdGUgPSAxIDw8IEJfUkVBRElORzsKKwkJKm5lZWRfc3VibWl0ID0gMTsKIAl9 CiAKLQliLT5zdGF0ZSA9IDEgPDwgQl9SRUFESU5HOwotCSpuZWVkX3N1Ym1pdCA9IDE7CisJLyoK KwkgKiBXZSBtdXN0bid0IGluc2VydCBpbnRvIHRoZSBjYWNoZSB1bnRpbCB0aGUgQl9SRUFESU5H IHN0YXRlCisJICogaXMgc2V0LiAgT3RoZXJ3aXNlIGFub3RoZXIgdGhyZWFkIGNvdWxkIGdldCBp dCBhbmQgdXNlCisJICogaXQgYmVmb3JlIGl0IGhhZCBiZWVuIHJlYWQuCisJICovCisJY2FjaGVf aW5zZXJ0KCZjLT5jYWNoZSwgYik7CiAKIAlyZXR1cm4gYjsKIAogZm91bmRfYnVmZmVyOgotCWlm IChuZiA9PSBORl9QUkVGRVRDSCkKKwlpZiAobmYgPT0gTkZfUFJFRkVUQ0gpIHsKKwkJY2FjaGVf cHV0X2FuZF93YWtlKGMsIGIpOwogCQlyZXR1cm4gTlVMTDsKKwl9CisKIAkvKgogCSAqIE5vdGU6 IGl0IGlzIGVzc2VudGlhbCB0aGF0IHdlIGRvbid0IHdhaXQgZm9yIHRoZSBidWZmZXIgdG8gYmUK IAkgKiByZWFkIGlmIGRtX2J1ZmlvX2dldCBmdW5jdGlvbiBpcyB1c2VkLiBCb3RoIGRtX2J1Zmlv X2dldCBhbmQKQEAgLTE3NzIsMTIgKzE3MDMsMTEgQEAgc3RhdGljIHN0cnVjdCBkbV9idWZmZXIg Kl9fYnVmaW9fbmV3KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMsIHNlY3Rvcl90IGJsb2NrLAog CSAqIElmIHRoZSB1c2VyIGNhbGxlZCBib3RoIGRtX2J1ZmlvX3ByZWZldGNoIGFuZCBkbV9idWZp b19nZXQgb24KIAkgKiB0aGUgc2FtZSBidWZmZXIsIGl0IHdvdWxkIGRlYWRsb2NrIGlmIHdlIHdh aXRlZC4KIAkgKi8KLQlpZiAobmYgPT0gTkZfR0VUICYmIHVubGlrZWx5KHRlc3RfYml0X2FjcXVp cmUoQl9SRUFESU5HLCAmYi0+c3RhdGUpKSkKKwlpZiAobmYgPT0gTkZfR0VUICYmIHVubGlrZWx5 KHRlc3RfYml0X2FjcXVpcmUoQl9SRUFESU5HLCAmYi0+c3RhdGUpKSkgeworCQljYWNoZV9wdXRf YW5kX3dha2UoYywgYik7CiAJCXJldHVybiBOVUxMOworCX0KIAotCWItPl9faG9sZF9jb3VudCsr OwotCV9fcmVsaW5rX2xydShiLCB0ZXN0X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUpIHx8Ci0JCSAg ICAgdGVzdF9iaXQoQl9XUklUSU5HLCAmYi0+c3RhdGUpKTsKIAlyZXR1cm4gYjsKIH0KIApAQCAt MTgwNywxOCArMTczNyw1MCBAQCBzdGF0aWMgdm9pZCByZWFkX2VuZGlvKHN0cnVjdCBkbV9idWZm ZXIgKmIsIGJsa19zdGF0dXNfdCBzdGF0dXMpCiBzdGF0aWMgdm9pZCAqbmV3X3JlYWQoc3RydWN0 IGRtX2J1ZmlvX2NsaWVudCAqYywgc2VjdG9yX3QgYmxvY2ssCiAJCSAgICAgIGVudW0gbmV3X2Zs YWcgbmYsIHN0cnVjdCBkbV9idWZmZXIgKipicCkKIHsKLQlpbnQgbmVlZF9zdWJtaXQ7CisJaW50 IG5lZWRfc3VibWl0ID0gMDsKIAlzdHJ1Y3QgZG1fYnVmZmVyICpiOwogCiAJTElTVF9IRUFEKHdy aXRlX2xpc3QpOwogCi0JZG1fYnVmaW9fbG9jayhjKTsKLQliID0gX19idWZpb19uZXcoYywgYmxv Y2ssIG5mLCAmbmVlZF9zdWJtaXQsICZ3cml0ZV9saXN0KTsKKwkqYnAgPSBOVUxMOworCisJLyoK KwkgKiBGYXN0IHBhdGgsIGhvcGVmdWxseSB0aGUgYmxvY2sgaXMgYWxyZWFkeSBpbiB0aGUgY2Fj aGUuICBObyBuZWVkCisJICogdG8gZ2V0IHRoZSBjbGllbnQgbG9jayBmb3IgdGhpcy4KKwkgKi8K KwliID0gY2FjaGVfZ2V0KCZjLT5jYWNoZSwgYmxvY2spOworCWlmIChiKSB7CisJCWlmIChuZiA9 PSBORl9QUkVGRVRDSCkgeworCQkJY2FjaGVfcHV0X2FuZF93YWtlKGMsIGIpOworCQkJcmV0dXJu IE5VTEw7CisJCX0KKworCQkvKgorCQkgKiBOb3RlOiBpdCBpcyBlc3NlbnRpYWwgdGhhdCB3ZSBk b24ndCB3YWl0IGZvciB0aGUgYnVmZmVyIHRvIGJlCisJCSAqIHJlYWQgaWYgZG1fYnVmaW9fZ2V0 IGZ1bmN0aW9uIGlzIHVzZWQuIEJvdGggZG1fYnVmaW9fZ2V0IGFuZAorCQkgKiBkbV9idWZpb19w cmVmZXRjaCBjYW4gYmUgdXNlZCBpbiB0aGUgZHJpdmVyIHJlcXVlc3Qgcm91dGluZS4KKwkJICog SWYgdGhlIHVzZXIgY2FsbGVkIGJvdGggZG1fYnVmaW9fcHJlZmV0Y2ggYW5kIGRtX2J1ZmlvX2dl dCBvbgorCQkgKiB0aGUgc2FtZSBidWZmZXIsIGl0IHdvdWxkIGRlYWRsb2NrIGlmIHdlIHdhaXRl ZC4KKwkJICovCisJCWlmIChuZiA9PSBORl9HRVQgJiYgdW5saWtlbHkodGVzdF9iaXRfYWNxdWly ZShCX1JFQURJTkcsICZiLT5zdGF0ZSkpKSB7CisJCQljYWNoZV9wdXRfYW5kX3dha2UoYywgYik7 CisJCQlyZXR1cm4gTlVMTDsKKwkJfQorCX0KKworCWlmICghYikgeworCQlpZiAobmYgPT0gTkZf R0VUKQorCQkJcmV0dXJuIE5VTEw7CisKKwkJZG1fYnVmaW9fbG9jayhjKTsKKwkJYiA9IF9fYnVm aW9fbmV3KGMsIGJsb2NrLCBuZiwgJm5lZWRfc3VibWl0LCAmd3JpdGVfbGlzdCk7CisJCWRtX2J1 ZmlvX3VubG9jayhjKTsKKwl9CisKICNpZmRlZiBDT05GSUdfRE1fREVCVUdfQkxPQ0tfU1RBQ0tf VFJBQ0lORwotCWlmIChiICYmIGItPmhvbGRfY291bnQgPT0gMSkKKwlpZiAoYiAmJiAoYXRvbWlj X3JlYWQoJmItPmhvbGRfY291bnQpID09IDEpKQogCQlidWZmZXJfcmVjb3JkX3N0YWNrKGIpOwog I2VuZGlmCi0JZG1fYnVmaW9fdW5sb2NrKGMpOwogCiAJX19mbHVzaF93cml0ZV9saXN0KCZ3cml0 ZV9saXN0KTsKIApAQCAtMTg4MSwxMiArMTg0MywxOSBAQCB2b2lkIGRtX2J1ZmlvX3ByZWZldGNo KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMsCiAJCXJldHVybjsgLyogc2hvdWxkIG5ldmVyIGhh cHBlbiAqLwogCiAJYmxrX3N0YXJ0X3BsdWcoJnBsdWcpOwotCWRtX2J1ZmlvX2xvY2soYyk7CiAK IAlmb3IgKDsgbl9ibG9ja3MtLTsgYmxvY2srKykgewogCQlpbnQgbmVlZF9zdWJtaXQ7CiAJCXN0 cnVjdCBkbV9idWZmZXIgKmI7CiAKKwkJYiA9IGNhY2hlX2dldCgmYy0+Y2FjaGUsIGJsb2NrKTsK KwkJaWYgKGIpIHsKKwkJCS8qIGFscmVhZHkgaW4gY2FjaGUgKi8KKwkJCWNhY2hlX3B1dF9hbmRf d2FrZShjLCBiKTsKKwkJCWNvbnRpbnVlOworCQl9CisKKwkJZG1fYnVmaW9fbG9jayhjKTsKIAkJ YiA9IF9fYnVmaW9fbmV3KGMsIGJsb2NrLCBORl9QUkVGRVRDSCwgJm5lZWRfc3VibWl0LAogCQkJ CSZ3cml0ZV9saXN0KTsKIAkJaWYgKHVubGlrZWx5KCFsaXN0X2VtcHR5KCZ3cml0ZV9saXN0KSkp IHsKQEAgLTE5MDksMTAgKzE4NzgsOSBAQCB2b2lkIGRtX2J1ZmlvX3ByZWZldGNoKHN0cnVjdCBk bV9idWZpb19jbGllbnQgKmMsCiAJCQkJZ290byBmbHVzaF9wbHVnOwogCQkJZG1fYnVmaW9fbG9j ayhjKTsKIAkJfQorCQlkbV9idWZpb191bmxvY2soYyk7CiAJfQogCi0JZG1fYnVmaW9fdW5sb2Nr KGMpOwotCiBmbHVzaF9wbHVnOgogCWJsa19maW5pc2hfcGx1ZygmcGx1Zyk7CiB9CkBAIC0xOTIy LDI5ICsxODkwLDI4IEBAIHZvaWQgZG1fYnVmaW9fcmVsZWFzZShzdHJ1Y3QgZG1fYnVmZmVyICpi KQogewogCXN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMgPSBiLT5jOwogCi0JZG1fYnVmaW9fbG9j ayhjKTsKKwkvKgorCSAqIElmIHRoZXJlIHdlcmUgZXJyb3JzIG9uIHRoZSBidWZmZXIsIGFuZCB0 aGUgYnVmZmVyIGlzIG5vdAorCSAqIHRvIGJlIHdyaXR0ZW4sIGZyZWUgdGhlIGJ1ZmZlci4gVGhl cmUgaXMgbm8gcG9pbnQgaW4gY2FjaGluZworCSAqIGludmFsaWQgYnVmZmVyLgorCSAqLworCWlm ICgoYi0+cmVhZF9lcnJvciB8fCBiLT53cml0ZV9lcnJvcikgJiYKKwkgICAgIXRlc3RfYml0X2Fj cXVpcmUoQl9SRUFESU5HLCAmYi0+c3RhdGUpICYmCisJICAgICF0ZXN0X2JpdChCX1dSSVRJTkcs ICZiLT5zdGF0ZSkgJiYKKwkgICAgIXRlc3RfYml0KEJfRElSVFksICZiLT5zdGF0ZSkpIHsKKwkJ ZG1fYnVmaW9fbG9jayhjKTsKIAotCUJVR19PTighYi0+X19ob2xkX2NvdW50KTsKLQotCWItPl9f aG9sZF9jb3VudC0tOwotCWlmICghYi0+X19ob2xkX2NvdW50KSB7Ci0JCXdha2VfdXAoJmMtPmZy ZWVfYnVmZmVyX3dhaXQpOwotCi0JCS8qCi0JCSAqIElmIHRoZXJlIHdlcmUgZXJyb3JzIG9uIHRo ZSBidWZmZXIsIGFuZCB0aGUgYnVmZmVyIGlzIG5vdAotCQkgKiB0byBiZSB3cml0dGVuLCBmcmVl IHRoZSBidWZmZXIuIFRoZXJlIGlzIG5vIHBvaW50IGluIGNhY2hpbmcKLQkJICogaW52YWxpZCBi dWZmZXIuCi0JCSAqLwotCQlpZiAoKGItPnJlYWRfZXJyb3IgfHwgYi0+d3JpdGVfZXJyb3IpICYm Ci0JCSAgICAhdGVzdF9iaXRfYWNxdWlyZShCX1JFQURJTkcsICZiLT5zdGF0ZSkgJiYKLQkJICAg ICF0ZXN0X2JpdChCX1dSSVRJTkcsICZiLT5zdGF0ZSkgJiYKLQkJICAgICF0ZXN0X2JpdChCX0RJ UlRZLCAmYi0+c3RhdGUpKSB7Ci0JCQlfX3VubGlua19idWZmZXIoYik7CisJCS8qIGNhY2hlIHJl bW92ZSBjYW4gZmFpbCBpZiB0aGVyZSBhcmUgb3RoZXIgaG9sZGVycyAqLworCQlpZiAoY2FjaGVf cmVtb3ZlKCZjLT5jYWNoZSwgYikpIHsKIAkJCV9fZnJlZV9idWZmZXJfd2FrZShiKTsKKwkJCWRt X2J1ZmlvX3VubG9jayhjKTsKKwkJCXJldHVybjsKIAkJfQorCisJCWRtX2J1ZmlvX3VubG9jayhj KTsKIAl9CiAKLQlkbV9idWZpb191bmxvY2soYyk7CisJY2FjaGVfcHV0X2FuZF93YWtlKGMsIGIp OwogfQogRVhQT1JUX1NZTUJPTF9HUEwoZG1fYnVmaW9fcmVsZWFzZSk7CiAKQEAgLTE5NjMsNyAr MTkzMCw3IEBAIHZvaWQgZG1fYnVmaW9fbWFya19wYXJ0aWFsX2J1ZmZlcl9kaXJ0eShzdHJ1Y3Qg ZG1fYnVmZmVyICpiLAogCWlmICghdGVzdF9hbmRfc2V0X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUp KSB7CiAJCWItPmRpcnR5X3N0YXJ0ID0gc3RhcnQ7CiAJCWItPmRpcnR5X2VuZCA9IGVuZDsKLQkJ X19yZWxpbmtfbHJ1KGIsIExJU1RfRElSVFkpOworCQljYWNoZV9tYXJrKCZjLT5jYWNoZSwgYiwg TElTVF9ESVJUWSk7CiAJfSBlbHNlIHsKIAkJaWYgKHN0YXJ0IDwgYi0+ZGlydHlfc3RhcnQpCiAJ CQliLT5kaXJ0eV9zdGFydCA9IHN0YXJ0OwpAQCAtMjAwMiwxMSArMTk2OSwxOSBAQCBFWFBPUlRf U1lNQk9MX0dQTChkbV9idWZpb193cml0ZV9kaXJ0eV9idWZmZXJzX2FzeW5jKTsKICAqCiAgKiBG aW5hbGx5LCB3ZSBmbHVzaCBoYXJkd2FyZSBkaXNrIGNhY2hlLgogICovCitzdGF0aWMgYm9vbCBp c193cml0aW5nKHN0cnVjdCBscnVfZW50cnkgKmUsIHZvaWQgKmNvbnRleHQpCit7CisJc3RydWN0 IGRtX2J1ZmZlciAqYiA9IGxlX3RvX2J1ZmZlcihlKTsKKworCXJldHVybiB0ZXN0X2JpdChCX1dS SVRJTkcsICZiLT5zdGF0ZSk7Cit9CisKIGludCBkbV9idWZpb193cml0ZV9kaXJ0eV9idWZmZXJz KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMpCiB7CiAJaW50IGEsIGY7Ci0JdW5zaWduZWQgbG9u ZyBidWZmZXJzX3Byb2Nlc3NlZCA9IDA7Ci0Jc3RydWN0IGRtX2J1ZmZlciAqYiwgKnRtcDsKKwl1 bnNpZ25lZCBsb25nIG5yX2J1ZmZlcnM7CisJc3RydWN0IGxydV9lbnRyeSAqZTsKKwlzdHJ1Y3Qg bHJ1X2l0ZXIgaXQ7CiAKIAlMSVNUX0hFQUQod3JpdGVfbGlzdCk7CiAKQEAgLTIwMTYsNTIgKzE5 OTEsMzIgQEAgaW50IGRtX2J1ZmlvX3dyaXRlX2RpcnR5X2J1ZmZlcnMoc3RydWN0IGRtX2J1Zmlv X2NsaWVudCAqYykKIAlfX2ZsdXNoX3dyaXRlX2xpc3QoJndyaXRlX2xpc3QpOwogCWRtX2J1Zmlv X2xvY2soYyk7CiAKLWFnYWluOgotCWxpc3RfZm9yX2VhY2hfZW50cnlfc2FmZV9yZXZlcnNlKGIs IHRtcCwgJmMtPmxydVtMSVNUX0RJUlRZXSwgbHJ1X2xpc3QpIHsKLQkJaW50IGRyb3BwZWRfbG9j ayA9IDA7Ci0KLQkJaWYgKGJ1ZmZlcnNfcHJvY2Vzc2VkIDwgYy0+bl9idWZmZXJzW0xJU1RfRElS VFldKQotCQkJYnVmZmVyc19wcm9jZXNzZWQrKzsKKwlucl9idWZmZXJzID0gY2FjaGVfY291bnQo JmMtPmNhY2hlLCBMSVNUX0RJUlRZKTsKKwlscnVfaXRlcl9iZWdpbigmYy0+Y2FjaGUubHJ1W0xJ U1RfRElSVFldLCAmaXQpOworCXdoaWxlICgoZSA9IGxydV9pdGVyX25leHQoJml0LCBpc193cml0 aW5nLCBjKSkpIHsKKwkJc3RydWN0IGRtX2J1ZmZlciAqYiA9IGxlX3RvX2J1ZmZlcihlKTsKKwkJ X19jYWNoZV9pbmNfYnVmZmVyKGIpOwogCiAJCUJVR19PTih0ZXN0X2JpdChCX1JFQURJTkcsICZi LT5zdGF0ZSkpOwogCi0JCWlmICh0ZXN0X2JpdChCX1dSSVRJTkcsICZiLT5zdGF0ZSkpIHsKLQkJ CWlmIChidWZmZXJzX3Byb2Nlc3NlZCA8IGMtPm5fYnVmZmVyc1tMSVNUX0RJUlRZXSkgewotCQkJ CWRyb3BwZWRfbG9jayA9IDE7Ci0JCQkJYi0+X19ob2xkX2NvdW50Kys7Ci0JCQkJZG1fYnVmaW9f dW5sb2NrKGMpOwotCQkJCXdhaXRfb25fYml0X2lvKCZiLT5zdGF0ZSwgQl9XUklUSU5HLAotCQkJ CQkgICAgICAgVEFTS19VTklOVEVSUlVQVElCTEUpOwotCQkJCWRtX2J1ZmlvX2xvY2soYyk7Ci0J CQkJYi0+X19ob2xkX2NvdW50LS07Ci0JCQl9IGVsc2UKLQkJCQl3YWl0X29uX2JpdF9pbygmYi0+ c3RhdGUsIEJfV1JJVElORywKLQkJCQkJICAgICAgIFRBU0tfVU5JTlRFUlJVUFRJQkxFKTsKKwkJ aWYgKG5yX2J1ZmZlcnMpIHsKKwkJCW5yX2J1ZmZlcnMtLTsKKwkJCWRtX2J1ZmlvX3VubG9jayhj KTsKKwkJCXdhaXRfb25fYml0X2lvKCZiLT5zdGF0ZSwgQl9XUklUSU5HLCBUQVNLX1VOSU5URVJS VVBUSUJMRSk7CisJCQlkbV9idWZpb19sb2NrKGMpOworCQl9IGVsc2UgeworCQkJd2FpdF9vbl9i aXRfaW8oJmItPnN0YXRlLCBCX1dSSVRJTkcsIFRBU0tfVU5JTlRFUlJVUFRJQkxFKTsKIAkJfQog Ci0JCWlmICghdGVzdF9iaXQoQl9ESVJUWSwgJmItPnN0YXRlKSAmJgotCQkgICAgIXRlc3RfYml0 KEJfV1JJVElORywgJmItPnN0YXRlKSkKLQkJCV9fcmVsaW5rX2xydShiLCBMSVNUX0NMRUFOKTsK KwkJaWYgKCF0ZXN0X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUpICYmICF0ZXN0X2JpdChCX1dSSVRJ TkcsICZiLT5zdGF0ZSkpCisJCQljYWNoZV9tYXJrKCZjLT5jYWNoZSwgYiwgTElTVF9DTEVBTik7 CisKKwkJY2FjaGVfcHV0X2FuZF93YWtlKGMsIGIpOwogCiAJCWNvbmRfcmVzY2hlZCgpOwotCi0J CS8qCi0JCSAqIElmIHdlIGRyb3BwZWQgdGhlIGxvY2ssIHRoZSBsaXN0IGlzIG5vIGxvbmdlciBj b25zaXN0ZW50LAotCQkgKiBzbyB3ZSBtdXN0IHJlc3RhcnQgdGhlIHNlYXJjaC4KLQkJICoKLQkJ ICogSW4gdGhlIG1vc3QgY29tbW9uIGNhc2UsIHRoZSBidWZmZXIganVzdCBwcm9jZXNzZWQgaXMK LQkJICogcmVsaW5rZWQgdG8gdGhlIGNsZWFuIGxpc3QsIHNvIHdlIHdvbid0IGxvb3Agc2Nhbm5p bmcgdGhlCi0JCSAqIHNhbWUgYnVmZmVyIGFnYWluIGFuZCBhZ2Fpbi4KLQkJICoKLQkJICogVGhp cyBtYXkgbGl2ZWxvY2sgaWYgdGhlcmUgaXMgYW5vdGhlciB0aHJlYWQgc2ltdWx0YW5lb3VzbHkK LQkJICogZGlydHlpbmcgYnVmZmVycywgc28gd2UgY291bnQgdGhlIG51bWJlciBvZiBidWZmZXJz IHdhbGtlZAotCQkgKiBhbmQgaWYgaXQgZXhjZWVkcyB0aGUgdG90YWwgbnVtYmVyIG9mIGJ1ZmZl cnMsIGl0IG1lYW5zIHRoYXQKLQkJICogc29tZW9uZSBpcyBkb2luZyBzb21lIHdyaXRlcyBzaW11 bHRhbmVvdXNseSB3aXRoIHVzLiAgSW4KLQkJICogdGhpcyBjYXNlLCBzdG9wLCBkcm9wcGluZyB0 aGUgbG9jay4KLQkJICovCi0JCWlmIChkcm9wcGVkX2xvY2spCi0JCQlnb3RvIGFnYWluOwogCX0K KwlscnVfaXRlcl9lbmQoJml0KTsKKwogCXdha2VfdXAoJmMtPmZyZWVfYnVmZmVyX3dhaXQpOwog CWRtX2J1ZmlvX3VubG9jayhjKTsKIApAQCAtMjEyMiwxMiArMjA3NywyMyBAQCBpbnQgZG1fYnVm aW9faXNzdWVfZGlzY2FyZChzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjLCBzZWN0b3JfdCBibG9j aywgc2VjdG9yX3QgYwogfQogRVhQT1JUX1NZTUJPTF9HUEwoZG1fYnVmaW9faXNzdWVfZGlzY2Fy ZCk7CiAKLXN0YXRpYyB2b2lkIGZvcmdldF9idWZmZXJfbG9ja2VkKHN0cnVjdCBkbV9idWZmZXIg KmIpCitzdGF0aWMgYm9vbCBmb3JnZXRfYnVmZmVyKHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMs IHNlY3Rvcl90IGJsb2NrKQogewotCWlmIChsaWtlbHkoIWItPl9faG9sZF9jb3VudCkgJiYgbGlr ZWx5KCFzbXBfbG9hZF9hY3F1aXJlKCZiLT5zdGF0ZSkpKSB7Ci0JCV9fdW5saW5rX2J1ZmZlcihi KTsKLQkJX19mcmVlX2J1ZmZlcl93YWtlKGIpOworCXN0cnVjdCBkbV9idWZmZXIgKmI7CisKKwli ID0gY2FjaGVfZ2V0KCZjLT5jYWNoZSwgYmxvY2spOworCWlmIChiKSB7CisJCWlmIChsaWtlbHko IXNtcF9sb2FkX2FjcXVpcmUoJmItPnN0YXRlKSkpIHsKKwkJCWlmIChjYWNoZV9yZW1vdmUoJmMt PmNhY2hlLCBiKSkKKwkJCQlfX2ZyZWVfYnVmZmVyX3dha2UoYik7CisJCQllbHNlCisJCQkJY2Fj aGVfcHV0X2FuZF93YWtlKGMsIGIpOworCQl9IGVsc2UgeworCQkJY2FjaGVfcHV0X2FuZF93YWtl KGMsIGIpOworCQl9CiAJfQorCisJcmV0dXJuIGIgPyB0cnVlIDogZmFsc2U7CiB9CiAKIC8qCkBA IC0yMTM4LDM4ICsyMTA0LDIyIEBAIHN0YXRpYyB2b2lkIGZvcmdldF9idWZmZXJfbG9ja2VkKHN0 cnVjdCBkbV9idWZmZXIgKmIpCiAgKi8KIHZvaWQgZG1fYnVmaW9fZm9yZ2V0KHN0cnVjdCBkbV9i dWZpb19jbGllbnQgKmMsIHNlY3Rvcl90IGJsb2NrKQogewotCXN0cnVjdCBkbV9idWZmZXIgKmI7 Ci0KIAlkbV9idWZpb19sb2NrKGMpOwotCi0JYiA9IF9fZmluZChjLCBibG9jayk7Ci0JaWYgKGIp Ci0JCWZvcmdldF9idWZmZXJfbG9ja2VkKGIpOwotCisJZm9yZ2V0X2J1ZmZlcihjLCBibG9jayk7 CiAJZG1fYnVmaW9fdW5sb2NrKGMpOwogfQogRVhQT1JUX1NZTUJPTF9HUEwoZG1fYnVmaW9fZm9y Z2V0KTsKIAorc3RhdGljIGVudW0gZXZpY3RfcmVzdWx0IGlkbGUoc3RydWN0IGRtX2J1ZmZlciAq Yiwgdm9pZCAqY29udGV4dCkKK3sKKwlyZXR1cm4gYi0+c3RhdGUgPyBFUl9ET05UX0VWSUNUIDog RVJfRVZJQ1Q7Cit9CisKIHZvaWQgZG1fYnVmaW9fZm9yZ2V0X2J1ZmZlcnMoc3RydWN0IGRtX2J1 ZmlvX2NsaWVudCAqYywgc2VjdG9yX3QgYmxvY2ssIHNlY3Rvcl90IG5fYmxvY2tzKQogewotCXN0 cnVjdCBkbV9idWZmZXIgKmI7Ci0Jc2VjdG9yX3QgZW5kX2Jsb2NrID0gYmxvY2sgKyBuX2Jsb2Nr czsKLQotCXdoaWxlIChibG9jayA8IGVuZF9ibG9jaykgewotCQlkbV9idWZpb19sb2NrKGMpOwot Ci0JCWIgPSBfX2ZpbmRfbmV4dF9vbGQoYywgYmxvY2spOwotCQlpZiAoYikgewotCQkJYmxvY2sg PSBiLT5ibG9jayArIDE7Ci0JCQlmb3JnZXRfYnVmZmVyX2xvY2tlZChiKTsKLQkJfQotCi0JCWRt X2J1ZmlvX3VubG9jayhjKTsKLQotCQlpZiAoIWIpCi0JCQlicmVhazsKLQl9Ci0KKwlkbV9idWZp b19sb2NrKGMpOworCWNhY2hlX3JlbW92ZV9yYW5nZSgmYy0+Y2FjaGUsIGJsb2NrLCBibG9jayAr IG5fYmxvY2tzLCBpZGxlLCBfX2ZyZWVfYnVmZmVyX3dha2UpOworCWRtX2J1ZmlvX3VubG9jayhj KTsKIH0KIEVYUE9SVF9TWU1CT0xfR1BMKGRtX2J1ZmlvX2ZvcmdldF9idWZmZXJzKTsKIApAQCAt MjIzMSwxMSArMjE4MSwyNiBAQCBzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpkbV9idWZpb19nZXRf Y2xpZW50KHN0cnVjdCBkbV9idWZmZXIgKmIpCiB9CiBFWFBPUlRfU1lNQk9MX0dQTChkbV9idWZp b19nZXRfY2xpZW50KTsKIAorc3RhdGljIGVudW0gaXRfYWN0aW9uIHdhcm5fbGVhayhzdHJ1Y3Qg ZG1fYnVmZmVyICpiLCB2b2lkICpjb250ZXh0KQoreworCWJvb2wgKndhcm5lZCA9IGNvbnRleHQ7 CisKKwlXQVJOX09OKCEoKndhcm5lZCkpOworCSp3YXJuZWQgPSB0cnVlOworCURNRVJSKCJsZWFr ZWQgYnVmZmVyICVsbHgsIGhvbGQgY291bnQgJXUsIGxpc3QgJWQiLAorCSAgICAgICh1bnNpZ25l ZCBsb25nIGxvbmcpYi0+YmxvY2ssIGF0b21pY19yZWFkKCZiLT5ob2xkX2NvdW50KSwgYi0+bGlz dF9tb2RlKTsKKyNpZmRlZiBDT05GSUdfRE1fREVCVUdfQkxPQ0tfU1RBQ0tfVFJBQ0lORworCXN0 YWNrX3RyYWNlX3ByaW50KGItPnN0YWNrX2VudHJpZXMsIGItPnN0YWNrX2xlbiwgMSk7CisJLyog bWFyayB1bmNsYWltZWQgdG8gYXZvaWQgV0FSTl9PTiBhdCBlbmQgb2YgZHJvcF9idWZmZXJzKCkg Ki8KKwlhdG9taWNfc2V0KCZiLT5ob2xkX2NvdW50LCAwKTsKKyNlbmRpZgorCXJldHVybiBJVF9O RVhUOworfQorCiBzdGF0aWMgdm9pZCBkcm9wX2J1ZmZlcnMoc3RydWN0IGRtX2J1ZmlvX2NsaWVu dCAqYykKIHsKLQlzdHJ1Y3QgZG1fYnVmZmVyICpiOwogCWludCBpOwotCWJvb2wgd2FybmVkID0g ZmFsc2U7CisJc3RydWN0IGRtX2J1ZmZlciAqYjsKIAogCWlmIChXQVJOX09OKGRtX2J1ZmlvX2lu X3JlcXVlc3QoKSkpCiAJCXJldHVybjsgLyogc2hvdWxkIG5ldmVyIGhhcHBlbiAqLwpAQCAtMjI1 MCwxOCArMjIxNSwxMSBAQCBzdGF0aWMgdm9pZCBkcm9wX2J1ZmZlcnMoc3RydWN0IGRtX2J1Zmlv X2NsaWVudCAqYykKIAl3aGlsZSAoKGIgPSBfX2dldF91bmNsYWltZWRfYnVmZmVyKGMpKSkKIAkJ X19mcmVlX2J1ZmZlcl93YWtlKGIpOwogCi0JZm9yIChpID0gMDsgaSA8IExJU1RfU0laRTsgaSsr KQotCQlsaXN0X2Zvcl9lYWNoX2VudHJ5KGIsICZjLT5scnVbaV0sIGxydV9saXN0KSB7Ci0JCQlX QVJOX09OKCF3YXJuZWQpOwotCQkJd2FybmVkID0gdHJ1ZTsKLQkJCURNRVJSKCJsZWFrZWQgYnVm ZmVyICVsbHgsIGhvbGQgY291bnQgJXUsIGxpc3QgJWQiLAotCQkJICAgICAgKHVuc2lnbmVkIGxv bmcgbG9uZyliLT5ibG9jaywgYi0+X19ob2xkX2NvdW50LCBpKTsKLSNpZmRlZiBDT05GSUdfRE1f REVCVUdfQkxPQ0tfU1RBQ0tfVFJBQ0lORwotCQkJc3RhY2tfdHJhY2VfcHJpbnQoYi0+c3RhY2tf ZW50cmllcywgYi0+c3RhY2tfbGVuLCAxKTsKLQkJCS8qIG1hcmsgdW5jbGFpbWVkIHRvIGF2b2lk IFdBUk5fT04gYmVsb3cgKi8KLQkJCWItPmhvbGRfY291bnQgPSAwOwotI2VuZGlmCi0JCX0KKwlm b3IgKGkgPSAwOyBpIDwgTElTVF9TSVpFOyBpKyspIHsKKwkJYm9vbCB3YXJuZWQgPSBmYWxzZTsK KworCQljYWNoZV9pdGVyYXRlKCZjLT5jYWNoZSwgaSwgd2Fybl9sZWFrLCAmd2FybmVkKTsKKwl9 CiAKICNpZmRlZiBDT05GSUdfRE1fREVCVUdfQkxPQ0tfU1RBQ0tfVFJBQ0lORwogCXdoaWxlICgo YiA9IF9fZ2V0X3VuY2xhaW1lZF9idWZmZXIoYykpKQpAQCAtMjI2OSwzOSArMjIyNywxMSBAQCBz dGF0aWMgdm9pZCBkcm9wX2J1ZmZlcnMoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYykKICNlbmRp ZgogCiAJZm9yIChpID0gMDsgaSA8IExJU1RfU0laRTsgaSsrKQotCQlXQVJOX09OKCFsaXN0X2Vt cHR5KCZjLT5scnVbaV0pKTsKKwkJV0FSTl9PTihjYWNoZV9jb3VudCgmYy0+Y2FjaGUsIGkpKTsK IAogCWRtX2J1ZmlvX3VubG9jayhjKTsKIH0KIAotLyoKLSAqIFdlIG1heSBub3QgYmUgYWJsZSB0 byBldmljdCB0aGlzIGJ1ZmZlciBpZiBJTyBwZW5kaW5nIG9yIHRoZSBjbGllbnQKLSAqIGlzIHN0 aWxsIHVzaW5nIGl0LiAgQ2FsbGVyIGlzIGV4cGVjdGVkIHRvIGtub3cgYnVmZmVyIGlzIHRvbyBv bGQuCi0gKgotICogQW5kIGlmIEdGUF9OT0ZTIGlzIHVzZWQsIHdlIG11c3Qgbm90IGRvIGFueSBJ L08gYmVjYXVzZSB3ZSBob2xkCi0gKiBkbV9idWZpb19jbGllbnRzX2xvY2sgYW5kIHdlIHdvdWxk IHJpc2sgZGVhZGxvY2sgaWYgdGhlIEkvTyBnZXRzCi0gKiByZXJvdXRlZCB0byBkaWZmZXJlbnQg YnVmaW8gY2xpZW50LgotICovCi1zdGF0aWMgYm9vbCBfX3RyeV9ldmljdF9idWZmZXIoc3RydWN0 IGRtX2J1ZmZlciAqYiwgZ2ZwX3QgZ2ZwKQotewotCWlmICghKGdmcCAmIF9fR0ZQX0ZTKSB8fAot CSAgICAoc3RhdGljX2JyYW5jaF91bmxpa2VseSgmbm9fc2xlZXBfZW5hYmxlZCkgJiYgYi0+Yy0+ bm9fc2xlZXApKSB7Ci0JCWlmICh0ZXN0X2JpdF9hY3F1aXJlKEJfUkVBRElORywgJmItPnN0YXRl KSB8fAotCQkgICAgdGVzdF9iaXQoQl9XUklUSU5HLCAmYi0+c3RhdGUpIHx8Ci0JCSAgICB0ZXN0 X2JpdChCX0RJUlRZLCAmYi0+c3RhdGUpKQotCQkJcmV0dXJuIGZhbHNlOwotCX0KLQotCWlmIChi LT5fX2hvbGRfY291bnQpCi0JCXJldHVybiBmYWxzZTsKLQotCV9fbWFrZV9idWZmZXJfY2xlYW4o Yik7Ci0JX191bmxpbmtfYnVmZmVyKGIpOwotCV9fZnJlZV9idWZmZXJfd2FrZShiKTsKLQotCXJl dHVybiB0cnVlOwotfQotCiBzdGF0aWMgdW5zaWduZWQgbG9uZyBnZXRfcmV0YWluX2J1ZmZlcnMo c3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYykKIHsKIAl1bnNpZ25lZCBsb25nIHJldGFpbl9ieXRl cyA9IFJFQURfT05DRShkbV9idWZpb19yZXRhaW5fYnl0ZXMpOwpAQCAtMjMxNywyMiArMjI0Nywy OCBAQCBzdGF0aWMgdW5zaWduZWQgbG9uZyBnZXRfcmV0YWluX2J1ZmZlcnMoc3RydWN0IGRtX2J1 ZmlvX2NsaWVudCAqYykKIHN0YXRpYyB2b2lkIF9fc2NhbihzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50 ICpjKQogewogCWludCBsOwotCXN0cnVjdCBkbV9idWZmZXIgKmIsICp0bXA7CisJc3RydWN0IGRt X2J1ZmZlciAqYjsKIAl1bnNpZ25lZCBsb25nIGZyZWVkID0gMDsKLQl1bnNpZ25lZCBsb25nIGNv dW50ID0gYy0+bl9idWZmZXJzW0xJU1RfQ0xFQU5dICsKLQkJCSAgICAgIGMtPm5fYnVmZmVyc1tM SVNUX0RJUlRZXTsKIAl1bnNpZ25lZCBsb25nIHJldGFpbl90YXJnZXQgPSBnZXRfcmV0YWluX2J1 ZmZlcnMoYyk7CisJdW5zaWduZWQgbG9uZyBjb3VudCA9IGNhY2hlX3RvdGFsKCZjLT5jYWNoZSk7 CiAKIAlmb3IgKGwgPSAwOyBsIDwgTElTVF9TSVpFOyBsKyspIHsKLQkJbGlzdF9mb3JfZWFjaF9l bnRyeV9zYWZlX3JldmVyc2UoYiwgdG1wLCAmYy0+bHJ1W2xdLCBscnVfbGlzdCkgeworCQl3aGls ZSAodHJ1ZSkgewogCQkJaWYgKGNvdW50IC0gZnJlZWQgPD0gcmV0YWluX3RhcmdldCkKIAkJCQlh dG9taWNfbG9uZ19zZXQoJmMtPm5lZWRfc2hyaW5rLCAwKTsKIAkJCWlmICghYXRvbWljX2xvbmdf cmVhZCgmYy0+bmVlZF9zaHJpbmspKQotCQkJCXJldHVybjsKLQkJCWlmIChfX3RyeV9ldmljdF9i dWZmZXIoYiwgR0ZQX0tFUk5FTCkpIHsKLQkJCQlhdG9taWNfbG9uZ19kZWMoJmMtPm5lZWRfc2hy aW5rKTsKLQkJCQlmcmVlZCsrOwotCQkJfQorCQkJCWJyZWFrOworCisJCQliID0gY2FjaGVfZXZp Y3QoJmMtPmNhY2hlLCBsLAorCQkJCQlsID09IExJU1RfQ0xFQU4gPyBpc19jbGVhbiA6IGlzX2Rp cnR5LCBjKTsKKwkJCWlmICghYikKKwkJCQlicmVhazsKKworCQkJX19tYWtlX2J1ZmZlcl9jbGVh bihiKTsKKwkJCV9fZnJlZV9idWZmZXJfd2FrZShiKTsKKworCQkJYXRvbWljX2xvbmdfZGVjKCZj LT5uZWVkX3Nocmluayk7CisJCQlmcmVlZCsrOwogCQkJY29uZF9yZXNjaGVkKCk7CiAJCX0KIAl9 CkBAIC0yMzYxLDggKzIyOTcsNyBAQCBzdGF0aWMgdW5zaWduZWQgbG9uZyBkbV9idWZpb19zaHJp bmtfc2NhbihzdHJ1Y3Qgc2hyaW5rZXIgKnNocmluaywgc3RydWN0IHNocmluawogc3RhdGljIHVu c2lnbmVkIGxvbmcgZG1fYnVmaW9fc2hyaW5rX2NvdW50KHN0cnVjdCBzaHJpbmtlciAqc2hyaW5r LCBzdHJ1Y3Qgc2hyaW5rX2NvbnRyb2wgKnNjKQogewogCXN0cnVjdCBkbV9idWZpb19jbGllbnQg KmMgPSBjb250YWluZXJfb2Yoc2hyaW5rLCBzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50LCBzaHJpbmtl cik7Ci0JdW5zaWduZWQgbG9uZyBjb3VudCA9IFJFQURfT05DRShjLT5uX2J1ZmZlcnNbTElTVF9D TEVBTl0pICsKLQkJCSAgICAgIFJFQURfT05DRShjLT5uX2J1ZmZlcnNbTElTVF9ESVJUWV0pOwor CXVuc2lnbmVkIGxvbmcgY291bnQgPSBjYWNoZV90b3RhbCgmYy0+Y2FjaGUpOwogCXVuc2lnbmVk IGxvbmcgcmV0YWluX3RhcmdldCA9IGdldF9yZXRhaW5fYnVmZmVycyhjKTsKIAl1bnNpZ25lZCBs b25nIHF1ZXVlZF9mb3JfY2xlYW51cCA9IGF0b21pY19sb25nX3JlYWQoJmMtPm5lZWRfc2hyaW5r KTsKIApAQCAtMjM5MCw3ICsyMzI1LDYgQEAgc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqZG1fYnVm aW9fY2xpZW50X2NyZWF0ZShzdHJ1Y3QgYmxvY2tfZGV2aWNlICpiZGV2LCB1bnNpZ24KIHsKIAlp bnQgcjsKIAlzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjOwotCXVuc2lnbmVkIGludCBpOwogCWNo YXIgc2xhYl9uYW1lWzI3XTsKIAogCWlmICghYmxvY2tfc2l6ZSB8fCBibG9ja19zaXplICYgKCgx IDw8IFNFQ1RPUl9TSElGVCkgLSAxKSkgewpAQCAtMjQwNCw3ICsyMzM4LDcgQEAgc3RydWN0IGRt X2J1ZmlvX2NsaWVudCAqZG1fYnVmaW9fY2xpZW50X2NyZWF0ZShzdHJ1Y3QgYmxvY2tfZGV2aWNl ICpiZGV2LCB1bnNpZ24KIAkJciA9IC1FTk9NRU07CiAJCWdvdG8gYmFkX2NsaWVudDsKIAl9Ci0J Yy0+YnVmZmVyX3RyZWUgPSBSQl9ST09UOworCWNhY2hlX2luaXQoJmMtPmNhY2hlKTsKIAogCWMt PmJkZXYgPSBiZGV2OwogCWMtPmJsb2NrX3NpemUgPSBibG9ja19zaXplOwpAQCAtMjQyMSwxMSAr MjM1NSw2IEBAIHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmRtX2J1ZmlvX2NsaWVudF9jcmVhdGUo c3RydWN0IGJsb2NrX2RldmljZSAqYmRldiwgdW5zaWduCiAJCXN0YXRpY19icmFuY2hfaW5jKCZu b19zbGVlcF9lbmFibGVkKTsKIAl9CiAKLQlmb3IgKGkgPSAwOyBpIDwgTElTVF9TSVpFOyBpKysp IHsKLQkJSU5JVF9MSVNUX0hFQUQoJmMtPmxydVtpXSk7Ci0JCWMtPm5fYnVmZmVyc1tpXSA9IDA7 Ci0JfQotCiAJbXV0ZXhfaW5pdCgmYy0+bG9jayk7CiAJc3Bpbl9sb2NrX2luaXQoJmMtPnNwaW5s b2NrKTsKIAlJTklUX0xJU1RfSEVBRCgmYy0+cmVzZXJ2ZWRfYnVmZmVycyk7CkBAIC0yNDk3LDkg KzI0MjYsOSBAQCBzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpkbV9idWZpb19jbGllbnRfY3JlYXRl KHN0cnVjdCBibG9ja19kZXZpY2UgKmJkZXYsIHVuc2lnbgogCiBiYWQ6CiAJd2hpbGUgKCFsaXN0 X2VtcHR5KCZjLT5yZXNlcnZlZF9idWZmZXJzKSkgewotCQlzdHJ1Y3QgZG1fYnVmZmVyICpiID0g bGlzdF9lbnRyeShjLT5yZXNlcnZlZF9idWZmZXJzLm5leHQsCi0JCQkJCQkgc3RydWN0IGRtX2J1 ZmZlciwgbHJ1X2xpc3QpOwotCQlsaXN0X2RlbCgmYi0+bHJ1X2xpc3QpOworCQlzdHJ1Y3QgZG1f YnVmZmVyICpiID0gbGlzdF90b19idWZmZXIoYy0+cmVzZXJ2ZWRfYnVmZmVycy5uZXh0KTsKKwor CQlsaXN0X2RlbCgmYi0+bHJ1Lmxpc3QpOwogCQlmcmVlX2J1ZmZlcihiKTsKIAl9CiAJa21lbV9j YWNoZV9kZXN0cm95KGMtPnNsYWJfY2FjaGUpOwpAQCAtMjUzNiwyMyArMjQ2NSwyMyBAQCB2b2lk IGRtX2J1ZmlvX2NsaWVudF9kZXN0cm95KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMpCiAKIAlt dXRleF91bmxvY2soJmRtX2J1ZmlvX2NsaWVudHNfbG9jayk7CiAKLQlXQVJOX09OKCFSQl9FTVBU WV9ST09UKCZjLT5idWZmZXJfdHJlZSkpOwogCVdBUk5fT04oYy0+bmVlZF9yZXNlcnZlZF9idWZm ZXJzKTsKIAogCXdoaWxlICghbGlzdF9lbXB0eSgmYy0+cmVzZXJ2ZWRfYnVmZmVycykpIHsKLQkJ c3RydWN0IGRtX2J1ZmZlciAqYiA9IGxpc3RfZW50cnkoYy0+cmVzZXJ2ZWRfYnVmZmVycy5uZXh0 LAotCQkJCQkJIHN0cnVjdCBkbV9idWZmZXIsIGxydV9saXN0KTsKLQkJbGlzdF9kZWwoJmItPmxy dV9saXN0KTsKKwkJc3RydWN0IGRtX2J1ZmZlciAqYiA9IGxpc3RfdG9fYnVmZmVyKGMtPnJlc2Vy dmVkX2J1ZmZlcnMubmV4dCk7CisKKwkJbGlzdF9kZWwoJmItPmxydS5saXN0KTsKIAkJZnJlZV9i dWZmZXIoYik7CiAJfQogCiAJZm9yIChpID0gMDsgaSA8IExJU1RfU0laRTsgaSsrKQotCQlpZiAo Yy0+bl9idWZmZXJzW2ldKQotCQkJRE1FUlIoImxlYWtlZCBidWZmZXIgY291bnQgJWQ6ICVsZCIs IGksIGMtPm5fYnVmZmVyc1tpXSk7CisJCWlmIChjYWNoZV9jb3VudCgmYy0+Y2FjaGUsIGkpKQor CQkJRE1FUlIoImxlYWtlZCBidWZmZXIgY291bnQgJWQ6ICVsdSIsIGksIGNhY2hlX2NvdW50KCZj LT5jYWNoZSwgaSkpOwogCiAJZm9yIChpID0gMDsgaSA8IExJU1RfU0laRTsgaSsrKQotCQlXQVJO X09OKGMtPm5fYnVmZmVyc1tpXSk7CisJCVdBUk5fT04oY2FjaGVfY291bnQoJmMtPmNhY2hlLCBp KSk7CiAKKwljYWNoZV9kZXN0cm95KCZjLT5jYWNoZSk7CiAJa21lbV9jYWNoZV9kZXN0cm95KGMt PnNsYWJfY2FjaGUpOwogCWttZW1fY2FjaGVfZGVzdHJveShjLT5zbGFiX2J1ZmZlcik7CiAJZG1f aW9fY2xpZW50X2Rlc3Ryb3koYy0+ZG1faW8pOwpAQCAtMjU2OSw2ICsyNDk4LDggQEAgdm9pZCBk bV9idWZpb19zZXRfc2VjdG9yX29mZnNldChzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjLCBzZWN0 b3JfdCBzdGFydCkKIH0KIEVYUE9SVF9TWU1CT0xfR1BMKGRtX2J1ZmlvX3NldF9zZWN0b3Jfb2Zm c2V0KTsKIAorLyotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLSovCisKIHN0YXRpYyB1bnNpZ25lZCBpbnQgZ2V0X21heF9hZ2VfaHoo dm9pZCkKIHsKIAl1bnNpZ25lZCBpbnQgbWF4X2FnZSA9IFJFQURfT05DRShkbV9idWZpb19tYXhf YWdlKTsKQEAgLTI1ODEsMTMgKzI1MTIsNzQgQEAgc3RhdGljIHVuc2lnbmVkIGludCBnZXRfbWF4 X2FnZV9oeih2b2lkKQogCiBzdGF0aWMgYm9vbCBvbGRlcl90aGFuKHN0cnVjdCBkbV9idWZmZXIg KmIsIHVuc2lnbmVkIGxvbmcgYWdlX2h6KQogewotCXJldHVybiB0aW1lX2FmdGVyX2VxKGppZmZp ZXMsIGItPmxhc3RfYWNjZXNzZWQgKyBhZ2VfaHopOworCXJldHVybiB0aW1lX2FmdGVyX2VxKGpp ZmZpZXMsIFJFQURfT05DRShiLT5sYXN0X2FjY2Vzc2VkKSArIGFnZV9oeik7CiB9CiAKLXN0YXRp YyB2b2lkIF9fZXZpY3Rfb2xkX2J1ZmZlcnMoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywgdW5z aWduZWQgbG9uZyBhZ2VfaHopCitzdHJ1Y3QgZXZpY3RfcGFyYW1zIHsKKwlnZnBfdCBnZnA7CisJ dW5zaWduZWQgbG9uZyBhZ2VfaHo7CisKKwkvKgorCSAqIFRoaXMgZ2V0cyB1cGRhdGVkIHdpdGgg dGhlIGxhcmdlc3QgbGFzdF9hY2Nlc3NlZCAoaWUuIG1vc3QKKwkgKiByZWNlbnRseSB1c2VkKSBv ZiB0aGUgZXZpY3RlZCBidWZmZXJzLiAgSXQgd2lsbCBub3QgYmUgcmVpbml0aWFsaXNlZAorCSAq IGJ5IF9fZXZpY3RfbWFueSgpLCBzbyB5b3UgY2FuIHVzZSBpdCBhY3Jvc3MgbXVsdGlwbGUgaW52 b2NhdGlvbnMuCisJICovCisJdW5zaWduZWQgbG9uZyBsYXN0X2FjY2Vzc2VkOworfTsKKworLyoK KyAqIFdlIG1heSBub3QgYmUgYWJsZSB0byBldmljdCB0aGlzIGJ1ZmZlciBpZiBJTyBwZW5kaW5n IG9yIHRoZSBjbGllbnQKKyAqIGlzIHN0aWxsIHVzaW5nIGl0LgorICoKKyAqIEFuZCBpZiBHRlBf Tk9GUyBpcyB1c2VkLCB3ZSBtdXN0IG5vdCBkbyBhbnkgSS9PIGJlY2F1c2Ugd2UgaG9sZAorICog ZG1fYnVmaW9fY2xpZW50c19sb2NrIGFuZCB3ZSB3b3VsZCByaXNrIGRlYWRsb2NrIGlmIHRoZSBJ L08gZ2V0cworICogcmVyb3V0ZWQgdG8gZGlmZmVyZW50IGJ1ZmlvIGNsaWVudC4KKyAqLworc3Rh dGljIGVudW0gZXZpY3RfcmVzdWx0IHNlbGVjdF9mb3JfZXZpY3Qoc3RydWN0IGRtX2J1ZmZlciAq Yiwgdm9pZCAqY29udGV4dCkKK3sKKwlzdHJ1Y3QgZXZpY3RfcGFyYW1zICpwYXJhbXMgPSBjb250 ZXh0OworCisJaWYgKCEocGFyYW1zLT5nZnAgJiBfX0dGUF9GUykgfHwKKwkgICAgKHN0YXRpY19i cmFuY2hfdW5saWtlbHkoJm5vX3NsZWVwX2VuYWJsZWQpICYmIGItPmMtPm5vX3NsZWVwKSkgewor CQlpZiAodGVzdF9iaXRfYWNxdWlyZShCX1JFQURJTkcsICZiLT5zdGF0ZSkgfHwKKwkJICAgIHRl c3RfYml0KEJfV1JJVElORywgJmItPnN0YXRlKSB8fAorCQkgICAgdGVzdF9iaXQoQl9ESVJUWSwg JmItPnN0YXRlKSkKKwkJCXJldHVybiBFUl9ET05UX0VWSUNUOworCX0KKworCXJldHVybiBvbGRl cl90aGFuKGIsIHBhcmFtcy0+YWdlX2h6KSA/IEVSX0VWSUNUIDogRVJfU1RPUDsKK30KKworc3Rh dGljIHVuc2lnbmVkIGxvbmcgX19ldmljdF9tYW55KHN0cnVjdCBkbV9idWZpb19jbGllbnQgKmMs CisJCQkJICBzdHJ1Y3QgZXZpY3RfcGFyYW1zICpwYXJhbXMsCisJCQkJICBpbnQgbGlzdF9tb2Rl LCB1bnNpZ25lZCBsb25nIG1heF9jb3VudCkKK3sKKwl1bnNpZ25lZCBsb25nIGNvdW50OworCXVu c2lnbmVkIGxvbmcgbGFzdF9hY2Nlc3NlZDsKKwlzdHJ1Y3QgZG1fYnVmZmVyICpiOworCisJZm9y IChjb3VudCA9IDA7IGNvdW50IDwgbWF4X2NvdW50OyBjb3VudCsrKSB7CisJCWIgPSBjYWNoZV9l dmljdCgmYy0+Y2FjaGUsIGxpc3RfbW9kZSwgc2VsZWN0X2Zvcl9ldmljdCwgcGFyYW1zKTsKKwkJ aWYgKCFiKQorCQkJYnJlYWs7CisKKwkJbGFzdF9hY2Nlc3NlZCA9IFJFQURfT05DRShiLT5sYXN0 X2FjY2Vzc2VkKTsKKwkJaWYgKHRpbWVfYWZ0ZXJfZXEocGFyYW1zLT5sYXN0X2FjY2Vzc2VkLCBs YXN0X2FjY2Vzc2VkKSkKKwkJCXBhcmFtcy0+bGFzdF9hY2Nlc3NlZCA9IGxhc3RfYWNjZXNzZWQ7 CisKKwkJX19tYWtlX2J1ZmZlcl9jbGVhbihiKTsKKwkJX19mcmVlX2J1ZmZlcl93YWtlKGIpOwor CisJCWNvbmRfcmVzY2hlZCgpOworCX0KKworCXJldHVybiBjb3VudDsKK30KKworc3RhdGljIHZv aWQgZXZpY3Rfb2xkX2J1ZmZlcnMoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYywgdW5zaWduZWQg bG9uZyBhZ2VfaHopCiB7Ci0Jc3RydWN0IGRtX2J1ZmZlciAqYiwgKnRtcDsKLQl1bnNpZ25lZCBs b25nIHJldGFpbl90YXJnZXQgPSBnZXRfcmV0YWluX2J1ZmZlcnMoYyk7CisJc3RydWN0IGV2aWN0 X3BhcmFtcyBwYXJhbXMgPSB7LmdmcCA9IDAsIC5hZ2VfaHogPSBhZ2VfaHosIC5sYXN0X2FjY2Vz c2VkID0gMH07CisJdW5zaWduZWQgbG9uZyByZXRhaW4gPSBnZXRfcmV0YWluX2J1ZmZlcnMoYyk7 CiAJdW5zaWduZWQgbG9uZyBjb3VudDsKIAlMSVNUX0hFQUQod3JpdGVfbGlzdCk7CiAKQEAgLTI2 MDAsOTEgKzI1OTIsMTMgQEAgc3RhdGljIHZvaWQgX19ldmljdF9vbGRfYnVmZmVycyhzdHJ1Y3Qg ZG1fYnVmaW9fY2xpZW50ICpjLCB1bnNpZ25lZCBsb25nIGFnZV9oeikKIAkJZG1fYnVmaW9fbG9j ayhjKTsKIAl9CiAKLQljb3VudCA9IGMtPm5fYnVmZmVyc1tMSVNUX0NMRUFOXSArIGMtPm5fYnVm ZmVyc1tMSVNUX0RJUlRZXTsKLQlsaXN0X2Zvcl9lYWNoX2VudHJ5X3NhZmVfcmV2ZXJzZShiLCB0 bXAsICZjLT5scnVbTElTVF9DTEVBTl0sIGxydV9saXN0KSB7Ci0JCWlmIChjb3VudCA8PSByZXRh aW5fdGFyZ2V0KQotCQkJYnJlYWs7Ci0KLQkJaWYgKCFvbGRlcl90aGFuKGIsIGFnZV9oeikpCi0J CQlicmVhazsKLQotCQlpZiAoX190cnlfZXZpY3RfYnVmZmVyKGIsIDApKQotCQkJY291bnQtLTsK LQotCQljb25kX3Jlc2NoZWQoKTsKLQl9CisJY291bnQgPSBjYWNoZV90b3RhbCgmYy0+Y2FjaGUp OworCWlmIChjb3VudCA+IHJldGFpbikKKwkJX19ldmljdF9tYW55KGMsICZwYXJhbXMsIExJU1Rf Q0xFQU4sIGNvdW50IC0gcmV0YWluKTsKIAogCWRtX2J1ZmlvX3VubG9jayhjKTsKIH0KIAotc3Rh dGljIHZvaWQgZG9fZ2xvYmFsX2NsZWFudXAoc3RydWN0IHdvcmtfc3RydWN0ICp3KQotewotCXN0 cnVjdCBkbV9idWZpb19jbGllbnQgKmxvY2tlZF9jbGllbnQgPSBOVUxMOwotCXN0cnVjdCBkbV9i dWZpb19jbGllbnQgKmN1cnJlbnRfY2xpZW50OwotCXN0cnVjdCBkbV9idWZmZXIgKmI7Ci0JdW5z aWduZWQgaW50IHNwaW5sb2NrX2hvbGRfY291bnQ7Ci0JdW5zaWduZWQgbG9uZyB0aHJlc2hvbGQg PSBkbV9idWZpb19jYWNoZV9zaXplIC0KLQkJZG1fYnVmaW9fY2FjaGVfc2l6ZSAvIERNX0JVRklP X0xPV19XQVRFUk1BUktfUkFUSU87Ci0JdW5zaWduZWQgbG9uZyBsb29wcyA9IGdsb2JhbF9udW0g KiAyOwotCi0JbXV0ZXhfbG9jaygmZG1fYnVmaW9fY2xpZW50c19sb2NrKTsKLQotCXdoaWxlICgx KSB7Ci0JCWNvbmRfcmVzY2hlZCgpOwotCi0JCXNwaW5fbG9jaygmZ2xvYmFsX3NwaW5sb2NrKTsK LQkJaWYgKHVubGlrZWx5KGRtX2J1ZmlvX2N1cnJlbnRfYWxsb2NhdGVkIDw9IHRocmVzaG9sZCkp Ci0JCQlicmVhazsKLQotCQlzcGlubG9ja19ob2xkX2NvdW50ID0gMDsKLWdldF9uZXh0OgotCQlp ZiAoIWxvb3BzLS0pCi0JCQlicmVhazsKLQkJaWYgKHVubGlrZWx5KGxpc3RfZW1wdHkoJmdsb2Jh bF9xdWV1ZSkpKQotCQkJYnJlYWs7Ci0JCWIgPSBsaXN0X2VudHJ5KGdsb2JhbF9xdWV1ZS5wcmV2 LCBzdHJ1Y3QgZG1fYnVmZmVyLCBnbG9iYWxfbGlzdCk7Ci0KLQkJaWYgKGItPmFjY2Vzc2VkKSB7 Ci0JCQliLT5hY2Nlc3NlZCA9IDA7Ci0JCQlsaXN0X21vdmUoJmItPmdsb2JhbF9saXN0LCAmZ2xv YmFsX3F1ZXVlKTsKLQkJCWlmIChsaWtlbHkoKytzcGlubG9ja19ob2xkX2NvdW50IDwgMTYpKQot CQkJCWdvdG8gZ2V0X25leHQ7Ci0JCQlzcGluX3VubG9jaygmZ2xvYmFsX3NwaW5sb2NrKTsKLQkJ CWNvbnRpbnVlOwotCQl9Ci0KLQkJY3VycmVudF9jbGllbnQgPSBiLT5jOwotCQlpZiAodW5saWtl bHkoY3VycmVudF9jbGllbnQgIT0gbG9ja2VkX2NsaWVudCkpIHsKLQkJCWlmIChsb2NrZWRfY2xp ZW50KQotCQkJCWRtX2J1ZmlvX3VubG9jayhsb2NrZWRfY2xpZW50KTsKLQotCQkJaWYgKCFkbV9i dWZpb190cnlsb2NrKGN1cnJlbnRfY2xpZW50KSkgewotCQkJCXNwaW5fdW5sb2NrKCZnbG9iYWxf c3BpbmxvY2spOwotCQkJCWRtX2J1ZmlvX2xvY2soY3VycmVudF9jbGllbnQpOwotCQkJCWxvY2tl ZF9jbGllbnQgPSBjdXJyZW50X2NsaWVudDsKLQkJCQljb250aW51ZTsKLQkJCX0KLQotCQkJbG9j a2VkX2NsaWVudCA9IGN1cnJlbnRfY2xpZW50OwotCQl9Ci0KLQkJc3Bpbl91bmxvY2soJmdsb2Jh bF9zcGlubG9jayk7Ci0KLQkJaWYgKHVubGlrZWx5KCFfX3RyeV9ldmljdF9idWZmZXIoYiwgR0ZQ X0tFUk5FTCkpKSB7Ci0JCQlzcGluX2xvY2soJmdsb2JhbF9zcGlubG9jayk7Ci0JCQlsaXN0X21v dmUoJmItPmdsb2JhbF9saXN0LCAmZ2xvYmFsX3F1ZXVlKTsKLQkJCXNwaW5fdW5sb2NrKCZnbG9i YWxfc3BpbmxvY2spOwotCQl9Ci0JfQotCi0Jc3Bpbl91bmxvY2soJmdsb2JhbF9zcGlubG9jayk7 Ci0KLQlpZiAobG9ja2VkX2NsaWVudCkKLQkJZG1fYnVmaW9fdW5sb2NrKGxvY2tlZF9jbGllbnQp OwotCi0JbXV0ZXhfdW5sb2NrKCZkbV9idWZpb19jbGllbnRzX2xvY2spOwotfQotCiBzdGF0aWMg dm9pZCBjbGVhbnVwX29sZF9idWZmZXJzKHZvaWQpCiB7CiAJdW5zaWduZWQgbG9uZyBtYXhfYWdl X2h6ID0gZ2V0X21heF9hZ2VfaHooKTsKQEAgLTI2OTUsNyArMjYwOSw3IEBAIHN0YXRpYyB2b2lk IGNsZWFudXBfb2xkX2J1ZmZlcnModm9pZCkKIAlfX2NhY2hlX3NpemVfcmVmcmVzaCgpOwogCiAJ bGlzdF9mb3JfZWFjaF9lbnRyeShjLCAmZG1fYnVmaW9fYWxsX2NsaWVudHMsIGNsaWVudF9saXN0 KQotCQlfX2V2aWN0X29sZF9idWZmZXJzKGMsIG1heF9hZ2VfaHopOworCQlldmljdF9vbGRfYnVm ZmVycyhjLCBtYXhfYWdlX2h6KTsKIAogCW11dGV4X3VubG9jaygmZG1fYnVmaW9fY2xpZW50c19s b2NrKTsKIH0KQEAgLTI3MDgsNiArMjYyMiwxMDcgQEAgc3RhdGljIHZvaWQgd29ya19mbihzdHJ1 Y3Qgd29ya19zdHJ1Y3QgKncpCiAJCQkgICBETV9CVUZJT19XT1JLX1RJTUVSX1NFQ1MgKiBIWik7 CiB9CiAKKy8qLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0qLworCisvKgorICogR2xvYmFsIGNsZWFudXAgdHJpZXMgdG8gZXZpY3Qg dGhlIG9sZGVzdCBidWZmZXJzIGZyb20gYWNyb3NzIF9hbGxfCisgKiB0aGUgY2xpZW50cy4gIEl0 IGRvZXMgdGhpcyBieSByZXBlYXRlZGx5IGV2aWN0aW5nIGEgZmV3IGJ1ZmZlcnMgZnJvbQorICog dGhlIGNsaWVudCB0aGF0IGhvbGRzIHRoZSBvbGRlc3QgYnVmZmVyLiAgSXQncyBhcHByb3hpbWF0 ZSwgYnV0IGhvcGVmdWxseQorICogZ29vZCBlbm91Z2guCisgKi8KK3N0YXRpYyBzdHJ1Y3QgZG1f YnVmaW9fY2xpZW50ICpfX3BvcF9jbGllbnQodm9pZCkKK3sKKwlzdHJ1Y3QgbGlzdF9oZWFkICpo OworCisJaWYgKGxpc3RfZW1wdHkoJmRtX2J1ZmlvX2FsbF9jbGllbnRzKSkKKwkJcmV0dXJuIE5V TEw7CisKKwloID0gZG1fYnVmaW9fYWxsX2NsaWVudHMubmV4dDsKKwlsaXN0X2RlbChoKTsKKwly ZXR1cm4gY29udGFpbmVyX29mKGgsIHN0cnVjdCBkbV9idWZpb19jbGllbnQsIGNsaWVudF9saXN0 KTsKK30KKworLyoKKyAqIEluc2VydHMgdGhlIGNsaWVudCBpbiB0aGUgZ2xvYmFsIGNsaWVudCBs aXN0IGJhc2VkIG9uIGl0cworICogJ29sZGVzdF9idWZmZXInIGZpZWxkLgorICovCitzdGF0aWMg dm9pZCBfX2luc2VydF9jbGllbnQoc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqbmV3X2NsaWVudCkK K3sKKwlzdHJ1Y3QgZG1fYnVmaW9fY2xpZW50ICpjOworCXN0cnVjdCBsaXN0X2hlYWQgKmggPSBk bV9idWZpb19hbGxfY2xpZW50cy5uZXh0OworCisJd2hpbGUgKGggIT0gJmRtX2J1ZmlvX2FsbF9j bGllbnRzKSB7CisJCWMgPSBjb250YWluZXJfb2YoaCwgc3RydWN0IGRtX2J1ZmlvX2NsaWVudCwg Y2xpZW50X2xpc3QpOworCQlpZiAodGltZV9hZnRlcl9lcShjLT5vbGRlc3RfYnVmZmVyLCBuZXdf Y2xpZW50LT5vbGRlc3RfYnVmZmVyKSkKKwkJCWJyZWFrOworCQloID0gaC0+bmV4dDsKKwl9CisK KwlsaXN0X2FkZF90YWlsKCZuZXdfY2xpZW50LT5jbGllbnRfbGlzdCwgaCk7Cit9CisKK3N0YXRp YyB1bnNpZ25lZCBsb25nIF9fZXZpY3RfYV9mZXcodW5zaWduZWQgbG9uZyBucl9idWZmZXJzKQor eworCXVuc2lnbmVkIGxvbmcgY291bnQ7CisJc3RydWN0IGRtX2J1ZmlvX2NsaWVudCAqYzsKKwlz dHJ1Y3QgZXZpY3RfcGFyYW1zIHBhcmFtcyA9IHsKKwkJLmdmcCA9IEdGUF9LRVJORUwsCisJCS5h Z2VfaHogPSAwLAorCQkvKiBzZXQgdG8gamlmZmllcyBpbiBjYXNlIHRoZXJlIGFyZSBubyBidWZm ZXJzIGluIHRoaXMgY2xpZW50ICovCisJCS5sYXN0X2FjY2Vzc2VkID0gamlmZmllcworCX07CisK KwljID0gX19wb3BfY2xpZW50KCk7CisJaWYgKCFjKQorCQlyZXR1cm4gMDsKKworCWRtX2J1Zmlv X2xvY2soYyk7CisJY291bnQgPSBfX2V2aWN0X21hbnkoYywgJnBhcmFtcywgTElTVF9DTEVBTiwg bnJfYnVmZmVycyk7CisJZG1fYnVmaW9fdW5sb2NrKGMpOworCisJaWYgKGNvdW50KQorCQljLT5v bGRlc3RfYnVmZmVyID0gcGFyYW1zLmxhc3RfYWNjZXNzZWQ7CisJX19pbnNlcnRfY2xpZW50KGMp OworCisJcmV0dXJuIGNvdW50OworfQorCitzdGF0aWMgdm9pZCBjaGVja193YXRlcm1hcmtzKHZv aWQpCit7CisJTElTVF9IRUFEKHdyaXRlX2xpc3QpOworCXN0cnVjdCBkbV9idWZpb19jbGllbnQg KmM7CisKKwltdXRleF9sb2NrKCZkbV9idWZpb19jbGllbnRzX2xvY2spOworCWxpc3RfZm9yX2Vh Y2hfZW50cnkoYywgJmRtX2J1ZmlvX2FsbF9jbGllbnRzLCBjbGllbnRfbGlzdCkgeworCQlkbV9i dWZpb19sb2NrKGMpOworCQlfX2NoZWNrX3dhdGVybWFyayhjLCAmd3JpdGVfbGlzdCk7CisJCWRt X2J1ZmlvX3VubG9jayhjKTsKKwl9CisJbXV0ZXhfdW5sb2NrKCZkbV9idWZpb19jbGllbnRzX2xv Y2spOworCisJX19mbHVzaF93cml0ZV9saXN0KCZ3cml0ZV9saXN0KTsKK30KKworc3RhdGljIHZv aWQgZXZpY3Rfb2xkKHZvaWQpCit7CisJdW5zaWduZWQgbG9uZyB0aHJlc2hvbGQgPSBkbV9idWZp b19jYWNoZV9zaXplIC0KKwkJZG1fYnVmaW9fY2FjaGVfc2l6ZSAvIERNX0JVRklPX0xPV19XQVRF Uk1BUktfUkFUSU87CisKKwltdXRleF9sb2NrKCZkbV9idWZpb19jbGllbnRzX2xvY2spOworCXdo aWxlIChkbV9idWZpb19jdXJyZW50X2FsbG9jYXRlZCA+IHRocmVzaG9sZCkgeworCQlpZiAoIV9f ZXZpY3RfYV9mZXcoNjQpKQorCQkJYnJlYWs7CisJCWNvbmRfcmVzY2hlZCgpOworCX0KKwltdXRl eF91bmxvY2soJmRtX2J1ZmlvX2NsaWVudHNfbG9jayk7Cit9CisKK3N0YXRpYyB2b2lkIGRvX2ds b2JhbF9jbGVhbnVwKHN0cnVjdCB3b3JrX3N0cnVjdCAqdykKK3sKKwljaGVja193YXRlcm1hcmtz KCk7CisJZXZpY3Rfb2xkKCk7Cit9CisKIC8qCiAgKi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiAgKiBNb2R1bGUgc2V0dXAKLS0g CjIuNDAuMAoKLS0KZG0tZGV2ZWwgbWFpbGluZyBsaXN0CmRtLWRldmVsQHJlZGhhdC5jb20KaHR0 cHM6Ly9saXN0bWFuLnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9kbS1kZXZlbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE085C76195 for ; Mon, 27 Mar 2023 20:14:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232354AbjC0UOO (ORCPT ); Mon, 27 Mar 2023 16:14:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231929AbjC0UOL (ORCPT ); Mon, 27 Mar 2023 16:14:11 -0400 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77C791BF6 for ; Mon, 27 Mar 2023 13:13:19 -0700 (PDT) Received: by mail-qt1-f169.google.com with SMTP id t19so9790941qta.12 for ; Mon, 27 Mar 2023 13:13:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679947998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KZEmP2Lma0r81Bos8w4U89OEq/V+OweIm/gue2zysiY=; b=zatRm5vcsBOJvKzHkYbhsp4LXTmtC97qp1dB94djR7hu2Zyq8pvmP8wWZT5f//RZhr hRXoPY80mMFPz8DZjlu8QhPok+2Gd9eUkw3WxSfJWgEi3AuKe9nbzLnTLFuQn01Tjxug jUWS1CoHapGBWEDmfJPJqVIiQg47mCg5/ANjIqLBsnPESr5KuhRTE9ivq6NFa6MPgN4P 20WnCSN1fhTSMwvsrCJDL9gvRa6G6N6RkQ78KKduIip6+eVomTNRI+H8kB/jy9Fjuwkt 5McdXjAo7Fi5dGHXwctBIL9LIQokfZF3mzlyBPFgxUjpHz1srjr2Q+psm+XkFRI7XN7A au+w== X-Gm-Message-State: AO0yUKUBJWgwAtAkMsa6p4tDFNEsHv7tZMHr7ty1na2UsScN1CyoxdIH 5Bu/DtRQy+TCNH+KOtDoXjBV X-Google-Smtp-Source: AK7set/4u5wtNp1+7i337s6/269TkCLiJgXwRPcmr5VJh9wfIB7rZepSb8dvmBloMyznyDlN/1yP2Q== X-Received: by 2002:a05:622a:1109:b0:3e3:82c4:db44 with SMTP id e9-20020a05622a110900b003e382c4db44mr23802990qty.52.1679947998246; Mon, 27 Mar 2023 13:13:18 -0700 (PDT) Received: from localhost (pool-68-160-166-30.bstnma.fios.verizon.net. [68.160.166.30]) by smtp.gmail.com with ESMTPSA id u23-20020a37ab17000000b0071f0d0aaef7sm13735581qke.80.2023.03.27.13.13.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Mar 2023 13:13:17 -0700 (PDT) From: Mike Snitzer To: dm-devel@redhat.com Cc: linux-block@vger.kernel.org, axboe@kernel.dk, ejt@redhat.com, mpatocka@redhat.com, heinzm@redhat.com, nhuck@google.com, ebiggers@kernel.org, keescook@chromium.org, luomeng12@huawei.com, Mike Snitzer Subject: [dm-6.4 PATCH v3 07/20] dm bufio: improve concurrent IO performance Date: Mon, 27 Mar 2023 16:11:30 -0400 Message-Id: <20230327201143.51026-8-snitzer@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230327201143.51026-1-snitzer@kernel.org> References: <20230327201143.51026-1-snitzer@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Joe Thornber When multiple threads perform IO to a thin device, the underlying dm_bufio object can become a bottleneck; slowing down access to btree nodes that store the thin metadata. Prior to this commit, each bufio instance had a single mutex that was taken for every bufio operation. This commit concentrates on improving the common case where: a user of dm_bufio wishes to access, but not modify, a buffer which is already within the dm_bufio cache. Implementation:: The code has been refactored; pulling out an 'lru' abstraction and a 'buffer cache' abstraction (see 2 previous commits). This commit updates higher level bufio code (that performs allocation of buffers, IO and eviction/cache sizing) to leverage both abstractions. It also deals with the delicate locking requirements of both abstractions to provide finer grained locking. The result is significantly better concurrent IO performance. Before this commit, bufio has a global lru list it used to evict the oldest, clean buffers from _all_ clients. With the new locking we don’t want different ways to access the same buffer, so instead do_global_cleanup() loops around the clients asking them to free buffers older than a certain time. This commit also converts many old BUG_ONs to WARN_ON_ONCE, see the lru_evict and cache_evict code in particular. They will return ER_DONT_EVICT if a given buffer somehow meets the invariants that should _never_ happen. [Aside from revising this commit's header and fixing coding style and whitespace nits: this switching to WARN_ON_ONCE is Mike Snitzer's lone contribution to this commit] Testing:: Some of the low level functions have been unit tested using dm-unit: https://github.com/jthornber/dm-unit/blob/main/src/tests/bufio.rs Higher level concurrency and IO is tested via a test only target found here: https://github.com/jthornber/linux/blob/2023-03-24-thin-concurrency-9/drivers/md/dm-bufio-test.c The associated userland side of these tests is here: https://github.com/jthornber/dmtest-python/blob/main/src/dmtest/bufio/bufio_tests.py In addition the full dmtest suite of tests (dm-thin, dm-cache, etc) has been run (~450 tests). Performance:: Most bufio operations have unchanged performance. But if multiple threads are attempting to get buffers concurrently, and these buffers are already in the cache then there's a big speed up. Eg, one test has 16 'hotspot' threads simulating btree lookups while another thread dirties the whole device. In this case the hotspot threads acquire the buffers about 25 times faster. Signed-off-by: Joe Thornber Signed-off-by: Mike Snitzer --- drivers/md/dm-bufio.c | 959 +++++++++++++++++++++--------------------- 1 file changed, 487 insertions(+), 472 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 9daff9b77cee..1e000ec73bd6 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -321,37 +321,42 @@ enum data_mode { }; struct dm_buffer { + /* protected by the locks in dm_buffer_cache */ struct rb_node node; - struct list_head lru_list; - struct list_head global_list; + /* immutable, so don't need protecting */ sector_t block; void *data; unsigned char data_mode; /* DATA_MODE_* */ - unsigned int accessed; - unsigned int __hold_count; + /* + * These two fields are used in isolation, so do not need + * a surrounding lock. + */ + atomic_t hold_count; unsigned long last_accessed; + /* + * Everything else is protected by the mutex in + * dm_bufio_client + */ + unsigned long state; + struct lru_entry lru; unsigned char list_mode; /* LIST_* */ blk_status_t read_error; blk_status_t write_error; - unsigned long state; unsigned int dirty_start; unsigned int dirty_end; unsigned int write_start; unsigned int write_end; - struct dm_bufio_client *c; struct list_head write_list; + struct dm_bufio_client *c; void (*end_io)(struct dm_buffer *b, blk_status_t bs); #ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING #define MAX_STACK 10 unsigned int stack_len; unsigned long stack_entries[MAX_STACK]; #endif - /* Temp members to allow dm_buffer_cache code to compile */ - atomic_t hold_count; - struct lru_entry lru; }; /*--------------------------------------------------------------*/ @@ -814,7 +819,7 @@ static void cache_remove_range(struct dm_buffer_cache *bc, /* * Linking of buffers: - * All buffers are linked to buffer_tree with their node field. + * All buffers are linked to buffer_cache with their node field. * * Clean buffers that are not being written (B_WRITING not set) * are linked to lru[LIST_CLEAN] with their lru_list field. @@ -832,9 +837,6 @@ struct dm_bufio_client { spinlock_t spinlock; bool no_sleep; - struct list_head lru[LIST_SIZE]; - unsigned long n_buffers[LIST_SIZE]; - struct block_device *bdev; unsigned int block_size; s8 sectors_per_block_bits; @@ -849,7 +851,7 @@ struct dm_bufio_client { unsigned int minimum_buffers; - struct rb_root buffer_tree; + struct dm_buffer_cache cache; wait_queue_head_t free_buffer_wait; sector_t start; @@ -861,6 +863,11 @@ struct dm_bufio_client { struct shrinker shrinker; struct work_struct shrink_work; atomic_long_t need_shrink; + + /* + * Used by global_cleanup to sort the clients list. + */ + unsigned long oldest_buffer; }; static DEFINE_STATIC_KEY_FALSE(no_sleep_enabled); @@ -877,14 +884,6 @@ static void dm_bufio_lock(struct dm_bufio_client *c) mutex_lock_nested(&c->lock, dm_bufio_in_request()); } -static int dm_bufio_trylock(struct dm_bufio_client *c) -{ - if (static_branch_unlikely(&no_sleep_enabled) && c->no_sleep) - return spin_trylock_bh(&c->spinlock); - else - return mutex_trylock(&c->lock); -} - static void dm_bufio_unlock(struct dm_bufio_client *c) { if (static_branch_unlikely(&no_sleep_enabled) && c->no_sleep) @@ -913,10 +912,6 @@ static unsigned long dm_bufio_cache_size_latch; static DEFINE_SPINLOCK(global_spinlock); -static LIST_HEAD(global_queue); - -static unsigned long global_num; - /* * Buffers are freed after this timeout */ @@ -958,78 +953,6 @@ static void buffer_record_stack(struct dm_buffer *b) } #endif -/* - *---------------------------------------------------------------- - * A red/black tree acts as an index for all the buffers. - *---------------------------------------------------------------- - */ -static struct dm_buffer *__find(struct dm_bufio_client *c, sector_t block) -{ - struct rb_node *n = c->buffer_tree.rb_node; - struct dm_buffer *b; - - while (n) { - b = container_of(n, struct dm_buffer, node); - - if (b->block == block) - return b; - - n = block < b->block ? n->rb_left : n->rb_right; - } - - return NULL; -} - -static struct dm_buffer *__find_next_old(struct dm_bufio_client *c, sector_t block) -{ - struct rb_node *n = c->buffer_tree.rb_node; - struct dm_buffer *b; - struct dm_buffer *best = NULL; - - while (n) { - b = container_of(n, struct dm_buffer, node); - - if (b->block == block) - return b; - - if (block <= b->block) { - n = n->rb_left; - best = b; - } else { - n = n->rb_right; - } - } - - return best; -} - -static void __insert(struct dm_bufio_client *c, struct dm_buffer *b) -{ - struct rb_node **new = &c->buffer_tree.rb_node, *parent = NULL; - struct dm_buffer *found; - - while (*new) { - found = container_of(*new, struct dm_buffer, node); - - if (found->block == b->block) { - BUG_ON(found != b); - return; - } - - parent = *new; - new = b->block < found->block ? - &found->node.rb_left : &found->node.rb_right; - } - - rb_link_node(&b->node, parent, new); - rb_insert_color(&b->node, &c->buffer_tree); -} - -static void __remove(struct dm_bufio_client *c, struct dm_buffer *b) -{ - rb_erase(&b->node, &c->buffer_tree); -} - /*----------------------------------------------------------------*/ static void adjust_total_allocated(struct dm_buffer *b, bool unlink) @@ -1057,16 +980,9 @@ static void adjust_total_allocated(struct dm_buffer *b, bool unlink) if (dm_bufio_current_allocated > dm_bufio_peak_allocated) dm_bufio_peak_allocated = dm_bufio_current_allocated; - b->accessed = 1; - if (!unlink) { - list_add(&b->global_list, &global_queue); - global_num++; if (dm_bufio_current_allocated > dm_bufio_cache_size) queue_work(dm_bufio_wq, &dm_bufio_replacement_work); - } else { - list_del(&b->global_list); - global_num--; } spin_unlock(&global_spinlock); @@ -1196,6 +1112,7 @@ static struct dm_buffer *alloc_buffer(struct dm_bufio_client *c, gfp_t gfp_mask) kmem_cache_free(c->slab_buffer, b); return NULL; } + adjust_total_allocated(b, false); #ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING b->stack_len = 0; @@ -1210,61 +1127,11 @@ static void free_buffer(struct dm_buffer *b) { struct dm_bufio_client *c = b->c; + adjust_total_allocated(b, true); free_buffer_data(c, b->data, b->data_mode); kmem_cache_free(c->slab_buffer, b); } -/* - * Link buffer to the buffer tree and clean or dirty queue. - */ -static void __link_buffer(struct dm_buffer *b, sector_t block, int dirty) -{ - struct dm_bufio_client *c = b->c; - - c->n_buffers[dirty]++; - b->block = block; - b->list_mode = dirty; - list_add(&b->lru_list, &c->lru[dirty]); - __insert(b->c, b); - b->last_accessed = jiffies; - - adjust_total_allocated(b, false); -} - -/* - * Unlink buffer from the buffer tree and dirty or clean queue. - */ -static void __unlink_buffer(struct dm_buffer *b) -{ - struct dm_bufio_client *c = b->c; - - BUG_ON(!c->n_buffers[b->list_mode]); - - c->n_buffers[b->list_mode]--; - __remove(b->c, b); - list_del(&b->lru_list); - - adjust_total_allocated(b, true); -} - -/* - * Place the buffer to the head of dirty or clean LRU queue. - */ -static void __relink_lru(struct dm_buffer *b, int dirty) -{ - struct dm_bufio_client *c = b->c; - - b->accessed = 1; - - BUG_ON(!c->n_buffers[b->list_mode]); - - c->n_buffers[b->list_mode]--; - c->n_buffers[dirty]++; - b->list_mode = dirty; - list_move(&b->lru_list, &c->lru[dirty]); - b->last_accessed = jiffies; -} - /* *-------------------------------------------------------------------------- * Submit I/O on the buffer. @@ -1504,7 +1371,7 @@ static void __flush_write_list(struct list_head *write_list) */ static void __make_buffer_clean(struct dm_buffer *b) { - BUG_ON(b->__hold_count); + BUG_ON(atomic_read(&b->hold_count)); /* smp_load_acquire() pairs with read_endio()'s smp_mb__before_atomic() */ if (!smp_load_acquire(&b->state)) /* fast case */ @@ -1515,6 +1382,36 @@ static void __make_buffer_clean(struct dm_buffer *b) wait_on_bit_io(&b->state, B_WRITING, TASK_UNINTERRUPTIBLE); } +static enum evict_result is_clean(struct dm_buffer *b, void *context) +{ + struct dm_bufio_client *c = context; + + /* These should never happen */ + if (WARN_ON_ONCE(test_bit(B_WRITING, &b->state))) + return ER_DONT_EVICT; + if (WARN_ON_ONCE(test_bit(B_DIRTY, &b->state))) + return ER_DONT_EVICT; + if (WARN_ON_ONCE(b->list_mode != LIST_CLEAN)) + return ER_DONT_EVICT; + + if (static_branch_unlikely(&no_sleep_enabled) && c->no_sleep && + unlikely(test_bit(B_READING, &b->state))) + return ER_DONT_EVICT; + + return ER_EVICT; +} + +static enum evict_result is_dirty(struct dm_buffer *b, void *context) +{ + /* These should never happen */ + if (WARN_ON_ONCE(test_bit(B_READING, &b->state))) + return ER_DONT_EVICT; + if (WARN_ON_ONCE(b->list_mode != LIST_DIRTY)) + return ER_DONT_EVICT; + + return ER_EVICT; +} + /* * Find some buffer that is not held by anybody, clean it, unlink it and * return it. @@ -1523,34 +1420,20 @@ static struct dm_buffer *__get_unclaimed_buffer(struct dm_bufio_client *c) { struct dm_buffer *b; - list_for_each_entry_reverse(b, &c->lru[LIST_CLEAN], lru_list) { - BUG_ON(test_bit(B_WRITING, &b->state)); - BUG_ON(test_bit(B_DIRTY, &b->state)); - - if (static_branch_unlikely(&no_sleep_enabled) && c->no_sleep && - unlikely(test_bit_acquire(B_READING, &b->state))) - continue; - - if (!b->__hold_count) { - __make_buffer_clean(b); - __unlink_buffer(b); - return b; - } - cond_resched(); + b = cache_evict(&c->cache, LIST_CLEAN, is_clean, c); + if (b) { + /* this also waits for pending reads */ + __make_buffer_clean(b); + return b; } if (static_branch_unlikely(&no_sleep_enabled) && c->no_sleep) return NULL; - list_for_each_entry_reverse(b, &c->lru[LIST_DIRTY], lru_list) { - BUG_ON(test_bit(B_READING, &b->state)); - - if (!b->__hold_count) { - __make_buffer_clean(b); - __unlink_buffer(b); - return b; - } - cond_resched(); + b = cache_evict(&c->cache, LIST_DIRTY, is_dirty, NULL); + if (b) { + __make_buffer_clean(b); + return b; } return NULL; @@ -1571,7 +1454,12 @@ static void __wait_for_free_buffer(struct dm_bufio_client *c) set_current_state(TASK_UNINTERRUPTIBLE); dm_bufio_unlock(c); - io_schedule(); + /* + * It's possible to miss a wake up event since we don't always + * hold c->lock when wake_up is called. So we have a timeout here, + * just in case. + */ + io_schedule_timeout(5 * HZ); remove_wait_queue(&c->free_buffer_wait, &wait); @@ -1629,9 +1517,8 @@ static struct dm_buffer *__alloc_buffer_wait_no_callback(struct dm_bufio_client } if (!list_empty(&c->reserved_buffers)) { - b = list_entry(c->reserved_buffers.next, - struct dm_buffer, lru_list); - list_del(&b->lru_list); + b = list_to_buffer(c->reserved_buffers.next); + list_del(&b->lru.list); c->need_reserved_buffers++; return b; @@ -1665,36 +1552,56 @@ static void __free_buffer_wake(struct dm_buffer *b) { struct dm_bufio_client *c = b->c; + b->block = -1; if (!c->need_reserved_buffers) free_buffer(b); else { - list_add(&b->lru_list, &c->reserved_buffers); + list_add(&b->lru.list, &c->reserved_buffers); c->need_reserved_buffers--; } wake_up(&c->free_buffer_wait); } +static enum evict_result cleaned(struct dm_buffer *b, void *context) +{ + if (WARN_ON_ONCE(test_bit(B_READING, &b->state))) + return ER_DONT_EVICT; /* should never happen */ + + if (test_bit(B_DIRTY, &b->state) || test_bit(B_WRITING, &b->state)) + return ER_DONT_EVICT; + else + return ER_EVICT; +} + +static void __move_clean_buffers(struct dm_bufio_client *c) +{ + cache_mark_many(&c->cache, LIST_DIRTY, LIST_CLEAN, cleaned, NULL); +} + +struct write_context { + int no_wait; + struct list_head *write_list; +}; + +static enum it_action write_one(struct dm_buffer *b, void *context) +{ + struct write_context *wc = context; + + if (wc->no_wait && test_bit(B_WRITING, &b->state)) + return IT_COMPLETE; + + __write_dirty_buffer(b, wc->write_list); + return IT_NEXT; +} + static void __write_dirty_buffers_async(struct dm_bufio_client *c, int no_wait, struct list_head *write_list) { - struct dm_buffer *b, *tmp; + struct write_context wc = {.no_wait = no_wait, .write_list = write_list}; - list_for_each_entry_safe_reverse(b, tmp, &c->lru[LIST_DIRTY], lru_list) { - BUG_ON(test_bit(B_READING, &b->state)); - - if (!test_bit(B_DIRTY, &b->state) && - !test_bit(B_WRITING, &b->state)) { - __relink_lru(b, LIST_CLEAN); - continue; - } - - if (no_wait && test_bit(B_WRITING, &b->state)) - return; - - __write_dirty_buffer(b, write_list); - cond_resched(); - } + __move_clean_buffers(c); + cache_iterate(&c->cache, LIST_DIRTY, write_one, &wc); } /* @@ -1705,7 +1612,8 @@ static void __write_dirty_buffers_async(struct dm_bufio_client *c, int no_wait, static void __check_watermark(struct dm_bufio_client *c, struct list_head *write_list) { - if (c->n_buffers[LIST_DIRTY] > c->n_buffers[LIST_CLEAN] * DM_BUFIO_WRITEBACK_RATIO) + if (cache_count(&c->cache, LIST_DIRTY) > + cache_count(&c->cache, LIST_CLEAN) * DM_BUFIO_WRITEBACK_RATIO) __write_dirty_buffers_async(c, 1, write_list); } @@ -1715,6 +1623,21 @@ static void __check_watermark(struct dm_bufio_client *c, *-------------------------------------------------------------- */ +static void cache_put_and_wake(struct dm_bufio_client *c, struct dm_buffer *b) +{ + /* + * Relying on waitqueue_active() is racey, but we sleep + * with schedule_timeout anyway. + */ + if (cache_put(&c->cache, b) && + unlikely(waitqueue_active(&c->free_buffer_wait))) + wake_up(&c->free_buffer_wait); +} + +/* + * This assumes you have already checked the cache to see if the buffer + * is already present (it will recheck after dropping the lock for allocation). + */ static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block, enum new_flag nf, int *need_submit, struct list_head *write_list) @@ -1723,11 +1646,8 @@ static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block, *need_submit = 0; - b = __find(c, block); - if (b) - goto found_buffer; - - if (nf == NF_GET) + /* This can't be called with NF_GET */ + if (WARN_ON_ONCE(nf == NF_GET)) return NULL; new_b = __alloc_buffer_wait(c, nf); @@ -1738,7 +1658,7 @@ static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block, * We've had a period where the mutex was unlocked, so need to * recheck the buffer tree. */ - b = __find(c, block); + b = cache_get(&c->cache, block); if (b) { __free_buffer_wake(new_b); goto found_buffer; @@ -1747,24 +1667,35 @@ static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block, __check_watermark(c, write_list); b = new_b; - b->__hold_count = 1; + atomic_set(&b->hold_count, 1); + WRITE_ONCE(b->last_accessed, jiffies); + b->block = block; b->read_error = 0; b->write_error = 0; - __link_buffer(b, block, LIST_CLEAN); + b->list_mode = LIST_CLEAN; - if (nf == NF_FRESH) { + if (nf == NF_FRESH) b->state = 0; - return b; + else { + b->state = 1 << B_READING; + *need_submit = 1; } - b->state = 1 << B_READING; - *need_submit = 1; + /* + * We mustn't insert into the cache until the B_READING state + * is set. Otherwise another thread could get it and use + * it before it had been read. + */ + cache_insert(&c->cache, b); return b; found_buffer: - if (nf == NF_PREFETCH) + if (nf == NF_PREFETCH) { + cache_put_and_wake(c, b); return NULL; + } + /* * Note: it is essential that we don't wait for the buffer to be * read if dm_bufio_get function is used. Both dm_bufio_get and @@ -1772,12 +1703,11 @@ static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block, * If the user called both dm_bufio_prefetch and dm_bufio_get on * the same buffer, it would deadlock if we waited. */ - if (nf == NF_GET && unlikely(test_bit_acquire(B_READING, &b->state))) + if (nf == NF_GET && unlikely(test_bit_acquire(B_READING, &b->state))) { + cache_put_and_wake(c, b); return NULL; + } - b->__hold_count++; - __relink_lru(b, test_bit(B_DIRTY, &b->state) || - test_bit(B_WRITING, &b->state)); return b; } @@ -1807,18 +1737,50 @@ static void read_endio(struct dm_buffer *b, blk_status_t status) static void *new_read(struct dm_bufio_client *c, sector_t block, enum new_flag nf, struct dm_buffer **bp) { - int need_submit; + int need_submit = 0; struct dm_buffer *b; LIST_HEAD(write_list); - dm_bufio_lock(c); - b = __bufio_new(c, block, nf, &need_submit, &write_list); + *bp = NULL; + + /* + * Fast path, hopefully the block is already in the cache. No need + * to get the client lock for this. + */ + b = cache_get(&c->cache, block); + if (b) { + if (nf == NF_PREFETCH) { + cache_put_and_wake(c, b); + return NULL; + } + + /* + * Note: it is essential that we don't wait for the buffer to be + * read if dm_bufio_get function is used. Both dm_bufio_get and + * dm_bufio_prefetch can be used in the driver request routine. + * If the user called both dm_bufio_prefetch and dm_bufio_get on + * the same buffer, it would deadlock if we waited. + */ + if (nf == NF_GET && unlikely(test_bit_acquire(B_READING, &b->state))) { + cache_put_and_wake(c, b); + return NULL; + } + } + + if (!b) { + if (nf == NF_GET) + return NULL; + + dm_bufio_lock(c); + b = __bufio_new(c, block, nf, &need_submit, &write_list); + dm_bufio_unlock(c); + } + #ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING - if (b && b->hold_count == 1) + if (b && (atomic_read(&b->hold_count) == 1)) buffer_record_stack(b); #endif - dm_bufio_unlock(c); __flush_write_list(&write_list); @@ -1881,12 +1843,19 @@ void dm_bufio_prefetch(struct dm_bufio_client *c, return; /* should never happen */ blk_start_plug(&plug); - dm_bufio_lock(c); for (; n_blocks--; block++) { int need_submit; struct dm_buffer *b; + b = cache_get(&c->cache, block); + if (b) { + /* already in cache */ + cache_put_and_wake(c, b); + continue; + } + + dm_bufio_lock(c); b = __bufio_new(c, block, NF_PREFETCH, &need_submit, &write_list); if (unlikely(!list_empty(&write_list))) { @@ -1909,10 +1878,9 @@ void dm_bufio_prefetch(struct dm_bufio_client *c, goto flush_plug; dm_bufio_lock(c); } + dm_bufio_unlock(c); } - dm_bufio_unlock(c); - flush_plug: blk_finish_plug(&plug); } @@ -1922,29 +1890,28 @@ void dm_bufio_release(struct dm_buffer *b) { struct dm_bufio_client *c = b->c; - dm_bufio_lock(c); + /* + * If there were errors on the buffer, and the buffer is not + * to be written, free the buffer. There is no point in caching + * invalid buffer. + */ + if ((b->read_error || b->write_error) && + !test_bit_acquire(B_READING, &b->state) && + !test_bit(B_WRITING, &b->state) && + !test_bit(B_DIRTY, &b->state)) { + dm_bufio_lock(c); - BUG_ON(!b->__hold_count); - - b->__hold_count--; - if (!b->__hold_count) { - wake_up(&c->free_buffer_wait); - - /* - * If there were errors on the buffer, and the buffer is not - * to be written, free the buffer. There is no point in caching - * invalid buffer. - */ - if ((b->read_error || b->write_error) && - !test_bit_acquire(B_READING, &b->state) && - !test_bit(B_WRITING, &b->state) && - !test_bit(B_DIRTY, &b->state)) { - __unlink_buffer(b); + /* cache remove can fail if there are other holders */ + if (cache_remove(&c->cache, b)) { __free_buffer_wake(b); + dm_bufio_unlock(c); + return; } + + dm_bufio_unlock(c); } - dm_bufio_unlock(c); + cache_put_and_wake(c, b); } EXPORT_SYMBOL_GPL(dm_bufio_release); @@ -1963,7 +1930,7 @@ void dm_bufio_mark_partial_buffer_dirty(struct dm_buffer *b, if (!test_and_set_bit(B_DIRTY, &b->state)) { b->dirty_start = start; b->dirty_end = end; - __relink_lru(b, LIST_DIRTY); + cache_mark(&c->cache, b, LIST_DIRTY); } else { if (start < b->dirty_start) b->dirty_start = start; @@ -2002,11 +1969,19 @@ EXPORT_SYMBOL_GPL(dm_bufio_write_dirty_buffers_async); * * Finally, we flush hardware disk cache. */ +static bool is_writing(struct lru_entry *e, void *context) +{ + struct dm_buffer *b = le_to_buffer(e); + + return test_bit(B_WRITING, &b->state); +} + int dm_bufio_write_dirty_buffers(struct dm_bufio_client *c) { int a, f; - unsigned long buffers_processed = 0; - struct dm_buffer *b, *tmp; + unsigned long nr_buffers; + struct lru_entry *e; + struct lru_iter it; LIST_HEAD(write_list); @@ -2016,52 +1991,32 @@ int dm_bufio_write_dirty_buffers(struct dm_bufio_client *c) __flush_write_list(&write_list); dm_bufio_lock(c); -again: - list_for_each_entry_safe_reverse(b, tmp, &c->lru[LIST_DIRTY], lru_list) { - int dropped_lock = 0; - - if (buffers_processed < c->n_buffers[LIST_DIRTY]) - buffers_processed++; + nr_buffers = cache_count(&c->cache, LIST_DIRTY); + lru_iter_begin(&c->cache.lru[LIST_DIRTY], &it); + while ((e = lru_iter_next(&it, is_writing, c))) { + struct dm_buffer *b = le_to_buffer(e); + __cache_inc_buffer(b); BUG_ON(test_bit(B_READING, &b->state)); - if (test_bit(B_WRITING, &b->state)) { - if (buffers_processed < c->n_buffers[LIST_DIRTY]) { - dropped_lock = 1; - b->__hold_count++; - dm_bufio_unlock(c); - wait_on_bit_io(&b->state, B_WRITING, - TASK_UNINTERRUPTIBLE); - dm_bufio_lock(c); - b->__hold_count--; - } else - wait_on_bit_io(&b->state, B_WRITING, - TASK_UNINTERRUPTIBLE); + if (nr_buffers) { + nr_buffers--; + dm_bufio_unlock(c); + wait_on_bit_io(&b->state, B_WRITING, TASK_UNINTERRUPTIBLE); + dm_bufio_lock(c); + } else { + wait_on_bit_io(&b->state, B_WRITING, TASK_UNINTERRUPTIBLE); } - if (!test_bit(B_DIRTY, &b->state) && - !test_bit(B_WRITING, &b->state)) - __relink_lru(b, LIST_CLEAN); + if (!test_bit(B_DIRTY, &b->state) && !test_bit(B_WRITING, &b->state)) + cache_mark(&c->cache, b, LIST_CLEAN); + + cache_put_and_wake(c, b); cond_resched(); - - /* - * If we dropped the lock, the list is no longer consistent, - * so we must restart the search. - * - * In the most common case, the buffer just processed is - * relinked to the clean list, so we won't loop scanning the - * same buffer again and again. - * - * This may livelock if there is another thread simultaneously - * dirtying buffers, so we count the number of buffers walked - * and if it exceeds the total number of buffers, it means that - * someone is doing some writes simultaneously with us. In - * this case, stop, dropping the lock. - */ - if (dropped_lock) - goto again; } + lru_iter_end(&it); + wake_up(&c->free_buffer_wait); dm_bufio_unlock(c); @@ -2122,12 +2077,23 @@ int dm_bufio_issue_discard(struct dm_bufio_client *c, sector_t block, sector_t c } EXPORT_SYMBOL_GPL(dm_bufio_issue_discard); -static void forget_buffer_locked(struct dm_buffer *b) +static bool forget_buffer(struct dm_bufio_client *c, sector_t block) { - if (likely(!b->__hold_count) && likely(!smp_load_acquire(&b->state))) { - __unlink_buffer(b); - __free_buffer_wake(b); + struct dm_buffer *b; + + b = cache_get(&c->cache, block); + if (b) { + if (likely(!smp_load_acquire(&b->state))) { + if (cache_remove(&c->cache, b)) + __free_buffer_wake(b); + else + cache_put_and_wake(c, b); + } else { + cache_put_and_wake(c, b); + } } + + return b ? true : false; } /* @@ -2138,38 +2104,22 @@ static void forget_buffer_locked(struct dm_buffer *b) */ void dm_bufio_forget(struct dm_bufio_client *c, sector_t block) { - struct dm_buffer *b; - dm_bufio_lock(c); - - b = __find(c, block); - if (b) - forget_buffer_locked(b); - + forget_buffer(c, block); dm_bufio_unlock(c); } EXPORT_SYMBOL_GPL(dm_bufio_forget); +static enum evict_result idle(struct dm_buffer *b, void *context) +{ + return b->state ? ER_DONT_EVICT : ER_EVICT; +} + void dm_bufio_forget_buffers(struct dm_bufio_client *c, sector_t block, sector_t n_blocks) { - struct dm_buffer *b; - sector_t end_block = block + n_blocks; - - while (block < end_block) { - dm_bufio_lock(c); - - b = __find_next_old(c, block); - if (b) { - block = b->block + 1; - forget_buffer_locked(b); - } - - dm_bufio_unlock(c); - - if (!b) - break; - } - + dm_bufio_lock(c); + cache_remove_range(&c->cache, block, block + n_blocks, idle, __free_buffer_wake); + dm_bufio_unlock(c); } EXPORT_SYMBOL_GPL(dm_bufio_forget_buffers); @@ -2231,11 +2181,26 @@ struct dm_bufio_client *dm_bufio_get_client(struct dm_buffer *b) } EXPORT_SYMBOL_GPL(dm_bufio_get_client); +static enum it_action warn_leak(struct dm_buffer *b, void *context) +{ + bool *warned = context; + + WARN_ON(!(*warned)); + *warned = true; + DMERR("leaked buffer %llx, hold count %u, list %d", + (unsigned long long)b->block, atomic_read(&b->hold_count), b->list_mode); +#ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING + stack_trace_print(b->stack_entries, b->stack_len, 1); + /* mark unclaimed to avoid WARN_ON at end of drop_buffers() */ + atomic_set(&b->hold_count, 0); +#endif + return IT_NEXT; +} + static void drop_buffers(struct dm_bufio_client *c) { - struct dm_buffer *b; int i; - bool warned = false; + struct dm_buffer *b; if (WARN_ON(dm_bufio_in_request())) return; /* should never happen */ @@ -2250,18 +2215,11 @@ static void drop_buffers(struct dm_bufio_client *c) while ((b = __get_unclaimed_buffer(c))) __free_buffer_wake(b); - for (i = 0; i < LIST_SIZE; i++) - list_for_each_entry(b, &c->lru[i], lru_list) { - WARN_ON(!warned); - warned = true; - DMERR("leaked buffer %llx, hold count %u, list %d", - (unsigned long long)b->block, b->__hold_count, i); -#ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING - stack_trace_print(b->stack_entries, b->stack_len, 1); - /* mark unclaimed to avoid WARN_ON below */ - b->hold_count = 0; -#endif - } + for (i = 0; i < LIST_SIZE; i++) { + bool warned = false; + + cache_iterate(&c->cache, i, warn_leak, &warned); + } #ifdef CONFIG_DM_DEBUG_BLOCK_STACK_TRACING while ((b = __get_unclaimed_buffer(c))) @@ -2269,39 +2227,11 @@ static void drop_buffers(struct dm_bufio_client *c) #endif for (i = 0; i < LIST_SIZE; i++) - WARN_ON(!list_empty(&c->lru[i])); + WARN_ON(cache_count(&c->cache, i)); dm_bufio_unlock(c); } -/* - * We may not be able to evict this buffer if IO pending or the client - * is still using it. Caller is expected to know buffer is too old. - * - * And if GFP_NOFS is used, we must not do any I/O because we hold - * dm_bufio_clients_lock and we would risk deadlock if the I/O gets - * rerouted to different bufio client. - */ -static bool __try_evict_buffer(struct dm_buffer *b, gfp_t gfp) -{ - if (!(gfp & __GFP_FS) || - (static_branch_unlikely(&no_sleep_enabled) && b->c->no_sleep)) { - if (test_bit_acquire(B_READING, &b->state) || - test_bit(B_WRITING, &b->state) || - test_bit(B_DIRTY, &b->state)) - return false; - } - - if (b->__hold_count) - return false; - - __make_buffer_clean(b); - __unlink_buffer(b); - __free_buffer_wake(b); - - return true; -} - static unsigned long get_retain_buffers(struct dm_bufio_client *c) { unsigned long retain_bytes = READ_ONCE(dm_bufio_retain_bytes); @@ -2317,22 +2247,28 @@ static unsigned long get_retain_buffers(struct dm_bufio_client *c) static void __scan(struct dm_bufio_client *c) { int l; - struct dm_buffer *b, *tmp; + struct dm_buffer *b; unsigned long freed = 0; - unsigned long count = c->n_buffers[LIST_CLEAN] + - c->n_buffers[LIST_DIRTY]; unsigned long retain_target = get_retain_buffers(c); + unsigned long count = cache_total(&c->cache); for (l = 0; l < LIST_SIZE; l++) { - list_for_each_entry_safe_reverse(b, tmp, &c->lru[l], lru_list) { + while (true) { if (count - freed <= retain_target) atomic_long_set(&c->need_shrink, 0); if (!atomic_long_read(&c->need_shrink)) - return; - if (__try_evict_buffer(b, GFP_KERNEL)) { - atomic_long_dec(&c->need_shrink); - freed++; - } + break; + + b = cache_evict(&c->cache, l, + l == LIST_CLEAN ? is_clean : is_dirty, c); + if (!b) + break; + + __make_buffer_clean(b); + __free_buffer_wake(b); + + atomic_long_dec(&c->need_shrink); + freed++; cond_resched(); } } @@ -2361,8 +2297,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker); - unsigned long count = READ_ONCE(c->n_buffers[LIST_CLEAN]) + - READ_ONCE(c->n_buffers[LIST_DIRTY]); + unsigned long count = cache_total(&c->cache); unsigned long retain_target = get_retain_buffers(c); unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink); @@ -2390,7 +2325,6 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign { int r; struct dm_bufio_client *c; - unsigned int i; char slab_name[27]; if (!block_size || block_size & ((1 << SECTOR_SHIFT) - 1)) { @@ -2404,7 +2338,7 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign r = -ENOMEM; goto bad_client; } - c->buffer_tree = RB_ROOT; + cache_init(&c->cache); c->bdev = bdev; c->block_size = block_size; @@ -2421,11 +2355,6 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign static_branch_inc(&no_sleep_enabled); } - for (i = 0; i < LIST_SIZE; i++) { - INIT_LIST_HEAD(&c->lru[i]); - c->n_buffers[i] = 0; - } - mutex_init(&c->lock); spin_lock_init(&c->spinlock); INIT_LIST_HEAD(&c->reserved_buffers); @@ -2497,9 +2426,9 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign bad: while (!list_empty(&c->reserved_buffers)) { - struct dm_buffer *b = list_entry(c->reserved_buffers.next, - struct dm_buffer, lru_list); - list_del(&b->lru_list); + struct dm_buffer *b = list_to_buffer(c->reserved_buffers.next); + + list_del(&b->lru.list); free_buffer(b); } kmem_cache_destroy(c->slab_cache); @@ -2536,23 +2465,23 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c) mutex_unlock(&dm_bufio_clients_lock); - WARN_ON(!RB_EMPTY_ROOT(&c->buffer_tree)); WARN_ON(c->need_reserved_buffers); while (!list_empty(&c->reserved_buffers)) { - struct dm_buffer *b = list_entry(c->reserved_buffers.next, - struct dm_buffer, lru_list); - list_del(&b->lru_list); + struct dm_buffer *b = list_to_buffer(c->reserved_buffers.next); + + list_del(&b->lru.list); free_buffer(b); } for (i = 0; i < LIST_SIZE; i++) - if (c->n_buffers[i]) - DMERR("leaked buffer count %d: %ld", i, c->n_buffers[i]); + if (cache_count(&c->cache, i)) + DMERR("leaked buffer count %d: %lu", i, cache_count(&c->cache, i)); for (i = 0; i < LIST_SIZE; i++) - WARN_ON(c->n_buffers[i]); + WARN_ON(cache_count(&c->cache, i)); + cache_destroy(&c->cache); kmem_cache_destroy(c->slab_cache); kmem_cache_destroy(c->slab_buffer); dm_io_client_destroy(c->dm_io); @@ -2569,6 +2498,8 @@ void dm_bufio_set_sector_offset(struct dm_bufio_client *c, sector_t start) } EXPORT_SYMBOL_GPL(dm_bufio_set_sector_offset); +/*--------------------------------------------------------------*/ + static unsigned int get_max_age_hz(void) { unsigned int max_age = READ_ONCE(dm_bufio_max_age); @@ -2581,13 +2512,74 @@ static unsigned int get_max_age_hz(void) static bool older_than(struct dm_buffer *b, unsigned long age_hz) { - return time_after_eq(jiffies, b->last_accessed + age_hz); + return time_after_eq(jiffies, READ_ONCE(b->last_accessed) + age_hz); } -static void __evict_old_buffers(struct dm_bufio_client *c, unsigned long age_hz) +struct evict_params { + gfp_t gfp; + unsigned long age_hz; + + /* + * This gets updated with the largest last_accessed (ie. most + * recently used) of the evicted buffers. It will not be reinitialised + * by __evict_many(), so you can use it across multiple invocations. + */ + unsigned long last_accessed; +}; + +/* + * We may not be able to evict this buffer if IO pending or the client + * is still using it. + * + * And if GFP_NOFS is used, we must not do any I/O because we hold + * dm_bufio_clients_lock and we would risk deadlock if the I/O gets + * rerouted to different bufio client. + */ +static enum evict_result select_for_evict(struct dm_buffer *b, void *context) +{ + struct evict_params *params = context; + + if (!(params->gfp & __GFP_FS) || + (static_branch_unlikely(&no_sleep_enabled) && b->c->no_sleep)) { + if (test_bit_acquire(B_READING, &b->state) || + test_bit(B_WRITING, &b->state) || + test_bit(B_DIRTY, &b->state)) + return ER_DONT_EVICT; + } + + return older_than(b, params->age_hz) ? ER_EVICT : ER_STOP; +} + +static unsigned long __evict_many(struct dm_bufio_client *c, + struct evict_params *params, + int list_mode, unsigned long max_count) +{ + unsigned long count; + unsigned long last_accessed; + struct dm_buffer *b; + + for (count = 0; count < max_count; count++) { + b = cache_evict(&c->cache, list_mode, select_for_evict, params); + if (!b) + break; + + last_accessed = READ_ONCE(b->last_accessed); + if (time_after_eq(params->last_accessed, last_accessed)) + params->last_accessed = last_accessed; + + __make_buffer_clean(b); + __free_buffer_wake(b); + + cond_resched(); + } + + return count; +} + +static void evict_old_buffers(struct dm_bufio_client *c, unsigned long age_hz) { - struct dm_buffer *b, *tmp; - unsigned long retain_target = get_retain_buffers(c); + struct evict_params params = {.gfp = 0, .age_hz = age_hz, .last_accessed = 0}; + unsigned long retain = get_retain_buffers(c); unsigned long count; LIST_HEAD(write_list); @@ -2600,91 +2592,13 @@ static void __evict_old_buffers(struct dm_bufio_client *c, unsigned long age_hz) dm_bufio_lock(c); } - count = c->n_buffers[LIST_CLEAN] + c->n_buffers[LIST_DIRTY]; - list_for_each_entry_safe_reverse(b, tmp, &c->lru[LIST_CLEAN], lru_list) { - if (count <= retain_target) - break; - - if (!older_than(b, age_hz)) - break; - - if (__try_evict_buffer(b, 0)) - count--; - - cond_resched(); - } + count = cache_total(&c->cache); + if (count > retain) + __evict_many(c, ¶ms, LIST_CLEAN, count - retain); dm_bufio_unlock(c); } -static void do_global_cleanup(struct work_struct *w) -{ - struct dm_bufio_client *locked_client = NULL; - struct dm_bufio_client *current_client; - struct dm_buffer *b; - unsigned int spinlock_hold_count; - unsigned long threshold = dm_bufio_cache_size - - dm_bufio_cache_size / DM_BUFIO_LOW_WATERMARK_RATIO; - unsigned long loops = global_num * 2; - - mutex_lock(&dm_bufio_clients_lock); - - while (1) { - cond_resched(); - - spin_lock(&global_spinlock); - if (unlikely(dm_bufio_current_allocated <= threshold)) - break; - - spinlock_hold_count = 0; -get_next: - if (!loops--) - break; - if (unlikely(list_empty(&global_queue))) - break; - b = list_entry(global_queue.prev, struct dm_buffer, global_list); - - if (b->accessed) { - b->accessed = 0; - list_move(&b->global_list, &global_queue); - if (likely(++spinlock_hold_count < 16)) - goto get_next; - spin_unlock(&global_spinlock); - continue; - } - - current_client = b->c; - if (unlikely(current_client != locked_client)) { - if (locked_client) - dm_bufio_unlock(locked_client); - - if (!dm_bufio_trylock(current_client)) { - spin_unlock(&global_spinlock); - dm_bufio_lock(current_client); - locked_client = current_client; - continue; - } - - locked_client = current_client; - } - - spin_unlock(&global_spinlock); - - if (unlikely(!__try_evict_buffer(b, GFP_KERNEL))) { - spin_lock(&global_spinlock); - list_move(&b->global_list, &global_queue); - spin_unlock(&global_spinlock); - } - } - - spin_unlock(&global_spinlock); - - if (locked_client) - dm_bufio_unlock(locked_client); - - mutex_unlock(&dm_bufio_clients_lock); -} - static void cleanup_old_buffers(void) { unsigned long max_age_hz = get_max_age_hz(); @@ -2695,7 +2609,7 @@ static void cleanup_old_buffers(void) __cache_size_refresh(); list_for_each_entry(c, &dm_bufio_all_clients, client_list) - __evict_old_buffers(c, max_age_hz); + evict_old_buffers(c, max_age_hz); mutex_unlock(&dm_bufio_clients_lock); } @@ -2708,6 +2622,107 @@ static void work_fn(struct work_struct *w) DM_BUFIO_WORK_TIMER_SECS * HZ); } +/*--------------------------------------------------------------*/ + +/* + * Global cleanup tries to evict the oldest buffers from across _all_ + * the clients. It does this by repeatedly evicting a few buffers from + * the client that holds the oldest buffer. It's approximate, but hopefully + * good enough. + */ +static struct dm_bufio_client *__pop_client(void) +{ + struct list_head *h; + + if (list_empty(&dm_bufio_all_clients)) + return NULL; + + h = dm_bufio_all_clients.next; + list_del(h); + return container_of(h, struct dm_bufio_client, client_list); +} + +/* + * Inserts the client in the global client list based on its + * 'oldest_buffer' field. + */ +static void __insert_client(struct dm_bufio_client *new_client) +{ + struct dm_bufio_client *c; + struct list_head *h = dm_bufio_all_clients.next; + + while (h != &dm_bufio_all_clients) { + c = container_of(h, struct dm_bufio_client, client_list); + if (time_after_eq(c->oldest_buffer, new_client->oldest_buffer)) + break; + h = h->next; + } + + list_add_tail(&new_client->client_list, h); +} + +static unsigned long __evict_a_few(unsigned long nr_buffers) +{ + unsigned long count; + struct dm_bufio_client *c; + struct evict_params params = { + .gfp = GFP_KERNEL, + .age_hz = 0, + /* set to jiffies in case there are no buffers in this client */ + .last_accessed = jiffies + }; + + c = __pop_client(); + if (!c) + return 0; + + dm_bufio_lock(c); + count = __evict_many(c, ¶ms, LIST_CLEAN, nr_buffers); + dm_bufio_unlock(c); + + if (count) + c->oldest_buffer = params.last_accessed; + __insert_client(c); + + return count; +} + +static void check_watermarks(void) +{ + LIST_HEAD(write_list); + struct dm_bufio_client *c; + + mutex_lock(&dm_bufio_clients_lock); + list_for_each_entry(c, &dm_bufio_all_clients, client_list) { + dm_bufio_lock(c); + __check_watermark(c, &write_list); + dm_bufio_unlock(c); + } + mutex_unlock(&dm_bufio_clients_lock); + + __flush_write_list(&write_list); +} + +static void evict_old(void) +{ + unsigned long threshold = dm_bufio_cache_size - + dm_bufio_cache_size / DM_BUFIO_LOW_WATERMARK_RATIO; + + mutex_lock(&dm_bufio_clients_lock); + while (dm_bufio_current_allocated > threshold) { + if (!__evict_a_few(64)) + break; + cond_resched(); + } + mutex_unlock(&dm_bufio_clients_lock); +} + +static void do_global_cleanup(struct work_struct *w) +{ + check_watermarks(); + evict_old(); +} + /* *-------------------------------------------------------------- * Module setup -- 2.40.0