From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0DF3C47095 for ; Wed, 9 Jun 2021 06:35:11 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8589561181 for ; Wed, 9 Jun 2021 06:35:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8589561181 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DDD196E235; Wed, 9 Jun 2021 06:35:04 +0000 (UTC) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by gabe.freedesktop.org (Postfix) with ESMTPS id 281606E212; Wed, 9 Jun 2021 06:35:02 +0000 (UTC) IronPort-SDR: T6UYHeithRGmwm75uuJhlCstpSbFvssdr37t8LVKuUzmD1ZUnXoZbIeaJKfhdR6eHD/QE4ud97 OhaXMkktgboQ== X-IronPort-AV: E=McAfee;i="6200,9189,10009"; a="268868907" X-IronPort-AV: E=Sophos;i="5.83,260,1616482800"; d="scan'208";a="268868907" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 23:35:01 -0700 IronPort-SDR: 19dPAPIKm6JiCEhqV887FPrdtc9DgeWVTRQrEWjLCIEIEj/WfDoOo6qVtPwsbiavU2vGXKSK6L QAqszx6CVPIg== X-IronPort-AV: E=Sophos;i="5.83,260,1616482800"; d="scan'208";a="482265833" Received: from ekolpasx-mobl.ccr.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.109]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 23:34:59 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Wed, 9 Jun 2021 08:34:34 +0200 Message-Id: <20210609063436.284332-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609063436.284332-1-thomas.hellstrom@linux.intel.com> References: <20210609063436.284332-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 7/9] drm/i915/gt: Pipelined page migration X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" RnJvbTogQ2hyaXMgV2lsc29uIDxjaHJpc0BjaHJpcy13aWxzb24uY28udWs+CgpJZiB3ZSBwaXBl bGluZSB0aGUgUFRFIHVwZGF0ZXMgYW5kIHRoZW4gZG8gdGhlIGNvcHkgb2YgdGhvc2UgcGFnZXMK d2l0aGluIGEgc2luZ2xlIHVucHJlZW1wdGlibGUgY29tbWFuZCBwYWNrZXQsIHdlIGNhbiBzdWJt aXQgdGhlIGNvcGllcwphbmQgbGVhdmUgdGhlbSB0byBiZSBzY2hlZHVsZWQgd2l0aG91dCBoYXZp bmcgdG8gc3luY2hyb25vdXNseSB3YWl0CnVuZGVyIGEgZ2xvYmFsIGxvY2suIEluIG9yZGVyIHRv IG1hbmFnZSBtaWdyYXRpb24sIHdlIG5lZWQgdG8KcHJlYWxsb2NhdGUgdGhlIHBhZ2UgdGFibGVz IChhbmQga2VlcCB0aGVtIHBpbm5lZCBhbmQgYXZhaWxhYmxlIGZvciB1c2UKYXQgYW55IHRpbWUp LCBjYXVzaW5nIGEgYm90dGxlbmVjayBmb3IgbWlncmF0aW9ucyBhcyBhbGwgY2xpZW50cyBtdXN0 CmNvbnRlbmQgb24gdGhlIGxpbWl0ZWQgcmVzb3VyY2VzLiBCeSBpbmxpbmluZyB0aGUgcHBHVFQg dXBkYXRlcyBhbmQKcGVyZm9ybWluZyB0aGUgYmxpdCBhdG9taWNhbGx5LCBlYWNoIGNsaWVudCBv bmx5IG93bnMgdGhlIFBURSB3aGlsZSBpbgp1c2UsIGFuZCBzbyB3ZSBjYW4gcmVzY2hlZHVsZSBp bmRpdmlkdWFsIG9wZXJhdGlvbnMgaG93ZXZlciB3ZSBzZWUgZml0LgpBbmQgbW9zdCBpbXBvcnRh bnRseSwgd2UgZG8gbm90IG5lZWQgdG8gdGFrZSBhIGdsb2JhbCBsb2NrIG9uIHRoZSBzaGFyZWQK dm0sIGFuZCB3YWl0IHVudGlsIHRoZSBvcGVyYXRpb24gaXMgY29tcGxldGUgYmVmb3JlIHJlbGVh c2luZyB0aGUgbG9jawpmb3Igb3RoZXJzIHRvIGNsYWltIHRoZSBQVEUgZm9yIHRoZW1zZWx2ZXMu CgpTaWduZWQtb2ZmLWJ5OiBDaHJpcyBXaWxzb24gPGNocmlzQGNocmlzLXdpbHNvbi5jby51az4K Q28tZGV2ZWxvcGVkLWJ5OiBUaG9tYXMgSGVsbHN0csO2bSA8dGhvbWFzLmhlbGxzdHJvbUBsaW51 eC5pbnRlbC5jb20+ClNpZ25lZC1vZmYtYnk6IFRob21hcyBIZWxsc3Ryw7ZtIDx0aG9tYXMuaGVs bHN0cm9tQGxpbnV4LmludGVsLmNvbT4KLS0tCnYyOgotIEFkZCBhIFRPRE8gZm9yIGh1Z2UgTE1F TSBwdGVzIChQb2ludGVkIG91dCBieSBNYXR0aGV3IEF1bGQpCi0gVXNlIGludGVsX2VuZ2luZV9k ZXN0cm95X3Bpbm5lZF9jb250ZXh0KCkgdG8gcHJvcGVybHkgdGFrZSB0aGUgcGlubmVkCiAgY29u dGV4dCB0aW1lbGluZSBvZmYgdGhlIGVuZ2luZSBsaXN0LiAoQ0kgd2FybmluZykuCi0tLQogZHJp dmVycy9ncHUvZHJtL2k5MTUvTWFrZWZpbGUgICAgICAgICAgICAgICAgIHwgICAxICsKIGRyaXZl cnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2VuZ2luZS5oICAgICAgICB8ICAgMSArCiBkcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9ncHVfY29tbWFuZHMuaCAgfCAgIDIgKwogZHJpdmVycy9n cHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5jICAgICAgIHwgNTQ0ICsrKysrKysrKysrKysr KysrKwogZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5oICAgICAgIHwgIDQ1 ICsrCiBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmggfCAgMTUg KwogZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfcmluZy5oICAgICAgICAgIHwgICAxICsK IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L3NlbGZ0ZXN0X21pZ3JhdGUuYyAgICB8IDI5MSArKysr KysrKysrCiAuLi4vZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfbGl2ZV9zZWxmdGVzdHMuaCAgfCAg IDEgKwogOSBmaWxlcyBjaGFuZ2VkLCA5MDEgaW5zZXJ0aW9ucygrKQogY3JlYXRlIG1vZGUgMTAw NjQ0IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuYwogY3JlYXRlIG1vZGUg MTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuaAogY3JlYXRlIG1v ZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaAog Y3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L3NlbGZ0ZXN0X21pZ3Jh dGUuYwoKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L01ha2VmaWxlIGIvZHJpdmVy cy9ncHUvZHJtL2k5MTUvTWFrZWZpbGUKaW5kZXggMTZhNWEwMDZjZjdjLi45NWJkMzhlODQ2MjUg MTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L01ha2VmaWxlCisrKyBiL2RyaXZlcnMv Z3B1L2RybS9pOTE1L01ha2VmaWxlCkBAIC0xMDgsNiArMTA4LDcgQEAgZ3QteSArPSBcCiAJZ3Qv aW50ZWxfZ3R0Lm8gXAogCWd0L2ludGVsX2xsYy5vIFwKIAlndC9pbnRlbF9scmMubyBcCisJZ3Qv aW50ZWxfbWlncmF0ZS5vIFwKIAlndC9pbnRlbF9tb2NzLm8gXAogCWd0L2ludGVsX3BwZ3R0Lm8g XAogCWd0L2ludGVsX3JjNi5vIFwKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0 L2ludGVsX2VuZ2luZS5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZW5naW5lLmgK aW5kZXggMzZlYTllYjUyYmI1Li42MmY3NDQwYmMxMTEgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1 L2RybS9pOTE1L2d0L2ludGVsX2VuZ2luZS5oCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0 L2ludGVsX2VuZ2luZS5oCkBAIC0xODgsNiArMTg4LDcgQEAgaW50ZWxfd3JpdGVfc3RhdHVzX3Bh Z2Uoc3RydWN0IGludGVsX2VuZ2luZV9jcyAqZW5naW5lLCBpbnQgcmVnLCB1MzIgdmFsdWUpCiAj ZGVmaW5lIEk5MTVfR0VNX0hXU19QUkVFTVBUX0FERFIJKEk5MTVfR0VNX0hXU19QUkVFTVBUICog c2l6ZW9mKHUzMikpCiAjZGVmaW5lIEk5MTVfR0VNX0hXU19TRVFOTwkJMHg0MAogI2RlZmluZSBJ OTE1X0dFTV9IV1NfU0VRTk9fQUREUgkJKEk5MTVfR0VNX0hXU19TRVFOTyAqIHNpemVvZih1MzIp KQorI2RlZmluZSBJOTE1X0dFTV9IV1NfTUlHUkFURQkJKDB4NDIgKiBzaXplb2YodTMyKSkKICNk ZWZpbmUgSTkxNV9HRU1fSFdTX1NDUkFUQ0gJCTB4ODAKIAogI2RlZmluZSBJOTE1X0hXU19DU0Jf QlVGMF9JTkRFWAkJMHgxMApkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50 ZWxfZ3B1X2NvbW1hbmRzLmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9ncHVfY29t bWFuZHMuaAppbmRleCAyNjk0ZGJiOTk2N2UuLjFjM2FmMGZjMDQ1NiAxMDA2NDQKLS0tIGEvZHJp dmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1hbmRzLmgKKysrIGIvZHJpdmVycy9n cHUvZHJtL2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1hbmRzLmgKQEAgLTEyMyw4ICsxMjMsMTAgQEAK ICNkZWZpbmUgICBNSV9TRU1BUEhPUkVfU0FEX05FUV9TREQJKDUgPDwgMTIpCiAjZGVmaW5lICAg TUlfU0VNQVBIT1JFX1RPS0VOX01BU0sJUkVHX0dFTk1BU0soOSwgNSkKICNkZWZpbmUgICBNSV9T RU1BUEhPUkVfVE9LRU5fU0hJRlQJNQorI2RlZmluZSBNSV9TVE9SRV9EQVRBX0lNTQlNSV9JTlNU UigweDIwLCAwKQogI2RlZmluZSBNSV9TVE9SRV9EV09SRF9JTU0JTUlfSU5TVFIoMHgyMCwgMSkK ICNkZWZpbmUgTUlfU1RPUkVfRFdPUkRfSU1NX0dFTjQJTUlfSU5TVFIoMHgyMCwgMikKKyNkZWZp bmUgTUlfU1RPUkVfUVdPUkRfSU1NX0dFTjggKE1JX0lOU1RSKDB4MjAsIDMpIHwgUkVHX0JJVCgy MSkpCiAjZGVmaW5lICAgTUlfTUVNX1ZJUlRVQUwJKDEgPDwgMjIpIC8qIDk0NSxnMzMsOTY1ICov CiAjZGVmaW5lICAgTUlfVVNFX0dHVFQJCSgxIDw8IDIyKSAvKiBnNHgrICovCiAjZGVmaW5lIE1J X1NUT1JFX0RXT1JEX0lOREVYCU1JX0lOU1RSKDB4MjEsIDEpCmRpZmYgLS1naXQgYS9kcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlLmMgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9n dC9pbnRlbF9taWdyYXRlLmMKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMDAwMDAw Li43MDc3NjMxNjg2M2QKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9n dC9pbnRlbF9taWdyYXRlLmMKQEAgLTAsMCArMSw1NDQgQEAKKy8vIFNQRFgtTGljZW5zZS1JZGVu dGlmaWVyOiBNSVQKKy8qCisgKiBDb3B5cmlnaHQgwqkgMjAyMCBJbnRlbCBDb3Jwb3JhdGlvbgor ICovCisKKyNpbmNsdWRlICJpOTE1X2Rydi5oIgorI2luY2x1ZGUgImludGVsX2NvbnRleHQuaCIK KyNpbmNsdWRlICJpbnRlbF9ncHVfY29tbWFuZHMuaCIKKyNpbmNsdWRlICJpbnRlbF9ndC5oIgor I2luY2x1ZGUgImludGVsX2d0dC5oIgorI2luY2x1ZGUgImludGVsX21pZ3JhdGUuaCIKKyNpbmNs dWRlICJpbnRlbF9yaW5nLmgiCisKK3N0cnVjdCBpbnNlcnRfcHRlX2RhdGEgeworCXU2NCBvZmZz ZXQ7CisJYm9vbCBpc19sbWVtOworfTsKKworI2RlZmluZSBDSFVOS19TWiBTWl84TSAvKiB+MW1z IGF0IDhHaUIvcyBwcmVlbXB0aW9uIGRlbGF5ICovCisKK3N0YXRpYyBib29sIGVuZ2luZV9zdXBw b3J0c19taWdyYXRpb24oc3RydWN0IGludGVsX2VuZ2luZV9jcyAqZW5naW5lKQoreworCWlmICgh ZW5naW5lKQorCQlyZXR1cm4gZmFsc2U7CisKKwkvKgorCSAqIFdlIG5lZWQgdGhlIGFiaWxpdHkg dG8gcHJldmVudCBhcmlidHJhdGlvbiAoTUlfQVJCX09OX09GRiksCisJICogdGhlIGFiaWxpdHkg dG8gd3JpdGUgUFRFIHVzaW5nIGlubGluZSBkYXRhIChNSV9TVE9SRV9EQVRBKQorCSAqIGFuZCBv ZiBjb3Vyc2UgdGhlIGFiaWxpdHkgdG8gZG8gdGhlIGJsb2NrIHRyYW5zZmVyIChibGl0cykuCisJ ICovCisJR0VNX0JVR19PTihlbmdpbmUtPmNsYXNzICE9IENPUFlfRU5HSU5FX0NMQVNTKTsKKwor CXJldHVybiB0cnVlOworfQorCitzdGF0aWMgdm9pZCBpbnNlcnRfcHRlKHN0cnVjdCBpOTE1X2Fk ZHJlc3Nfc3BhY2UgKnZtLAorCQkgICAgICAgc3RydWN0IGk5MTVfcGFnZV90YWJsZSAqcHQsCisJ CSAgICAgICB2b2lkICpkYXRhKQoreworCXN0cnVjdCBpbnNlcnRfcHRlX2RhdGEgKmQgPSBkYXRh OworCisJdm0tPmluc2VydF9wYWdlKHZtLCBweF9kbWEocHQpLCBkLT5vZmZzZXQsIEk5MTVfQ0FD SEVfTk9ORSwKKwkJCWQtPmlzX2xtZW0gPyBQVEVfTE0gOiAwKTsKKwlkLT5vZmZzZXQgKz0gUEFH RV9TSVpFOworfQorCitzdGF0aWMgc3RydWN0IGk5MTVfYWRkcmVzc19zcGFjZSAqbWlncmF0ZV92 bShzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0cnVjdCBpOTE1X3ZtX3B0X3N0YXNoIHN0YXNo ID0ge307CisJc3RydWN0IGk5MTVfcHBndHQgKnZtOworCWludCBlcnI7CisJaW50IGk7CisKKwkv KgorCSAqIFdlIGNvbnN0cnVjdCBhIHZlcnkgc3BlY2lhbCBWTSBmb3IgdXNlIGJ5IGFsbCBtaWdy YXRpb24gY29udGV4dHMsCisJICogaXQgaXMga2VwdCBwaW5uZWQgc28gdGhhdCBpdCBjYW4gYmUg dXNlZCBhdCBhbnkgdGltZS4gQXMgd2UgbmVlZAorCSAqIHRvIHByZS1hbGxvY2F0ZSB0aGUgcGFn ZSBkaXJlY3RvcmllcyBmb3IgdGhlIG1pZ3JhdGlvbiBWTSwgdGhpcworCSAqIGxpbWl0cyB1cyB0 byBvbmx5IHVzaW5nIGEgc21hbGwgbnVtYmVyIG9mIHByZXBhcmVkIHZtYS4KKwkgKgorCSAqIFRv IGJlIGFibGUgdG8gcGlwZWxpbmUgYW5kIHJlc2NoZWR1bGUgbWlncmF0aW9uIG9wZXJhdGlvbnMg d2hpbGUKKwkgKiBhdm9pZGluZyB1bm5lY2Vzc2FyeSBjb250ZW50aW9uIG9uIHRoZSB2bSBpdHNl bGYsIHRoZSBQVEUgdXBkYXRlcworCSAqIGFyZSBpbmxpbmUgd2l0aCB0aGUgYmxpdHMuIEFsbCB0 aGUgYmxpdHMgdXNlIHRoZSBzYW1lIGZpeGVkCisJICogYWRkcmVzc2VzLCB3aXRoIHRoZSBiYWNr aW5nIHN0b3JlIHJlZGlyZWN0aW9uIGJlaW5nIHVwZGF0ZWQgb24gdGhlCisJICogZmx5LiBPbmx5 IDIgaW1wbGljaXQgdm1hIGFyZSB1c2VkIGZvciBhbGwgbWlncmF0aW9uIG9wZXJhdGlvbnMuCisJ ICoKKwkgKiBXZSBsYXkgdGhlIHBwR1RUIG91dCBhczoKKwkgKgorCSAqCVswLCBDSFVOS19TWikg LT4gZmlyc3Qgb2JqZWN0CisJICoJW0NIVU5LX1NaLCAyICogQ0hVTktfU1opIC0+IHNlY29uZCBv YmplY3QKKwkgKglbMiAqIENIVU5LX1NaLCAyICogQ0hVTktfU1ogKyAyICogQ0hVTktfU1ogPj4g OV0gLT4gUFRFCisJICoKKwkgKiBCeSBleHBvc2luZyB0aGUgZG1hIGFkZHJlc3NlcyBvZiB0aGUg cGFnZSBkaXJlY3RvcmllcyB0aGVtc2VsdmVzCisJICogd2l0aGluIHRoZSBwcEdUVCwgd2UgYXJl IHRoZW4gYWJsZSB0byByZXdyaXRlIHRoZSBQVEUgcHJpb3IgdG8gdXNlLgorCSAqIEJ1dCB0aGUg UFRFIHVwZGF0ZSBhbmQgc3Vic2VxdWVudCBtaWdyYXRpb24gb3BlcmF0aW9uIG11c3QgYmUgYXRv bWljLAorCSAqIGkuZS4gd2l0aGluIHRoZSBzYW1lIG5vbi1wcmVlbXB0aWJsZSB3aW5kb3cgc28g dGhhdCB3ZSBkbyBub3Qgc3dpdGNoCisJICogdG8gYW5vdGhlciBtaWdyYXRpb24gY29udGV4dCB0 aGF0IG92ZXJ3cml0ZXMgdGhlIFBURS4KKwkgKgorCSAqIFRPRE86IEFkZCBzdXBwb3J0IGZvciBo dWdlIExNRU0gUFRFcworCSAqLworCisJdm0gPSBpOTE1X3BwZ3R0X2NyZWF0ZShndCk7CisJaWYg KElTX0VSUih2bSkpCisJCXJldHVybiBFUlJfQ0FTVCh2bSk7CisKKwlpZiAoIXZtLT52bS5hbGxv Y2F0ZV92YV9yYW5nZSB8fCAhdm0tPnZtLmZvcmVhY2gpIHsKKwkJZXJyID0gLUVOT0RFVjsKKwkJ Z290byBlcnJfdm07CisJfQorCisJLyoKKwkgKiBFYWNoIGVuZ2luZSBpbnN0YW5jZSBpcyBhc3Np Z25lZCBpdHMgb3duIGNodW5rIGluIHRoZSBWTSwgc28KKwkgKiB0aGF0IHdlIGNhbiBydW4gbXVs dGlwbGUgaW5zdGFuY2VzIGNvbmN1cnJlbnRseQorCSAqLworCWZvciAoaSA9IDA7IGkgPCBBUlJB WV9TSVpFKGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xBU1NdKTsgaSsrKSB7CisJCXN0 cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwkJdTY0IGJhc2UgPSAodTY0KWkgPDwgMzI7 CisJCXN0cnVjdCBpbnNlcnRfcHRlX2RhdGEgZCA9IHt9OworCQlzdHJ1Y3QgaTkxNV9nZW1fd3df Y3R4IHd3OworCQl1NjQgc3o7CisKKwkJZW5naW5lID0gZ3QtPmVuZ2luZV9jbGFzc1tDT1BZX0VO R0lORV9DTEFTU11baV07CisJCWlmICghZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihlbmdpbmUp KQorCQkJY29udGludWU7CisKKwkJLyoKKwkJICogV2UgY29weSBpbiA4TWlCIGNodW5rcy4gRWFj aCBQREUgY292ZXJzIDJNaUIsIHNvIHdlIG5lZWQKKwkJICogNHgyIHBhZ2UgZGlyZWN0b3JpZXMg Zm9yIHNvdXJjZS9kZXN0aW5hdGlvbi4KKwkJICovCisJCXN6ID0gMiAqIENIVU5LX1NaOworCQlk Lm9mZnNldCA9IGJhc2UgKyBzejsKKworCQkvKgorCQkgKiBXZSBuZWVkIGFub3RoZXIgcGFnZSBk aXJlY3Rvcnkgc2V0dXAgc28gdGhhdCB3ZSBjYW4gd3JpdGUKKwkJICogdGhlIDh4NTEyIFBURSBp biBlYWNoIGNodW5rLgorCQkgKi8KKwkJc3ogKz0gKHN6ID4+IDEyKSAqIHNpemVvZih1NjQpOwor CisJCWVyciA9IGk5MTVfdm1fYWxsb2NfcHRfc3Rhc2goJnZtLT52bSwgJnN0YXNoLCBzeik7CisJ CWlmIChlcnIpCisJCQlnb3RvIGVycl92bTsKKworCQlmb3JfaTkxNV9nZW1fd3coJnd3LCBlcnIs IHRydWUpIHsKKwkJCWVyciA9IGk5MTVfdm1fbG9ja19vYmplY3RzKCZ2bS0+dm0sICZ3dyk7CisJ CQlpZiAoZXJyKQorCQkJCWNvbnRpbnVlOworCQkJZXJyID0gaTkxNV92bV9tYXBfcHRfc3Rhc2go JnZtLT52bSwgJnN0YXNoKTsKKwkJCWlmIChlcnIpCisJCQkJY29udGludWU7CisKKwkJCXZtLT52 bS5hbGxvY2F0ZV92YV9yYW5nZSgmdm0tPnZtLCAmc3Rhc2gsIGJhc2UsIGJhc2UgKyBzeik7CisJ CX0KKwkJaTkxNV92bV9mcmVlX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCk7CisJCWlmIChlcnIp CisJCQlnb3RvIGVycl92bTsKKworCQkvKiBOb3cgYWxsb3cgdGhlIEdQVSB0byByZXdyaXRlIHRo ZSBQVEUgdmlhIGl0cyBvd24gcHBHVFQgKi8KKwkJZC5pc19sbWVtID0gaTkxNV9nZW1fb2JqZWN0 X2lzX2xtZW0odm0tPnZtLnNjcmF0Y2hbMF0pOworCQl2bS0+dm0uZm9yZWFjaCgmdm0tPnZtLCBi YXNlLCBiYXNlICsgc3osIGluc2VydF9wdGUsICZkKTsKKwl9CisKKwlyZXR1cm4gJnZtLT52bTsK KworZXJyX3ZtOgorCWk5MTVfdm1fcHV0KCZ2bS0+dm0pOworCXJldHVybiBFUlJfUFRSKGVycik7 Cit9CisKK3N0YXRpYyBzdHJ1Y3QgaW50ZWxfZW5naW5lX2NzICpmaXJzdF9jb3B5X2VuZ2luZShz dHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsK KwlpbnQgaTsKKworCWZvciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKGd0LT5lbmdpbmVfY2xhc3Nb Q09QWV9FTkdJTkVfQ0xBU1NdKTsgaSsrKSB7CisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3Nb Q09QWV9FTkdJTkVfQ0xBU1NdW2ldOworCQlpZiAoZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihl bmdpbmUpKQorCQkJcmV0dXJuIGVuZ2luZTsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworc3Rh dGljIHN0cnVjdCBpbnRlbF9jb250ZXh0ICpwaW5uZWRfY29udGV4dChzdHJ1Y3QgaW50ZWxfZ3Qg Kmd0KQoreworCXN0YXRpYyBzdHJ1Y3QgbG9ja19jbGFzc19rZXkga2V5OworCXN0cnVjdCBpbnRl bF9lbmdpbmVfY3MgKmVuZ2luZTsKKwlzdHJ1Y3QgaTkxNV9hZGRyZXNzX3NwYWNlICp2bTsKKwlz dHJ1Y3QgaW50ZWxfY29udGV4dCAqY2U7CisKKwllbmdpbmUgPSBmaXJzdF9jb3B5X2VuZ2luZShn dCk7CisJaWYgKCFlbmdpbmUpCisJCXJldHVybiBFUlJfUFRSKC1FTk9ERVYpOworCisJdm0gPSBt aWdyYXRlX3ZtKGd0KTsKKwlpZiAoSVNfRVJSKHZtKSkKKwkJcmV0dXJuIEVSUl9DQVNUKHZtKTsK KworCWNlID0gaW50ZWxfZW5naW5lX2NyZWF0ZV9waW5uZWRfY29udGV4dChlbmdpbmUsIHZtLCBT Wl81MTJLLAorCQkJCQkJSTkxNV9HRU1fSFdTX01JR1JBVEUsCisJCQkJCQkma2V5LCAibWlncmF0 ZSIpOworCWk5MTVfdm1fcHV0KGNlLT52bSk7CisJcmV0dXJuIGNlOworfQorCitpbnQgaW50ZWxf bWlncmF0ZV9pbml0KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0 KQoreworCXN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZTsKKworCW1lbXNldChtLCAwLCBzaXplb2Yo Km0pKTsKKworCWNlID0gcGlubmVkX2NvbnRleHQoZ3QpOworCWlmIChJU19FUlIoY2UpKQorCQly ZXR1cm4gUFRSX0VSUihjZSk7CisKKwltLT5jb250ZXh0ID0gY2U7CisJcmV0dXJuIDA7Cit9CisK K3N0YXRpYyBpbnQgcmFuZG9tX2luZGV4KHVuc2lnbmVkIGludCBtYXgpCit7CisJcmV0dXJuIHVw cGVyXzMyX2JpdHMobXVsX3UzMl91MzIoZ2V0X3JhbmRvbV91MzIoKSwgbWF4KSk7Cit9CisKK3N0 YXRpYyBzdHJ1Y3QgaW50ZWxfY29udGV4dCAqX19taWdyYXRlX2VuZ2luZXMoc3RydWN0IGludGVs X2d0ICpndCkKK3sKKwlzdHJ1Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmVzW01BWF9FTkdJTkVf SU5TVEFOQ0VdOworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwl1bnNpZ25lZCBp bnQgY291bnQsIGk7CisKKwljb3VudCA9IDA7CisJZm9yIChpID0gMDsgaSA8IEFSUkFZX1NJWkUo Z3QtPmVuZ2luZV9jbGFzc1tDT1BZX0VOR0lORV9DTEFTU10pOyBpKyspIHsKKwkJZW5naW5lID0g Z3QtPmVuZ2luZV9jbGFzc1tDT1BZX0VOR0lORV9DTEFTU11baV07CisJCWlmIChlbmdpbmVfc3Vw cG9ydHNfbWlncmF0aW9uKGVuZ2luZSkpCisJCQllbmdpbmVzW2NvdW50KytdID0gZW5naW5lOwor CX0KKworCXJldHVybiBpbnRlbF9jb250ZXh0X2NyZWF0ZShlbmdpbmVzW3JhbmRvbV9pbmRleChj b3VudCldKTsKK30KKworc3RydWN0IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRl X2NvbnRleHQoc3RydWN0IGludGVsX21pZ3JhdGUgKm0pCit7CisJc3RydWN0IGludGVsX2NvbnRl eHQgKmNlOworCisJLyoKKwkgKiBXZSByYW5kb21seSBkaXN0cmlidXRlIGNvbnRleHRzIGFjcm9z cyB0aGUgZW5naW5lcyB1cG9uIGNvbnN0cmN0aW9uLAorCSAqIGFzIHRoZXkgYWxsIHNoYXJlIHRo ZSBzYW1lIHBpbm5lZCB2bSwgYW5kIHNvIGluIG9yZGVyIHRvIGFsbG93CisJICogbXVsdGlwbGUg YmxpdHMgdG8gcnVuIGluIHBhcmFsbGVsLCB3ZSBtdXN0IGNvbnN0cnVjdCBlYWNoIGJsaXQKKwkg KiB0byB1c2UgYSBkaWZmZXJlbnQgcmFuZ2Ugb2YgdGhlIHZtIGZvciBpdHMgR1RULiBUaGlzIGhh cyB0byBiZQorCSAqIGtub3duIGF0IGNvbnN0cnVjdGlvbiwgc28gd2UgY2FuIG5vdCB1c2UgdGhl IGxhdGUgZ3JlZWR5IGxvYWQKKwkgKiBiYWxhbmNpbmcgb2YgdGhlIHZpcnR1YWwtZW5naW5lLgor CSAqLworCWNlID0gX19taWdyYXRlX2VuZ2luZXMobS0+Y29udGV4dC0+ZW5naW5lLT5ndCk7CisJ aWYgKElTX0VSUihjZSkpCisJCXJldHVybiBjZTsKKworCWNlLT5yaW5nID0gX19pbnRlbF9jb250 ZXh0X3Jpbmdfc2l6ZShTWl8yNTZLKTsKKworCWk5MTVfdm1fcHV0KGNlLT52bSk7CisJY2UtPnZt ID0gaTkxNV92bV9nZXQobS0+Y29udGV4dC0+dm0pOworCisJcmV0dXJuIGNlOworfQorCitzdGF0 aWMgaW5saW5lIHN0cnVjdCBzZ3RfZG1hIHNnX3NndChzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnKQor eworCWRtYV9hZGRyX3QgYWRkciA9IHNnX2RtYV9hZGRyZXNzKHNnKTsKKworCXJldHVybiAoc3Ry dWN0IHNndF9kbWEpeyBzZywgYWRkciwgYWRkciArIHNnX2RtYV9sZW4oc2cpIH07Cit9CisKK3N0 YXRpYyBpbnQgZW1pdF9ub19hcmJpdHJhdGlvbihzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSkKK3sK Kwl1MzIgKmNzOworCisJY3MgPSBpbnRlbF9yaW5nX2JlZ2luKHJxLCAyKTsKKwlpZiAoSVNfRVJS KGNzKSkKKwkJcmV0dXJuIFBUUl9FUlIoY3MpOworCisJLyogRXhwbGljaXRseSBkaXNhYmxlIHBy ZWVtcHRpb24gZm9yIHRoaXMgcmVxdWVzdC4gKi8KKwkqY3MrKyA9IE1JX0FSQl9PTl9PRkY7CisJ KmNzKysgPSBNSV9OT09QOworCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCisJcmV0dXJu IDA7Cit9CisKK3N0YXRpYyBpbnQgZW1pdF9wdGUoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsCisJ CSAgICBzdHJ1Y3Qgc2d0X2RtYSAqaXQsCisJCSAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgY2Fj aGVfbGV2ZWwsCisJCSAgICBib29sIGlzX2xtZW0sCisJCSAgICB1NjQgb2Zmc2V0LAorCQkgICAg aW50IGxlbmd0aCkKK3sKKwljb25zdCB1NjQgZW5jb2RlID0gcnEtPmNvbnRleHQtPnZtLT5wdGVf ZW5jb2RlKDAsIGNhY2hlX2xldmVsLAorCQkJCQkJICAgICAgIGlzX2xtZW0gPyBQVEVfTE0gOiAw KTsKKwlzdHJ1Y3QgaW50ZWxfcmluZyAqcmluZyA9IHJxLT5yaW5nOworCWludCB0b3RhbCA9IDA7 CisJdTMyICpoZHIsICpjczsKKwlpbnQgcGt0OworCisJR0VNX0JVR19PTihJTlRFTF9HRU4ocnEt PmVuZ2luZS0+aTkxNSkgPCA4KTsKKworCS8qIENvbXB1dGUgdGhlIHBhZ2UgZGlyZWN0b3J5IG9m ZnNldCBmb3IgdGhlIHRhcmdldCBhZGRyZXNzIHJhbmdlICovCisJb2Zmc2V0ICs9ICh1NjQpcnEt PmVuZ2luZS0+aW5zdGFuY2UgPDwgMzI7CisJb2Zmc2V0ID4+PSAxMjsKKwlvZmZzZXQgKj0gc2l6 ZW9mKHU2NCk7CisJb2Zmc2V0ICs9IDIgKiBDSFVOS19TWjsKKworCWNzID0gaW50ZWxfcmluZ19i ZWdpbihycSwgNik7CisJaWYgKElTX0VSUihjcykpCisJCXJldHVybiBQVFJfRVJSKGNzKTsKKwor CS8qIFBhY2sgYXMgbWFueSBQVEUgdXBkYXRlcyBhcyBwb3NzaWJsZSBpbnRvIGEgc2luZ2xlIE1J IGNvbW1hbmQgKi8KKwlwa3QgPSBtaW5fdChpbnQsIDB4NDAwLCByaW5nLT5zcGFjZSAvIHNpemVv Zih1MzIpICsgNSk7CisJcGt0ID0gbWluX3QoaW50LCBwa3QsIChyaW5nLT5zaXplIC0gcmluZy0+ ZW1pdCkgLyBzaXplb2YodTMyKSArIDUpOworCisJaGRyID0gY3M7CisJKmNzKysgPSBNSV9TVE9S RV9EQVRBX0lNTSB8IFJFR19CSVQoMjEpOyAvKiBhcyBxd29yZCBlbGVtZW50cyAqLworCSpjcysr ID0gbG93ZXJfMzJfYml0cyhvZmZzZXQpOworCSpjcysrID0gdXBwZXJfMzJfYml0cyhvZmZzZXQp OworCisJZG8geworCQlpZiAoY3MgLSBoZHIgPj0gcGt0KSB7CisJCQkqaGRyICs9IGNzIC0gaGRy IC0gMjsKKwkJCSpjcysrID0gTUlfTk9PUDsKKworCQkJcmluZy0+ZW1pdCA9ICh2b2lkICopY3Mg LSByaW5nLT52YWRkcjsKKwkJCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCQkJaW50ZWxf cmluZ191cGRhdGVfc3BhY2UocmluZyk7CisKKwkJCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwg Nik7CisJCQlpZiAoSVNfRVJSKGNzKSkKKwkJCQlyZXR1cm4gUFRSX0VSUihjcyk7CisKKwkJCXBr dCA9IG1pbl90KGludCwgMHg0MDAsIHJpbmctPnNwYWNlIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwkJ CXBrdCA9IG1pbl90KGludCwgcGt0LCAocmluZy0+c2l6ZSAtIHJpbmctPmVtaXQpIC8gc2l6ZW9m KHUzMikgKyA1KTsKKworCQkJaGRyID0gY3M7CisJCQkqY3MrKyA9IE1JX1NUT1JFX0RBVEFfSU1N IHwgUkVHX0JJVCgyMSk7CisJCQkqY3MrKyA9IGxvd2VyXzMyX2JpdHMob2Zmc2V0KTsKKwkJCSpj cysrID0gdXBwZXJfMzJfYml0cyhvZmZzZXQpOworCQl9CisKKwkJKmNzKysgPSBsb3dlcl8zMl9i aXRzKGVuY29kZSB8IGl0LT5kbWEpOworCQkqY3MrKyA9IHVwcGVyXzMyX2JpdHMoZW5jb2RlIHwg aXQtPmRtYSk7CisKKwkJb2Zmc2V0ICs9IDg7CisJCXRvdGFsICs9IEk5MTVfR1RUX1BBR0VfU0la RTsKKworCQlpdC0+ZG1hICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKwkJaWYgKGl0LT5kbWEgPj0g aXQtPm1heCkgeworCQkJaXQtPnNnID0gX19zZ19uZXh0KGl0LT5zZyk7CisJCQlpZiAoIWl0LT5z ZyB8fCBzZ19kbWFfbGVuKGl0LT5zZykgPT0gMCkKKwkJCQlicmVhazsKKworCQkJaXQtPmRtYSA9 IHNnX2RtYV9hZGRyZXNzKGl0LT5zZyk7CisJCQlpdC0+bWF4ID0gaXQtPmRtYSArIHNnX2RtYV9s ZW4oaXQtPnNnKTsKKwkJfQorCX0gd2hpbGUgKHRvdGFsIDwgbGVuZ3RoKTsKKworCSpoZHIgKz0g Y3MgLSBoZHIgLSAyOworCSpjcysrID0gTUlfTk9PUDsKKworCXJpbmctPmVtaXQgPSAodm9pZCAq KWNzIC0gcmluZy0+dmFkZHI7CisJaW50ZWxfcmluZ19hZHZhbmNlKHJxLCBjcyk7CisJaW50ZWxf cmluZ191cGRhdGVfc3BhY2UocmluZyk7CisKKwlyZXR1cm4gdG90YWw7Cit9CisKK3N0YXRpYyBi b29sIHdhXzEyMDk2NDQ2MTFfYXBwbGllcyhpbnQgZ2VuLCB1MzIgc2l6ZSkKK3sKKwl1MzIgaGVp Z2h0ID0gc2l6ZSA+PiBQQUdFX1NISUZUOworCisJaWYgKGdlbiAhPSAxMSkKKwkJcmV0dXJuIGZh bHNlOworCisJcmV0dXJuIGhlaWdodCAlIDQgPT0gMyAmJiBoZWlnaHQgPD0gODsKK30KKworc3Rh dGljIGludCBlbWl0X2NvcHkoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsIGludCBzaXplKQorewor CWNvbnN0IGludCBnZW4gPSBJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSk7CisJdTMyIGluc3Rh bmNlID0gcnEtPmVuZ2luZS0+aW5zdGFuY2U7CisJdTMyICpjczsKKworCWNzID0gaW50ZWxfcmlu Z19iZWdpbihycSwgZ2VuID49IDggPyAxMCA6IDYpOworCWlmIChJU19FUlIoY3MpKQorCQlyZXR1 cm4gUFRSX0VSUihjcyk7CisKKwlpZiAoZ2VuID49IDkgJiYgIXdhXzEyMDk2NDQ2MTFfYXBwbGll cyhnZW4sIHNpemUpKSB7CisJCSpjcysrID0gR0VOOV9YWV9GQVNUX0NPUFlfQkxUX0NNRCB8ICgx MCAtIDIpOworCQkqY3MrKyA9IEJMVF9ERVBUSF8zMiB8IFBBR0VfU0laRTsKKwkJKmNzKysgPSAw OworCQkqY3MrKyA9IHNpemUgPj4gUEFHRV9TSElGVCA8PCAxNiB8IFBBR0VfU0laRSAvIDQ7CisJ CSpjcysrID0gQ0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBpbnN0YW5jZTsK KwkJKmNzKysgPSAwOworCQkqY3MrKyA9IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOyAvKiBzcmMg b2Zmc2V0ICovCisJCSpjcysrID0gaW5zdGFuY2U7CisJfSBlbHNlIGlmIChnZW4gPj0gOCkgewor CQkqY3MrKyA9IFhZX1NSQ19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICgxMCAtIDIp OworCQkqY3MrKyA9IEJMVF9ERVBUSF8zMiB8IEJMVF9ST1BfU1JDX0NPUFkgfCBQQUdFX1NJWkU7 CisJCSpjcysrID0gMDsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdF X1NJWkUgLyA0OworCQkqY3MrKyA9IENIVU5LX1NaOyAvKiBkc3Qgb2Zmc2V0ICovCisJCSpjcysr ID0gaW5zdGFuY2U7CisJCSpjcysrID0gMDsKKwkJKmNzKysgPSBQQUdFX1NJWkU7CisJCSpjcysr ID0gMDsgLyogc3JjIG9mZnNldCAqLworCQkqY3MrKyA9IGluc3RhbmNlOworCX0gZWxzZSB7CisJ CUdFTV9CVUdfT04oaW5zdGFuY2UpOworCQkqY3MrKyA9IFNSQ19DT1BZX0JMVF9DTUQgfCBCTFRf V1JJVEVfUkdCQSB8ICg2IC0gMik7CisJCSpjcysrID0gQkxUX0RFUFRIXzMyIHwgQkxUX1JPUF9T UkNfQ09QWSB8IFBBR0VfU0laRTsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYg fCBQQUdFX1NJWkU7CisJCSpjcysrID0gQ0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNz KysgPSBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsgLyogc3JjIG9mZnNldCAqLworCX0KKworCWlu dGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCXJldHVybiAwOworfQorCitpbnQKK2ludGVsX2Nv bnRleHRfbWlncmF0ZV9jb3B5KHN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZSwKKwkJCSAgIHN0cnVj dCBkbWFfZmVuY2UgKmF3YWl0LAorCQkJICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkg ICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgc3JjX2NhY2hlX2xldmVsLAorCQkJICAgYm9vbCBzcmNf aXNfbG1lbSwKKwkJCSAgIHN0cnVjdCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgZW51bSBpOTE1 X2NhY2hlX2xldmVsIGRzdF9jYWNoZV9sZXZlbCwKKwkJCSAgIGJvb2wgZHN0X2lzX2xtZW0sCisJ CQkgICBzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KQoreworCXN0cnVjdCBzZ3RfZG1hIGl0X3Ny YyA9IHNnX3NndChzcmMpLCBpdF9kc3QgPSBzZ19zZ3QoZHN0KTsKKwlzdHJ1Y3QgaTkxNV9yZXF1 ZXN0ICpycTsKKwlpbnQgZXJyOworCisJKm91dCA9IE5VTEw7CisKKwkvKiBHRU1fQlVHX09OKGNl LT52bSAhPSBtaWdyYXRlX3ZtKTsgKi8KKworCUdFTV9CVUdfT04oY2UtPnJpbmctPnNpemUgPCBT Wl82NEspOworCisJZG8geworCQlpbnQgbGVuOworCisJCXJxID0gaTkxNV9yZXF1ZXN0X2NyZWF0 ZShjZSk7CisJCWlmIChJU19FUlIocnEpKSB7CisJCQllcnIgPSBQVFJfRVJSKHJxKTsKKwkJCWdv dG8gb3V0X2NlOworCQl9CisKKwkJaWYgKGF3YWl0KSB7CisJCQllcnIgPSBpOTE1X3JlcXVlc3Rf YXdhaXRfZG1hX2ZlbmNlKHJxLCBhd2FpdCk7CisJCQlpZiAoZXJyKQorCQkJCWdvdG8gb3V0X3Jx OworCisJCQlpZiAocnEtPmVuZ2luZS0+ZW1pdF9pbml0X2JyZWFkY3J1bWIpIHsKKwkJCQllcnIg PSBycS0+ZW5naW5lLT5lbWl0X2luaXRfYnJlYWRjcnVtYihycSk7CisJCQkJaWYgKGVycikKKwkJ CQkJZ290byBvdXRfcnE7CisJCQl9CisKKwkJCWF3YWl0ID0gTlVMTDsKKwkJfQorCisJCS8qIFRo ZSBQVEUgdXBkYXRlcyArIGNvcHkgbXVzdCBub3QgYmUgaW50ZXJydXB0ZWQuICovCisJCWVyciA9 IGVtaXRfbm9fYXJiaXRyYXRpb24ocnEpOworCQlpZiAoZXJyKQorCQkJZ290byBvdXRfcnE7CisK KwkJbGVuID0gZW1pdF9wdGUocnEsICZpdF9zcmMsIHNyY19jYWNoZV9sZXZlbCwgc3JjX2lzX2xt ZW0sIDAsCisJCQkgICAgICAgQ0hVTktfU1opOworCQlpZiAobGVuIDw9IDApIHsKKwkJCWVyciA9 IGxlbjsKKwkJCWdvdG8gb3V0X3JxOworCQl9CisKKwkJZXJyID0gZW1pdF9wdGUocnEsICZpdF9k c3QsIGRzdF9jYWNoZV9sZXZlbCwgZHN0X2lzX2xtZW0sCisJCQkgICAgICAgQ0hVTktfU1osIGxl bik7CisJCWlmIChlcnIgPCAwKQorCQkJZ290byBvdXRfcnE7CisJCWlmIChlcnIgPCBsZW4pIHsK KwkJCWVyciA9IC1FSU5WQUw7CisJCQlnb3RvIG91dF9ycTsKKwkJfQorCisJCWVyciA9IHJxLT5l bmdpbmUtPmVtaXRfZmx1c2gocnEsIEVNSVRfSU5WQUxJREFURSk7CisJCWlmIChlcnIpCisJCQln b3RvIG91dF9ycTsKKworCQllcnIgPSBlbWl0X2NvcHkocnEsIGxlbik7CisKKwkJLyogQXJiaXRy YXRpb24gaXMgcmUtZW5hYmxlZCBiZXR3ZWVuIHJlcXVlc3RzLiAqLworb3V0X3JxOgorCQlpZiAo Km91dCkKKwkJCWk5MTVfcmVxdWVzdF9wdXQoKm91dCk7CisJCSpvdXQgPSBpOTE1X3JlcXVlc3Rf Z2V0KHJxKTsKKwkJaTkxNV9yZXF1ZXN0X2FkZChycSk7CisJCWlmIChlcnIgfHwgIWl0X3NyYy5z ZyB8fCAhc2dfZG1hX2xlbihpdF9zcmMuc2cpKQorCQkJYnJlYWs7CisKKwkJY29uZF9yZXNjaGVk KCk7CisJfSB3aGlsZSAoMSk7CisKK291dF9jZToKKwlyZXR1cm4gZXJyOworfQorCitpbnQgaW50 ZWxfbWlncmF0ZV9jb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLAorCQkgICAgICAgc3RydWN0 IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCSAgICAgICBzdHJ1Y3QgZG1hX2ZlbmNlICphd2FpdCwK KwkJICAgICAgIHN0cnVjdCBzY2F0dGVybGlzdCAqc3JjLAorCQkgICAgICAgZW51bSBpOTE1X2Nh Y2hlX2xldmVsIHNyY19jYWNoZV9sZXZlbCwKKwkJICAgICAgIGJvb2wgc3JjX2lzX2xtZW0sCisJ CSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKmRzdCwKKwkJICAgICAgIGVudW0gaTkxNV9jYWNo ZV9sZXZlbCBkc3RfY2FjaGVfbGV2ZWwsCisJCSAgICAgICBib29sIGRzdF9pc19sbWVtLAorCQkg ICAgICAgc3RydWN0IGk5MTVfcmVxdWVzdCAqKm91dCkKK3sKKwlzdHJ1Y3QgaW50ZWxfY29udGV4 dCAqY2U7CisJaW50IGVycjsKKworCSpvdXQgPSBOVUxMOworCWlmICghbS0+Y29udGV4dCkKKwkJ cmV0dXJuIC1FTk9ERVY7CisKKwljZSA9IGludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQobSk7 CisJaWYgKElTX0VSUihjZSkpCisJCWNlID0gaW50ZWxfY29udGV4dF9nZXQobS0+Y29udGV4dCk7 CisJR0VNX0JVR19PTihJU19FUlIoY2UpKTsKKworCWVyciA9IGludGVsX2NvbnRleHRfcGluX3d3 KGNlLCB3dyk7CisJaWYgKGVycikKKwkJZ290byBvdXQ7CisKKwllcnIgPSBpbnRlbF9jb250ZXh0 X21pZ3JhdGVfY29weShjZSwgYXdhaXQsCisJCQkJCSBzcmMsIHNyY19jYWNoZV9sZXZlbCwgc3Jj X2lzX2xtZW0sCisJCQkJCSBkc3QsIGRzdF9jYWNoZV9sZXZlbCwgZHN0X2lzX2xtZW0sCisJCQkJ CSBvdXQpOworCisJaW50ZWxfY29udGV4dF91bnBpbihjZSk7CitvdXQ6CisJaW50ZWxfY29udGV4 dF9wdXQoY2UpOworCXJldHVybiBlcnI7Cit9CisKK3ZvaWQgaW50ZWxfbWlncmF0ZV9maW5pKHN0 cnVjdCBpbnRlbF9taWdyYXRlICptKQoreworCXN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZTsKKwor CWNlID0gZmV0Y2hfYW5kX3plcm8oJm0tPmNvbnRleHQpOworCWlmICghY2UpCisJCXJldHVybjsK KworCWludGVsX2VuZ2luZV9kZXN0cm95X3Bpbm5lZF9jb250ZXh0KGNlKTsKK30KKworI2lmIElT X0VOQUJMRUQoQ09ORklHX0RSTV9JOTE1X1NFTEZURVNUKQorI2luY2x1ZGUgInNlbGZ0ZXN0X21p Z3JhdGUuYyIKKyNlbmRpZgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50 ZWxfbWlncmF0ZS5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5oCm5l dyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwMC4uMzJjNjExOTBlZDczCi0tLSAv ZGV2L251bGwKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5oCkBA IC0wLDAgKzEsNDUgQEAKKy8qIFNQRFgtTGljZW5zZS1JZGVudGlmaWVyOiBNSVQgKi8KKy8qCisg KiBDb3B5cmlnaHQgwqkgMjAyMCBJbnRlbCBDb3Jwb3JhdGlvbgorICovCisKKyNpZm5kZWYgX19J TlRFTF9NSUdSQVRFX18KKyNkZWZpbmUgX19JTlRFTF9NSUdSQVRFX18KKworI2luY2x1ZGUgImlu dGVsX21pZ3JhdGVfdHlwZXMuaCIKKworc3RydWN0IGRtYV9mZW5jZTsKK3N0cnVjdCBpOTE1X3Jl cXVlc3Q7CitzdHJ1Y3QgaTkxNV9nZW1fd3dfY3R4Oworc3RydWN0IGludGVsX2d0Oworc3RydWN0 IHNjYXR0ZXJsaXN0OworZW51bSBpOTE1X2NhY2hlX2xldmVsOworCitpbnQgaW50ZWxfbWlncmF0 ZV9pbml0KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0KTsKKwor c3RydWN0IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQoc3RydWN0 IGludGVsX21pZ3JhdGUgKm0pOworCitpbnQgaW50ZWxfbWlncmF0ZV9jb3B5KHN0cnVjdCBpbnRl bF9taWdyYXRlICptLAorCQkgICAgICAgc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCSAg ICAgICBzdHJ1Y3QgZG1hX2ZlbmNlICphd2FpdCwKKwkJICAgICAgIHN0cnVjdCBzY2F0dGVybGlz dCAqc3JjLAorCQkgICAgICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIHNyY19jYWNoZV9sZXZlbCwK KwkJICAgICAgIGJvb2wgc3JjX2lzX2xtZW0sCisJCSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3Qg KmRzdCwKKwkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBkc3RfY2FjaGVfbGV2ZWwsCisJ CSAgICAgICBib29sIGRzdF9pc19sbWVtLAorCQkgICAgICAgc3RydWN0IGk5MTVfcmVxdWVzdCAq Km91dCk7CisKK2ludCBpbnRlbF9jb250ZXh0X21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfY29u dGV4dCAqY2UsCisJCQkgICAgICAgc3RydWN0IGRtYV9mZW5jZSAqYXdhaXQsCisJCQkgICAgICAg c3RydWN0IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkgICAgICAgZW51bSBpOTE1X2NhY2hlX2xldmVs IHNyY19jYWNoZV9sZXZlbCwKKwkJCSAgICAgICBib29sIHNyY19pc19sbWVtLAorCQkJICAgICAg IHN0cnVjdCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZl bCBkc3RfY2FjaGVfbGV2ZWwsCisJCQkgICAgICAgYm9vbCBkc3RfaXNfbG1lbSwKKwkJCSAgICAg ICBzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KTsKKwordm9pZCBpbnRlbF9taWdyYXRlX2Zpbmko c3RydWN0IGludGVsX21pZ3JhdGUgKm0pOworCisjZW5kaWYgLyogX19JTlRFTF9NSUdSQVRFX18g Ki8KZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlw ZXMuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaApuZXcg ZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLmQ5ODIzMDU5N2Y0MgotLS0gL2Rl di9udWxsCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMu aApAQCAtMCwwICsxLDE1IEBACisvKiBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUICovCisv KgorICogQ29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaWZuZGVm IF9fSU5URUxfTUlHUkFURV9UWVBFU19fCisjZGVmaW5lIF9fSU5URUxfTUlHUkFURV9UWVBFU19f CisKK3N0cnVjdCBpbnRlbF9jb250ZXh0OworCitzdHJ1Y3QgaW50ZWxfbWlncmF0ZSB7CisJc3Ry dWN0IGludGVsX2NvbnRleHQgKmNvbnRleHQ7Cit9OworCisjZW5kaWYgLyogX19JTlRFTF9NSUdS QVRFX1RZUEVTX18gKi8KZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVs X3JpbmcuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3JpbmcuaAppbmRleCBkYmY1 ZjE0YTEzNmYuLjFiMzJkYWRmYjhjMyAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJtL2k5MTUv Z3QvaW50ZWxfcmluZy5oCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3Jpbmcu aApAQCAtNDksNiArNDksNyBAQCBzdGF0aWMgaW5saW5lIHZvaWQgaW50ZWxfcmluZ19hZHZhbmNl KHN0cnVjdCBpOTE1X3JlcXVlc3QgKnJxLCB1MzIgKmNzKQogCSAqIGludGVsX3JpbmdfYmVnaW4o KSkuCiAJICovCiAJR0VNX0JVR19PTigocnEtPnJpbmctPnZhZGRyICsgcnEtPnJpbmctPmVtaXQp ICE9IGNzKTsKKwlHRU1fQlVHX09OKCFJU19BTElHTkVEKHJxLT5yaW5nLT5lbWl0LCA4KSk7IC8q IFJJTkdfVEFJTCBxd29yZCBhbGlnbiAqLwogfQogCiBzdGF0aWMgaW5saW5lIHUzMiBpbnRlbF9y aW5nX3dyYXAoY29uc3Qgc3RydWN0IGludGVsX3JpbmcgKnJpbmcsIHUzMiBwb3MpCmRpZmYgLS1n aXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMgYi9kcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMKbmV3IGZpbGUgbW9kZSAxMDA2NDQK aW5kZXggMDAwMDAwMDAwMDAwLi45Nzg0ZDE0OWViZjEKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2 ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMKQEAgLTAsMCArMSwyOTEgQEAK Ky8vIFNQRFgtTGljZW5zZS1JZGVudGlmaWVyOiBNSVQKKy8qCisgKiBDb3B5cmlnaHQgwqkgMjAy MCBJbnRlbCBDb3Jwb3JhdGlvbgorICovCisKKyNpbmNsdWRlICJzZWxmdGVzdHMvaTkxNV9yYW5k b20uaCIKKworc3RhdGljIGNvbnN0IHVuc2lnbmVkIGludCBzaXplc1tdID0geworCVNaXzRLLAor CVNaXzY0SywKKwlTWl8yTSwKKwlDSFVOS19TWiAtIFNaXzRLLAorCUNIVU5LX1NaLAorCUNIVU5L X1NaICsgU1pfNEssCisJU1pfNjRNLAorfTsKKworc3RhdGljIHN0cnVjdCBkcm1faTkxNV9nZW1f b2JqZWN0ICoKK2NyZWF0ZV9sbWVtX29yX2ludGVybmFsKHN0cnVjdCBkcm1faTkxNV9wcml2YXRl ICppOTE1LCBzaXplX3Qgc2l6ZSkKK3sKKwlpZiAoSEFTX0xNRU0oaTkxNSkpIHsKKwkJc3RydWN0 IGRybV9pOTE1X2dlbV9vYmplY3QgKm9iajsKKworCQlvYmogPSBpOTE1X2dlbV9vYmplY3RfY3Jl YXRlX2xtZW0oaTkxNSwgc2l6ZSwgMCk7CisJCWlmICghSVNfRVJSKG9iaikpCisJCQlyZXR1cm4g b2JqOworCX0KKworCXJldHVybiBpOTE1X2dlbV9vYmplY3RfY3JlYXRlX2ludGVybmFsKGk5MTUs IHNpemUpOworfQorCitzdGF0aWMgaW50IGNvcHkoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3Jh dGUsCisJCWludCAoKmZuKShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSAgc3Ry dWN0IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0 ICpzcmMsCisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpkc3QsCisJCQkgIHN0cnVj dCBpOTE1X3JlcXVlc3QgKipvdXQpLAorCQl1MzIgc3osIHN0cnVjdCBybmRfc3RhdGUgKnBybmcp Cit7CisJc3RydWN0IGRybV9pOTE1X3ByaXZhdGUgKmk5MTUgPSBtaWdyYXRlLT5jb250ZXh0LT5l bmdpbmUtPmk5MTU7CisJc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKnNyYywgKmRzdDsKKwlz dHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycTsKKwlzdHJ1Y3QgaTkxNV9nZW1fd3dfY3R4IHd3OworCXUz MiAqdmFkZHI7CisJaW50IGVyciA9IDA7CisJaW50IGk7CisKKwlzcmMgPSBjcmVhdGVfbG1lbV9v cl9pbnRlcm5hbChpOTE1LCBzeik7CisJaWYgKElTX0VSUihzcmMpKQorCQlyZXR1cm4gMDsKKwor CWRzdCA9IGk5MTVfZ2VtX29iamVjdF9jcmVhdGVfaW50ZXJuYWwoaTkxNSwgc3opOworCWlmIChJ U19FUlIoZHN0KSkKKwkJZ290byBlcnJfZnJlZV9zcmM7CisKKwlmb3JfaTkxNV9nZW1fd3coJnd3 LCBlcnIsIHRydWUpIHsKKwkJZXJyID0gaTkxNV9nZW1fb2JqZWN0X2xvY2soc3JjLCAmd3cpOwor CQlpZiAoZXJyKQorCQkJY29udGludWU7CisKKwkJZXJyID0gaTkxNV9nZW1fb2JqZWN0X2xvY2so ZHN0LCAmd3cpOworCQlpZiAoZXJyKQorCQkJY29udGludWU7CisKKwkJdmFkZHIgPSBpOTE1X2dl bV9vYmplY3RfcGluX21hcChzcmMsIEk5MTVfTUFQX1dDKTsKKwkJaWYgKElTX0VSUih2YWRkcikp IHsKKwkJCWVyciA9IFBUUl9FUlIodmFkZHIpOworCQkJY29udGludWU7CisJCX0KKworCQlmb3Ig KGkgPSAwOyBpIDwgc3ogLyBzaXplb2YodTMyKTsgaSsrKQorCQkJdmFkZHJbaV0gPSBpOworCQlp OTE1X2dlbV9vYmplY3RfZmx1c2hfbWFwKHNyYyk7CisKKwkJdmFkZHIgPSBpOTE1X2dlbV9vYmpl Y3RfcGluX21hcChkc3QsIEk5MTVfTUFQX1dDKTsKKwkJaWYgKElTX0VSUih2YWRkcikpIHsKKwkJ CWVyciA9IFBUUl9FUlIodmFkZHIpOworCQkJZ290byB1bnBpbl9zcmM7CisJCX0KKworCQlmb3Ig KGkgPSAwOyBpIDwgc3ogLyBzaXplb2YodTMyKTsgaSsrKQorCQkJdmFkZHJbaV0gPSB+aTsKKwkJ aTkxNV9nZW1fb2JqZWN0X2ZsdXNoX21hcChkc3QpOworCisJCWVyciA9IGZuKG1pZ3JhdGUsICZ3 dywgc3JjLCBkc3QsICZycSk7CisJCWlmICghZXJyKQorCQkJY29udGludWU7CisKKwkJaWYgKGVy ciAhPSAtRURFQURMSyAmJiBlcnIgIT0gLUVJTlRSICYmIGVyciAhPSAtRVJFU1RBUlRTWVMpCisJ CQlwcl9lcnIoIiVwcyBmYWlsZWQsIHNpemU6ICV1XG4iLCBmbiwgc3opOworCQlpZiAocnEpIHsK KwkJCWk5MTVfcmVxdWVzdF93YWl0KHJxLCAwLCBIWik7CisJCQlpOTE1X3JlcXVlc3RfcHV0KHJx KTsKKwkJfQorCQlpOTE1X2dlbV9vYmplY3RfdW5waW5fbWFwKGRzdCk7Cit1bnBpbl9zcmM6CisJ CWk5MTVfZ2VtX29iamVjdF91bnBpbl9tYXAoc3JjKTsKKwl9CisJaWYgKGVycikKKwkJZ290byBl cnJfb3V0OworCisJaWYgKHJxKSB7CisJCWlmIChpOTE1X3JlcXVlc3Rfd2FpdChycSwgMCwgSFop IDwgMCkgeworCQkJcHJfZXJyKCIlcHMgdGltZWQgb3V0LCBzaXplOiAldVxuIiwgZm4sIHN6KTsK KwkJCWVyciA9IC1FVElNRTsKKwkJfQorCQlpOTE1X3JlcXVlc3RfcHV0KHJxKTsKKwl9CisKKwlm b3IgKGkgPSAwOyAhZXJyICYmIGkgPCBzeiAvIFBBR0VfU0laRTsgaSsrKSB7CisJCWludCB4ID0g aSAqIDEwMjQgKyBpOTE1X3ByYW5kb21fdTMyX21heF9zdGF0ZSgxMDI0LCBwcm5nKTsKKworCQlp ZiAodmFkZHJbeF0gIT0geCkgeworCQkJcHJfZXJyKCIlcHMgZmFpbGVkLCBzaXplOiAldSwgb2Zm c2V0OiAlenVcbiIsCisJCQkgICAgICAgZm4sIHN6LCB4ICogc2l6ZW9mKHUzMikpOworCQkJaWd0 X2hleGR1bXAodmFkZHIgKyBpICogMTAyNCwgNDA5Nik7CisJCQllcnIgPSAtRUlOVkFMOworCQl9 CisJfQorCisJaTkxNV9nZW1fb2JqZWN0X3VucGluX21hcChkc3QpOworCWk5MTVfZ2VtX29iamVj dF91bnBpbl9tYXAoc3JjKTsKKworZXJyX291dDoKKwlpOTE1X2dlbV9vYmplY3RfcHV0KGRzdCk7 CitlcnJfZnJlZV9zcmM6CisJaTkxNV9nZW1fb2JqZWN0X3B1dChzcmMpOworCisJcmV0dXJuIGVy cjsKK30KKworc3RhdGljIGludCBfX21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAq bWlncmF0ZSwKKwkJCSAgc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCQkgIHN0cnVjdCBk cm1faTkxNV9nZW1fb2JqZWN0ICpzcmMsCisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0 ICpkc3QsCisJCQkgIHN0cnVjdCBpOTE1X3JlcXVlc3QgKipvdXQpCit7CisJcmV0dXJuIGludGVs X21pZ3JhdGVfY29weShtaWdyYXRlLCB3dywgTlVMTCwKKwkJCQkgIHNyYy0+bW0ucGFnZXMtPnNn bCwgc3JjLT5jYWNoZV9sZXZlbCwKKwkJCQkgIGk5MTVfZ2VtX29iamVjdF9pc19sbWVtKHNyYyks CisJCQkJICBkc3QtPm1tLnBhZ2VzLT5zZ2wsIGRzdC0+Y2FjaGVfbGV2ZWwsCisJCQkJICBpOTE1 X2dlbV9vYmplY3RfaXNfbG1lbShkc3QpLAorCQkJCSAgb3V0KTsKK30KKworc3RhdGljIGludCBf X2dsb2JhbF9jb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlLAorCQkJIHN0cnVjdCBp OTE1X2dlbV93d19jdHggKnd3LAorCQkJIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpzcmMs CisJCQkgc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKmRzdCwKKwkJCSBzdHJ1Y3QgaTkxNV9y ZXF1ZXN0ICoqb3V0KQoreworCXJldHVybiBpbnRlbF9jb250ZXh0X21pZ3JhdGVfY29weShtaWdy YXRlLT5jb250ZXh0LCBOVUxMLAorCQkJCQkgIHNyYy0+bW0ucGFnZXMtPnNnbCwgc3JjLT5jYWNo ZV9sZXZlbCwKKwkJCQkJICBpOTE1X2dlbV9vYmplY3RfaXNfbG1lbShzcmMpLAorCQkJCQkgIGRz dC0+bW0ucGFnZXMtPnNnbCwgZHN0LT5jYWNoZV9sZXZlbCwKKwkJCQkJICBpOTE1X2dlbV9vYmpl Y3RfaXNfbG1lbShkc3QpLAorCQkJCQkgIG91dCk7Cit9CisKK3N0YXRpYyBpbnQKK21pZ3JhdGVf Y29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwgdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0 YXRlICpwcm5nKQoreworCXJldHVybiBjb3B5KG1pZ3JhdGUsIF9fbWlncmF0ZV9jb3B5LCBzeiwg cHJuZyk7Cit9CisKK3N0YXRpYyBpbnQKK2dsb2JhbF9jb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRl ICptaWdyYXRlLCB1MzIgc3osIHN0cnVjdCBybmRfc3RhdGUgKnBybmcpCit7CisJcmV0dXJuIGNv cHkobWlncmF0ZSwgX19nbG9iYWxfY29weSwgc3osIHBybmcpOworfQorCitzdGF0aWMgaW50IGxp dmVfbWlncmF0ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWln cmF0ZSA9IGFyZzsKKwlzdHJ1Y3QgZHJtX2k5MTVfcHJpdmF0ZSAqaTkxNSA9IG1pZ3JhdGUtPmNv bnRleHQtPmVuZ2luZS0+aTkxNTsKKwlJOTE1X1JORF9TVEFURShwcm5nKTsKKwlpbnQgaTsKKwor CWZvciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKHNpemVzKTsgaSsrKSB7CisJCWludCBlcnI7CisK KwkJZXJyID0gbWlncmF0ZV9jb3B5KG1pZ3JhdGUsIHNpemVzW2ldLCAmcHJuZyk7CisJCWlmIChl cnIgPT0gMCkKKwkJCWVyciA9IGdsb2JhbF9jb3B5KG1pZ3JhdGUsIHNpemVzW2ldLCAmcHJuZyk7 CisJCWk5MTVfZ2VtX2RyYWluX2ZyZWVkX29iamVjdHMoaTkxNSk7CisJCWlmIChlcnIpCisJCQly ZXR1cm4gZXJyOworCX0KKworCXJldHVybiAwOworfQorCitzdHJ1Y3QgdGhyZWFkZWRfbWlncmF0 ZSB7CisJc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGU7CisJc3RydWN0IHRhc2tfc3RydWN0 ICp0c2s7CisJc3RydWN0IHJuZF9zdGF0ZSBwcm5nOworfTsKKworc3RhdGljIGludCB0aHJlYWRl ZF9taWdyYXRlKHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlLAorCQkJICAgIGludCAoKmZu KSh2b2lkICphcmcpLAorCQkJICAgIHVuc2lnbmVkIGludCBmbGFncykKK3sKKwljb25zdCB1bnNp Z25lZCBpbnQgbl9jcHVzID0gbnVtX29ubGluZV9jcHVzKCkgKyAxOworCXN0cnVjdCB0aHJlYWRl ZF9taWdyYXRlICp0aHJlYWQ7CisJSTkxNV9STkRfU1RBVEUocHJuZyk7CisJdW5zaWduZWQgaW50 IGk7CisJaW50IGVyciA9IDA7CisKKwl0aHJlYWQgPSBrY2FsbG9jKG5fY3B1cywgc2l6ZW9mKCp0 aHJlYWQpLCBHRlBfS0VSTkVMKTsKKwlpZiAoIXRocmVhZCkKKwkJcmV0dXJuIDA7CisKKwlmb3Ig KGkgPSAwOyBpIDwgbl9jcHVzOyArK2kpIHsKKwkJc3RydWN0IHRhc2tfc3RydWN0ICp0c2s7CisK KwkJdGhyZWFkW2ldLm1pZ3JhdGUgPSBtaWdyYXRlOworCQl0aHJlYWRbaV0ucHJuZyA9CisJCQlJ OTE1X1JORF9TVEFURV9JTklUSUFMSVpFUihwcmFuZG9tX3UzMl9zdGF0ZSgmcHJuZykpOworCisJ CXRzayA9IGt0aHJlYWRfcnVuKGZuLCAmdGhyZWFkW2ldLCAiaWd0LSVkIiwgaSk7CisJCWlmIChJ U19FUlIodHNrKSkgeworCQkJZXJyID0gUFRSX0VSUih0c2spOworCQkJYnJlYWs7CisJCX0KKwor CQlnZXRfdGFza19zdHJ1Y3QodHNrKTsKKwkJdGhyZWFkW2ldLnRzayA9IHRzazsKKwl9CisKKwlt c2xlZXAoMTApOyAvKiBzdGFydCBhbGwgdGhyZWFkcyBiZWZvcmUgd2Uga3RocmVhZF9zdG9wKCkg Ki8KKworCWZvciAoaSA9IDA7IGkgPCBuX2NwdXM7ICsraSkgeworCQlzdHJ1Y3QgdGFza19zdHJ1 Y3QgKnRzayA9IHRocmVhZFtpXS50c2s7CisJCWludCBzdGF0dXM7CisKKwkJaWYgKElTX0VSUl9P Ul9OVUxMKHRzaykpCisJCQljb250aW51ZTsKKworCQlzdGF0dXMgPSBrdGhyZWFkX3N0b3AodHNr KTsKKwkJaWYgKHN0YXR1cyAmJiAhZXJyKQorCQkJZXJyID0gc3RhdHVzOworCisJCXB1dF90YXNr X3N0cnVjdCh0c2spOworCX0KKworCWtmcmVlKHRocmVhZCk7CisJcmV0dXJuIGVycjsKK30KKwor c3RhdGljIGludCBfX3RocmVhZF9taWdyYXRlX2NvcHkodm9pZCAqYXJnKQoreworCXN0cnVjdCB0 aHJlYWRlZF9taWdyYXRlICp0bSA9IGFyZzsKKworCXJldHVybiBtaWdyYXRlX2NvcHkodG0tPm1p Z3JhdGUsIDIgKiBDSFVOS19TWiwgJnRtLT5wcm5nKTsKK30KKworc3RhdGljIGludCB0aHJlYWRf bWlncmF0ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlyZXR1cm4gdGhyZWFkZWRfbWlncmF0ZShhcmcs IF9fdGhyZWFkX21pZ3JhdGVfY29weSwgMCk7Cit9CisKK3N0YXRpYyBpbnQgX190aHJlYWRfZ2xv YmFsX2NvcHkodm9pZCAqYXJnKQoreworCXN0cnVjdCB0aHJlYWRlZF9taWdyYXRlICp0bSA9IGFy ZzsKKworCXJldHVybiBnbG9iYWxfY29weSh0bS0+bWlncmF0ZSwgMiAqIENIVU5LX1NaLCAmdG0t PnBybmcpOworfQorCitzdGF0aWMgaW50IHRocmVhZF9nbG9iYWxfY29weSh2b2lkICphcmcpCit7 CisJcmV0dXJuIHRocmVhZGVkX21pZ3JhdGUoYXJnLCBfX3RocmVhZF9nbG9iYWxfY29weSwgMCk7 Cit9CisKK2ludCBpbnRlbF9taWdyYXRlX2xpdmVfc2VsZnRlc3RzKHN0cnVjdCBkcm1faTkxNV9w cml2YXRlICppOTE1KQoreworCXN0YXRpYyBjb25zdCBzdHJ1Y3QgaTkxNV9zdWJ0ZXN0IHRlc3Rz W10gPSB7CisJCVNVQlRFU1QobGl2ZV9taWdyYXRlX2NvcHkpLAorCQlTVUJURVNUKHRocmVhZF9t aWdyYXRlX2NvcHkpLAorCQlTVUJURVNUKHRocmVhZF9nbG9iYWxfY29weSksCisJfTsKKwlzdHJ1 Y3QgaW50ZWxfbWlncmF0ZSBtOworCWludCBlcnI7CisKKwlpZiAoaW50ZWxfbWlncmF0ZV9pbml0 KCZtLCAmaTkxNS0+Z3QpKQorCQlyZXR1cm4gMDsKKworCWVyciA9IGk5MTVfc3VidGVzdHModGVz dHMsICZtKTsKKwlpbnRlbF9taWdyYXRlX2ZpbmkoJm0pOworCisJcmV0dXJuIGVycjsKK30KZGlm ZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRl c3RzLmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0 cy5oCmluZGV4IGE5MmMwZTliN2U2Yi4uYmU1ZTAxOTFlYWVhIDEwMDY0NAotLS0gYS9kcml2ZXJz L2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oCisrKyBiL2RyaXZl cnMvZ3B1L2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRlc3RzLmgKQEAgLTI2LDYg KzI2LDcgQEAgc2VsZnRlc3QoZ3RfbW9jcywgaW50ZWxfbW9jc19saXZlX3NlbGZ0ZXN0cykKIHNl bGZ0ZXN0KGd0X3BtLCBpbnRlbF9ndF9wbV9saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KGd0X2hl YXJ0YmVhdCwgaW50ZWxfaGVhcnRiZWF0X2xpdmVfc2VsZnRlc3RzKQogc2VsZnRlc3QocmVxdWVz dHMsIGk5MTVfcmVxdWVzdF9saXZlX3NlbGZ0ZXN0cykKK3NlbGZ0ZXN0KG1pZ3JhdGUsIGludGVs X21pZ3JhdGVfbGl2ZV9zZWxmdGVzdHMpCiBzZWxmdGVzdChhY3RpdmUsIGk5MTVfYWN0aXZlX2xp dmVfc2VsZnRlc3RzKQogc2VsZnRlc3Qob2JqZWN0cywgaTkxNV9nZW1fb2JqZWN0X2xpdmVfc2Vs ZnRlc3RzKQogc2VsZnRlc3QobW1hbiwgaTkxNV9nZW1fbW1hbl9saXZlX3NlbGZ0ZXN0cykKLS0g CjIuMzEuMQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K SW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0 dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30C7DC48BCD for ; Wed, 9 Jun 2021 06:35:13 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0285B61181 for ; Wed, 9 Jun 2021 06:35:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0285B61181 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C96CE6E221; Wed, 9 Jun 2021 06:35:05 +0000 (UTC) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by gabe.freedesktop.org (Postfix) with ESMTPS id 281606E212; Wed, 9 Jun 2021 06:35:02 +0000 (UTC) IronPort-SDR: T6UYHeithRGmwm75uuJhlCstpSbFvssdr37t8LVKuUzmD1ZUnXoZbIeaJKfhdR6eHD/QE4ud97 OhaXMkktgboQ== X-IronPort-AV: E=McAfee;i="6200,9189,10009"; a="268868907" X-IronPort-AV: E=Sophos;i="5.83,260,1616482800"; d="scan'208";a="268868907" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 23:35:01 -0700 IronPort-SDR: 19dPAPIKm6JiCEhqV887FPrdtc9DgeWVTRQrEWjLCIEIEj/WfDoOo6qVtPwsbiavU2vGXKSK6L QAqszx6CVPIg== X-IronPort-AV: E=Sophos;i="5.83,260,1616482800"; d="scan'208";a="482265833" Received: from ekolpasx-mobl.ccr.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.109]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 23:34:59 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 7/9] drm/i915/gt: Pipelined page migration Date: Wed, 9 Jun 2021 08:34:34 +0200 Message-Id: <20210609063436.284332-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609063436.284332-1-thomas.hellstrom@linux.intel.com> References: <20210609063436.284332-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Chris Wilson If we pipeline the PTE updates and then do the copy of those pages within a single unpreemptible command packet, we can submit the copies and leave them to be scheduled without having to synchronously wait under a global lock. In order to manage migration, we need to preallocate the page tables (and keep them pinned and available for use at any time), causing a bottleneck for migrations as all clients must contend on the limited resources. By inlining the ppGTT updates and performing the blit atomically, each client only owns the PTE while in use, and so we can reschedule individual operations however we see fit. And most importantly, we do not need to take a global lock on the shared vm, and wait until the operation is complete before releasing the lock for others to claim the PTE for themselves. Signed-off-by: Chris Wilson Co-developed-by: Thomas Hellström Signed-off-by: Thomas Hellström --- v2: - Add a TODO for huge LMEM ptes (Pointed out by Matthew Auld) - Use intel_engine_destroy_pinned_context() to properly take the pinned context timeline off the engine list. (CI warning). --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_gpu_commands.h | 2 + drivers/gpu/drm/i915/gt/intel_migrate.c | 544 ++++++++++++++++++ drivers/gpu/drm/i915/gt/intel_migrate.h | 45 ++ drivers/gpu/drm/i915/gt/intel_migrate_types.h | 15 + drivers/gpu/drm/i915/gt/intel_ring.h | 1 + drivers/gpu/drm/i915/gt/selftest_migrate.c | 291 ++++++++++ .../drm/i915/selftests/i915_live_selftests.h | 1 + 9 files changed, 901 insertions(+) create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.c create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.h create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate_types.h create mode 100644 drivers/gpu/drm/i915/gt/selftest_migrate.c diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index 16a5a006cf7c..95bd38e84625 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -108,6 +108,7 @@ gt-y += \ gt/intel_gtt.o \ gt/intel_llc.o \ gt/intel_lrc.o \ + gt/intel_migrate.o \ gt/intel_mocs.o \ gt/intel_ppgtt.o \ gt/intel_rc6.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 36ea9eb52bb5..62f7440bc111 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -188,6 +188,7 @@ intel_write_status_page(struct intel_engine_cs *engine, int reg, u32 value) #define I915_GEM_HWS_PREEMPT_ADDR (I915_GEM_HWS_PREEMPT * sizeof(u32)) #define I915_GEM_HWS_SEQNO 0x40 #define I915_GEM_HWS_SEQNO_ADDR (I915_GEM_HWS_SEQNO * sizeof(u32)) +#define I915_GEM_HWS_MIGRATE (0x42 * sizeof(u32)) #define I915_GEM_HWS_SCRATCH 0x80 #define I915_HWS_CSB_BUF0_INDEX 0x10 diff --git a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h index 2694dbb9967e..1c3af0fc0456 100644 --- a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h +++ b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h @@ -123,8 +123,10 @@ #define MI_SEMAPHORE_SAD_NEQ_SDD (5 << 12) #define MI_SEMAPHORE_TOKEN_MASK REG_GENMASK(9, 5) #define MI_SEMAPHORE_TOKEN_SHIFT 5 +#define MI_STORE_DATA_IMM MI_INSTR(0x20, 0) #define MI_STORE_DWORD_IMM MI_INSTR(0x20, 1) #define MI_STORE_DWORD_IMM_GEN4 MI_INSTR(0x20, 2) +#define MI_STORE_QWORD_IMM_GEN8 (MI_INSTR(0x20, 3) | REG_BIT(21)) #define MI_MEM_VIRTUAL (1 << 22) /* 945,g33,965 */ #define MI_USE_GGTT (1 << 22) /* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1) diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.c b/drivers/gpu/drm/i915/gt/intel_migrate.c new file mode 100644 index 000000000000..70776316863d --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.c @@ -0,0 +1,544 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "i915_drv.h" +#include "intel_context.h" +#include "intel_gpu_commands.h" +#include "intel_gt.h" +#include "intel_gtt.h" +#include "intel_migrate.h" +#include "intel_ring.h" + +struct insert_pte_data { + u64 offset; + bool is_lmem; +}; + +#define CHUNK_SZ SZ_8M /* ~1ms at 8GiB/s preemption delay */ + +static bool engine_supports_migration(struct intel_engine_cs *engine) +{ + if (!engine) + return false; + + /* + * We need the ability to prevent aribtration (MI_ARB_ON_OFF), + * the ability to write PTE using inline data (MI_STORE_DATA) + * and of course the ability to do the block transfer (blits). + */ + GEM_BUG_ON(engine->class != COPY_ENGINE_CLASS); + + return true; +} + +static void insert_pte(struct i915_address_space *vm, + struct i915_page_table *pt, + void *data) +{ + struct insert_pte_data *d = data; + + vm->insert_page(vm, px_dma(pt), d->offset, I915_CACHE_NONE, + d->is_lmem ? PTE_LM : 0); + d->offset += PAGE_SIZE; +} + +static struct i915_address_space *migrate_vm(struct intel_gt *gt) +{ + struct i915_vm_pt_stash stash = {}; + struct i915_ppgtt *vm; + int err; + int i; + + /* + * We construct a very special VM for use by all migration contexts, + * it is kept pinned so that it can be used at any time. As we need + * to pre-allocate the page directories for the migration VM, this + * limits us to only using a small number of prepared vma. + * + * To be able to pipeline and reschedule migration operations while + * avoiding unnecessary contention on the vm itself, the PTE updates + * are inline with the blits. All the blits use the same fixed + * addresses, with the backing store redirection being updated on the + * fly. Only 2 implicit vma are used for all migration operations. + * + * We lay the ppGTT out as: + * + * [0, CHUNK_SZ) -> first object + * [CHUNK_SZ, 2 * CHUNK_SZ) -> second object + * [2 * CHUNK_SZ, 2 * CHUNK_SZ + 2 * CHUNK_SZ >> 9] -> PTE + * + * By exposing the dma addresses of the page directories themselves + * within the ppGTT, we are then able to rewrite the PTE prior to use. + * But the PTE update and subsequent migration operation must be atomic, + * i.e. within the same non-preemptible window so that we do not switch + * to another migration context that overwrites the PTE. + * + * TODO: Add support for huge LMEM PTEs + */ + + vm = i915_ppgtt_create(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + if (!vm->vm.allocate_va_range || !vm->vm.foreach) { + err = -ENODEV; + goto err_vm; + } + + /* + * Each engine instance is assigned its own chunk in the VM, so + * that we can run multiple instances concurrently + */ + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + struct intel_engine_cs *engine; + u64 base = (u64)i << 32; + struct insert_pte_data d = {}; + struct i915_gem_ww_ctx ww; + u64 sz; + + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (!engine_supports_migration(engine)) + continue; + + /* + * We copy in 8MiB chunks. Each PDE covers 2MiB, so we need + * 4x2 page directories for source/destination. + */ + sz = 2 * CHUNK_SZ; + d.offset = base + sz; + + /* + * We need another page directory setup so that we can write + * the 8x512 PTE in each chunk. + */ + sz += (sz >> 12) * sizeof(u64); + + err = i915_vm_alloc_pt_stash(&vm->vm, &stash, sz); + if (err) + goto err_vm; + + for_i915_gem_ww(&ww, err, true) { + err = i915_vm_lock_objects(&vm->vm, &ww); + if (err) + continue; + err = i915_vm_map_pt_stash(&vm->vm, &stash); + if (err) + continue; + + vm->vm.allocate_va_range(&vm->vm, &stash, base, base + sz); + } + i915_vm_free_pt_stash(&vm->vm, &stash); + if (err) + goto err_vm; + + /* Now allow the GPU to rewrite the PTE via its own ppGTT */ + d.is_lmem = i915_gem_object_is_lmem(vm->vm.scratch[0]); + vm->vm.foreach(&vm->vm, base, base + sz, insert_pte, &d); + } + + return &vm->vm; + +err_vm: + i915_vm_put(&vm->vm); + return ERR_PTR(err); +} + +static struct intel_engine_cs *first_copy_engine(struct intel_gt *gt) +{ + struct intel_engine_cs *engine; + int i; + + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + return engine; + } + + return NULL; +} + +static struct intel_context *pinned_context(struct intel_gt *gt) +{ + static struct lock_class_key key; + struct intel_engine_cs *engine; + struct i915_address_space *vm; + struct intel_context *ce; + + engine = first_copy_engine(gt); + if (!engine) + return ERR_PTR(-ENODEV); + + vm = migrate_vm(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + ce = intel_engine_create_pinned_context(engine, vm, SZ_512K, + I915_GEM_HWS_MIGRATE, + &key, "migrate"); + i915_vm_put(ce->vm); + return ce; +} + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt) +{ + struct intel_context *ce; + + memset(m, 0, sizeof(*m)); + + ce = pinned_context(gt); + if (IS_ERR(ce)) + return PTR_ERR(ce); + + m->context = ce; + return 0; +} + +static int random_index(unsigned int max) +{ + return upper_32_bits(mul_u32_u32(get_random_u32(), max)); +} + +static struct intel_context *__migrate_engines(struct intel_gt *gt) +{ + struct intel_engine_cs *engines[MAX_ENGINE_INSTANCE]; + struct intel_engine_cs *engine; + unsigned int count, i; + + count = 0; + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + engines[count++] = engine; + } + + return intel_context_create(engines[random_index(count)]); +} + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m) +{ + struct intel_context *ce; + + /* + * We randomly distribute contexts across the engines upon constrction, + * as they all share the same pinned vm, and so in order to allow + * multiple blits to run in parallel, we must construct each blit + * to use a different range of the vm for its GTT. This has to be + * known at construction, so we can not use the late greedy load + * balancing of the virtual-engine. + */ + ce = __migrate_engines(m->context->engine->gt); + if (IS_ERR(ce)) + return ce; + + ce->ring = __intel_context_ring_size(SZ_256K); + + i915_vm_put(ce->vm); + ce->vm = i915_vm_get(m->context->vm); + + return ce; +} + +static inline struct sgt_dma sg_sgt(struct scatterlist *sg) +{ + dma_addr_t addr = sg_dma_address(sg); + + return (struct sgt_dma){ sg, addr, addr + sg_dma_len(sg) }; +} + +static int emit_no_arbitration(struct i915_request *rq) +{ + u32 *cs; + + cs = intel_ring_begin(rq, 2); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Explicitly disable preemption for this request. */ + *cs++ = MI_ARB_ON_OFF; + *cs++ = MI_NOOP; + intel_ring_advance(rq, cs); + + return 0; +} + +static int emit_pte(struct i915_request *rq, + struct sgt_dma *it, + enum i915_cache_level cache_level, + bool is_lmem, + u64 offset, + int length) +{ + const u64 encode = rq->context->vm->pte_encode(0, cache_level, + is_lmem ? PTE_LM : 0); + struct intel_ring *ring = rq->ring; + int total = 0; + u32 *hdr, *cs; + int pkt; + + GEM_BUG_ON(INTEL_GEN(rq->engine->i915) < 8); + + /* Compute the page directory offset for the target address range */ + offset += (u64)rq->engine->instance << 32; + offset >>= 12; + offset *= sizeof(u64); + offset += 2 * CHUNK_SZ; + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Pack as many PTE updates as possible into a single MI command */ + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); /* as qword elements */ + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + + do { + if (cs - hdr >= pkt) { + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + } + + *cs++ = lower_32_bits(encode | it->dma); + *cs++ = upper_32_bits(encode | it->dma); + + offset += 8; + total += I915_GTT_PAGE_SIZE; + + it->dma += I915_GTT_PAGE_SIZE; + if (it->dma >= it->max) { + it->sg = __sg_next(it->sg); + if (!it->sg || sg_dma_len(it->sg) == 0) + break; + + it->dma = sg_dma_address(it->sg); + it->max = it->dma + sg_dma_len(it->sg); + } + } while (total < length); + + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + return total; +} + +static bool wa_1209644611_applies(int gen, u32 size) +{ + u32 height = size >> PAGE_SHIFT; + + if (gen != 11) + return false; + + return height % 4 == 3 && height <= 8; +} + +static int emit_copy(struct i915_request *rq, int size) +{ + const int gen = INTEL_GEN(rq->engine->i915); + u32 instance = rq->engine->instance; + u32 *cs; + + cs = intel_ring_begin(rq, gen >= 8 ? 10 : 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + if (gen >= 9 && !wa_1209644611_applies(gen, size)) { + *cs++ = GEN9_XY_FAST_COPY_BLT_CMD | (10 - 2); + *cs++ = BLT_DEPTH_32 | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else if (gen >= 8) { + *cs++ = XY_SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (10 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else { + GEM_BUG_ON(instance); + *cs++ = SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (6 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + } + + intel_ring_advance(rq, cs); + return 0; +} + +int +intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct sgt_dma it_src = sg_sgt(src), it_dst = sg_sgt(dst); + struct i915_request *rq; + int err; + + *out = NULL; + + /* GEM_BUG_ON(ce->vm != migrate_vm); */ + + GEM_BUG_ON(ce->ring->size < SZ_64K); + + do { + int len; + + rq = i915_request_create(ce); + if (IS_ERR(rq)) { + err = PTR_ERR(rq); + goto out_ce; + } + + if (await) { + err = i915_request_await_dma_fence(rq, await); + if (err) + goto out_rq; + + if (rq->engine->emit_init_breadcrumb) { + err = rq->engine->emit_init_breadcrumb(rq); + if (err) + goto out_rq; + } + + await = NULL; + } + + /* The PTE updates + copy must not be interrupted. */ + err = emit_no_arbitration(rq); + if (err) + goto out_rq; + + len = emit_pte(rq, &it_src, src_cache_level, src_is_lmem, 0, + CHUNK_SZ); + if (len <= 0) { + err = len; + goto out_rq; + } + + err = emit_pte(rq, &it_dst, dst_cache_level, dst_is_lmem, + CHUNK_SZ, len); + if (err < 0) + goto out_rq; + if (err < len) { + err = -EINVAL; + goto out_rq; + } + + err = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (err) + goto out_rq; + + err = emit_copy(rq, len); + + /* Arbitration is re-enabled between requests. */ +out_rq: + if (*out) + i915_request_put(*out); + *out = i915_request_get(rq); + i915_request_add(rq); + if (err || !it_src.sg || !sg_dma_len(it_src.sg)) + break; + + cond_resched(); + } while (1); + +out_ce: + return err; +} + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct intel_context *ce; + int err; + + *out = NULL; + if (!m->context) + return -ENODEV; + + ce = intel_migrate_create_context(m); + if (IS_ERR(ce)) + ce = intel_context_get(m->context); + GEM_BUG_ON(IS_ERR(ce)); + + err = intel_context_pin_ww(ce, ww); + if (err) + goto out; + + err = intel_context_migrate_copy(ce, await, + src, src_cache_level, src_is_lmem, + dst, dst_cache_level, dst_is_lmem, + out); + + intel_context_unpin(ce); +out: + intel_context_put(ce); + return err; +} + +void intel_migrate_fini(struct intel_migrate *m) +{ + struct intel_context *ce; + + ce = fetch_and_zero(&m->context); + if (!ce) + return; + + intel_engine_destroy_pinned_context(ce); +} + +#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) +#include "selftest_migrate.c" +#endif diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.h b/drivers/gpu/drm/i915/gt/intel_migrate.h new file mode 100644 index 000000000000..32c61190ed73 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE__ +#define __INTEL_MIGRATE__ + +#include "intel_migrate_types.h" + +struct dma_fence; +struct i915_request; +struct i915_gem_ww_ctx; +struct intel_gt; +struct scatterlist; +enum i915_cache_level; + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt); + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m); + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +int intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +void intel_migrate_fini(struct intel_migrate *m); + +#endif /* __INTEL_MIGRATE__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_migrate_types.h b/drivers/gpu/drm/i915/gt/intel_migrate_types.h new file mode 100644 index 000000000000..d98230597f42 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate_types.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE_TYPES__ +#define __INTEL_MIGRATE_TYPES__ + +struct intel_context; + +struct intel_migrate { + struct intel_context *context; +}; + +#endif /* __INTEL_MIGRATE_TYPES__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_ring.h b/drivers/gpu/drm/i915/gt/intel_ring.h index dbf5f14a136f..1b32dadfb8c3 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.h +++ b/drivers/gpu/drm/i915/gt/intel_ring.h @@ -49,6 +49,7 @@ static inline void intel_ring_advance(struct i915_request *rq, u32 *cs) * intel_ring_begin()). */ GEM_BUG_ON((rq->ring->vaddr + rq->ring->emit) != cs); + GEM_BUG_ON(!IS_ALIGNED(rq->ring->emit, 8)); /* RING_TAIL qword align */ } static inline u32 intel_ring_wrap(const struct intel_ring *ring, u32 pos) diff --git a/drivers/gpu/drm/i915/gt/selftest_migrate.c b/drivers/gpu/drm/i915/gt/selftest_migrate.c new file mode 100644 index 000000000000..9784d149ebf1 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/selftest_migrate.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "selftests/i915_random.h" + +static const unsigned int sizes[] = { + SZ_4K, + SZ_64K, + SZ_2M, + CHUNK_SZ - SZ_4K, + CHUNK_SZ, + CHUNK_SZ + SZ_4K, + SZ_64M, +}; + +static struct drm_i915_gem_object * +create_lmem_or_internal(struct drm_i915_private *i915, size_t size) +{ + if (HAS_LMEM(i915)) { + struct drm_i915_gem_object *obj; + + obj = i915_gem_object_create_lmem(i915, size, 0); + if (!IS_ERR(obj)) + return obj; + } + + return i915_gem_object_create_internal(i915, size); +} + +static int copy(struct intel_migrate *migrate, + int (*fn)(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out), + u32 sz, struct rnd_state *prng) +{ + struct drm_i915_private *i915 = migrate->context->engine->i915; + struct drm_i915_gem_object *src, *dst; + struct i915_request *rq; + struct i915_gem_ww_ctx ww; + u32 *vaddr; + int err = 0; + int i; + + src = create_lmem_or_internal(i915, sz); + if (IS_ERR(src)) + return 0; + + dst = i915_gem_object_create_internal(i915, sz); + if (IS_ERR(dst)) + goto err_free_src; + + for_i915_gem_ww(&ww, err, true) { + err = i915_gem_object_lock(src, &ww); + if (err) + continue; + + err = i915_gem_object_lock(dst, &ww); + if (err) + continue; + + vaddr = i915_gem_object_pin_map(src, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + continue; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = i; + i915_gem_object_flush_map(src); + + vaddr = i915_gem_object_pin_map(dst, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto unpin_src; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = ~i; + i915_gem_object_flush_map(dst); + + err = fn(migrate, &ww, src, dst, &rq); + if (!err) + continue; + + if (err != -EDEADLK && err != -EINTR && err != -ERESTARTSYS) + pr_err("%ps failed, size: %u\n", fn, sz); + if (rq) { + i915_request_wait(rq, 0, HZ); + i915_request_put(rq); + } + i915_gem_object_unpin_map(dst); +unpin_src: + i915_gem_object_unpin_map(src); + } + if (err) + goto err_out; + + if (rq) { + if (i915_request_wait(rq, 0, HZ) < 0) { + pr_err("%ps timed out, size: %u\n", fn, sz); + err = -ETIME; + } + i915_request_put(rq); + } + + for (i = 0; !err && i < sz / PAGE_SIZE; i++) { + int x = i * 1024 + i915_prandom_u32_max_state(1024, prng); + + if (vaddr[x] != x) { + pr_err("%ps failed, size: %u, offset: %zu\n", + fn, sz, x * sizeof(u32)); + igt_hexdump(vaddr + i * 1024, 4096); + err = -EINVAL; + } + } + + i915_gem_object_unpin_map(dst); + i915_gem_object_unpin_map(src); + +err_out: + i915_gem_object_put(dst); +err_free_src: + i915_gem_object_put(src); + + return err; +} + +static int __migrate_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_migrate_copy(migrate, ww, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int __global_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_context_migrate_copy(migrate->context, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int +migrate_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __migrate_copy, sz, prng); +} + +static int +global_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __global_copy, sz, prng); +} + +static int live_migrate_copy(void *arg) +{ + struct intel_migrate *migrate = arg; + struct drm_i915_private *i915 = migrate->context->engine->i915; + I915_RND_STATE(prng); + int i; + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + int err; + + err = migrate_copy(migrate, sizes[i], &prng); + if (err == 0) + err = global_copy(migrate, sizes[i], &prng); + i915_gem_drain_freed_objects(i915); + if (err) + return err; + } + + return 0; +} + +struct threaded_migrate { + struct intel_migrate *migrate; + struct task_struct *tsk; + struct rnd_state prng; +}; + +static int threaded_migrate(struct intel_migrate *migrate, + int (*fn)(void *arg), + unsigned int flags) +{ + const unsigned int n_cpus = num_online_cpus() + 1; + struct threaded_migrate *thread; + I915_RND_STATE(prng); + unsigned int i; + int err = 0; + + thread = kcalloc(n_cpus, sizeof(*thread), GFP_KERNEL); + if (!thread) + return 0; + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk; + + thread[i].migrate = migrate; + thread[i].prng = + I915_RND_STATE_INITIALIZER(prandom_u32_state(&prng)); + + tsk = kthread_run(fn, &thread[i], "igt-%d", i); + if (IS_ERR(tsk)) { + err = PTR_ERR(tsk); + break; + } + + get_task_struct(tsk); + thread[i].tsk = tsk; + } + + msleep(10); /* start all threads before we kthread_stop() */ + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk = thread[i].tsk; + int status; + + if (IS_ERR_OR_NULL(tsk)) + continue; + + status = kthread_stop(tsk); + if (status && !err) + err = status; + + put_task_struct(tsk); + } + + kfree(thread); + return err; +} + +static int __thread_migrate_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return migrate_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_migrate_copy(void *arg) +{ + return threaded_migrate(arg, __thread_migrate_copy, 0); +} + +static int __thread_global_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return global_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_global_copy(void *arg) +{ + return threaded_migrate(arg, __thread_global_copy, 0); +} + +int intel_migrate_live_selftests(struct drm_i915_private *i915) +{ + static const struct i915_subtest tests[] = { + SUBTEST(live_migrate_copy), + SUBTEST(thread_migrate_copy), + SUBTEST(thread_global_copy), + }; + struct intel_migrate m; + int err; + + if (intel_migrate_init(&m, &i915->gt)) + return 0; + + err = i915_subtests(tests, &m); + intel_migrate_fini(&m); + + return err; +} diff --git a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h index a92c0e9b7e6b..be5e0191eaea 100644 --- a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h +++ b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h @@ -26,6 +26,7 @@ selftest(gt_mocs, intel_mocs_live_selftests) selftest(gt_pm, intel_gt_pm_live_selftests) selftest(gt_heartbeat, intel_heartbeat_live_selftests) selftest(requests, i915_request_live_selftests) +selftest(migrate, intel_migrate_live_selftests) selftest(active, i915_active_live_selftests) selftest(objects, i915_gem_object_live_selftests) selftest(mman, i915_gem_mman_live_selftests) -- 2.31.1