From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 407DDC2B9F4 for ; Mon, 14 Jun 2021 16:27:03 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0B71961159 for ; Mon, 14 Jun 2021 16:27:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B71961159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8D3E589F19; Mon, 14 Jun 2021 16:26:51 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 19AF589EEB; Mon, 14 Jun 2021 16:26:46 +0000 (UTC) IronPort-SDR: mIflQ/xD4M6UJ/8Pw9Z9rrhSvWagEMPcdbbP2sCI+jlHtVK2NNKD09YHCm8K3Qhjn3AIIeUm1K bri8ZsucYvAA== X-IronPort-AV: E=McAfee;i="6200,9189,10015"; a="204008331" X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="204008331" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2021 09:26:38 -0700 IronPort-SDR: lDC7B4uIzlPC2GTH/9L+WjXb2Kn2EA98ojhSHtjCbuTaT4nr5COa+c/P/XHkbrfNjZKoNKkJt6 L2P2EAUfte9A== X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="449946715" Received: from fnygreen-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.50]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2021 09:26:35 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Mon, 14 Jun 2021 18:26:07 +0200 Message-Id: <20210614162612.294869-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210614162612.294869-1-thomas.hellstrom@linux.intel.com> References: <20210614162612.294869-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 07/12] drm/i915/gt: Pipelined page migration X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" RnJvbTogQ2hyaXMgV2lsc29uIDxjaHJpc0BjaHJpcy13aWxzb24uY28udWs+CgpJZiB3ZSBwaXBl bGluZSB0aGUgUFRFIHVwZGF0ZXMgYW5kIHRoZW4gZG8gdGhlIGNvcHkgb2YgdGhvc2UgcGFnZXMK d2l0aGluIGEgc2luZ2xlIHVucHJlZW1wdGlibGUgY29tbWFuZCBwYWNrZXQsIHdlIGNhbiBzdWJt aXQgdGhlIGNvcGllcwphbmQgbGVhdmUgdGhlbSB0byBiZSBzY2hlZHVsZWQgd2l0aG91dCBoYXZp bmcgdG8gc3luY2hyb25vdXNseSB3YWl0CnVuZGVyIGEgZ2xvYmFsIGxvY2suIEluIG9yZGVyIHRv IG1hbmFnZSBtaWdyYXRpb24sIHdlIG5lZWQgdG8KcHJlYWxsb2NhdGUgdGhlIHBhZ2UgdGFibGVz IChhbmQga2VlcCB0aGVtIHBpbm5lZCBhbmQgYXZhaWxhYmxlIGZvciB1c2UKYXQgYW55IHRpbWUp LCBjYXVzaW5nIGEgYm90dGxlbmVjayBmb3IgbWlncmF0aW9ucyBhcyBhbGwgY2xpZW50cyBtdXN0 CmNvbnRlbmQgb24gdGhlIGxpbWl0ZWQgcmVzb3VyY2VzLiBCeSBpbmxpbmluZyB0aGUgcHBHVFQg dXBkYXRlcyBhbmQKcGVyZm9ybWluZyB0aGUgYmxpdCBhdG9taWNhbGx5LCBlYWNoIGNsaWVudCBv bmx5IG93bnMgdGhlIFBURSB3aGlsZSBpbgp1c2UsIGFuZCBzbyB3ZSBjYW4gcmVzY2hlZHVsZSBp bmRpdmlkdWFsIG9wZXJhdGlvbnMgaG93ZXZlciB3ZSBzZWUgZml0LgpBbmQgbW9zdCBpbXBvcnRh bnRseSwgd2UgZG8gbm90IG5lZWQgdG8gdGFrZSBhIGdsb2JhbCBsb2NrIG9uIHRoZSBzaGFyZWQK dm0sIGFuZCB3YWl0IHVudGlsIHRoZSBvcGVyYXRpb24gaXMgY29tcGxldGUgYmVmb3JlIHJlbGVh c2luZyB0aGUgbG9jawpmb3Igb3RoZXJzIHRvIGNsYWltIHRoZSBQVEUgZm9yIHRoZW1zZWx2ZXMu CgpTaWduZWQtb2ZmLWJ5OiBDaHJpcyBXaWxzb24gPGNocmlzQGNocmlzLXdpbHNvbi5jby51az4K Q28tZGV2ZWxvcGVkLWJ5OiBUaG9tYXMgSGVsbHN0csO2bSA8dGhvbWFzLmhlbGxzdHJvbUBsaW51 eC5pbnRlbC5jb20+ClNpZ25lZC1vZmYtYnk6IFRob21hcyBIZWxsc3Ryw7ZtIDx0aG9tYXMuaGVs bHN0cm9tQGxpbnV4LmludGVsLmNvbT4KLS0tCnYyOgotIEFkZCBhIFRPRE8gZm9yIGh1Z2UgTE1F TSBwdGVzIChQb2ludGVkIG91dCBieSBNYXR0aGV3IEF1bGQpCi0gVXNlIGludGVsX2VuZ2luZV9k ZXN0cm95X3Bpbm5lZF9jb250ZXh0KCkgdG8gcHJvcGVybHkgdGFrZSB0aGUgcGlubmVkCiAgY29u dGV4dCB0aW1lbGluZSBvZmYgdGhlIGVuZ2luZSBsaXN0LiAoQ0kgd2FybmluZykuCnYzOgotIFJl bW92ZSBhbiBvYnNvbGV0ZSBHRU1fQlVHX09OIChQb2ludGVkIG91dCBieSBNYXR0aGV3IEF1bGQp Ci0gRml4IHRoZSBzaXplIGFyZ3VtZW50IGluIGFsbG9jYXRlX3ZhX3JhbmdlKCkgdG8gbm90IGlu Y2x1ZGUgdGhlIGJhc2UKICAoUG9pbnRlZCBvdXQgYnkgTWF0dGhldyBBdWxkKQotLS0KIGRyaXZl cnMvZ3B1L2RybS9pOTE1L01ha2VmaWxlICAgICAgICAgICAgICAgICB8ICAgMSArCiBkcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9lbmdpbmUuaCAgICAgICAgfCAgIDEgKwogZHJpdmVycy9n cHUvZHJtL2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1hbmRzLmggIHwgICAyICsKIGRyaXZlcnMvZ3B1 L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuYyAgICAgICB8IDU0MiArKysrKysrKysrKysrKysr KysKIGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuaCAgICAgICB8ICA0NSAr KwogZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZV90eXBlcy5oIHwgIDE1ICsK IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3JpbmcuaCAgICAgICAgICB8ICAgMSArCiBk cml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMgICAgfCAyOTEgKysrKysr KysrKwogLi4uL2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRlc3RzLmggIHwgICAx ICsKIDkgZmlsZXMgY2hhbmdlZCwgODk5IGluc2VydGlvbnMoKykKIGNyZWF0ZSBtb2RlIDEwMDY0 NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlLmMKIGNyZWF0ZSBtb2RlIDEw MDY0NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlLmgKIGNyZWF0ZSBtb2Rl IDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmgKIGNy ZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRl LmMKCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZSBiL2RyaXZlcnMv Z3B1L2RybS9pOTE1L01ha2VmaWxlCmluZGV4IDdlMDFlYTJjMGYwMC4uZGU0Y2I5YzUyNTg1IDEw MDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZQorKysgYi9kcml2ZXJzL2dw dS9kcm0vaTkxNS9NYWtlZmlsZQpAQCAtMTA4LDYgKzEwOCw3IEBAIGd0LXkgKz0gXAogCWd0L2lu dGVsX2d0dC5vIFwKIAlndC9pbnRlbF9sbGMubyBcCiAJZ3QvaW50ZWxfbHJjLm8gXAorCWd0L2lu dGVsX21pZ3JhdGUubyBcCiAJZ3QvaW50ZWxfbW9jcy5vIFwKIAlndC9pbnRlbF9wcGd0dC5vIFwK IAlndC9pbnRlbF9yYzYubyBcCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9p bnRlbF9lbmdpbmUuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2VuZ2luZS5oCmlu ZGV4IDM2ZWE5ZWI1MmJiNS4uNjJmNzQ0MGJjMTExIDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9k cm0vaTkxNS9ndC9pbnRlbF9lbmdpbmUuaAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9p bnRlbF9lbmdpbmUuaApAQCAtMTg4LDYgKzE4OCw3IEBAIGludGVsX3dyaXRlX3N0YXR1c19wYWdl KHN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZSwgaW50IHJlZywgdTMyIHZhbHVlKQogI2Rl ZmluZSBJOTE1X0dFTV9IV1NfUFJFRU1QVF9BRERSCShJOTE1X0dFTV9IV1NfUFJFRU1QVCAqIHNp emVvZih1MzIpKQogI2RlZmluZSBJOTE1X0dFTV9IV1NfU0VRTk8JCTB4NDAKICNkZWZpbmUgSTkx NV9HRU1fSFdTX1NFUU5PX0FERFIJCShJOTE1X0dFTV9IV1NfU0VRTk8gKiBzaXplb2YodTMyKSkK KyNkZWZpbmUgSTkxNV9HRU1fSFdTX01JR1JBVEUJCSgweDQyICogc2l6ZW9mKHUzMikpCiAjZGVm aW5lIEk5MTVfR0VNX0hXU19TQ1JBVENICQkweDgwCiAKICNkZWZpbmUgSTkxNV9IV1NfQ1NCX0JV RjBfSU5ERVgJCTB4MTAKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVs X2dwdV9jb21tYW5kcy5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1h bmRzLmgKaW5kZXggMjY5NGRiYjk5NjdlLi4xYzNhZjBmYzA0NTYgMTAwNjQ0Ci0tLSBhL2RyaXZl cnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2dwdV9jb21tYW5kcy5oCisrKyBiL2RyaXZlcnMvZ3B1 L2RybS9pOTE1L2d0L2ludGVsX2dwdV9jb21tYW5kcy5oCkBAIC0xMjMsOCArMTIzLDEwIEBACiAj ZGVmaW5lICAgTUlfU0VNQVBIT1JFX1NBRF9ORVFfU0RECSg1IDw8IDEyKQogI2RlZmluZSAgIE1J X1NFTUFQSE9SRV9UT0tFTl9NQVNLCVJFR19HRU5NQVNLKDksIDUpCiAjZGVmaW5lICAgTUlfU0VN QVBIT1JFX1RPS0VOX1NISUZUCTUKKyNkZWZpbmUgTUlfU1RPUkVfREFUQV9JTU0JTUlfSU5TVFIo MHgyMCwgMCkKICNkZWZpbmUgTUlfU1RPUkVfRFdPUkRfSU1NCU1JX0lOU1RSKDB4MjAsIDEpCiAj ZGVmaW5lIE1JX1NUT1JFX0RXT1JEX0lNTV9HRU40CU1JX0lOU1RSKDB4MjAsIDIpCisjZGVmaW5l IE1JX1NUT1JFX1FXT1JEX0lNTV9HRU44IChNSV9JTlNUUigweDIwLCAzKSB8IFJFR19CSVQoMjEp KQogI2RlZmluZSAgIE1JX01FTV9WSVJUVUFMCSgxIDw8IDIyKSAvKiA5NDUsZzMzLDk2NSAqLwog I2RlZmluZSAgIE1JX1VTRV9HR1RUCQkoMSA8PCAyMikgLyogZzR4KyAqLwogI2RlZmluZSBNSV9T VE9SRV9EV09SRF9JTkRFWAlNSV9JTlNUUigweDIxLCAxKQpkaWZmIC0tZ2l0IGEvZHJpdmVycy9n cHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5jIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qv aW50ZWxfbWlncmF0ZS5jCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwMC4u ZTJlODYwMDYzZTdiCi0tLSAvZGV2L251bGwKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qv aW50ZWxfbWlncmF0ZS5jCkBAIC0wLDAgKzEsNTQyIEBACisvLyBTUERYLUxpY2Vuc2UtSWRlbnRp ZmllcjogTUlUCisvKgorICogQ29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAq LworCisjaW5jbHVkZSAiaTkxNV9kcnYuaCIKKyNpbmNsdWRlICJpbnRlbF9jb250ZXh0LmgiCisj aW5jbHVkZSAiaW50ZWxfZ3B1X2NvbW1hbmRzLmgiCisjaW5jbHVkZSAiaW50ZWxfZ3QuaCIKKyNp bmNsdWRlICJpbnRlbF9ndHQuaCIKKyNpbmNsdWRlICJpbnRlbF9taWdyYXRlLmgiCisjaW5jbHVk ZSAiaW50ZWxfcmluZy5oIgorCitzdHJ1Y3QgaW5zZXJ0X3B0ZV9kYXRhIHsKKwl1NjQgb2Zmc2V0 OworCWJvb2wgaXNfbG1lbTsKK307CisKKyNkZWZpbmUgQ0hVTktfU1ogU1pfOE0gLyogfjFtcyBh dCA4R2lCL3MgcHJlZW1wdGlvbiBkZWxheSAqLworCitzdGF0aWMgYm9vbCBlbmdpbmVfc3VwcG9y dHNfbWlncmF0aW9uKHN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZSkKK3sKKwlpZiAoIWVu Z2luZSkKKwkJcmV0dXJuIGZhbHNlOworCisJLyoKKwkgKiBXZSBuZWVkIHRoZSBhYmlsaXR5IHRv IHByZXZlbnQgYXJpYnRyYXRpb24gKE1JX0FSQl9PTl9PRkYpLAorCSAqIHRoZSBhYmlsaXR5IHRv IHdyaXRlIFBURSB1c2luZyBpbmxpbmUgZGF0YSAoTUlfU1RPUkVfREFUQSkKKwkgKiBhbmQgb2Yg Y291cnNlIHRoZSBhYmlsaXR5IHRvIGRvIHRoZSBibG9jayB0cmFuc2ZlciAoYmxpdHMpLgorCSAq LworCUdFTV9CVUdfT04oZW5naW5lLT5jbGFzcyAhPSBDT1BZX0VOR0lORV9DTEFTUyk7CisKKwly ZXR1cm4gdHJ1ZTsKK30KKworc3RhdGljIHZvaWQgaW5zZXJ0X3B0ZShzdHJ1Y3QgaTkxNV9hZGRy ZXNzX3NwYWNlICp2bSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X3BhZ2VfdGFibGUgKnB0LAorCQkg ICAgICAgdm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgaW5zZXJ0X3B0ZV9kYXRhICpkID0gZGF0YTsK KworCXZtLT5pbnNlcnRfcGFnZSh2bSwgcHhfZG1hKHB0KSwgZC0+b2Zmc2V0LCBJOTE1X0NBQ0hF X05PTkUsCisJCQlkLT5pc19sbWVtID8gUFRFX0xNIDogMCk7CisJZC0+b2Zmc2V0ICs9IFBBR0Vf U0laRTsKK30KKworc3RhdGljIHN0cnVjdCBpOTE1X2FkZHJlc3Nfc3BhY2UgKm1pZ3JhdGVfdm0o c3RydWN0IGludGVsX2d0ICpndCkKK3sKKwlzdHJ1Y3QgaTkxNV92bV9wdF9zdGFzaCBzdGFzaCA9 IHt9OworCXN0cnVjdCBpOTE1X3BwZ3R0ICp2bTsKKwlpbnQgZXJyOworCWludCBpOworCisJLyoK KwkgKiBXZSBjb25zdHJ1Y3QgYSB2ZXJ5IHNwZWNpYWwgVk0gZm9yIHVzZSBieSBhbGwgbWlncmF0 aW9uIGNvbnRleHRzLAorCSAqIGl0IGlzIGtlcHQgcGlubmVkIHNvIHRoYXQgaXQgY2FuIGJlIHVz ZWQgYXQgYW55IHRpbWUuIEFzIHdlIG5lZWQKKwkgKiB0byBwcmUtYWxsb2NhdGUgdGhlIHBhZ2Ug ZGlyZWN0b3JpZXMgZm9yIHRoZSBtaWdyYXRpb24gVk0sIHRoaXMKKwkgKiBsaW1pdHMgdXMgdG8g b25seSB1c2luZyBhIHNtYWxsIG51bWJlciBvZiBwcmVwYXJlZCB2bWEuCisJICoKKwkgKiBUbyBi ZSBhYmxlIHRvIHBpcGVsaW5lIGFuZCByZXNjaGVkdWxlIG1pZ3JhdGlvbiBvcGVyYXRpb25zIHdo aWxlCisJICogYXZvaWRpbmcgdW5uZWNlc3NhcnkgY29udGVudGlvbiBvbiB0aGUgdm0gaXRzZWxm LCB0aGUgUFRFIHVwZGF0ZXMKKwkgKiBhcmUgaW5saW5lIHdpdGggdGhlIGJsaXRzLiBBbGwgdGhl IGJsaXRzIHVzZSB0aGUgc2FtZSBmaXhlZAorCSAqIGFkZHJlc3Nlcywgd2l0aCB0aGUgYmFja2lu ZyBzdG9yZSByZWRpcmVjdGlvbiBiZWluZyB1cGRhdGVkIG9uIHRoZQorCSAqIGZseS4gT25seSAy IGltcGxpY2l0IHZtYSBhcmUgdXNlZCBmb3IgYWxsIG1pZ3JhdGlvbiBvcGVyYXRpb25zLgorCSAq CisJICogV2UgbGF5IHRoZSBwcEdUVCBvdXQgYXM6CisJICoKKwkgKglbMCwgQ0hVTktfU1opIC0+ IGZpcnN0IG9iamVjdAorCSAqCVtDSFVOS19TWiwgMiAqIENIVU5LX1NaKSAtPiBzZWNvbmQgb2Jq ZWN0CisJICoJWzIgKiBDSFVOS19TWiwgMiAqIENIVU5LX1NaICsgMiAqIENIVU5LX1NaID4+IDld IC0+IFBURQorCSAqCisJICogQnkgZXhwb3NpbmcgdGhlIGRtYSBhZGRyZXNzZXMgb2YgdGhlIHBh Z2UgZGlyZWN0b3JpZXMgdGhlbXNlbHZlcworCSAqIHdpdGhpbiB0aGUgcHBHVFQsIHdlIGFyZSB0 aGVuIGFibGUgdG8gcmV3cml0ZSB0aGUgUFRFIHByaW9yIHRvIHVzZS4KKwkgKiBCdXQgdGhlIFBU RSB1cGRhdGUgYW5kIHN1YnNlcXVlbnQgbWlncmF0aW9uIG9wZXJhdGlvbiBtdXN0IGJlIGF0b21p YywKKwkgKiBpLmUuIHdpdGhpbiB0aGUgc2FtZSBub24tcHJlZW1wdGlibGUgd2luZG93IHNvIHRo YXQgd2UgZG8gbm90IHN3aXRjaAorCSAqIHRvIGFub3RoZXIgbWlncmF0aW9uIGNvbnRleHQgdGhh dCBvdmVyd3JpdGVzIHRoZSBQVEUuCisJICoKKwkgKiBUT0RPOiBBZGQgc3VwcG9ydCBmb3IgaHVn ZSBMTUVNIFBURXMKKwkgKi8KKworCXZtID0gaTkxNV9wcGd0dF9jcmVhdGUoZ3QpOworCWlmIChJ U19FUlIodm0pKQorCQlyZXR1cm4gRVJSX0NBU1Qodm0pOworCisJaWYgKCF2bS0+dm0uYWxsb2Nh dGVfdmFfcmFuZ2UgfHwgIXZtLT52bS5mb3JlYWNoKSB7CisJCWVyciA9IC1FTk9ERVY7CisJCWdv dG8gZXJyX3ZtOworCX0KKworCS8qCisJICogRWFjaCBlbmdpbmUgaW5zdGFuY2UgaXMgYXNzaWdu ZWQgaXRzIG93biBjaHVuayBpbiB0aGUgVk0sIHNvCisJICogdGhhdCB3ZSBjYW4gcnVuIG11bHRp cGxlIGluc3RhbmNlcyBjb25jdXJyZW50bHkKKwkgKi8KKwlmb3IgKGkgPSAwOyBpIDwgQVJSQVlf U0laRShndC0+ZW5naW5lX2NsYXNzW0NPUFlfRU5HSU5FX0NMQVNTXSk7IGkrKykgeworCQlzdHJ1 Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmU7CisJCXU2NCBiYXNlID0gKHU2NClpIDw8IDMyOwor CQlzdHJ1Y3QgaW5zZXJ0X3B0ZV9kYXRhIGQgPSB7fTsKKwkJc3RydWN0IGk5MTVfZ2VtX3d3X2N0 eCB3dzsKKwkJdTY0IHN6OworCisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJ TkVfQ0xBU1NdW2ldOworCQlpZiAoIWVuZ2luZV9zdXBwb3J0c19taWdyYXRpb24oZW5naW5lKSkK KwkJCWNvbnRpbnVlOworCisJCS8qCisJCSAqIFdlIGNvcHkgaW4gOE1pQiBjaHVua3MuIEVhY2gg UERFIGNvdmVycyAyTWlCLCBzbyB3ZSBuZWVkCisJCSAqIDR4MiBwYWdlIGRpcmVjdG9yaWVzIGZv ciBzb3VyY2UvZGVzdGluYXRpb24uCisJCSAqLworCQlzeiA9IDIgKiBDSFVOS19TWjsKKwkJZC5v ZmZzZXQgPSBiYXNlICsgc3o7CisKKwkJLyoKKwkJICogV2UgbmVlZCBhbm90aGVyIHBhZ2UgZGly ZWN0b3J5IHNldHVwIHNvIHRoYXQgd2UgY2FuIHdyaXRlCisJCSAqIHRoZSA4eDUxMiBQVEUgaW4g ZWFjaCBjaHVuay4KKwkJICovCisJCXN6ICs9IChzeiA+PiAxMikgKiBzaXplb2YodTY0KTsKKwor CQllcnIgPSBpOTE1X3ZtX2FsbG9jX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCwgc3opOworCQlp ZiAoZXJyKQorCQkJZ290byBlcnJfdm07CisKKwkJZm9yX2k5MTVfZ2VtX3d3KCZ3dywgZXJyLCB0 cnVlKSB7CisJCQllcnIgPSBpOTE1X3ZtX2xvY2tfb2JqZWN0cygmdm0tPnZtLCAmd3cpOworCQkJ aWYgKGVycikKKwkJCQljb250aW51ZTsKKwkJCWVyciA9IGk5MTVfdm1fbWFwX3B0X3N0YXNoKCZ2 bS0+dm0sICZzdGFzaCk7CisJCQlpZiAoZXJyKQorCQkJCWNvbnRpbnVlOworCisJCQl2bS0+dm0u YWxsb2NhdGVfdmFfcmFuZ2UoJnZtLT52bSwgJnN0YXNoLCBiYXNlLCBzeik7CisJCX0KKwkJaTkx NV92bV9mcmVlX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCk7CisJCWlmIChlcnIpCisJCQlnb3Rv IGVycl92bTsKKworCQkvKiBOb3cgYWxsb3cgdGhlIEdQVSB0byByZXdyaXRlIHRoZSBQVEUgdmlh IGl0cyBvd24gcHBHVFQgKi8KKwkJZC5pc19sbWVtID0gaTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0o dm0tPnZtLnNjcmF0Y2hbMF0pOworCQl2bS0+dm0uZm9yZWFjaCgmdm0tPnZtLCBiYXNlLCBiYXNl ICsgc3osIGluc2VydF9wdGUsICZkKTsKKwl9CisKKwlyZXR1cm4gJnZtLT52bTsKKworZXJyX3Zt OgorCWk5MTVfdm1fcHV0KCZ2bS0+dm0pOworCXJldHVybiBFUlJfUFRSKGVycik7Cit9CisKK3N0 YXRpYyBzdHJ1Y3QgaW50ZWxfZW5naW5lX2NzICpmaXJzdF9jb3B5X2VuZ2luZShzdHJ1Y3QgaW50 ZWxfZ3QgKmd0KQoreworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwlpbnQgaTsK KworCWZvciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJ TkVfQ0xBU1NdKTsgaSsrKSB7CisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJ TkVfQ0xBU1NdW2ldOworCQlpZiAoZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihlbmdpbmUpKQor CQkJcmV0dXJuIGVuZ2luZTsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworc3RhdGljIHN0cnVj dCBpbnRlbF9jb250ZXh0ICpwaW5uZWRfY29udGV4dChzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQorewor CXN0YXRpYyBzdHJ1Y3QgbG9ja19jbGFzc19rZXkga2V5OworCXN0cnVjdCBpbnRlbF9lbmdpbmVf Y3MgKmVuZ2luZTsKKwlzdHJ1Y3QgaTkxNV9hZGRyZXNzX3NwYWNlICp2bTsKKwlzdHJ1Y3QgaW50 ZWxfY29udGV4dCAqY2U7CisKKwllbmdpbmUgPSBmaXJzdF9jb3B5X2VuZ2luZShndCk7CisJaWYg KCFlbmdpbmUpCisJCXJldHVybiBFUlJfUFRSKC1FTk9ERVYpOworCisJdm0gPSBtaWdyYXRlX3Zt KGd0KTsKKwlpZiAoSVNfRVJSKHZtKSkKKwkJcmV0dXJuIEVSUl9DQVNUKHZtKTsKKworCWNlID0g aW50ZWxfZW5naW5lX2NyZWF0ZV9waW5uZWRfY29udGV4dChlbmdpbmUsIHZtLCBTWl81MTJLLAor CQkJCQkJSTkxNV9HRU1fSFdTX01JR1JBVEUsCisJCQkJCQkma2V5LCAibWlncmF0ZSIpOworCWk5 MTVfdm1fcHV0KGNlLT52bSk7CisJcmV0dXJuIGNlOworfQorCitpbnQgaW50ZWxfbWlncmF0ZV9p bml0KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0 cnVjdCBpbnRlbF9jb250ZXh0ICpjZTsKKworCW1lbXNldChtLCAwLCBzaXplb2YoKm0pKTsKKwor CWNlID0gcGlubmVkX2NvbnRleHQoZ3QpOworCWlmIChJU19FUlIoY2UpKQorCQlyZXR1cm4gUFRS X0VSUihjZSk7CisKKwltLT5jb250ZXh0ID0gY2U7CisJcmV0dXJuIDA7Cit9CisKK3N0YXRpYyBp bnQgcmFuZG9tX2luZGV4KHVuc2lnbmVkIGludCBtYXgpCit7CisJcmV0dXJuIHVwcGVyXzMyX2Jp dHMobXVsX3UzMl91MzIoZ2V0X3JhbmRvbV91MzIoKSwgbWF4KSk7Cit9CisKK3N0YXRpYyBzdHJ1 Y3QgaW50ZWxfY29udGV4dCAqX19taWdyYXRlX2VuZ2luZXMoc3RydWN0IGludGVsX2d0ICpndCkK K3sKKwlzdHJ1Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmVzW01BWF9FTkdJTkVfSU5TVEFOQ0Vd OworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwl1bnNpZ25lZCBpbnQgY291bnQs IGk7CisKKwljb3VudCA9IDA7CisJZm9yIChpID0gMDsgaSA8IEFSUkFZX1NJWkUoZ3QtPmVuZ2lu ZV9jbGFzc1tDT1BZX0VOR0lORV9DTEFTU10pOyBpKyspIHsKKwkJZW5naW5lID0gZ3QtPmVuZ2lu ZV9jbGFzc1tDT1BZX0VOR0lORV9DTEFTU11baV07CisJCWlmIChlbmdpbmVfc3VwcG9ydHNfbWln cmF0aW9uKGVuZ2luZSkpCisJCQllbmdpbmVzW2NvdW50KytdID0gZW5naW5lOworCX0KKworCXJl dHVybiBpbnRlbF9jb250ZXh0X2NyZWF0ZShlbmdpbmVzW3JhbmRvbV9pbmRleChjb3VudCldKTsK K30KKworc3RydWN0IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQo c3RydWN0IGludGVsX21pZ3JhdGUgKm0pCit7CisJc3RydWN0IGludGVsX2NvbnRleHQgKmNlOwor CisJLyoKKwkgKiBXZSByYW5kb21seSBkaXN0cmlidXRlIGNvbnRleHRzIGFjcm9zcyB0aGUgZW5n aW5lcyB1cG9uIGNvbnN0cmN0aW9uLAorCSAqIGFzIHRoZXkgYWxsIHNoYXJlIHRoZSBzYW1lIHBp bm5lZCB2bSwgYW5kIHNvIGluIG9yZGVyIHRvIGFsbG93CisJICogbXVsdGlwbGUgYmxpdHMgdG8g cnVuIGluIHBhcmFsbGVsLCB3ZSBtdXN0IGNvbnN0cnVjdCBlYWNoIGJsaXQKKwkgKiB0byB1c2Ug YSBkaWZmZXJlbnQgcmFuZ2Ugb2YgdGhlIHZtIGZvciBpdHMgR1RULiBUaGlzIGhhcyB0byBiZQor CSAqIGtub3duIGF0IGNvbnN0cnVjdGlvbiwgc28gd2UgY2FuIG5vdCB1c2UgdGhlIGxhdGUgZ3Jl ZWR5IGxvYWQKKwkgKiBiYWxhbmNpbmcgb2YgdGhlIHZpcnR1YWwtZW5naW5lLgorCSAqLworCWNl ID0gX19taWdyYXRlX2VuZ2luZXMobS0+Y29udGV4dC0+ZW5naW5lLT5ndCk7CisJaWYgKElTX0VS UihjZSkpCisJCXJldHVybiBjZTsKKworCWNlLT5yaW5nID0gX19pbnRlbF9jb250ZXh0X3Jpbmdf c2l6ZShTWl8yNTZLKTsKKworCWk5MTVfdm1fcHV0KGNlLT52bSk7CisJY2UtPnZtID0gaTkxNV92 bV9nZXQobS0+Y29udGV4dC0+dm0pOworCisJcmV0dXJuIGNlOworfQorCitzdGF0aWMgaW5saW5l IHN0cnVjdCBzZ3RfZG1hIHNnX3NndChzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnKQoreworCWRtYV9h ZGRyX3QgYWRkciA9IHNnX2RtYV9hZGRyZXNzKHNnKTsKKworCXJldHVybiAoc3RydWN0IHNndF9k bWEpeyBzZywgYWRkciwgYWRkciArIHNnX2RtYV9sZW4oc2cpIH07Cit9CisKK3N0YXRpYyBpbnQg ZW1pdF9ub19hcmJpdHJhdGlvbihzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSkKK3sKKwl1MzIgKmNz OworCisJY3MgPSBpbnRlbF9yaW5nX2JlZ2luKHJxLCAyKTsKKwlpZiAoSVNfRVJSKGNzKSkKKwkJ cmV0dXJuIFBUUl9FUlIoY3MpOworCisJLyogRXhwbGljaXRseSBkaXNhYmxlIHByZWVtcHRpb24g Zm9yIHRoaXMgcmVxdWVzdC4gKi8KKwkqY3MrKyA9IE1JX0FSQl9PTl9PRkY7CisJKmNzKysgPSBN SV9OT09QOworCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCisJcmV0dXJuIDA7Cit9CisK K3N0YXRpYyBpbnQgZW1pdF9wdGUoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsCisJCSAgICBzdHJ1 Y3Qgc2d0X2RtYSAqaXQsCisJCSAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgY2FjaGVfbGV2ZWws CisJCSAgICBib29sIGlzX2xtZW0sCisJCSAgICB1NjQgb2Zmc2V0LAorCQkgICAgaW50IGxlbmd0 aCkKK3sKKwljb25zdCB1NjQgZW5jb2RlID0gcnEtPmNvbnRleHQtPnZtLT5wdGVfZW5jb2RlKDAs IGNhY2hlX2xldmVsLAorCQkJCQkJICAgICAgIGlzX2xtZW0gPyBQVEVfTE0gOiAwKTsKKwlzdHJ1 Y3QgaW50ZWxfcmluZyAqcmluZyA9IHJxLT5yaW5nOworCWludCB0b3RhbCA9IDA7CisJdTMyICpo ZHIsICpjczsKKwlpbnQgcGt0OworCisJR0VNX0JVR19PTihJTlRFTF9HRU4ocnEtPmVuZ2luZS0+ aTkxNSkgPCA4KTsKKworCS8qIENvbXB1dGUgdGhlIHBhZ2UgZGlyZWN0b3J5IG9mZnNldCBmb3Ig dGhlIHRhcmdldCBhZGRyZXNzIHJhbmdlICovCisJb2Zmc2V0ICs9ICh1NjQpcnEtPmVuZ2luZS0+ aW5zdGFuY2UgPDwgMzI7CisJb2Zmc2V0ID4+PSAxMjsKKwlvZmZzZXQgKj0gc2l6ZW9mKHU2NCk7 CisJb2Zmc2V0ICs9IDIgKiBDSFVOS19TWjsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwg Nik7CisJaWYgKElTX0VSUihjcykpCisJCXJldHVybiBQVFJfRVJSKGNzKTsKKworCS8qIFBhY2sg YXMgbWFueSBQVEUgdXBkYXRlcyBhcyBwb3NzaWJsZSBpbnRvIGEgc2luZ2xlIE1JIGNvbW1hbmQg Ki8KKwlwa3QgPSBtaW5fdChpbnQsIDB4NDAwLCByaW5nLT5zcGFjZSAvIHNpemVvZih1MzIpICsg NSk7CisJcGt0ID0gbWluX3QoaW50LCBwa3QsIChyaW5nLT5zaXplIC0gcmluZy0+ZW1pdCkgLyBz aXplb2YodTMyKSArIDUpOworCisJaGRyID0gY3M7CisJKmNzKysgPSBNSV9TVE9SRV9EQVRBX0lN TSB8IFJFR19CSVQoMjEpOyAvKiBhcyBxd29yZCBlbGVtZW50cyAqLworCSpjcysrID0gbG93ZXJf MzJfYml0cyhvZmZzZXQpOworCSpjcysrID0gdXBwZXJfMzJfYml0cyhvZmZzZXQpOworCisJZG8g eworCQlpZiAoY3MgLSBoZHIgPj0gcGt0KSB7CisJCQkqaGRyICs9IGNzIC0gaGRyIC0gMjsKKwkJ CSpjcysrID0gTUlfTk9PUDsKKworCQkJcmluZy0+ZW1pdCA9ICh2b2lkICopY3MgLSByaW5nLT52 YWRkcjsKKwkJCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCQkJaW50ZWxfcmluZ191cGRh dGVfc3BhY2UocmluZyk7CisKKwkJCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgNik7CisJCQlp ZiAoSVNfRVJSKGNzKSkKKwkJCQlyZXR1cm4gUFRSX0VSUihjcyk7CisKKwkJCXBrdCA9IG1pbl90 KGludCwgMHg0MDAsIHJpbmctPnNwYWNlIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwkJCXBrdCA9IG1p bl90KGludCwgcGt0LCAocmluZy0+c2l6ZSAtIHJpbmctPmVtaXQpIC8gc2l6ZW9mKHUzMikgKyA1 KTsKKworCQkJaGRyID0gY3M7CisJCQkqY3MrKyA9IE1JX1NUT1JFX0RBVEFfSU1NIHwgUkVHX0JJ VCgyMSk7CisJCQkqY3MrKyA9IGxvd2VyXzMyX2JpdHMob2Zmc2V0KTsKKwkJCSpjcysrID0gdXBw ZXJfMzJfYml0cyhvZmZzZXQpOworCQl9CisKKwkJKmNzKysgPSBsb3dlcl8zMl9iaXRzKGVuY29k ZSB8IGl0LT5kbWEpOworCQkqY3MrKyA9IHVwcGVyXzMyX2JpdHMoZW5jb2RlIHwgaXQtPmRtYSk7 CisKKwkJb2Zmc2V0ICs9IDg7CisJCXRvdGFsICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKworCQlp dC0+ZG1hICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKwkJaWYgKGl0LT5kbWEgPj0gaXQtPm1heCkg eworCQkJaXQtPnNnID0gX19zZ19uZXh0KGl0LT5zZyk7CisJCQlpZiAoIWl0LT5zZyB8fCBzZ19k bWFfbGVuKGl0LT5zZykgPT0gMCkKKwkJCQlicmVhazsKKworCQkJaXQtPmRtYSA9IHNnX2RtYV9h ZGRyZXNzKGl0LT5zZyk7CisJCQlpdC0+bWF4ID0gaXQtPmRtYSArIHNnX2RtYV9sZW4oaXQtPnNn KTsKKwkJfQorCX0gd2hpbGUgKHRvdGFsIDwgbGVuZ3RoKTsKKworCSpoZHIgKz0gY3MgLSBoZHIg LSAyOworCSpjcysrID0gTUlfTk9PUDsKKworCXJpbmctPmVtaXQgPSAodm9pZCAqKWNzIC0gcmlu Zy0+dmFkZHI7CisJaW50ZWxfcmluZ19hZHZhbmNlKHJxLCBjcyk7CisJaW50ZWxfcmluZ191cGRh dGVfc3BhY2UocmluZyk7CisKKwlyZXR1cm4gdG90YWw7Cit9CisKK3N0YXRpYyBib29sIHdhXzEy MDk2NDQ2MTFfYXBwbGllcyhpbnQgZ2VuLCB1MzIgc2l6ZSkKK3sKKwl1MzIgaGVpZ2h0ID0gc2l6 ZSA+PiBQQUdFX1NISUZUOworCisJaWYgKGdlbiAhPSAxMSkKKwkJcmV0dXJuIGZhbHNlOworCisJ cmV0dXJuIGhlaWdodCAlIDQgPT0gMyAmJiBoZWlnaHQgPD0gODsKK30KKworc3RhdGljIGludCBl bWl0X2NvcHkoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsIGludCBzaXplKQoreworCWNvbnN0IGlu dCBnZW4gPSBJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSk7CisJdTMyIGluc3RhbmNlID0gcnEt PmVuZ2luZS0+aW5zdGFuY2U7CisJdTMyICpjczsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihy cSwgZ2VuID49IDggPyAxMCA6IDYpOworCWlmIChJU19FUlIoY3MpKQorCQlyZXR1cm4gUFRSX0VS Uihjcyk7CisKKwlpZiAoZ2VuID49IDkgJiYgIXdhXzEyMDk2NDQ2MTFfYXBwbGllcyhnZW4sIHNp emUpKSB7CisJCSpjcysrID0gR0VOOV9YWV9GQVNUX0NPUFlfQkxUX0NNRCB8ICgxMCAtIDIpOwor CQkqY3MrKyA9IEJMVF9ERVBUSF8zMiB8IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOworCQkqY3Mr KyA9IHNpemUgPj4gUEFHRV9TSElGVCA8PCAxNiB8IFBBR0VfU0laRSAvIDQ7CisJCSpjcysrID0g Q0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBpbnN0YW5jZTsKKwkJKmNzKysg PSAwOworCQkqY3MrKyA9IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOyAvKiBzcmMgb2Zmc2V0ICov CisJCSpjcysrID0gaW5zdGFuY2U7CisJfSBlbHNlIGlmIChnZW4gPj0gOCkgeworCQkqY3MrKyA9 IFhZX1NSQ19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICgxMCAtIDIpOworCQkqY3Mr KyA9IEJMVF9ERVBUSF8zMiB8IEJMVF9ST1BfU1JDX0NPUFkgfCBQQUdFX1NJWkU7CisJCSpjcysr ID0gMDsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJWkUgLyA0 OworCQkqY3MrKyA9IENIVU5LX1NaOyAvKiBkc3Qgb2Zmc2V0ICovCisJCSpjcysrID0gaW5zdGFu Y2U7CisJCSpjcysrID0gMDsKKwkJKmNzKysgPSBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsgLyog c3JjIG9mZnNldCAqLworCQkqY3MrKyA9IGluc3RhbmNlOworCX0gZWxzZSB7CisJCUdFTV9CVUdf T04oaW5zdGFuY2UpOworCQkqY3MrKyA9IFNSQ19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdC QSB8ICg2IC0gMik7CisJCSpjcysrID0gQkxUX0RFUFRIXzMyIHwgQkxUX1JPUF9TUkNfQ09QWSB8 IFBBR0VfU0laRTsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJ WkU7CisJCSpjcysrID0gQ0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBQQUdF X1NJWkU7CisJCSpjcysrID0gMDsgLyogc3JjIG9mZnNldCAqLworCX0KKworCWludGVsX3Jpbmdf YWR2YW5jZShycSwgY3MpOworCXJldHVybiAwOworfQorCitpbnQKK2ludGVsX2NvbnRleHRfbWln cmF0ZV9jb3B5KHN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZSwKKwkJCSAgIHN0cnVjdCBkbWFfZmVu Y2UgKmF3YWl0LAorCQkJICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkgICBlbnVtIGk5 MTVfY2FjaGVfbGV2ZWwgc3JjX2NhY2hlX2xldmVsLAorCQkJICAgYm9vbCBzcmNfaXNfbG1lbSwK KwkJCSAgIHN0cnVjdCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgZW51bSBpOTE1X2NhY2hlX2xl dmVsIGRzdF9jYWNoZV9sZXZlbCwKKwkJCSAgIGJvb2wgZHN0X2lzX2xtZW0sCisJCQkgICBzdHJ1 Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KQoreworCXN0cnVjdCBzZ3RfZG1hIGl0X3NyYyA9IHNnX3Nn dChzcmMpLCBpdF9kc3QgPSBzZ19zZ3QoZHN0KTsKKwlzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycTsK KwlpbnQgZXJyOworCisJKm91dCA9IE5VTEw7CisKKwlHRU1fQlVHX09OKGNlLT5yaW5nLT5zaXpl IDwgU1pfNjRLKTsKKworCWRvIHsKKwkJaW50IGxlbjsKKworCQlycSA9IGk5MTVfcmVxdWVzdF9j cmVhdGUoY2UpOworCQlpZiAoSVNfRVJSKHJxKSkgeworCQkJZXJyID0gUFRSX0VSUihycSk7CisJ CQlnb3RvIG91dF9jZTsKKwkJfQorCisJCWlmIChhd2FpdCkgeworCQkJZXJyID0gaTkxNV9yZXF1 ZXN0X2F3YWl0X2RtYV9mZW5jZShycSwgYXdhaXQpOworCQkJaWYgKGVycikKKwkJCQlnb3RvIG91 dF9ycTsKKworCQkJaWYgKHJxLT5lbmdpbmUtPmVtaXRfaW5pdF9icmVhZGNydW1iKSB7CisJCQkJ ZXJyID0gcnEtPmVuZ2luZS0+ZW1pdF9pbml0X2JyZWFkY3J1bWIocnEpOworCQkJCWlmIChlcnIp CisJCQkJCWdvdG8gb3V0X3JxOworCQkJfQorCisJCQlhd2FpdCA9IE5VTEw7CisJCX0KKworCQkv KiBUaGUgUFRFIHVwZGF0ZXMgKyBjb3B5IG11c3Qgbm90IGJlIGludGVycnVwdGVkLiAqLworCQll cnIgPSBlbWl0X25vX2FyYml0cmF0aW9uKHJxKTsKKwkJaWYgKGVycikKKwkJCWdvdG8gb3V0X3Jx OworCisJCWxlbiA9IGVtaXRfcHRlKHJxLCAmaXRfc3JjLCBzcmNfY2FjaGVfbGV2ZWwsIHNyY19p c19sbWVtLCAwLAorCQkJICAgICAgIENIVU5LX1NaKTsKKwkJaWYgKGxlbiA8PSAwKSB7CisJCQll cnIgPSBsZW47CisJCQlnb3RvIG91dF9ycTsKKwkJfQorCisJCWVyciA9IGVtaXRfcHRlKHJxLCAm aXRfZHN0LCBkc3RfY2FjaGVfbGV2ZWwsIGRzdF9pc19sbWVtLAorCQkJICAgICAgIENIVU5LX1Na LCBsZW4pOworCQlpZiAoZXJyIDwgMCkKKwkJCWdvdG8gb3V0X3JxOworCQlpZiAoZXJyIDwgbGVu KSB7CisJCQllcnIgPSAtRUlOVkFMOworCQkJZ290byBvdXRfcnE7CisJCX0KKworCQllcnIgPSBy cS0+ZW5naW5lLT5lbWl0X2ZsdXNoKHJxLCBFTUlUX0lOVkFMSURBVEUpOworCQlpZiAoZXJyKQor CQkJZ290byBvdXRfcnE7CisKKwkJZXJyID0gZW1pdF9jb3B5KHJxLCBsZW4pOworCisJCS8qIEFy Yml0cmF0aW9uIGlzIHJlLWVuYWJsZWQgYmV0d2VlbiByZXF1ZXN0cy4gKi8KK291dF9ycToKKwkJ aWYgKCpvdXQpCisJCQlpOTE1X3JlcXVlc3RfcHV0KCpvdXQpOworCQkqb3V0ID0gaTkxNV9yZXF1 ZXN0X2dldChycSk7CisJCWk5MTVfcmVxdWVzdF9hZGQocnEpOworCQlpZiAoZXJyIHx8ICFpdF9z cmMuc2cgfHwgIXNnX2RtYV9sZW4oaXRfc3JjLnNnKSkKKwkJCWJyZWFrOworCisJCWNvbmRfcmVz Y2hlZCgpOworCX0gd2hpbGUgKDEpOworCitvdXRfY2U6CisJcmV0dXJuIGVycjsKK30KKworaW50 IGludGVsX21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbSwKKwkJICAgICAgIHN0 cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkgICAgICAgc3RydWN0IGRtYV9mZW5jZSAqYXdh aXQsCisJCSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNyYywKKwkJICAgICAgIGVudW0gaTkx NV9jYWNoZV9sZXZlbCBzcmNfY2FjaGVfbGV2ZWwsCisJCSAgICAgICBib29sIHNyY19pc19sbWVt LAorCQkgICAgICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpkc3QsCisJCSAgICAgICBlbnVtIGk5MTVf Y2FjaGVfbGV2ZWwgZHN0X2NhY2hlX2xldmVsLAorCQkgICAgICAgYm9vbCBkc3RfaXNfbG1lbSwK KwkJICAgICAgIHN0cnVjdCBpOTE1X3JlcXVlc3QgKipvdXQpCit7CisJc3RydWN0IGludGVsX2Nv bnRleHQgKmNlOworCWludCBlcnI7CisKKwkqb3V0ID0gTlVMTDsKKwlpZiAoIW0tPmNvbnRleHQp CisJCXJldHVybiAtRU5PREVWOworCisJY2UgPSBpbnRlbF9taWdyYXRlX2NyZWF0ZV9jb250ZXh0 KG0pOworCWlmIChJU19FUlIoY2UpKQorCQljZSA9IGludGVsX2NvbnRleHRfZ2V0KG0tPmNvbnRl eHQpOworCUdFTV9CVUdfT04oSVNfRVJSKGNlKSk7CisKKwllcnIgPSBpbnRlbF9jb250ZXh0X3Bp bl93dyhjZSwgd3cpOworCWlmIChlcnIpCisJCWdvdG8gb3V0OworCisJZXJyID0gaW50ZWxfY29u dGV4dF9taWdyYXRlX2NvcHkoY2UsIGF3YWl0LAorCQkJCQkgc3JjLCBzcmNfY2FjaGVfbGV2ZWws IHNyY19pc19sbWVtLAorCQkJCQkgZHN0LCBkc3RfY2FjaGVfbGV2ZWwsIGRzdF9pc19sbWVtLAor CQkJCQkgb3V0KTsKKworCWludGVsX2NvbnRleHRfdW5waW4oY2UpOworb3V0OgorCWludGVsX2Nv bnRleHRfcHV0KGNlKTsKKwlyZXR1cm4gZXJyOworfQorCit2b2lkIGludGVsX21pZ3JhdGVfZmlu aShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbSkKK3sKKwlzdHJ1Y3QgaW50ZWxfY29udGV4dCAqY2U7 CisKKwljZSA9IGZldGNoX2FuZF96ZXJvKCZtLT5jb250ZXh0KTsKKwlpZiAoIWNlKQorCQlyZXR1 cm47CisKKwlpbnRlbF9lbmdpbmVfZGVzdHJveV9waW5uZWRfY29udGV4dChjZSk7Cit9CisKKyNp ZiBJU19FTkFCTEVEKENPTkZJR19EUk1fSTkxNV9TRUxGVEVTVCkKKyNpbmNsdWRlICJzZWxmdGVz dF9taWdyYXRlLmMiCisjZW5kaWYKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0 L2ludGVsX21pZ3JhdGUuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUu aApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLjMyYzYxMTkwZWQ3Mwot LS0gL2Rldi9udWxsCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUu aApAQCAtMCwwICsxLDQ1IEBACisvKiBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUICovCisv KgorICogQ29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaWZuZGVm IF9fSU5URUxfTUlHUkFURV9fCisjZGVmaW5lIF9fSU5URUxfTUlHUkFURV9fCisKKyNpbmNsdWRl ICJpbnRlbF9taWdyYXRlX3R5cGVzLmgiCisKK3N0cnVjdCBkbWFfZmVuY2U7CitzdHJ1Y3QgaTkx NV9yZXF1ZXN0Oworc3RydWN0IGk5MTVfZ2VtX3d3X2N0eDsKK3N0cnVjdCBpbnRlbF9ndDsKK3N0 cnVjdCBzY2F0dGVybGlzdDsKK2VudW0gaTkxNV9jYWNoZV9sZXZlbDsKKworaW50IGludGVsX21p Z3JhdGVfaW5pdChzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbSwgc3RydWN0IGludGVsX2d0ICpndCk7 CisKK3N0cnVjdCBpbnRlbF9jb250ZXh0ICppbnRlbF9taWdyYXRlX2NyZWF0ZV9jb250ZXh0KHN0 cnVjdCBpbnRlbF9taWdyYXRlICptKTsKKworaW50IGludGVsX21pZ3JhdGVfY29weShzdHJ1Y3Qg aW50ZWxfbWlncmF0ZSAqbSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAor CQkgICAgICAgc3RydWN0IGRtYV9mZW5jZSAqYXdhaXQsCisJCSAgICAgICBzdHJ1Y3Qgc2NhdHRl cmxpc3QgKnNyYywKKwkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBzcmNfY2FjaGVfbGV2 ZWwsCisJCSAgICAgICBib29sIHNyY19pc19sbWVtLAorCQkgICAgICAgc3RydWN0IHNjYXR0ZXJs aXN0ICpkc3QsCisJCSAgICAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgZHN0X2NhY2hlX2xldmVs LAorCQkgICAgICAgYm9vbCBkc3RfaXNfbG1lbSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X3JlcXVl c3QgKipvdXQpOworCitpbnQgaW50ZWxfY29udGV4dF9taWdyYXRlX2NvcHkoc3RydWN0IGludGVs X2NvbnRleHQgKmNlLAorCQkJICAgICAgIHN0cnVjdCBkbWFfZmVuY2UgKmF3YWl0LAorCQkJICAg ICAgIHN0cnVjdCBzY2F0dGVybGlzdCAqc3JjLAorCQkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9s ZXZlbCBzcmNfY2FjaGVfbGV2ZWwsCisJCQkgICAgICAgYm9vbCBzcmNfaXNfbG1lbSwKKwkJCSAg ICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKmRzdCwKKwkJCSAgICAgICBlbnVtIGk5MTVfY2FjaGVf bGV2ZWwgZHN0X2NhY2hlX2xldmVsLAorCQkJICAgICAgIGJvb2wgZHN0X2lzX2xtZW0sCisJCQkg ICAgICAgc3RydWN0IGk5MTVfcmVxdWVzdCAqKm91dCk7CisKK3ZvaWQgaW50ZWxfbWlncmF0ZV9m aW5pKHN0cnVjdCBpbnRlbF9taWdyYXRlICptKTsKKworI2VuZGlmIC8qIF9fSU5URUxfTUlHUkFU RV9fICovCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRl X3R5cGVzLmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmgK bmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMDAwMDAwLi5kOTgyMzA1OTdmNDIKLS0t IC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5 cGVzLmgKQEAgLTAsMCArMSwxNSBAQAorLyogU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6IE1JVCAq LworLyoKKyAqIENvcHlyaWdodCDCqSAyMDIwIEludGVsIENvcnBvcmF0aW9uCisgKi8KKworI2lm bmRlZiBfX0lOVEVMX01JR1JBVEVfVFlQRVNfXworI2RlZmluZSBfX0lOVEVMX01JR1JBVEVfVFlQ RVNfXworCitzdHJ1Y3QgaW50ZWxfY29udGV4dDsKKworc3RydWN0IGludGVsX21pZ3JhdGUgewor CXN0cnVjdCBpbnRlbF9jb250ZXh0ICpjb250ZXh0OworfTsKKworI2VuZGlmIC8qIF9fSU5URUxf TUlHUkFURV9UWVBFU19fICovCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9p bnRlbF9yaW5nLmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9yaW5nLmgKaW5kZXgg ZGJmNWYxNGExMzZmLi4xYjMyZGFkZmI4YzMgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9p OTE1L2d0L2ludGVsX3JpbmcuaAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9y aW5nLmgKQEAgLTQ5LDYgKzQ5LDcgQEAgc3RhdGljIGlubGluZSB2b2lkIGludGVsX3JpbmdfYWR2 YW5jZShzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSwgdTMyICpjcykKIAkgKiBpbnRlbF9yaW5nX2Jl Z2luKCkpLgogCSAqLwogCUdFTV9CVUdfT04oKHJxLT5yaW5nLT52YWRkciArIHJxLT5yaW5nLT5l bWl0KSAhPSBjcyk7CisJR0VNX0JVR19PTighSVNfQUxJR05FRChycS0+cmluZy0+ZW1pdCwgOCkp OyAvKiBSSU5HX1RBSUwgcXdvcmQgYWxpZ24gKi8KIH0KIAogc3RhdGljIGlubGluZSB1MzIgaW50 ZWxfcmluZ193cmFwKGNvbnN0IHN0cnVjdCBpbnRlbF9yaW5nICpyaW5nLCB1MzIgcG9zKQpkaWZm IC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jIGIvZHJp dmVycy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jCm5ldyBmaWxlIG1vZGUgMTAw NjQ0CmluZGV4IDAwMDAwMDAwMDAwMC4uOTc4NGQxNDllYmYxCi0tLSAvZGV2L251bGwKKysrIGIv ZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jCkBAIC0wLDAgKzEsMjkx IEBACisvLyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUCisvKgorICogQ29weXJpZ2h0IMKp IDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaW5jbHVkZSAic2VsZnRlc3RzL2k5MTVf cmFuZG9tLmgiCisKK3N0YXRpYyBjb25zdCB1bnNpZ25lZCBpbnQgc2l6ZXNbXSA9IHsKKwlTWl80 SywKKwlTWl82NEssCisJU1pfMk0sCisJQ0hVTktfU1ogLSBTWl80SywKKwlDSFVOS19TWiwKKwlD SFVOS19TWiArIFNaXzRLLAorCVNaXzY0TSwKK307CisKK3N0YXRpYyBzdHJ1Y3QgZHJtX2k5MTVf Z2VtX29iamVjdCAqCitjcmVhdGVfbG1lbV9vcl9pbnRlcm5hbChzdHJ1Y3QgZHJtX2k5MTVfcHJp dmF0ZSAqaTkxNSwgc2l6ZV90IHNpemUpCit7CisJaWYgKEhBU19MTUVNKGk5MTUpKSB7CisJCXN0 cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpvYmo7CisKKwkJb2JqID0gaTkxNV9nZW1fb2JqZWN0 X2NyZWF0ZV9sbWVtKGk5MTUsIHNpemUsIDApOworCQlpZiAoIUlTX0VSUihvYmopKQorCQkJcmV0 dXJuIG9iajsKKwl9CisKKwlyZXR1cm4gaTkxNV9nZW1fb2JqZWN0X2NyZWF0ZV9pbnRlcm5hbChp OTE1LCBzaXplKTsKK30KKworc3RhdGljIGludCBjb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICpt aWdyYXRlLAorCQlpbnQgKCpmbikoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGUsCisJCQkg IHN0cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29i amVjdCAqc3JjLAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAqZHN0LAorCQkJICBz dHJ1Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KSwKKwkJdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0YXRlICpw cm5nKQoreworCXN0cnVjdCBkcm1faTkxNV9wcml2YXRlICppOTE1ID0gbWlncmF0ZS0+Y29udGV4 dC0+ZW5naW5lLT5pOTE1OworCXN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpzcmMsICpkc3Q7 CisJc3RydWN0IGk5MTVfcmVxdWVzdCAqcnE7CisJc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCB3dzsK Kwl1MzIgKnZhZGRyOworCWludCBlcnIgPSAwOworCWludCBpOworCisJc3JjID0gY3JlYXRlX2xt ZW1fb3JfaW50ZXJuYWwoaTkxNSwgc3opOworCWlmIChJU19FUlIoc3JjKSkKKwkJcmV0dXJuIDA7 CisKKwlkc3QgPSBpOTE1X2dlbV9vYmplY3RfY3JlYXRlX2ludGVybmFsKGk5MTUsIHN6KTsKKwlp ZiAoSVNfRVJSKGRzdCkpCisJCWdvdG8gZXJyX2ZyZWVfc3JjOworCisJZm9yX2k5MTVfZ2VtX3d3 KCZ3dywgZXJyLCB0cnVlKSB7CisJCWVyciA9IGk5MTVfZ2VtX29iamVjdF9sb2NrKHNyYywgJnd3 KTsKKwkJaWYgKGVycikKKwkJCWNvbnRpbnVlOworCisJCWVyciA9IGk5MTVfZ2VtX29iamVjdF9s b2NrKGRzdCwgJnd3KTsKKwkJaWYgKGVycikKKwkJCWNvbnRpbnVlOworCisJCXZhZGRyID0gaTkx NV9nZW1fb2JqZWN0X3Bpbl9tYXAoc3JjLCBJOTE1X01BUF9XQyk7CisJCWlmIChJU19FUlIodmFk ZHIpKSB7CisJCQllcnIgPSBQVFJfRVJSKHZhZGRyKTsKKwkJCWNvbnRpbnVlOworCQl9CisKKwkJ Zm9yIChpID0gMDsgaSA8IHN6IC8gc2l6ZW9mKHUzMik7IGkrKykKKwkJCXZhZGRyW2ldID0gaTsK KwkJaTkxNV9nZW1fb2JqZWN0X2ZsdXNoX21hcChzcmMpOworCisJCXZhZGRyID0gaTkxNV9nZW1f b2JqZWN0X3Bpbl9tYXAoZHN0LCBJOTE1X01BUF9XQyk7CisJCWlmIChJU19FUlIodmFkZHIpKSB7 CisJCQllcnIgPSBQVFJfRVJSKHZhZGRyKTsKKwkJCWdvdG8gdW5waW5fc3JjOworCQl9CisKKwkJ Zm9yIChpID0gMDsgaSA8IHN6IC8gc2l6ZW9mKHUzMik7IGkrKykKKwkJCXZhZGRyW2ldID0gfmk7 CisJCWk5MTVfZ2VtX29iamVjdF9mbHVzaF9tYXAoZHN0KTsKKworCQllcnIgPSBmbihtaWdyYXRl LCAmd3csIHNyYywgZHN0LCAmcnEpOworCQlpZiAoIWVycikKKwkJCWNvbnRpbnVlOworCisJCWlm IChlcnIgIT0gLUVERUFETEsgJiYgZXJyICE9IC1FSU5UUiAmJiBlcnIgIT0gLUVSRVNUQVJUU1lT KQorCQkJcHJfZXJyKCIlcHMgZmFpbGVkLCBzaXplOiAldVxuIiwgZm4sIHN6KTsKKwkJaWYgKHJx KSB7CisJCQlpOTE1X3JlcXVlc3Rfd2FpdChycSwgMCwgSFopOworCQkJaTkxNV9yZXF1ZXN0X3B1 dChycSk7CisJCX0KKwkJaTkxNV9nZW1fb2JqZWN0X3VucGluX21hcChkc3QpOwordW5waW5fc3Jj OgorCQlpOTE1X2dlbV9vYmplY3RfdW5waW5fbWFwKHNyYyk7CisJfQorCWlmIChlcnIpCisJCWdv dG8gZXJyX291dDsKKworCWlmIChycSkgeworCQlpZiAoaTkxNV9yZXF1ZXN0X3dhaXQocnEsIDAs IEhaKSA8IDApIHsKKwkJCXByX2VycigiJXBzIHRpbWVkIG91dCwgc2l6ZTogJXVcbiIsIGZuLCBz eik7CisJCQllcnIgPSAtRVRJTUU7CisJCX0KKwkJaTkxNV9yZXF1ZXN0X3B1dChycSk7CisJfQor CisJZm9yIChpID0gMDsgIWVyciAmJiBpIDwgc3ogLyBQQUdFX1NJWkU7IGkrKykgeworCQlpbnQg eCA9IGkgKiAxMDI0ICsgaTkxNV9wcmFuZG9tX3UzMl9tYXhfc3RhdGUoMTAyNCwgcHJuZyk7CisK KwkJaWYgKHZhZGRyW3hdICE9IHgpIHsKKwkJCXByX2VycigiJXBzIGZhaWxlZCwgc2l6ZTogJXUs IG9mZnNldDogJXp1XG4iLAorCQkJICAgICAgIGZuLCBzeiwgeCAqIHNpemVvZih1MzIpKTsKKwkJ CWlndF9oZXhkdW1wKHZhZGRyICsgaSAqIDEwMjQsIDQwOTYpOworCQkJZXJyID0gLUVJTlZBTDsK KwkJfQorCX0KKworCWk5MTVfZ2VtX29iamVjdF91bnBpbl9tYXAoZHN0KTsKKwlpOTE1X2dlbV9v YmplY3RfdW5waW5fbWFwKHNyYyk7CisKK2Vycl9vdXQ6CisJaTkxNV9nZW1fb2JqZWN0X3B1dChk c3QpOworZXJyX2ZyZWVfc3JjOgorCWk5MTVfZ2VtX29iamVjdF9wdXQoc3JjKTsKKworCXJldHVy biBlcnI7Cit9CisKK3N0YXRpYyBpbnQgX19taWdyYXRlX2NvcHkoc3RydWN0IGludGVsX21pZ3Jh dGUgKm1pZ3JhdGUsCisJCQkgIHN0cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkJICBzdHJ1 Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAqc3JjLAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29i amVjdCAqZHN0LAorCQkJICBzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KQoreworCXJldHVybiBp bnRlbF9taWdyYXRlX2NvcHkobWlncmF0ZSwgd3csIE5VTEwsCisJCQkJICBzcmMtPm1tLnBhZ2Vz LT5zZ2wsIHNyYy0+Y2FjaGVfbGV2ZWwsCisJCQkJICBpOTE1X2dlbV9vYmplY3RfaXNfbG1lbShz cmMpLAorCQkJCSAgZHN0LT5tbS5wYWdlcy0+c2dsLCBkc3QtPmNhY2hlX2xldmVsLAorCQkJCSAg aTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0oZHN0KSwKKwkJCQkgIG91dCk7Cit9CisKK3N0YXRpYyBp bnQgX19nbG9iYWxfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSBzdHJ1 Y3QgaTkxNV9nZW1fd3dfY3R4ICp3dywKKwkJCSBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAq c3JjLAorCQkJIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpkc3QsCisJCQkgc3RydWN0IGk5 MTVfcmVxdWVzdCAqKm91dCkKK3sKKwlyZXR1cm4gaW50ZWxfY29udGV4dF9taWdyYXRlX2NvcHko bWlncmF0ZS0+Y29udGV4dCwgTlVMTCwKKwkJCQkJICBzcmMtPm1tLnBhZ2VzLT5zZ2wsIHNyYy0+ Y2FjaGVfbGV2ZWwsCisJCQkJCSAgaTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0oc3JjKSwKKwkJCQkJ ICBkc3QtPm1tLnBhZ2VzLT5zZ2wsIGRzdC0+Y2FjaGVfbGV2ZWwsCisJCQkJCSAgaTkxNV9nZW1f b2JqZWN0X2lzX2xtZW0oZHN0KSwKKwkJCQkJICBvdXQpOworfQorCitzdGF0aWMgaW50CittaWdy YXRlX2NvcHkoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGUsIHUzMiBzeiwgc3RydWN0IHJu ZF9zdGF0ZSAqcHJuZykKK3sKKwlyZXR1cm4gY29weShtaWdyYXRlLCBfX21pZ3JhdGVfY29weSwg c3osIHBybmcpOworfQorCitzdGF0aWMgaW50CitnbG9iYWxfY29weShzdHJ1Y3QgaW50ZWxfbWln cmF0ZSAqbWlncmF0ZSwgdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0YXRlICpwcm5nKQoreworCXJldHVy biBjb3B5KG1pZ3JhdGUsIF9fZ2xvYmFsX2NvcHksIHN6LCBwcm5nKTsKK30KKworc3RhdGljIGlu dCBsaXZlX21pZ3JhdGVfY29weSh2b2lkICphcmcpCit7CisJc3RydWN0IGludGVsX21pZ3JhdGUg Km1pZ3JhdGUgPSBhcmc7CisJc3RydWN0IGRybV9pOTE1X3ByaXZhdGUgKmk5MTUgPSBtaWdyYXRl LT5jb250ZXh0LT5lbmdpbmUtPmk5MTU7CisJSTkxNV9STkRfU1RBVEUocHJuZyk7CisJaW50IGk7 CisKKwlmb3IgKGkgPSAwOyBpIDwgQVJSQVlfU0laRShzaXplcyk7IGkrKykgeworCQlpbnQgZXJy OworCisJCWVyciA9IG1pZ3JhdGVfY29weShtaWdyYXRlLCBzaXplc1tpXSwgJnBybmcpOworCQlp ZiAoZXJyID09IDApCisJCQllcnIgPSBnbG9iYWxfY29weShtaWdyYXRlLCBzaXplc1tpXSwgJnBy bmcpOworCQlpOTE1X2dlbV9kcmFpbl9mcmVlZF9vYmplY3RzKGk5MTUpOworCQlpZiAoZXJyKQor CQkJcmV0dXJuIGVycjsKKwl9CisKKwlyZXR1cm4gMDsKK30KKworc3RydWN0IHRocmVhZGVkX21p Z3JhdGUgeworCXN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlOworCXN0cnVjdCB0YXNrX3N0 cnVjdCAqdHNrOworCXN0cnVjdCBybmRfc3RhdGUgcHJuZzsKK307CisKK3N0YXRpYyBpbnQgdGhy ZWFkZWRfbWlncmF0ZShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSAgICBpbnQg KCpmbikodm9pZCAqYXJnKSwKKwkJCSAgICB1bnNpZ25lZCBpbnQgZmxhZ3MpCit7CisJY29uc3Qg dW5zaWduZWQgaW50IG5fY3B1cyA9IG51bV9vbmxpbmVfY3B1cygpICsgMTsKKwlzdHJ1Y3QgdGhy ZWFkZWRfbWlncmF0ZSAqdGhyZWFkOworCUk5MTVfUk5EX1NUQVRFKHBybmcpOworCXVuc2lnbmVk IGludCBpOworCWludCBlcnIgPSAwOworCisJdGhyZWFkID0ga2NhbGxvYyhuX2NwdXMsIHNpemVv ZigqdGhyZWFkKSwgR0ZQX0tFUk5FTCk7CisJaWYgKCF0aHJlYWQpCisJCXJldHVybiAwOworCisJ Zm9yIChpID0gMDsgaSA8IG5fY3B1czsgKytpKSB7CisJCXN0cnVjdCB0YXNrX3N0cnVjdCAqdHNr OworCisJCXRocmVhZFtpXS5taWdyYXRlID0gbWlncmF0ZTsKKwkJdGhyZWFkW2ldLnBybmcgPQor CQkJSTkxNV9STkRfU1RBVEVfSU5JVElBTElaRVIocHJhbmRvbV91MzJfc3RhdGUoJnBybmcpKTsK KworCQl0c2sgPSBrdGhyZWFkX3J1bihmbiwgJnRocmVhZFtpXSwgImlndC0lZCIsIGkpOworCQlp ZiAoSVNfRVJSKHRzaykpIHsKKwkJCWVyciA9IFBUUl9FUlIodHNrKTsKKwkJCWJyZWFrOworCQl9 CisKKwkJZ2V0X3Rhc2tfc3RydWN0KHRzayk7CisJCXRocmVhZFtpXS50c2sgPSB0c2s7CisJfQor CisJbXNsZWVwKDEwKTsgLyogc3RhcnQgYWxsIHRocmVhZHMgYmVmb3JlIHdlIGt0aHJlYWRfc3Rv cCgpICovCisKKwlmb3IgKGkgPSAwOyBpIDwgbl9jcHVzOyArK2kpIHsKKwkJc3RydWN0IHRhc2tf c3RydWN0ICp0c2sgPSB0aHJlYWRbaV0udHNrOworCQlpbnQgc3RhdHVzOworCisJCWlmIChJU19F UlJfT1JfTlVMTCh0c2spKQorCQkJY29udGludWU7CisKKwkJc3RhdHVzID0ga3RocmVhZF9zdG9w KHRzayk7CisJCWlmIChzdGF0dXMgJiYgIWVycikKKwkJCWVyciA9IHN0YXR1czsKKworCQlwdXRf dGFza19zdHJ1Y3QodHNrKTsKKwl9CisKKwlrZnJlZSh0aHJlYWQpOworCXJldHVybiBlcnI7Cit9 CisKK3N0YXRpYyBpbnQgX190aHJlYWRfbWlncmF0ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1 Y3QgdGhyZWFkZWRfbWlncmF0ZSAqdG0gPSBhcmc7CisKKwlyZXR1cm4gbWlncmF0ZV9jb3B5KHRt LT5taWdyYXRlLCAyICogQ0hVTktfU1osICZ0bS0+cHJuZyk7Cit9CisKK3N0YXRpYyBpbnQgdGhy ZWFkX21pZ3JhdGVfY29weSh2b2lkICphcmcpCit7CisJcmV0dXJuIHRocmVhZGVkX21pZ3JhdGUo YXJnLCBfX3RocmVhZF9taWdyYXRlX2NvcHksIDApOworfQorCitzdGF0aWMgaW50IF9fdGhyZWFk X2dsb2JhbF9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgdGhyZWFkZWRfbWlncmF0ZSAqdG0g PSBhcmc7CisKKwlyZXR1cm4gZ2xvYmFsX2NvcHkodG0tPm1pZ3JhdGUsIDIgKiBDSFVOS19TWiwg JnRtLT5wcm5nKTsKK30KKworc3RhdGljIGludCB0aHJlYWRfZ2xvYmFsX2NvcHkodm9pZCAqYXJn KQoreworCXJldHVybiB0aHJlYWRlZF9taWdyYXRlKGFyZywgX190aHJlYWRfZ2xvYmFsX2NvcHks IDApOworfQorCitpbnQgaW50ZWxfbWlncmF0ZV9saXZlX3NlbGZ0ZXN0cyhzdHJ1Y3QgZHJtX2k5 MTVfcHJpdmF0ZSAqaTkxNSkKK3sKKwlzdGF0aWMgY29uc3Qgc3RydWN0IGk5MTVfc3VidGVzdCB0 ZXN0c1tdID0geworCQlTVUJURVNUKGxpdmVfbWlncmF0ZV9jb3B5KSwKKwkJU1VCVEVTVCh0aHJl YWRfbWlncmF0ZV9jb3B5KSwKKwkJU1VCVEVTVCh0aHJlYWRfZ2xvYmFsX2NvcHkpLAorCX07CisJ c3RydWN0IGludGVsX21pZ3JhdGUgbTsKKwlpbnQgZXJyOworCisJaWYgKGludGVsX21pZ3JhdGVf aW5pdCgmbSwgJmk5MTUtPmd0KSkKKwkJcmV0dXJuIDA7CisKKwllcnIgPSBpOTE1X3N1YnRlc3Rz KHRlc3RzLCAmbSk7CisJaW50ZWxfbWlncmF0ZV9maW5pKCZtKTsKKworCXJldHVybiBlcnI7Cit9 CmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3Nl bGZ0ZXN0cy5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfbGl2ZV9zZWxm dGVzdHMuaAppbmRleCBhOTJjMGU5YjdlNmIuLmJlNWUwMTkxZWFlYSAxMDA2NDQKLS0tIGEvZHJp dmVycy9ncHUvZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfbGl2ZV9zZWxmdGVzdHMuaAorKysgYi9k cml2ZXJzL2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oCkBAIC0y Niw2ICsyNiw3IEBAIHNlbGZ0ZXN0KGd0X21vY3MsIGludGVsX21vY3NfbGl2ZV9zZWxmdGVzdHMp CiBzZWxmdGVzdChndF9wbSwgaW50ZWxfZ3RfcG1fbGl2ZV9zZWxmdGVzdHMpCiBzZWxmdGVzdChn dF9oZWFydGJlYXQsIGludGVsX2hlYXJ0YmVhdF9saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KHJl cXVlc3RzLCBpOTE1X3JlcXVlc3RfbGl2ZV9zZWxmdGVzdHMpCitzZWxmdGVzdChtaWdyYXRlLCBp bnRlbF9taWdyYXRlX2xpdmVfc2VsZnRlc3RzKQogc2VsZnRlc3QoYWN0aXZlLCBpOTE1X2FjdGl2 ZV9saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KG9iamVjdHMsIGk5MTVfZ2VtX29iamVjdF9saXZl X3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KG1tYW4sIGk5MTVfZ2VtX21tYW5fbGl2ZV9zZWxmdGVzdHMp Ci0tIAoyLjMxLjEKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fCkludGVsLWdmeCBtYWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9y ZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdm eAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 547F9C4743C for ; Mon, 14 Jun 2021 16:27:00 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2397060234 for ; Mon, 14 Jun 2021 16:27:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2397060234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E5C4689EFF; Mon, 14 Jun 2021 16:26:47 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 19AF589EEB; Mon, 14 Jun 2021 16:26:46 +0000 (UTC) IronPort-SDR: mIflQ/xD4M6UJ/8Pw9Z9rrhSvWagEMPcdbbP2sCI+jlHtVK2NNKD09YHCm8K3Qhjn3AIIeUm1K bri8ZsucYvAA== X-IronPort-AV: E=McAfee;i="6200,9189,10015"; a="204008331" X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="204008331" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2021 09:26:38 -0700 IronPort-SDR: lDC7B4uIzlPC2GTH/9L+WjXb2Kn2EA98ojhSHtjCbuTaT4nr5COa+c/P/XHkbrfNjZKoNKkJt6 L2P2EAUfte9A== X-IronPort-AV: E=Sophos;i="5.83,273,1616482800"; d="scan'208";a="449946715" Received: from fnygreen-mobl1.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.50]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2021 09:26:35 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v3 07/12] drm/i915/gt: Pipelined page migration Date: Mon, 14 Jun 2021 18:26:07 +0200 Message-Id: <20210614162612.294869-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210614162612.294869-1-thomas.hellstrom@linux.intel.com> References: <20210614162612.294869-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Chris Wilson If we pipeline the PTE updates and then do the copy of those pages within a single unpreemptible command packet, we can submit the copies and leave them to be scheduled without having to synchronously wait under a global lock. In order to manage migration, we need to preallocate the page tables (and keep them pinned and available for use at any time), causing a bottleneck for migrations as all clients must contend on the limited resources. By inlining the ppGTT updates and performing the blit atomically, each client only owns the PTE while in use, and so we can reschedule individual operations however we see fit. And most importantly, we do not need to take a global lock on the shared vm, and wait until the operation is complete before releasing the lock for others to claim the PTE for themselves. Signed-off-by: Chris Wilson Co-developed-by: Thomas Hellström Signed-off-by: Thomas Hellström --- v2: - Add a TODO for huge LMEM ptes (Pointed out by Matthew Auld) - Use intel_engine_destroy_pinned_context() to properly take the pinned context timeline off the engine list. (CI warning). v3: - Remove an obsolete GEM_BUG_ON (Pointed out by Matthew Auld) - Fix the size argument in allocate_va_range() to not include the base (Pointed out by Matthew Auld) --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_gpu_commands.h | 2 + drivers/gpu/drm/i915/gt/intel_migrate.c | 542 ++++++++++++++++++ drivers/gpu/drm/i915/gt/intel_migrate.h | 45 ++ drivers/gpu/drm/i915/gt/intel_migrate_types.h | 15 + drivers/gpu/drm/i915/gt/intel_ring.h | 1 + drivers/gpu/drm/i915/gt/selftest_migrate.c | 291 ++++++++++ .../drm/i915/selftests/i915_live_selftests.h | 1 + 9 files changed, 899 insertions(+) create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.c create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.h create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate_types.h create mode 100644 drivers/gpu/drm/i915/gt/selftest_migrate.c diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index 7e01ea2c0f00..de4cb9c52585 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -108,6 +108,7 @@ gt-y += \ gt/intel_gtt.o \ gt/intel_llc.o \ gt/intel_lrc.o \ + gt/intel_migrate.o \ gt/intel_mocs.o \ gt/intel_ppgtt.o \ gt/intel_rc6.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 36ea9eb52bb5..62f7440bc111 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -188,6 +188,7 @@ intel_write_status_page(struct intel_engine_cs *engine, int reg, u32 value) #define I915_GEM_HWS_PREEMPT_ADDR (I915_GEM_HWS_PREEMPT * sizeof(u32)) #define I915_GEM_HWS_SEQNO 0x40 #define I915_GEM_HWS_SEQNO_ADDR (I915_GEM_HWS_SEQNO * sizeof(u32)) +#define I915_GEM_HWS_MIGRATE (0x42 * sizeof(u32)) #define I915_GEM_HWS_SCRATCH 0x80 #define I915_HWS_CSB_BUF0_INDEX 0x10 diff --git a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h index 2694dbb9967e..1c3af0fc0456 100644 --- a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h +++ b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h @@ -123,8 +123,10 @@ #define MI_SEMAPHORE_SAD_NEQ_SDD (5 << 12) #define MI_SEMAPHORE_TOKEN_MASK REG_GENMASK(9, 5) #define MI_SEMAPHORE_TOKEN_SHIFT 5 +#define MI_STORE_DATA_IMM MI_INSTR(0x20, 0) #define MI_STORE_DWORD_IMM MI_INSTR(0x20, 1) #define MI_STORE_DWORD_IMM_GEN4 MI_INSTR(0x20, 2) +#define MI_STORE_QWORD_IMM_GEN8 (MI_INSTR(0x20, 3) | REG_BIT(21)) #define MI_MEM_VIRTUAL (1 << 22) /* 945,g33,965 */ #define MI_USE_GGTT (1 << 22) /* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1) diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.c b/drivers/gpu/drm/i915/gt/intel_migrate.c new file mode 100644 index 000000000000..e2e860063e7b --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.c @@ -0,0 +1,542 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "i915_drv.h" +#include "intel_context.h" +#include "intel_gpu_commands.h" +#include "intel_gt.h" +#include "intel_gtt.h" +#include "intel_migrate.h" +#include "intel_ring.h" + +struct insert_pte_data { + u64 offset; + bool is_lmem; +}; + +#define CHUNK_SZ SZ_8M /* ~1ms at 8GiB/s preemption delay */ + +static bool engine_supports_migration(struct intel_engine_cs *engine) +{ + if (!engine) + return false; + + /* + * We need the ability to prevent aribtration (MI_ARB_ON_OFF), + * the ability to write PTE using inline data (MI_STORE_DATA) + * and of course the ability to do the block transfer (blits). + */ + GEM_BUG_ON(engine->class != COPY_ENGINE_CLASS); + + return true; +} + +static void insert_pte(struct i915_address_space *vm, + struct i915_page_table *pt, + void *data) +{ + struct insert_pte_data *d = data; + + vm->insert_page(vm, px_dma(pt), d->offset, I915_CACHE_NONE, + d->is_lmem ? PTE_LM : 0); + d->offset += PAGE_SIZE; +} + +static struct i915_address_space *migrate_vm(struct intel_gt *gt) +{ + struct i915_vm_pt_stash stash = {}; + struct i915_ppgtt *vm; + int err; + int i; + + /* + * We construct a very special VM for use by all migration contexts, + * it is kept pinned so that it can be used at any time. As we need + * to pre-allocate the page directories for the migration VM, this + * limits us to only using a small number of prepared vma. + * + * To be able to pipeline and reschedule migration operations while + * avoiding unnecessary contention on the vm itself, the PTE updates + * are inline with the blits. All the blits use the same fixed + * addresses, with the backing store redirection being updated on the + * fly. Only 2 implicit vma are used for all migration operations. + * + * We lay the ppGTT out as: + * + * [0, CHUNK_SZ) -> first object + * [CHUNK_SZ, 2 * CHUNK_SZ) -> second object + * [2 * CHUNK_SZ, 2 * CHUNK_SZ + 2 * CHUNK_SZ >> 9] -> PTE + * + * By exposing the dma addresses of the page directories themselves + * within the ppGTT, we are then able to rewrite the PTE prior to use. + * But the PTE update and subsequent migration operation must be atomic, + * i.e. within the same non-preemptible window so that we do not switch + * to another migration context that overwrites the PTE. + * + * TODO: Add support for huge LMEM PTEs + */ + + vm = i915_ppgtt_create(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + if (!vm->vm.allocate_va_range || !vm->vm.foreach) { + err = -ENODEV; + goto err_vm; + } + + /* + * Each engine instance is assigned its own chunk in the VM, so + * that we can run multiple instances concurrently + */ + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + struct intel_engine_cs *engine; + u64 base = (u64)i << 32; + struct insert_pte_data d = {}; + struct i915_gem_ww_ctx ww; + u64 sz; + + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (!engine_supports_migration(engine)) + continue; + + /* + * We copy in 8MiB chunks. Each PDE covers 2MiB, so we need + * 4x2 page directories for source/destination. + */ + sz = 2 * CHUNK_SZ; + d.offset = base + sz; + + /* + * We need another page directory setup so that we can write + * the 8x512 PTE in each chunk. + */ + sz += (sz >> 12) * sizeof(u64); + + err = i915_vm_alloc_pt_stash(&vm->vm, &stash, sz); + if (err) + goto err_vm; + + for_i915_gem_ww(&ww, err, true) { + err = i915_vm_lock_objects(&vm->vm, &ww); + if (err) + continue; + err = i915_vm_map_pt_stash(&vm->vm, &stash); + if (err) + continue; + + vm->vm.allocate_va_range(&vm->vm, &stash, base, sz); + } + i915_vm_free_pt_stash(&vm->vm, &stash); + if (err) + goto err_vm; + + /* Now allow the GPU to rewrite the PTE via its own ppGTT */ + d.is_lmem = i915_gem_object_is_lmem(vm->vm.scratch[0]); + vm->vm.foreach(&vm->vm, base, base + sz, insert_pte, &d); + } + + return &vm->vm; + +err_vm: + i915_vm_put(&vm->vm); + return ERR_PTR(err); +} + +static struct intel_engine_cs *first_copy_engine(struct intel_gt *gt) +{ + struct intel_engine_cs *engine; + int i; + + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + return engine; + } + + return NULL; +} + +static struct intel_context *pinned_context(struct intel_gt *gt) +{ + static struct lock_class_key key; + struct intel_engine_cs *engine; + struct i915_address_space *vm; + struct intel_context *ce; + + engine = first_copy_engine(gt); + if (!engine) + return ERR_PTR(-ENODEV); + + vm = migrate_vm(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + ce = intel_engine_create_pinned_context(engine, vm, SZ_512K, + I915_GEM_HWS_MIGRATE, + &key, "migrate"); + i915_vm_put(ce->vm); + return ce; +} + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt) +{ + struct intel_context *ce; + + memset(m, 0, sizeof(*m)); + + ce = pinned_context(gt); + if (IS_ERR(ce)) + return PTR_ERR(ce); + + m->context = ce; + return 0; +} + +static int random_index(unsigned int max) +{ + return upper_32_bits(mul_u32_u32(get_random_u32(), max)); +} + +static struct intel_context *__migrate_engines(struct intel_gt *gt) +{ + struct intel_engine_cs *engines[MAX_ENGINE_INSTANCE]; + struct intel_engine_cs *engine; + unsigned int count, i; + + count = 0; + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + engines[count++] = engine; + } + + return intel_context_create(engines[random_index(count)]); +} + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m) +{ + struct intel_context *ce; + + /* + * We randomly distribute contexts across the engines upon constrction, + * as they all share the same pinned vm, and so in order to allow + * multiple blits to run in parallel, we must construct each blit + * to use a different range of the vm for its GTT. This has to be + * known at construction, so we can not use the late greedy load + * balancing of the virtual-engine. + */ + ce = __migrate_engines(m->context->engine->gt); + if (IS_ERR(ce)) + return ce; + + ce->ring = __intel_context_ring_size(SZ_256K); + + i915_vm_put(ce->vm); + ce->vm = i915_vm_get(m->context->vm); + + return ce; +} + +static inline struct sgt_dma sg_sgt(struct scatterlist *sg) +{ + dma_addr_t addr = sg_dma_address(sg); + + return (struct sgt_dma){ sg, addr, addr + sg_dma_len(sg) }; +} + +static int emit_no_arbitration(struct i915_request *rq) +{ + u32 *cs; + + cs = intel_ring_begin(rq, 2); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Explicitly disable preemption for this request. */ + *cs++ = MI_ARB_ON_OFF; + *cs++ = MI_NOOP; + intel_ring_advance(rq, cs); + + return 0; +} + +static int emit_pte(struct i915_request *rq, + struct sgt_dma *it, + enum i915_cache_level cache_level, + bool is_lmem, + u64 offset, + int length) +{ + const u64 encode = rq->context->vm->pte_encode(0, cache_level, + is_lmem ? PTE_LM : 0); + struct intel_ring *ring = rq->ring; + int total = 0; + u32 *hdr, *cs; + int pkt; + + GEM_BUG_ON(INTEL_GEN(rq->engine->i915) < 8); + + /* Compute the page directory offset for the target address range */ + offset += (u64)rq->engine->instance << 32; + offset >>= 12; + offset *= sizeof(u64); + offset += 2 * CHUNK_SZ; + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Pack as many PTE updates as possible into a single MI command */ + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); /* as qword elements */ + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + + do { + if (cs - hdr >= pkt) { + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + } + + *cs++ = lower_32_bits(encode | it->dma); + *cs++ = upper_32_bits(encode | it->dma); + + offset += 8; + total += I915_GTT_PAGE_SIZE; + + it->dma += I915_GTT_PAGE_SIZE; + if (it->dma >= it->max) { + it->sg = __sg_next(it->sg); + if (!it->sg || sg_dma_len(it->sg) == 0) + break; + + it->dma = sg_dma_address(it->sg); + it->max = it->dma + sg_dma_len(it->sg); + } + } while (total < length); + + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + return total; +} + +static bool wa_1209644611_applies(int gen, u32 size) +{ + u32 height = size >> PAGE_SHIFT; + + if (gen != 11) + return false; + + return height % 4 == 3 && height <= 8; +} + +static int emit_copy(struct i915_request *rq, int size) +{ + const int gen = INTEL_GEN(rq->engine->i915); + u32 instance = rq->engine->instance; + u32 *cs; + + cs = intel_ring_begin(rq, gen >= 8 ? 10 : 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + if (gen >= 9 && !wa_1209644611_applies(gen, size)) { + *cs++ = GEN9_XY_FAST_COPY_BLT_CMD | (10 - 2); + *cs++ = BLT_DEPTH_32 | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else if (gen >= 8) { + *cs++ = XY_SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (10 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else { + GEM_BUG_ON(instance); + *cs++ = SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (6 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + } + + intel_ring_advance(rq, cs); + return 0; +} + +int +intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct sgt_dma it_src = sg_sgt(src), it_dst = sg_sgt(dst); + struct i915_request *rq; + int err; + + *out = NULL; + + GEM_BUG_ON(ce->ring->size < SZ_64K); + + do { + int len; + + rq = i915_request_create(ce); + if (IS_ERR(rq)) { + err = PTR_ERR(rq); + goto out_ce; + } + + if (await) { + err = i915_request_await_dma_fence(rq, await); + if (err) + goto out_rq; + + if (rq->engine->emit_init_breadcrumb) { + err = rq->engine->emit_init_breadcrumb(rq); + if (err) + goto out_rq; + } + + await = NULL; + } + + /* The PTE updates + copy must not be interrupted. */ + err = emit_no_arbitration(rq); + if (err) + goto out_rq; + + len = emit_pte(rq, &it_src, src_cache_level, src_is_lmem, 0, + CHUNK_SZ); + if (len <= 0) { + err = len; + goto out_rq; + } + + err = emit_pte(rq, &it_dst, dst_cache_level, dst_is_lmem, + CHUNK_SZ, len); + if (err < 0) + goto out_rq; + if (err < len) { + err = -EINVAL; + goto out_rq; + } + + err = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (err) + goto out_rq; + + err = emit_copy(rq, len); + + /* Arbitration is re-enabled between requests. */ +out_rq: + if (*out) + i915_request_put(*out); + *out = i915_request_get(rq); + i915_request_add(rq); + if (err || !it_src.sg || !sg_dma_len(it_src.sg)) + break; + + cond_resched(); + } while (1); + +out_ce: + return err; +} + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct intel_context *ce; + int err; + + *out = NULL; + if (!m->context) + return -ENODEV; + + ce = intel_migrate_create_context(m); + if (IS_ERR(ce)) + ce = intel_context_get(m->context); + GEM_BUG_ON(IS_ERR(ce)); + + err = intel_context_pin_ww(ce, ww); + if (err) + goto out; + + err = intel_context_migrate_copy(ce, await, + src, src_cache_level, src_is_lmem, + dst, dst_cache_level, dst_is_lmem, + out); + + intel_context_unpin(ce); +out: + intel_context_put(ce); + return err; +} + +void intel_migrate_fini(struct intel_migrate *m) +{ + struct intel_context *ce; + + ce = fetch_and_zero(&m->context); + if (!ce) + return; + + intel_engine_destroy_pinned_context(ce); +} + +#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) +#include "selftest_migrate.c" +#endif diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.h b/drivers/gpu/drm/i915/gt/intel_migrate.h new file mode 100644 index 000000000000..32c61190ed73 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE__ +#define __INTEL_MIGRATE__ + +#include "intel_migrate_types.h" + +struct dma_fence; +struct i915_request; +struct i915_gem_ww_ctx; +struct intel_gt; +struct scatterlist; +enum i915_cache_level; + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt); + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m); + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +int intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +void intel_migrate_fini(struct intel_migrate *m); + +#endif /* __INTEL_MIGRATE__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_migrate_types.h b/drivers/gpu/drm/i915/gt/intel_migrate_types.h new file mode 100644 index 000000000000..d98230597f42 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate_types.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE_TYPES__ +#define __INTEL_MIGRATE_TYPES__ + +struct intel_context; + +struct intel_migrate { + struct intel_context *context; +}; + +#endif /* __INTEL_MIGRATE_TYPES__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_ring.h b/drivers/gpu/drm/i915/gt/intel_ring.h index dbf5f14a136f..1b32dadfb8c3 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.h +++ b/drivers/gpu/drm/i915/gt/intel_ring.h @@ -49,6 +49,7 @@ static inline void intel_ring_advance(struct i915_request *rq, u32 *cs) * intel_ring_begin()). */ GEM_BUG_ON((rq->ring->vaddr + rq->ring->emit) != cs); + GEM_BUG_ON(!IS_ALIGNED(rq->ring->emit, 8)); /* RING_TAIL qword align */ } static inline u32 intel_ring_wrap(const struct intel_ring *ring, u32 pos) diff --git a/drivers/gpu/drm/i915/gt/selftest_migrate.c b/drivers/gpu/drm/i915/gt/selftest_migrate.c new file mode 100644 index 000000000000..9784d149ebf1 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/selftest_migrate.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "selftests/i915_random.h" + +static const unsigned int sizes[] = { + SZ_4K, + SZ_64K, + SZ_2M, + CHUNK_SZ - SZ_4K, + CHUNK_SZ, + CHUNK_SZ + SZ_4K, + SZ_64M, +}; + +static struct drm_i915_gem_object * +create_lmem_or_internal(struct drm_i915_private *i915, size_t size) +{ + if (HAS_LMEM(i915)) { + struct drm_i915_gem_object *obj; + + obj = i915_gem_object_create_lmem(i915, size, 0); + if (!IS_ERR(obj)) + return obj; + } + + return i915_gem_object_create_internal(i915, size); +} + +static int copy(struct intel_migrate *migrate, + int (*fn)(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out), + u32 sz, struct rnd_state *prng) +{ + struct drm_i915_private *i915 = migrate->context->engine->i915; + struct drm_i915_gem_object *src, *dst; + struct i915_request *rq; + struct i915_gem_ww_ctx ww; + u32 *vaddr; + int err = 0; + int i; + + src = create_lmem_or_internal(i915, sz); + if (IS_ERR(src)) + return 0; + + dst = i915_gem_object_create_internal(i915, sz); + if (IS_ERR(dst)) + goto err_free_src; + + for_i915_gem_ww(&ww, err, true) { + err = i915_gem_object_lock(src, &ww); + if (err) + continue; + + err = i915_gem_object_lock(dst, &ww); + if (err) + continue; + + vaddr = i915_gem_object_pin_map(src, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + continue; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = i; + i915_gem_object_flush_map(src); + + vaddr = i915_gem_object_pin_map(dst, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto unpin_src; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = ~i; + i915_gem_object_flush_map(dst); + + err = fn(migrate, &ww, src, dst, &rq); + if (!err) + continue; + + if (err != -EDEADLK && err != -EINTR && err != -ERESTARTSYS) + pr_err("%ps failed, size: %u\n", fn, sz); + if (rq) { + i915_request_wait(rq, 0, HZ); + i915_request_put(rq); + } + i915_gem_object_unpin_map(dst); +unpin_src: + i915_gem_object_unpin_map(src); + } + if (err) + goto err_out; + + if (rq) { + if (i915_request_wait(rq, 0, HZ) < 0) { + pr_err("%ps timed out, size: %u\n", fn, sz); + err = -ETIME; + } + i915_request_put(rq); + } + + for (i = 0; !err && i < sz / PAGE_SIZE; i++) { + int x = i * 1024 + i915_prandom_u32_max_state(1024, prng); + + if (vaddr[x] != x) { + pr_err("%ps failed, size: %u, offset: %zu\n", + fn, sz, x * sizeof(u32)); + igt_hexdump(vaddr + i * 1024, 4096); + err = -EINVAL; + } + } + + i915_gem_object_unpin_map(dst); + i915_gem_object_unpin_map(src); + +err_out: + i915_gem_object_put(dst); +err_free_src: + i915_gem_object_put(src); + + return err; +} + +static int __migrate_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_migrate_copy(migrate, ww, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int __global_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_context_migrate_copy(migrate->context, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int +migrate_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __migrate_copy, sz, prng); +} + +static int +global_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __global_copy, sz, prng); +} + +static int live_migrate_copy(void *arg) +{ + struct intel_migrate *migrate = arg; + struct drm_i915_private *i915 = migrate->context->engine->i915; + I915_RND_STATE(prng); + int i; + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + int err; + + err = migrate_copy(migrate, sizes[i], &prng); + if (err == 0) + err = global_copy(migrate, sizes[i], &prng); + i915_gem_drain_freed_objects(i915); + if (err) + return err; + } + + return 0; +} + +struct threaded_migrate { + struct intel_migrate *migrate; + struct task_struct *tsk; + struct rnd_state prng; +}; + +static int threaded_migrate(struct intel_migrate *migrate, + int (*fn)(void *arg), + unsigned int flags) +{ + const unsigned int n_cpus = num_online_cpus() + 1; + struct threaded_migrate *thread; + I915_RND_STATE(prng); + unsigned int i; + int err = 0; + + thread = kcalloc(n_cpus, sizeof(*thread), GFP_KERNEL); + if (!thread) + return 0; + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk; + + thread[i].migrate = migrate; + thread[i].prng = + I915_RND_STATE_INITIALIZER(prandom_u32_state(&prng)); + + tsk = kthread_run(fn, &thread[i], "igt-%d", i); + if (IS_ERR(tsk)) { + err = PTR_ERR(tsk); + break; + } + + get_task_struct(tsk); + thread[i].tsk = tsk; + } + + msleep(10); /* start all threads before we kthread_stop() */ + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk = thread[i].tsk; + int status; + + if (IS_ERR_OR_NULL(tsk)) + continue; + + status = kthread_stop(tsk); + if (status && !err) + err = status; + + put_task_struct(tsk); + } + + kfree(thread); + return err; +} + +static int __thread_migrate_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return migrate_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_migrate_copy(void *arg) +{ + return threaded_migrate(arg, __thread_migrate_copy, 0); +} + +static int __thread_global_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return global_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_global_copy(void *arg) +{ + return threaded_migrate(arg, __thread_global_copy, 0); +} + +int intel_migrate_live_selftests(struct drm_i915_private *i915) +{ + static const struct i915_subtest tests[] = { + SUBTEST(live_migrate_copy), + SUBTEST(thread_migrate_copy), + SUBTEST(thread_global_copy), + }; + struct intel_migrate m; + int err; + + if (intel_migrate_init(&m, &i915->gt)) + return 0; + + err = i915_subtests(tests, &m); + intel_migrate_fini(&m); + + return err; +} diff --git a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h index a92c0e9b7e6b..be5e0191eaea 100644 --- a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h +++ b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h @@ -26,6 +26,7 @@ selftest(gt_mocs, intel_mocs_live_selftests) selftest(gt_pm, intel_gt_pm_live_selftests) selftest(gt_heartbeat, intel_heartbeat_live_selftests) selftest(requests, i915_request_live_selftests) +selftest(migrate, intel_migrate_live_selftests) selftest(active, i915_active_live_selftests) selftest(objects, i915_gem_object_live_selftests) selftest(mman, i915_gem_mman_live_selftests) -- 2.31.1