From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB927C2B9F4 for ; Thu, 17 Jun 2021 06:30:51 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FEA1613E7 for ; Thu, 17 Jun 2021 06:30:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FEA1613E7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1B17A6E87B; Thu, 17 Jun 2021 06:30:50 +0000 (UTC) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by gabe.freedesktop.org (Postfix) with ESMTPS id 821436E87B; Thu, 17 Jun 2021 06:30:46 +0000 (UTC) IronPort-SDR: wX5uFlaYLbnLMQSh+zwnrRnfhQeL7JwmSYcDMc1ZevUGaTc0tVKg9WXicb5Jfx9bVhqInBeER5 k13/mPIsARMA== X-IronPort-AV: E=McAfee;i="6200,9189,10017"; a="203287834" X-IronPort-AV: E=Sophos;i="5.83,278,1616482800"; d="scan'208";a="203287834" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2021 23:30:46 -0700 IronPort-SDR: B07bbOGu03TVQCtYXyw/IipnBxK7MZ4FrJoRd3uwa0lkM4DPD1+2ENTJvDumVasu+5sRps/EvL DzYNvChAswZQ== X-IronPort-AV: E=Sophos;i="5.83,278,1616482800"; d="scan'208";a="554302720" Received: from vanderss-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.193]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2021 23:30:43 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Thu, 17 Jun 2021 08:30:13 +0200 Message-Id: <20210617063018.92802-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210617063018.92802-1-thomas.hellstrom@linux.intel.com> References: <20210617063018.92802-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v5 07/12] drm/i915/gt: Pipelined page migration X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" RnJvbTogQ2hyaXMgV2lsc29uIDxjaHJpc0BjaHJpcy13aWxzb24uY28udWs+CgpJZiB3ZSBwaXBl bGluZSB0aGUgUFRFIHVwZGF0ZXMgYW5kIHRoZW4gZG8gdGhlIGNvcHkgb2YgdGhvc2UgcGFnZXMK d2l0aGluIGEgc2luZ2xlIHVucHJlZW1wdGlibGUgY29tbWFuZCBwYWNrZXQsIHdlIGNhbiBzdWJt aXQgdGhlIGNvcGllcwphbmQgbGVhdmUgdGhlbSB0byBiZSBzY2hlZHVsZWQgd2l0aG91dCBoYXZp bmcgdG8gc3luY2hyb25vdXNseSB3YWl0CnVuZGVyIGEgZ2xvYmFsIGxvY2suIEluIG9yZGVyIHRv IG1hbmFnZSBtaWdyYXRpb24sIHdlIG5lZWQgdG8KcHJlYWxsb2NhdGUgdGhlIHBhZ2UgdGFibGVz IChhbmQga2VlcCB0aGVtIHBpbm5lZCBhbmQgYXZhaWxhYmxlIGZvciB1c2UKYXQgYW55IHRpbWUp LCBjYXVzaW5nIGEgYm90dGxlbmVjayBmb3IgbWlncmF0aW9ucyBhcyBhbGwgY2xpZW50cyBtdXN0 CmNvbnRlbmQgb24gdGhlIGxpbWl0ZWQgcmVzb3VyY2VzLiBCeSBpbmxpbmluZyB0aGUgcHBHVFQg dXBkYXRlcyBhbmQKcGVyZm9ybWluZyB0aGUgYmxpdCBhdG9taWNhbGx5LCBlYWNoIGNsaWVudCBv bmx5IG93bnMgdGhlIFBURSB3aGlsZSBpbgp1c2UsIGFuZCBzbyB3ZSBjYW4gcmVzY2hlZHVsZSBp bmRpdmlkdWFsIG9wZXJhdGlvbnMgaG93ZXZlciB3ZSBzZWUgZml0LgpBbmQgbW9zdCBpbXBvcnRh bnRseSwgd2UgZG8gbm90IG5lZWQgdG8gdGFrZSBhIGdsb2JhbCBsb2NrIG9uIHRoZSBzaGFyZWQK dm0sIGFuZCB3YWl0IHVudGlsIHRoZSBvcGVyYXRpb24gaXMgY29tcGxldGUgYmVmb3JlIHJlbGVh c2luZyB0aGUgbG9jawpmb3Igb3RoZXJzIHRvIGNsYWltIHRoZSBQVEUgZm9yIHRoZW1zZWx2ZXMu CgpTaWduZWQtb2ZmLWJ5OiBDaHJpcyBXaWxzb24gPGNocmlzQGNocmlzLXdpbHNvbi5jby51az4K Q28tZGV2ZWxvcGVkLWJ5OiBUaG9tYXMgSGVsbHN0csO2bSA8dGhvbWFzLmhlbGxzdHJvbUBsaW51 eC5pbnRlbC5jb20+ClNpZ25lZC1vZmYtYnk6IFRob21hcyBIZWxsc3Ryw7ZtIDx0aG9tYXMuaGVs bHN0cm9tQGxpbnV4LmludGVsLmNvbT4KUmV2aWV3ZWQtYnk6IE1hdHRoZXcgQXVsZCA8bWF0dGhl dy5hdWxkQGludGVsLmNvbT4KLS0tCnYyOgotIEFkZCBhIFRPRE8gZm9yIGh1Z2UgTE1FTSBwdGVz IChQb2ludGVkIG91dCBieSBNYXR0aGV3IEF1bGQpCi0gVXNlIGludGVsX2VuZ2luZV9kZXN0cm95 X3Bpbm5lZF9jb250ZXh0KCkgdG8gcHJvcGVybHkgdGFrZSB0aGUgcGlubmVkCiAgY29udGV4dCB0 aW1lbGluZSBvZmYgdGhlIGVuZ2luZSBsaXN0LiAoQ0kgd2FybmluZykuCnYzOgotIFJlbW92ZSBh biBvYnNvbGV0ZSBHRU1fQlVHX09OIChQb2ludGVkIG91dCBieSBNYXR0aGV3IEF1bGQpCi0gRml4 IHRoZSBzaXplIGFyZ3VtZW50IGluIGFsbG9jYXRlX3ZhX3JhbmdlKCkgdG8gbm90IGluY2x1ZGUg dGhlIGJhc2UKICAoUG9pbnRlZCBvdXQgYnkgTWF0dGhldyBBdWxkKQotLS0KIGRyaXZlcnMvZ3B1 L2RybS9pOTE1L01ha2VmaWxlICAgICAgICAgICAgICAgICB8ICAgMSArCiBkcml2ZXJzL2dwdS9k cm0vaTkxNS9ndC9pbnRlbF9lbmdpbmUuaCAgICAgICAgfCAgIDEgKwogZHJpdmVycy9ncHUvZHJt L2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1hbmRzLmggIHwgICAyICsKIGRyaXZlcnMvZ3B1L2RybS9p OTE1L2d0L2ludGVsX21pZ3JhdGUuYyAgICAgICB8IDU0MiArKysrKysrKysrKysrKysrKysKIGRy aXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuaCAgICAgICB8ICA0NSArKwogZHJp dmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZV90eXBlcy5oIHwgIDE1ICsKIGRyaXZl cnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3JpbmcuaCAgICAgICAgICB8ICAgMSArCiBkcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMgICAgfCAyOTEgKysrKysrKysrKwog Li4uL2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRlc3RzLmggIHwgICAxICsKIDkg ZmlsZXMgY2hhbmdlZCwgODk5IGluc2VydGlvbnMoKykKIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2 ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlLmMKIGNyZWF0ZSBtb2RlIDEwMDY0NCBk cml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlLmgKIGNyZWF0ZSBtb2RlIDEwMDY0 NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmgKIGNyZWF0ZSBt b2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMKCmRp ZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZSBiL2RyaXZlcnMvZ3B1L2Ry bS9pOTE1L01ha2VmaWxlCmluZGV4IGRkZTY5OGYzYmZmNC4uNWUxMGUwNjI4YzU2IDEwMDY0NAot LS0gYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZQorKysgYi9kcml2ZXJzL2dwdS9kcm0v aTkxNS9NYWtlZmlsZQpAQCAtMTA4LDYgKzEwOCw3IEBAIGd0LXkgKz0gXAogCWd0L2ludGVsX2d0 dC5vIFwKIAlndC9pbnRlbF9sbGMubyBcCiAJZ3QvaW50ZWxfbHJjLm8gXAorCWd0L2ludGVsX21p Z3JhdGUubyBcCiAJZ3QvaW50ZWxfbW9jcy5vIFwKIAlndC9pbnRlbF9wcGd0dC5vIFwKIAlndC9p bnRlbF9yYzYubyBcCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9l bmdpbmUuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2VuZ2luZS5oCmluZGV4IDM2 ZWE5ZWI1MmJiNS4uNjJmNzQ0MGJjMTExIDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vaTkx NS9ndC9pbnRlbF9lbmdpbmUuaAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9l bmdpbmUuaApAQCAtMTg4LDYgKzE4OCw3IEBAIGludGVsX3dyaXRlX3N0YXR1c19wYWdlKHN0cnVj dCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZSwgaW50IHJlZywgdTMyIHZhbHVlKQogI2RlZmluZSBJ OTE1X0dFTV9IV1NfUFJFRU1QVF9BRERSCShJOTE1X0dFTV9IV1NfUFJFRU1QVCAqIHNpemVvZih1 MzIpKQogI2RlZmluZSBJOTE1X0dFTV9IV1NfU0VRTk8JCTB4NDAKICNkZWZpbmUgSTkxNV9HRU1f SFdTX1NFUU5PX0FERFIJCShJOTE1X0dFTV9IV1NfU0VRTk8gKiBzaXplb2YodTMyKSkKKyNkZWZp bmUgSTkxNV9HRU1fSFdTX01JR1JBVEUJCSgweDQyICogc2l6ZW9mKHUzMikpCiAjZGVmaW5lIEk5 MTVfR0VNX0hXU19TQ1JBVENICQkweDgwCiAKICNkZWZpbmUgSTkxNV9IV1NfQ1NCX0JVRjBfSU5E RVgJCTB4MTAKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2dwdV9j b21tYW5kcy5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZ3B1X2NvbW1hbmRzLmgK aW5kZXggMjY5NGRiYjk5NjdlLi4xYzNhZjBmYzA0NTYgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1 L2RybS9pOTE1L2d0L2ludGVsX2dwdV9jb21tYW5kcy5oCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9p OTE1L2d0L2ludGVsX2dwdV9jb21tYW5kcy5oCkBAIC0xMjMsOCArMTIzLDEwIEBACiAjZGVmaW5l ICAgTUlfU0VNQVBIT1JFX1NBRF9ORVFfU0RECSg1IDw8IDEyKQogI2RlZmluZSAgIE1JX1NFTUFQ SE9SRV9UT0tFTl9NQVNLCVJFR19HRU5NQVNLKDksIDUpCiAjZGVmaW5lICAgTUlfU0VNQVBIT1JF X1RPS0VOX1NISUZUCTUKKyNkZWZpbmUgTUlfU1RPUkVfREFUQV9JTU0JTUlfSU5TVFIoMHgyMCwg MCkKICNkZWZpbmUgTUlfU1RPUkVfRFdPUkRfSU1NCU1JX0lOU1RSKDB4MjAsIDEpCiAjZGVmaW5l IE1JX1NUT1JFX0RXT1JEX0lNTV9HRU40CU1JX0lOU1RSKDB4MjAsIDIpCisjZGVmaW5lIE1JX1NU T1JFX1FXT1JEX0lNTV9HRU44IChNSV9JTlNUUigweDIwLCAzKSB8IFJFR19CSVQoMjEpKQogI2Rl ZmluZSAgIE1JX01FTV9WSVJUVUFMCSgxIDw8IDIyKSAvKiA5NDUsZzMzLDk2NSAqLwogI2RlZmlu ZSAgIE1JX1VTRV9HR1RUCQkoMSA8PCAyMikgLyogZzR4KyAqLwogI2RlZmluZSBNSV9TVE9SRV9E V09SRF9JTkRFWAlNSV9JTlNUUigweDIxLCAxKQpkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJt L2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5jIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxf bWlncmF0ZS5jCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwMC4uZTJlODYw MDYzZTdiCi0tLSAvZGV2L251bGwKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxf bWlncmF0ZS5jCkBAIC0wLDAgKzEsNTQyIEBACisvLyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjog TUlUCisvKgorICogQ29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisj aW5jbHVkZSAiaTkxNV9kcnYuaCIKKyNpbmNsdWRlICJpbnRlbF9jb250ZXh0LmgiCisjaW5jbHVk ZSAiaW50ZWxfZ3B1X2NvbW1hbmRzLmgiCisjaW5jbHVkZSAiaW50ZWxfZ3QuaCIKKyNpbmNsdWRl ICJpbnRlbF9ndHQuaCIKKyNpbmNsdWRlICJpbnRlbF9taWdyYXRlLmgiCisjaW5jbHVkZSAiaW50 ZWxfcmluZy5oIgorCitzdHJ1Y3QgaW5zZXJ0X3B0ZV9kYXRhIHsKKwl1NjQgb2Zmc2V0OworCWJv b2wgaXNfbG1lbTsKK307CisKKyNkZWZpbmUgQ0hVTktfU1ogU1pfOE0gLyogfjFtcyBhdCA4R2lC L3MgcHJlZW1wdGlvbiBkZWxheSAqLworCitzdGF0aWMgYm9vbCBlbmdpbmVfc3VwcG9ydHNfbWln cmF0aW9uKHN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZSkKK3sKKwlpZiAoIWVuZ2luZSkK KwkJcmV0dXJuIGZhbHNlOworCisJLyoKKwkgKiBXZSBuZWVkIHRoZSBhYmlsaXR5IHRvIHByZXZl bnQgYXJpYnRyYXRpb24gKE1JX0FSQl9PTl9PRkYpLAorCSAqIHRoZSBhYmlsaXR5IHRvIHdyaXRl IFBURSB1c2luZyBpbmxpbmUgZGF0YSAoTUlfU1RPUkVfREFUQSkKKwkgKiBhbmQgb2YgY291cnNl IHRoZSBhYmlsaXR5IHRvIGRvIHRoZSBibG9jayB0cmFuc2ZlciAoYmxpdHMpLgorCSAqLworCUdF TV9CVUdfT04oZW5naW5lLT5jbGFzcyAhPSBDT1BZX0VOR0lORV9DTEFTUyk7CisKKwlyZXR1cm4g dHJ1ZTsKK30KKworc3RhdGljIHZvaWQgaW5zZXJ0X3B0ZShzdHJ1Y3QgaTkxNV9hZGRyZXNzX3Nw YWNlICp2bSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X3BhZ2VfdGFibGUgKnB0LAorCQkgICAgICAg dm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgaW5zZXJ0X3B0ZV9kYXRhICpkID0gZGF0YTsKKworCXZt LT5pbnNlcnRfcGFnZSh2bSwgcHhfZG1hKHB0KSwgZC0+b2Zmc2V0LCBJOTE1X0NBQ0hFX05PTkUs CisJCQlkLT5pc19sbWVtID8gUFRFX0xNIDogMCk7CisJZC0+b2Zmc2V0ICs9IFBBR0VfU0laRTsK K30KKworc3RhdGljIHN0cnVjdCBpOTE1X2FkZHJlc3Nfc3BhY2UgKm1pZ3JhdGVfdm0oc3RydWN0 IGludGVsX2d0ICpndCkKK3sKKwlzdHJ1Y3QgaTkxNV92bV9wdF9zdGFzaCBzdGFzaCA9IHt9Owor CXN0cnVjdCBpOTE1X3BwZ3R0ICp2bTsKKwlpbnQgZXJyOworCWludCBpOworCisJLyoKKwkgKiBX ZSBjb25zdHJ1Y3QgYSB2ZXJ5IHNwZWNpYWwgVk0gZm9yIHVzZSBieSBhbGwgbWlncmF0aW9uIGNv bnRleHRzLAorCSAqIGl0IGlzIGtlcHQgcGlubmVkIHNvIHRoYXQgaXQgY2FuIGJlIHVzZWQgYXQg YW55IHRpbWUuIEFzIHdlIG5lZWQKKwkgKiB0byBwcmUtYWxsb2NhdGUgdGhlIHBhZ2UgZGlyZWN0 b3JpZXMgZm9yIHRoZSBtaWdyYXRpb24gVk0sIHRoaXMKKwkgKiBsaW1pdHMgdXMgdG8gb25seSB1 c2luZyBhIHNtYWxsIG51bWJlciBvZiBwcmVwYXJlZCB2bWEuCisJICoKKwkgKiBUbyBiZSBhYmxl IHRvIHBpcGVsaW5lIGFuZCByZXNjaGVkdWxlIG1pZ3JhdGlvbiBvcGVyYXRpb25zIHdoaWxlCisJ ICogYXZvaWRpbmcgdW5uZWNlc3NhcnkgY29udGVudGlvbiBvbiB0aGUgdm0gaXRzZWxmLCB0aGUg UFRFIHVwZGF0ZXMKKwkgKiBhcmUgaW5saW5lIHdpdGggdGhlIGJsaXRzLiBBbGwgdGhlIGJsaXRz IHVzZSB0aGUgc2FtZSBmaXhlZAorCSAqIGFkZHJlc3Nlcywgd2l0aCB0aGUgYmFja2luZyBzdG9y ZSByZWRpcmVjdGlvbiBiZWluZyB1cGRhdGVkIG9uIHRoZQorCSAqIGZseS4gT25seSAyIGltcGxp Y2l0IHZtYSBhcmUgdXNlZCBmb3IgYWxsIG1pZ3JhdGlvbiBvcGVyYXRpb25zLgorCSAqCisJICog V2UgbGF5IHRoZSBwcEdUVCBvdXQgYXM6CisJICoKKwkgKglbMCwgQ0hVTktfU1opIC0+IGZpcnN0 IG9iamVjdAorCSAqCVtDSFVOS19TWiwgMiAqIENIVU5LX1NaKSAtPiBzZWNvbmQgb2JqZWN0CisJ ICoJWzIgKiBDSFVOS19TWiwgMiAqIENIVU5LX1NaICsgMiAqIENIVU5LX1NaID4+IDldIC0+IFBU RQorCSAqCisJICogQnkgZXhwb3NpbmcgdGhlIGRtYSBhZGRyZXNzZXMgb2YgdGhlIHBhZ2UgZGly ZWN0b3JpZXMgdGhlbXNlbHZlcworCSAqIHdpdGhpbiB0aGUgcHBHVFQsIHdlIGFyZSB0aGVuIGFi bGUgdG8gcmV3cml0ZSB0aGUgUFRFIHByaW9yIHRvIHVzZS4KKwkgKiBCdXQgdGhlIFBURSB1cGRh dGUgYW5kIHN1YnNlcXVlbnQgbWlncmF0aW9uIG9wZXJhdGlvbiBtdXN0IGJlIGF0b21pYywKKwkg KiBpLmUuIHdpdGhpbiB0aGUgc2FtZSBub24tcHJlZW1wdGlibGUgd2luZG93IHNvIHRoYXQgd2Ug ZG8gbm90IHN3aXRjaAorCSAqIHRvIGFub3RoZXIgbWlncmF0aW9uIGNvbnRleHQgdGhhdCBvdmVy d3JpdGVzIHRoZSBQVEUuCisJICoKKwkgKiBUT0RPOiBBZGQgc3VwcG9ydCBmb3IgaHVnZSBMTUVN IFBURXMKKwkgKi8KKworCXZtID0gaTkxNV9wcGd0dF9jcmVhdGUoZ3QpOworCWlmIChJU19FUlIo dm0pKQorCQlyZXR1cm4gRVJSX0NBU1Qodm0pOworCisJaWYgKCF2bS0+dm0uYWxsb2NhdGVfdmFf cmFuZ2UgfHwgIXZtLT52bS5mb3JlYWNoKSB7CisJCWVyciA9IC1FTk9ERVY7CisJCWdvdG8gZXJy X3ZtOworCX0KKworCS8qCisJICogRWFjaCBlbmdpbmUgaW5zdGFuY2UgaXMgYXNzaWduZWQgaXRz IG93biBjaHVuayBpbiB0aGUgVk0sIHNvCisJICogdGhhdCB3ZSBjYW4gcnVuIG11bHRpcGxlIGlu c3RhbmNlcyBjb25jdXJyZW50bHkKKwkgKi8KKwlmb3IgKGkgPSAwOyBpIDwgQVJSQVlfU0laRShn dC0+ZW5naW5lX2NsYXNzW0NPUFlfRU5HSU5FX0NMQVNTXSk7IGkrKykgeworCQlzdHJ1Y3QgaW50 ZWxfZW5naW5lX2NzICplbmdpbmU7CisJCXU2NCBiYXNlID0gKHU2NClpIDw8IDMyOworCQlzdHJ1 Y3QgaW5zZXJ0X3B0ZV9kYXRhIGQgPSB7fTsKKwkJc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCB3dzsK KwkJdTY0IHN6OworCisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xB U1NdW2ldOworCQlpZiAoIWVuZ2luZV9zdXBwb3J0c19taWdyYXRpb24oZW5naW5lKSkKKwkJCWNv bnRpbnVlOworCisJCS8qCisJCSAqIFdlIGNvcHkgaW4gOE1pQiBjaHVua3MuIEVhY2ggUERFIGNv dmVycyAyTWlCLCBzbyB3ZSBuZWVkCisJCSAqIDR4MiBwYWdlIGRpcmVjdG9yaWVzIGZvciBzb3Vy Y2UvZGVzdGluYXRpb24uCisJCSAqLworCQlzeiA9IDIgKiBDSFVOS19TWjsKKwkJZC5vZmZzZXQg PSBiYXNlICsgc3o7CisKKwkJLyoKKwkJICogV2UgbmVlZCBhbm90aGVyIHBhZ2UgZGlyZWN0b3J5 IHNldHVwIHNvIHRoYXQgd2UgY2FuIHdyaXRlCisJCSAqIHRoZSA4eDUxMiBQVEUgaW4gZWFjaCBj aHVuay4KKwkJICovCisJCXN6ICs9IChzeiA+PiAxMikgKiBzaXplb2YodTY0KTsKKworCQllcnIg PSBpOTE1X3ZtX2FsbG9jX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCwgc3opOworCQlpZiAoZXJy KQorCQkJZ290byBlcnJfdm07CisKKwkJZm9yX2k5MTVfZ2VtX3d3KCZ3dywgZXJyLCB0cnVlKSB7 CisJCQllcnIgPSBpOTE1X3ZtX2xvY2tfb2JqZWN0cygmdm0tPnZtLCAmd3cpOworCQkJaWYgKGVy cikKKwkJCQljb250aW51ZTsKKwkJCWVyciA9IGk5MTVfdm1fbWFwX3B0X3N0YXNoKCZ2bS0+dm0s ICZzdGFzaCk7CisJCQlpZiAoZXJyKQorCQkJCWNvbnRpbnVlOworCisJCQl2bS0+dm0uYWxsb2Nh dGVfdmFfcmFuZ2UoJnZtLT52bSwgJnN0YXNoLCBiYXNlLCBzeik7CisJCX0KKwkJaTkxNV92bV9m cmVlX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCk7CisJCWlmIChlcnIpCisJCQlnb3RvIGVycl92 bTsKKworCQkvKiBOb3cgYWxsb3cgdGhlIEdQVSB0byByZXdyaXRlIHRoZSBQVEUgdmlhIGl0cyBv d24gcHBHVFQgKi8KKwkJZC5pc19sbWVtID0gaTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0odm0tPnZt LnNjcmF0Y2hbMF0pOworCQl2bS0+dm0uZm9yZWFjaCgmdm0tPnZtLCBiYXNlLCBiYXNlICsgc3os IGluc2VydF9wdGUsICZkKTsKKwl9CisKKwlyZXR1cm4gJnZtLT52bTsKKworZXJyX3ZtOgorCWk5 MTVfdm1fcHV0KCZ2bS0+dm0pOworCXJldHVybiBFUlJfUFRSKGVycik7Cit9CisKK3N0YXRpYyBz dHJ1Y3QgaW50ZWxfZW5naW5lX2NzICpmaXJzdF9jb3B5X2VuZ2luZShzdHJ1Y3QgaW50ZWxfZ3Qg Kmd0KQoreworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwlpbnQgaTsKKworCWZv ciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xB U1NdKTsgaSsrKSB7CisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xB U1NdW2ldOworCQlpZiAoZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihlbmdpbmUpKQorCQkJcmV0 dXJuIGVuZ2luZTsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworc3RhdGljIHN0cnVjdCBpbnRl bF9jb250ZXh0ICpwaW5uZWRfY29udGV4dChzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0YXRp YyBzdHJ1Y3QgbG9ja19jbGFzc19rZXkga2V5OworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVu Z2luZTsKKwlzdHJ1Y3QgaTkxNV9hZGRyZXNzX3NwYWNlICp2bTsKKwlzdHJ1Y3QgaW50ZWxfY29u dGV4dCAqY2U7CisKKwllbmdpbmUgPSBmaXJzdF9jb3B5X2VuZ2luZShndCk7CisJaWYgKCFlbmdp bmUpCisJCXJldHVybiBFUlJfUFRSKC1FTk9ERVYpOworCisJdm0gPSBtaWdyYXRlX3ZtKGd0KTsK KwlpZiAoSVNfRVJSKHZtKSkKKwkJcmV0dXJuIEVSUl9DQVNUKHZtKTsKKworCWNlID0gaW50ZWxf ZW5naW5lX2NyZWF0ZV9waW5uZWRfY29udGV4dChlbmdpbmUsIHZtLCBTWl81MTJLLAorCQkJCQkJ STkxNV9HRU1fSFdTX01JR1JBVEUsCisJCQkJCQkma2V5LCAibWlncmF0ZSIpOworCWk5MTVfdm1f cHV0KGNlLT52bSk7CisJcmV0dXJuIGNlOworfQorCitpbnQgaW50ZWxfbWlncmF0ZV9pbml0KHN0 cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0cnVjdCBp bnRlbF9jb250ZXh0ICpjZTsKKworCW1lbXNldChtLCAwLCBzaXplb2YoKm0pKTsKKworCWNlID0g cGlubmVkX2NvbnRleHQoZ3QpOworCWlmIChJU19FUlIoY2UpKQorCQlyZXR1cm4gUFRSX0VSUihj ZSk7CisKKwltLT5jb250ZXh0ID0gY2U7CisJcmV0dXJuIDA7Cit9CisKK3N0YXRpYyBpbnQgcmFu ZG9tX2luZGV4KHVuc2lnbmVkIGludCBtYXgpCit7CisJcmV0dXJuIHVwcGVyXzMyX2JpdHMobXVs X3UzMl91MzIoZ2V0X3JhbmRvbV91MzIoKSwgbWF4KSk7Cit9CisKK3N0YXRpYyBzdHJ1Y3QgaW50 ZWxfY29udGV4dCAqX19taWdyYXRlX2VuZ2luZXMoc3RydWN0IGludGVsX2d0ICpndCkKK3sKKwlz dHJ1Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmVzW01BWF9FTkdJTkVfSU5TVEFOQ0VdOworCXN0 cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwl1bnNpZ25lZCBpbnQgY291bnQsIGk7CisK Kwljb3VudCA9IDA7CisJZm9yIChpID0gMDsgaSA8IEFSUkFZX1NJWkUoZ3QtPmVuZ2luZV9jbGFz c1tDT1BZX0VOR0lORV9DTEFTU10pOyBpKyspIHsKKwkJZW5naW5lID0gZ3QtPmVuZ2luZV9jbGFz c1tDT1BZX0VOR0lORV9DTEFTU11baV07CisJCWlmIChlbmdpbmVfc3VwcG9ydHNfbWlncmF0aW9u KGVuZ2luZSkpCisJCQllbmdpbmVzW2NvdW50KytdID0gZW5naW5lOworCX0KKworCXJldHVybiBp bnRlbF9jb250ZXh0X2NyZWF0ZShlbmdpbmVzW3JhbmRvbV9pbmRleChjb3VudCldKTsKK30KKwor c3RydWN0IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQoc3RydWN0 IGludGVsX21pZ3JhdGUgKm0pCit7CisJc3RydWN0IGludGVsX2NvbnRleHQgKmNlOworCisJLyoK KwkgKiBXZSByYW5kb21seSBkaXN0cmlidXRlIGNvbnRleHRzIGFjcm9zcyB0aGUgZW5naW5lcyB1 cG9uIGNvbnN0cmN0aW9uLAorCSAqIGFzIHRoZXkgYWxsIHNoYXJlIHRoZSBzYW1lIHBpbm5lZCB2 bSwgYW5kIHNvIGluIG9yZGVyIHRvIGFsbG93CisJICogbXVsdGlwbGUgYmxpdHMgdG8gcnVuIGlu IHBhcmFsbGVsLCB3ZSBtdXN0IGNvbnN0cnVjdCBlYWNoIGJsaXQKKwkgKiB0byB1c2UgYSBkaWZm ZXJlbnQgcmFuZ2Ugb2YgdGhlIHZtIGZvciBpdHMgR1RULiBUaGlzIGhhcyB0byBiZQorCSAqIGtu b3duIGF0IGNvbnN0cnVjdGlvbiwgc28gd2UgY2FuIG5vdCB1c2UgdGhlIGxhdGUgZ3JlZWR5IGxv YWQKKwkgKiBiYWxhbmNpbmcgb2YgdGhlIHZpcnR1YWwtZW5naW5lLgorCSAqLworCWNlID0gX19t aWdyYXRlX2VuZ2luZXMobS0+Y29udGV4dC0+ZW5naW5lLT5ndCk7CisJaWYgKElTX0VSUihjZSkp CisJCXJldHVybiBjZTsKKworCWNlLT5yaW5nID0gX19pbnRlbF9jb250ZXh0X3Jpbmdfc2l6ZShT Wl8yNTZLKTsKKworCWk5MTVfdm1fcHV0KGNlLT52bSk7CisJY2UtPnZtID0gaTkxNV92bV9nZXQo bS0+Y29udGV4dC0+dm0pOworCisJcmV0dXJuIGNlOworfQorCitzdGF0aWMgaW5saW5lIHN0cnVj dCBzZ3RfZG1hIHNnX3NndChzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnKQoreworCWRtYV9hZGRyX3Qg YWRkciA9IHNnX2RtYV9hZGRyZXNzKHNnKTsKKworCXJldHVybiAoc3RydWN0IHNndF9kbWEpeyBz ZywgYWRkciwgYWRkciArIHNnX2RtYV9sZW4oc2cpIH07Cit9CisKK3N0YXRpYyBpbnQgZW1pdF9u b19hcmJpdHJhdGlvbihzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSkKK3sKKwl1MzIgKmNzOworCisJ Y3MgPSBpbnRlbF9yaW5nX2JlZ2luKHJxLCAyKTsKKwlpZiAoSVNfRVJSKGNzKSkKKwkJcmV0dXJu IFBUUl9FUlIoY3MpOworCisJLyogRXhwbGljaXRseSBkaXNhYmxlIHByZWVtcHRpb24gZm9yIHRo aXMgcmVxdWVzdC4gKi8KKwkqY3MrKyA9IE1JX0FSQl9PTl9PRkY7CisJKmNzKysgPSBNSV9OT09Q OworCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCisJcmV0dXJuIDA7Cit9CisKK3N0YXRp YyBpbnQgZW1pdF9wdGUoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsCisJCSAgICBzdHJ1Y3Qgc2d0 X2RtYSAqaXQsCisJCSAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgY2FjaGVfbGV2ZWwsCisJCSAg ICBib29sIGlzX2xtZW0sCisJCSAgICB1NjQgb2Zmc2V0LAorCQkgICAgaW50IGxlbmd0aCkKK3sK Kwljb25zdCB1NjQgZW5jb2RlID0gcnEtPmNvbnRleHQtPnZtLT5wdGVfZW5jb2RlKDAsIGNhY2hl X2xldmVsLAorCQkJCQkJICAgICAgIGlzX2xtZW0gPyBQVEVfTE0gOiAwKTsKKwlzdHJ1Y3QgaW50 ZWxfcmluZyAqcmluZyA9IHJxLT5yaW5nOworCWludCB0b3RhbCA9IDA7CisJdTMyICpoZHIsICpj czsKKwlpbnQgcGt0OworCisJR0VNX0JVR19PTihJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSkg PCA4KTsKKworCS8qIENvbXB1dGUgdGhlIHBhZ2UgZGlyZWN0b3J5IG9mZnNldCBmb3IgdGhlIHRh cmdldCBhZGRyZXNzIHJhbmdlICovCisJb2Zmc2V0ICs9ICh1NjQpcnEtPmVuZ2luZS0+aW5zdGFu Y2UgPDwgMzI7CisJb2Zmc2V0ID4+PSAxMjsKKwlvZmZzZXQgKj0gc2l6ZW9mKHU2NCk7CisJb2Zm c2V0ICs9IDIgKiBDSFVOS19TWjsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgNik7CisJ aWYgKElTX0VSUihjcykpCisJCXJldHVybiBQVFJfRVJSKGNzKTsKKworCS8qIFBhY2sgYXMgbWFu eSBQVEUgdXBkYXRlcyBhcyBwb3NzaWJsZSBpbnRvIGEgc2luZ2xlIE1JIGNvbW1hbmQgKi8KKwlw a3QgPSBtaW5fdChpbnQsIDB4NDAwLCByaW5nLT5zcGFjZSAvIHNpemVvZih1MzIpICsgNSk7CisJ cGt0ID0gbWluX3QoaW50LCBwa3QsIChyaW5nLT5zaXplIC0gcmluZy0+ZW1pdCkgLyBzaXplb2Yo dTMyKSArIDUpOworCisJaGRyID0gY3M7CisJKmNzKysgPSBNSV9TVE9SRV9EQVRBX0lNTSB8IFJF R19CSVQoMjEpOyAvKiBhcyBxd29yZCBlbGVtZW50cyAqLworCSpjcysrID0gbG93ZXJfMzJfYml0 cyhvZmZzZXQpOworCSpjcysrID0gdXBwZXJfMzJfYml0cyhvZmZzZXQpOworCisJZG8geworCQlp ZiAoY3MgLSBoZHIgPj0gcGt0KSB7CisJCQkqaGRyICs9IGNzIC0gaGRyIC0gMjsKKwkJCSpjcysr ID0gTUlfTk9PUDsKKworCQkJcmluZy0+ZW1pdCA9ICh2b2lkICopY3MgLSByaW5nLT52YWRkcjsK KwkJCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCQkJaW50ZWxfcmluZ191cGRhdGVfc3Bh Y2UocmluZyk7CisKKwkJCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgNik7CisJCQlpZiAoSVNf RVJSKGNzKSkKKwkJCQlyZXR1cm4gUFRSX0VSUihjcyk7CisKKwkJCXBrdCA9IG1pbl90KGludCwg MHg0MDAsIHJpbmctPnNwYWNlIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwkJCXBrdCA9IG1pbl90KGlu dCwgcGt0LCAocmluZy0+c2l6ZSAtIHJpbmctPmVtaXQpIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwor CQkJaGRyID0gY3M7CisJCQkqY3MrKyA9IE1JX1NUT1JFX0RBVEFfSU1NIHwgUkVHX0JJVCgyMSk7 CisJCQkqY3MrKyA9IGxvd2VyXzMyX2JpdHMob2Zmc2V0KTsKKwkJCSpjcysrID0gdXBwZXJfMzJf Yml0cyhvZmZzZXQpOworCQl9CisKKwkJKmNzKysgPSBsb3dlcl8zMl9iaXRzKGVuY29kZSB8IGl0 LT5kbWEpOworCQkqY3MrKyA9IHVwcGVyXzMyX2JpdHMoZW5jb2RlIHwgaXQtPmRtYSk7CisKKwkJ b2Zmc2V0ICs9IDg7CisJCXRvdGFsICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKworCQlpdC0+ZG1h ICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKwkJaWYgKGl0LT5kbWEgPj0gaXQtPm1heCkgeworCQkJ aXQtPnNnID0gX19zZ19uZXh0KGl0LT5zZyk7CisJCQlpZiAoIWl0LT5zZyB8fCBzZ19kbWFfbGVu KGl0LT5zZykgPT0gMCkKKwkJCQlicmVhazsKKworCQkJaXQtPmRtYSA9IHNnX2RtYV9hZGRyZXNz KGl0LT5zZyk7CisJCQlpdC0+bWF4ID0gaXQtPmRtYSArIHNnX2RtYV9sZW4oaXQtPnNnKTsKKwkJ fQorCX0gd2hpbGUgKHRvdGFsIDwgbGVuZ3RoKTsKKworCSpoZHIgKz0gY3MgLSBoZHIgLSAyOwor CSpjcysrID0gTUlfTk9PUDsKKworCXJpbmctPmVtaXQgPSAodm9pZCAqKWNzIC0gcmluZy0+dmFk ZHI7CisJaW50ZWxfcmluZ19hZHZhbmNlKHJxLCBjcyk7CisJaW50ZWxfcmluZ191cGRhdGVfc3Bh Y2UocmluZyk7CisKKwlyZXR1cm4gdG90YWw7Cit9CisKK3N0YXRpYyBib29sIHdhXzEyMDk2NDQ2 MTFfYXBwbGllcyhpbnQgZ2VuLCB1MzIgc2l6ZSkKK3sKKwl1MzIgaGVpZ2h0ID0gc2l6ZSA+PiBQ QUdFX1NISUZUOworCisJaWYgKGdlbiAhPSAxMSkKKwkJcmV0dXJuIGZhbHNlOworCisJcmV0dXJu IGhlaWdodCAlIDQgPT0gMyAmJiBoZWlnaHQgPD0gODsKK30KKworc3RhdGljIGludCBlbWl0X2Nv cHkoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsIGludCBzaXplKQoreworCWNvbnN0IGludCBnZW4g PSBJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSk7CisJdTMyIGluc3RhbmNlID0gcnEtPmVuZ2lu ZS0+aW5zdGFuY2U7CisJdTMyICpjczsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgZ2Vu ID49IDggPyAxMCA6IDYpOworCWlmIChJU19FUlIoY3MpKQorCQlyZXR1cm4gUFRSX0VSUihjcyk7 CisKKwlpZiAoZ2VuID49IDkgJiYgIXdhXzEyMDk2NDQ2MTFfYXBwbGllcyhnZW4sIHNpemUpKSB7 CisJCSpjcysrID0gR0VOOV9YWV9GQVNUX0NPUFlfQkxUX0NNRCB8ICgxMCAtIDIpOworCQkqY3Mr KyA9IEJMVF9ERVBUSF8zMiB8IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOworCQkqY3MrKyA9IHNp emUgPj4gUEFHRV9TSElGVCA8PCAxNiB8IFBBR0VfU0laRSAvIDQ7CisJCSpjcysrID0gQ0hVTktf U1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBpbnN0YW5jZTsKKwkJKmNzKysgPSAwOwor CQkqY3MrKyA9IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOyAvKiBzcmMgb2Zmc2V0ICovCisJCSpj cysrID0gaW5zdGFuY2U7CisJfSBlbHNlIGlmIChnZW4gPj0gOCkgeworCQkqY3MrKyA9IFhZX1NS Q19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICgxMCAtIDIpOworCQkqY3MrKyA9IEJM VF9ERVBUSF8zMiB8IEJMVF9ST1BfU1JDX0NPUFkgfCBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsK KwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJWkUgLyA0OworCQkq Y3MrKyA9IENIVU5LX1NaOyAvKiBkc3Qgb2Zmc2V0ICovCisJCSpjcysrID0gaW5zdGFuY2U7CisJ CSpjcysrID0gMDsKKwkJKmNzKysgPSBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsgLyogc3JjIG9m ZnNldCAqLworCQkqY3MrKyA9IGluc3RhbmNlOworCX0gZWxzZSB7CisJCUdFTV9CVUdfT04oaW5z dGFuY2UpOworCQkqY3MrKyA9IFNSQ19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICg2 IC0gMik7CisJCSpjcysrID0gQkxUX0RFUFRIXzMyIHwgQkxUX1JPUF9TUkNfQ09QWSB8IFBBR0Vf U0laRTsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJWkU7CisJ CSpjcysrID0gQ0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBQQUdFX1NJWkU7 CisJCSpjcysrID0gMDsgLyogc3JjIG9mZnNldCAqLworCX0KKworCWludGVsX3JpbmdfYWR2YW5j ZShycSwgY3MpOworCXJldHVybiAwOworfQorCitpbnQKK2ludGVsX2NvbnRleHRfbWlncmF0ZV9j b3B5KHN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZSwKKwkJCSAgIHN0cnVjdCBkbWFfZmVuY2UgKmF3 YWl0LAorCQkJICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkgICBlbnVtIGk5MTVfY2Fj aGVfbGV2ZWwgc3JjX2NhY2hlX2xldmVsLAorCQkJICAgYm9vbCBzcmNfaXNfbG1lbSwKKwkJCSAg IHN0cnVjdCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIGRz dF9jYWNoZV9sZXZlbCwKKwkJCSAgIGJvb2wgZHN0X2lzX2xtZW0sCisJCQkgICBzdHJ1Y3QgaTkx NV9yZXF1ZXN0ICoqb3V0KQoreworCXN0cnVjdCBzZ3RfZG1hIGl0X3NyYyA9IHNnX3NndChzcmMp LCBpdF9kc3QgPSBzZ19zZ3QoZHN0KTsKKwlzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycTsKKwlpbnQg ZXJyOworCisJKm91dCA9IE5VTEw7CisKKwlHRU1fQlVHX09OKGNlLT5yaW5nLT5zaXplIDwgU1pf NjRLKTsKKworCWRvIHsKKwkJaW50IGxlbjsKKworCQlycSA9IGk5MTVfcmVxdWVzdF9jcmVhdGUo Y2UpOworCQlpZiAoSVNfRVJSKHJxKSkgeworCQkJZXJyID0gUFRSX0VSUihycSk7CisJCQlnb3Rv IG91dF9jZTsKKwkJfQorCisJCWlmIChhd2FpdCkgeworCQkJZXJyID0gaTkxNV9yZXF1ZXN0X2F3 YWl0X2RtYV9mZW5jZShycSwgYXdhaXQpOworCQkJaWYgKGVycikKKwkJCQlnb3RvIG91dF9ycTsK KworCQkJaWYgKHJxLT5lbmdpbmUtPmVtaXRfaW5pdF9icmVhZGNydW1iKSB7CisJCQkJZXJyID0g cnEtPmVuZ2luZS0+ZW1pdF9pbml0X2JyZWFkY3J1bWIocnEpOworCQkJCWlmIChlcnIpCisJCQkJ CWdvdG8gb3V0X3JxOworCQkJfQorCisJCQlhd2FpdCA9IE5VTEw7CisJCX0KKworCQkvKiBUaGUg UFRFIHVwZGF0ZXMgKyBjb3B5IG11c3Qgbm90IGJlIGludGVycnVwdGVkLiAqLworCQllcnIgPSBl bWl0X25vX2FyYml0cmF0aW9uKHJxKTsKKwkJaWYgKGVycikKKwkJCWdvdG8gb3V0X3JxOworCisJ CWxlbiA9IGVtaXRfcHRlKHJxLCAmaXRfc3JjLCBzcmNfY2FjaGVfbGV2ZWwsIHNyY19pc19sbWVt LCAwLAorCQkJICAgICAgIENIVU5LX1NaKTsKKwkJaWYgKGxlbiA8PSAwKSB7CisJCQllcnIgPSBs ZW47CisJCQlnb3RvIG91dF9ycTsKKwkJfQorCisJCWVyciA9IGVtaXRfcHRlKHJxLCAmaXRfZHN0 LCBkc3RfY2FjaGVfbGV2ZWwsIGRzdF9pc19sbWVtLAorCQkJICAgICAgIENIVU5LX1NaLCBsZW4p OworCQlpZiAoZXJyIDwgMCkKKwkJCWdvdG8gb3V0X3JxOworCQlpZiAoZXJyIDwgbGVuKSB7CisJ CQllcnIgPSAtRUlOVkFMOworCQkJZ290byBvdXRfcnE7CisJCX0KKworCQllcnIgPSBycS0+ZW5n aW5lLT5lbWl0X2ZsdXNoKHJxLCBFTUlUX0lOVkFMSURBVEUpOworCQlpZiAoZXJyKQorCQkJZ290 byBvdXRfcnE7CisKKwkJZXJyID0gZW1pdF9jb3B5KHJxLCBsZW4pOworCisJCS8qIEFyYml0cmF0 aW9uIGlzIHJlLWVuYWJsZWQgYmV0d2VlbiByZXF1ZXN0cy4gKi8KK291dF9ycToKKwkJaWYgKCpv dXQpCisJCQlpOTE1X3JlcXVlc3RfcHV0KCpvdXQpOworCQkqb3V0ID0gaTkxNV9yZXF1ZXN0X2dl dChycSk7CisJCWk5MTVfcmVxdWVzdF9hZGQocnEpOworCQlpZiAoZXJyIHx8ICFpdF9zcmMuc2cg fHwgIXNnX2RtYV9sZW4oaXRfc3JjLnNnKSkKKwkJCWJyZWFrOworCisJCWNvbmRfcmVzY2hlZCgp OworCX0gd2hpbGUgKDEpOworCitvdXRfY2U6CisJcmV0dXJuIGVycjsKK30KKworaW50IGludGVs X21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbSwKKwkJICAgICAgIHN0cnVjdCBp OTE1X2dlbV93d19jdHggKnd3LAorCQkgICAgICAgc3RydWN0IGRtYV9mZW5jZSAqYXdhaXQsCisJ CSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNyYywKKwkJICAgICAgIGVudW0gaTkxNV9jYWNo ZV9sZXZlbCBzcmNfY2FjaGVfbGV2ZWwsCisJCSAgICAgICBib29sIHNyY19pc19sbWVtLAorCQkg ICAgICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpkc3QsCisJCSAgICAgICBlbnVtIGk5MTVfY2FjaGVf bGV2ZWwgZHN0X2NhY2hlX2xldmVsLAorCQkgICAgICAgYm9vbCBkc3RfaXNfbG1lbSwKKwkJICAg ICAgIHN0cnVjdCBpOTE1X3JlcXVlc3QgKipvdXQpCit7CisJc3RydWN0IGludGVsX2NvbnRleHQg KmNlOworCWludCBlcnI7CisKKwkqb3V0ID0gTlVMTDsKKwlpZiAoIW0tPmNvbnRleHQpCisJCXJl dHVybiAtRU5PREVWOworCisJY2UgPSBpbnRlbF9taWdyYXRlX2NyZWF0ZV9jb250ZXh0KG0pOwor CWlmIChJU19FUlIoY2UpKQorCQljZSA9IGludGVsX2NvbnRleHRfZ2V0KG0tPmNvbnRleHQpOwor CUdFTV9CVUdfT04oSVNfRVJSKGNlKSk7CisKKwllcnIgPSBpbnRlbF9jb250ZXh0X3Bpbl93dyhj ZSwgd3cpOworCWlmIChlcnIpCisJCWdvdG8gb3V0OworCisJZXJyID0gaW50ZWxfY29udGV4dF9t aWdyYXRlX2NvcHkoY2UsIGF3YWl0LAorCQkJCQkgc3JjLCBzcmNfY2FjaGVfbGV2ZWwsIHNyY19p c19sbWVtLAorCQkJCQkgZHN0LCBkc3RfY2FjaGVfbGV2ZWwsIGRzdF9pc19sbWVtLAorCQkJCQkg b3V0KTsKKworCWludGVsX2NvbnRleHRfdW5waW4oY2UpOworb3V0OgorCWludGVsX2NvbnRleHRf cHV0KGNlKTsKKwlyZXR1cm4gZXJyOworfQorCit2b2lkIGludGVsX21pZ3JhdGVfZmluaShzdHJ1 Y3QgaW50ZWxfbWlncmF0ZSAqbSkKK3sKKwlzdHJ1Y3QgaW50ZWxfY29udGV4dCAqY2U7CisKKwlj ZSA9IGZldGNoX2FuZF96ZXJvKCZtLT5jb250ZXh0KTsKKwlpZiAoIWNlKQorCQlyZXR1cm47CisK KwlpbnRlbF9lbmdpbmVfZGVzdHJveV9waW5uZWRfY29udGV4dChjZSk7Cit9CisKKyNpZiBJU19F TkFCTEVEKENPTkZJR19EUk1fSTkxNV9TRUxGVEVTVCkKKyNpbmNsdWRlICJzZWxmdGVzdF9taWdy YXRlLmMiCisjZW5kaWYKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVs X21pZ3JhdGUuaCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuaApuZXcg ZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLjMyYzYxMTkwZWQ3MwotLS0gL2Rl di9udWxsCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuaApAQCAt MCwwICsxLDQ1IEBACisvKiBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUICovCisvKgorICog Q29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaWZuZGVmIF9fSU5U RUxfTUlHUkFURV9fCisjZGVmaW5lIF9fSU5URUxfTUlHUkFURV9fCisKKyNpbmNsdWRlICJpbnRl bF9taWdyYXRlX3R5cGVzLmgiCisKK3N0cnVjdCBkbWFfZmVuY2U7CitzdHJ1Y3QgaTkxNV9yZXF1 ZXN0Oworc3RydWN0IGk5MTVfZ2VtX3d3X2N0eDsKK3N0cnVjdCBpbnRlbF9ndDsKK3N0cnVjdCBz Y2F0dGVybGlzdDsKK2VudW0gaTkxNV9jYWNoZV9sZXZlbDsKKworaW50IGludGVsX21pZ3JhdGVf aW5pdChzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbSwgc3RydWN0IGludGVsX2d0ICpndCk7CisKK3N0 cnVjdCBpbnRlbF9jb250ZXh0ICppbnRlbF9taWdyYXRlX2NyZWF0ZV9jb250ZXh0KHN0cnVjdCBp bnRlbF9taWdyYXRlICptKTsKKworaW50IGludGVsX21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxf bWlncmF0ZSAqbSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkgICAg ICAgc3RydWN0IGRtYV9mZW5jZSAqYXdhaXQsCisJCSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3Qg KnNyYywKKwkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBzcmNfY2FjaGVfbGV2ZWwsCisJ CSAgICAgICBib29sIHNyY19pc19sbWVtLAorCQkgICAgICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpk c3QsCisJCSAgICAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgZHN0X2NhY2hlX2xldmVsLAorCQkg ICAgICAgYm9vbCBkc3RfaXNfbG1lbSwKKwkJICAgICAgIHN0cnVjdCBpOTE1X3JlcXVlc3QgKipv dXQpOworCitpbnQgaW50ZWxfY29udGV4dF9taWdyYXRlX2NvcHkoc3RydWN0IGludGVsX2NvbnRl eHQgKmNlLAorCQkJICAgICAgIHN0cnVjdCBkbWFfZmVuY2UgKmF3YWl0LAorCQkJICAgICAgIHN0 cnVjdCBzY2F0dGVybGlzdCAqc3JjLAorCQkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBz cmNfY2FjaGVfbGV2ZWwsCisJCQkgICAgICAgYm9vbCBzcmNfaXNfbG1lbSwKKwkJCSAgICAgICBz dHJ1Y3Qgc2NhdHRlcmxpc3QgKmRzdCwKKwkJCSAgICAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwg ZHN0X2NhY2hlX2xldmVsLAorCQkJICAgICAgIGJvb2wgZHN0X2lzX2xtZW0sCisJCQkgICAgICAg c3RydWN0IGk5MTVfcmVxdWVzdCAqKm91dCk7CisKK3ZvaWQgaW50ZWxfbWlncmF0ZV9maW5pKHN0 cnVjdCBpbnRlbF9taWdyYXRlICptKTsKKworI2VuZGlmIC8qIF9fSU5URUxfTUlHUkFURV9fICov CmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVz LmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmgKbmV3IGZp bGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMDAwMDAwLi5kOTgyMzA1OTdmNDIKLS0tIC9kZXYv bnVsbAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdyYXRlX3R5cGVzLmgK QEAgLTAsMCArMSwxNSBAQAorLyogU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6IE1JVCAqLworLyoK KyAqIENvcHlyaWdodCDCqSAyMDIwIEludGVsIENvcnBvcmF0aW9uCisgKi8KKworI2lmbmRlZiBf X0lOVEVMX01JR1JBVEVfVFlQRVNfXworI2RlZmluZSBfX0lOVEVMX01JR1JBVEVfVFlQRVNfXwor CitzdHJ1Y3QgaW50ZWxfY29udGV4dDsKKworc3RydWN0IGludGVsX21pZ3JhdGUgeworCXN0cnVj dCBpbnRlbF9jb250ZXh0ICpjb250ZXh0OworfTsKKworI2VuZGlmIC8qIF9fSU5URUxfTUlHUkFU RV9UWVBFU19fICovCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9y aW5nLmggYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9yaW5nLmgKaW5kZXggZGJmNWYx NGExMzZmLi4xYjMyZGFkZmI4YzMgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0 L2ludGVsX3JpbmcuaAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9yaW5nLmgK QEAgLTQ5LDYgKzQ5LDcgQEAgc3RhdGljIGlubGluZSB2b2lkIGludGVsX3JpbmdfYWR2YW5jZShz dHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSwgdTMyICpjcykKIAkgKiBpbnRlbF9yaW5nX2JlZ2luKCkp LgogCSAqLwogCUdFTV9CVUdfT04oKHJxLT5yaW5nLT52YWRkciArIHJxLT5yaW5nLT5lbWl0KSAh PSBjcyk7CisJR0VNX0JVR19PTighSVNfQUxJR05FRChycS0+cmluZy0+ZW1pdCwgOCkpOyAvKiBS SU5HX1RBSUwgcXdvcmQgYWxpZ24gKi8KIH0KIAogc3RhdGljIGlubGluZSB1MzIgaW50ZWxfcmlu Z193cmFwKGNvbnN0IHN0cnVjdCBpbnRlbF9yaW5nICpyaW5nLCB1MzIgcG9zKQpkaWZmIC0tZ2l0 IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jIGIvZHJpdmVycy9n cHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jCm5ldyBmaWxlIG1vZGUgMTAwNjQ0Cmlu ZGV4IDAwMDAwMDAwMDAwMC4uOTc4NGQxNDllYmYxCi0tLSAvZGV2L251bGwKKysrIGIvZHJpdmVy cy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jCkBAIC0wLDAgKzEsMjkxIEBACisv LyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUCisvKgorICogQ29weXJpZ2h0IMKpIDIwMjAg SW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaW5jbHVkZSAic2VsZnRlc3RzL2k5MTVfcmFuZG9t LmgiCisKK3N0YXRpYyBjb25zdCB1bnNpZ25lZCBpbnQgc2l6ZXNbXSA9IHsKKwlTWl80SywKKwlT Wl82NEssCisJU1pfMk0sCisJQ0hVTktfU1ogLSBTWl80SywKKwlDSFVOS19TWiwKKwlDSFVOS19T WiArIFNaXzRLLAorCVNaXzY0TSwKK307CisKK3N0YXRpYyBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29i amVjdCAqCitjcmVhdGVfbG1lbV9vcl9pbnRlcm5hbChzdHJ1Y3QgZHJtX2k5MTVfcHJpdmF0ZSAq aTkxNSwgc2l6ZV90IHNpemUpCit7CisJaWYgKEhBU19MTUVNKGk5MTUpKSB7CisJCXN0cnVjdCBk cm1faTkxNV9nZW1fb2JqZWN0ICpvYmo7CisKKwkJb2JqID0gaTkxNV9nZW1fb2JqZWN0X2NyZWF0 ZV9sbWVtKGk5MTUsIHNpemUsIDApOworCQlpZiAoIUlTX0VSUihvYmopKQorCQkJcmV0dXJuIG9i ajsKKwl9CisKKwlyZXR1cm4gaTkxNV9nZW1fb2JqZWN0X2NyZWF0ZV9pbnRlcm5hbChpOTE1LCBz aXplKTsKK30KKworc3RhdGljIGludCBjb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRl LAorCQlpbnQgKCpmbikoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGUsCisJCQkgIHN0cnVj dCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAq c3JjLAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAqZHN0LAorCQkJICBzdHJ1Y3Qg aTkxNV9yZXF1ZXN0ICoqb3V0KSwKKwkJdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0YXRlICpwcm5nKQor eworCXN0cnVjdCBkcm1faTkxNV9wcml2YXRlICppOTE1ID0gbWlncmF0ZS0+Y29udGV4dC0+ZW5n aW5lLT5pOTE1OworCXN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpzcmMsICpkc3Q7CisJc3Ry dWN0IGk5MTVfcmVxdWVzdCAqcnE7CisJc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCB3dzsKKwl1MzIg KnZhZGRyOworCWludCBlcnIgPSAwOworCWludCBpOworCisJc3JjID0gY3JlYXRlX2xtZW1fb3Jf aW50ZXJuYWwoaTkxNSwgc3opOworCWlmIChJU19FUlIoc3JjKSkKKwkJcmV0dXJuIDA7CisKKwlk c3QgPSBpOTE1X2dlbV9vYmplY3RfY3JlYXRlX2ludGVybmFsKGk5MTUsIHN6KTsKKwlpZiAoSVNf RVJSKGRzdCkpCisJCWdvdG8gZXJyX2ZyZWVfc3JjOworCisJZm9yX2k5MTVfZ2VtX3d3KCZ3dywg ZXJyLCB0cnVlKSB7CisJCWVyciA9IGk5MTVfZ2VtX29iamVjdF9sb2NrKHNyYywgJnd3KTsKKwkJ aWYgKGVycikKKwkJCWNvbnRpbnVlOworCisJCWVyciA9IGk5MTVfZ2VtX29iamVjdF9sb2NrKGRz dCwgJnd3KTsKKwkJaWYgKGVycikKKwkJCWNvbnRpbnVlOworCisJCXZhZGRyID0gaTkxNV9nZW1f b2JqZWN0X3Bpbl9tYXAoc3JjLCBJOTE1X01BUF9XQyk7CisJCWlmIChJU19FUlIodmFkZHIpKSB7 CisJCQllcnIgPSBQVFJfRVJSKHZhZGRyKTsKKwkJCWNvbnRpbnVlOworCQl9CisKKwkJZm9yIChp ID0gMDsgaSA8IHN6IC8gc2l6ZW9mKHUzMik7IGkrKykKKwkJCXZhZGRyW2ldID0gaTsKKwkJaTkx NV9nZW1fb2JqZWN0X2ZsdXNoX21hcChzcmMpOworCisJCXZhZGRyID0gaTkxNV9nZW1fb2JqZWN0 X3Bpbl9tYXAoZHN0LCBJOTE1X01BUF9XQyk7CisJCWlmIChJU19FUlIodmFkZHIpKSB7CisJCQll cnIgPSBQVFJfRVJSKHZhZGRyKTsKKwkJCWdvdG8gdW5waW5fc3JjOworCQl9CisKKwkJZm9yIChp ID0gMDsgaSA8IHN6IC8gc2l6ZW9mKHUzMik7IGkrKykKKwkJCXZhZGRyW2ldID0gfmk7CisJCWk5 MTVfZ2VtX29iamVjdF9mbHVzaF9tYXAoZHN0KTsKKworCQllcnIgPSBmbihtaWdyYXRlLCAmd3cs IHNyYywgZHN0LCAmcnEpOworCQlpZiAoIWVycikKKwkJCWNvbnRpbnVlOworCisJCWlmIChlcnIg IT0gLUVERUFETEsgJiYgZXJyICE9IC1FSU5UUiAmJiBlcnIgIT0gLUVSRVNUQVJUU1lTKQorCQkJ cHJfZXJyKCIlcHMgZmFpbGVkLCBzaXplOiAldVxuIiwgZm4sIHN6KTsKKwkJaWYgKHJxKSB7CisJ CQlpOTE1X3JlcXVlc3Rfd2FpdChycSwgMCwgSFopOworCQkJaTkxNV9yZXF1ZXN0X3B1dChycSk7 CisJCX0KKwkJaTkxNV9nZW1fb2JqZWN0X3VucGluX21hcChkc3QpOwordW5waW5fc3JjOgorCQlp OTE1X2dlbV9vYmplY3RfdW5waW5fbWFwKHNyYyk7CisJfQorCWlmIChlcnIpCisJCWdvdG8gZXJy X291dDsKKworCWlmIChycSkgeworCQlpZiAoaTkxNV9yZXF1ZXN0X3dhaXQocnEsIDAsIEhaKSA8 IDApIHsKKwkJCXByX2VycigiJXBzIHRpbWVkIG91dCwgc2l6ZTogJXVcbiIsIGZuLCBzeik7CisJ CQllcnIgPSAtRVRJTUU7CisJCX0KKwkJaTkxNV9yZXF1ZXN0X3B1dChycSk7CisJfQorCisJZm9y IChpID0gMDsgIWVyciAmJiBpIDwgc3ogLyBQQUdFX1NJWkU7IGkrKykgeworCQlpbnQgeCA9IGkg KiAxMDI0ICsgaTkxNV9wcmFuZG9tX3UzMl9tYXhfc3RhdGUoMTAyNCwgcHJuZyk7CisKKwkJaWYg KHZhZGRyW3hdICE9IHgpIHsKKwkJCXByX2VycigiJXBzIGZhaWxlZCwgc2l6ZTogJXUsIG9mZnNl dDogJXp1XG4iLAorCQkJICAgICAgIGZuLCBzeiwgeCAqIHNpemVvZih1MzIpKTsKKwkJCWlndF9o ZXhkdW1wKHZhZGRyICsgaSAqIDEwMjQsIDQwOTYpOworCQkJZXJyID0gLUVJTlZBTDsKKwkJfQor CX0KKworCWk5MTVfZ2VtX29iamVjdF91bnBpbl9tYXAoZHN0KTsKKwlpOTE1X2dlbV9vYmplY3Rf dW5waW5fbWFwKHNyYyk7CisKK2Vycl9vdXQ6CisJaTkxNV9nZW1fb2JqZWN0X3B1dChkc3QpOwor ZXJyX2ZyZWVfc3JjOgorCWk5MTVfZ2VtX29iamVjdF9wdXQoc3JjKTsKKworCXJldHVybiBlcnI7 Cit9CisKK3N0YXRpYyBpbnQgX19taWdyYXRlX2NvcHkoc3RydWN0IGludGVsX21pZ3JhdGUgKm1p Z3JhdGUsCisJCQkgIHN0cnVjdCBpOTE1X2dlbV93d19jdHggKnd3LAorCQkJICBzdHJ1Y3QgZHJt X2k5MTVfZ2VtX29iamVjdCAqc3JjLAorCQkJICBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAq ZHN0LAorCQkJICBzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KQoreworCXJldHVybiBpbnRlbF9t aWdyYXRlX2NvcHkobWlncmF0ZSwgd3csIE5VTEwsCisJCQkJICBzcmMtPm1tLnBhZ2VzLT5zZ2ws IHNyYy0+Y2FjaGVfbGV2ZWwsCisJCQkJICBpOTE1X2dlbV9vYmplY3RfaXNfbG1lbShzcmMpLAor CQkJCSAgZHN0LT5tbS5wYWdlcy0+c2dsLCBkc3QtPmNhY2hlX2xldmVsLAorCQkJCSAgaTkxNV9n ZW1fb2JqZWN0X2lzX2xtZW0oZHN0KSwKKwkJCQkgIG91dCk7Cit9CisKK3N0YXRpYyBpbnQgX19n bG9iYWxfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSBzdHJ1Y3QgaTkx NV9nZW1fd3dfY3R4ICp3dywKKwkJCSBzdHJ1Y3QgZHJtX2k5MTVfZ2VtX29iamVjdCAqc3JjLAor CQkJIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpkc3QsCisJCQkgc3RydWN0IGk5MTVfcmVx dWVzdCAqKm91dCkKK3sKKwlyZXR1cm4gaW50ZWxfY29udGV4dF9taWdyYXRlX2NvcHkobWlncmF0 ZS0+Y29udGV4dCwgTlVMTCwKKwkJCQkJICBzcmMtPm1tLnBhZ2VzLT5zZ2wsIHNyYy0+Y2FjaGVf bGV2ZWwsCisJCQkJCSAgaTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0oc3JjKSwKKwkJCQkJICBkc3Qt Pm1tLnBhZ2VzLT5zZ2wsIGRzdC0+Y2FjaGVfbGV2ZWwsCisJCQkJCSAgaTkxNV9nZW1fb2JqZWN0 X2lzX2xtZW0oZHN0KSwKKwkJCQkJICBvdXQpOworfQorCitzdGF0aWMgaW50CittaWdyYXRlX2Nv cHkoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGUsIHUzMiBzeiwgc3RydWN0IHJuZF9zdGF0 ZSAqcHJuZykKK3sKKwlyZXR1cm4gY29weShtaWdyYXRlLCBfX21pZ3JhdGVfY29weSwgc3osIHBy bmcpOworfQorCitzdGF0aWMgaW50CitnbG9iYWxfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAq bWlncmF0ZSwgdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0YXRlICpwcm5nKQoreworCXJldHVybiBjb3B5 KG1pZ3JhdGUsIF9fZ2xvYmFsX2NvcHksIHN6LCBwcm5nKTsKK30KKworc3RhdGljIGludCBsaXZl X21pZ3JhdGVfY29weSh2b2lkICphcmcpCit7CisJc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3Jh dGUgPSBhcmc7CisJc3RydWN0IGRybV9pOTE1X3ByaXZhdGUgKmk5MTUgPSBtaWdyYXRlLT5jb250 ZXh0LT5lbmdpbmUtPmk5MTU7CisJSTkxNV9STkRfU1RBVEUocHJuZyk7CisJaW50IGk7CisKKwlm b3IgKGkgPSAwOyBpIDwgQVJSQVlfU0laRShzaXplcyk7IGkrKykgeworCQlpbnQgZXJyOworCisJ CWVyciA9IG1pZ3JhdGVfY29weShtaWdyYXRlLCBzaXplc1tpXSwgJnBybmcpOworCQlpZiAoZXJy ID09IDApCisJCQllcnIgPSBnbG9iYWxfY29weShtaWdyYXRlLCBzaXplc1tpXSwgJnBybmcpOwor CQlpOTE1X2dlbV9kcmFpbl9mcmVlZF9vYmplY3RzKGk5MTUpOworCQlpZiAoZXJyKQorCQkJcmV0 dXJuIGVycjsKKwl9CisKKwlyZXR1cm4gMDsKK30KKworc3RydWN0IHRocmVhZGVkX21pZ3JhdGUg eworCXN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlOworCXN0cnVjdCB0YXNrX3N0cnVjdCAq dHNrOworCXN0cnVjdCBybmRfc3RhdGUgcHJuZzsKK307CisKK3N0YXRpYyBpbnQgdGhyZWFkZWRf bWlncmF0ZShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSAgICBpbnQgKCpmbiko dm9pZCAqYXJnKSwKKwkJCSAgICB1bnNpZ25lZCBpbnQgZmxhZ3MpCit7CisJY29uc3QgdW5zaWdu ZWQgaW50IG5fY3B1cyA9IG51bV9vbmxpbmVfY3B1cygpICsgMTsKKwlzdHJ1Y3QgdGhyZWFkZWRf bWlncmF0ZSAqdGhyZWFkOworCUk5MTVfUk5EX1NUQVRFKHBybmcpOworCXVuc2lnbmVkIGludCBp OworCWludCBlcnIgPSAwOworCisJdGhyZWFkID0ga2NhbGxvYyhuX2NwdXMsIHNpemVvZigqdGhy ZWFkKSwgR0ZQX0tFUk5FTCk7CisJaWYgKCF0aHJlYWQpCisJCXJldHVybiAwOworCisJZm9yIChp ID0gMDsgaSA8IG5fY3B1czsgKytpKSB7CisJCXN0cnVjdCB0YXNrX3N0cnVjdCAqdHNrOworCisJ CXRocmVhZFtpXS5taWdyYXRlID0gbWlncmF0ZTsKKwkJdGhyZWFkW2ldLnBybmcgPQorCQkJSTkx NV9STkRfU1RBVEVfSU5JVElBTElaRVIocHJhbmRvbV91MzJfc3RhdGUoJnBybmcpKTsKKworCQl0 c2sgPSBrdGhyZWFkX3J1bihmbiwgJnRocmVhZFtpXSwgImlndC0lZCIsIGkpOworCQlpZiAoSVNf RVJSKHRzaykpIHsKKwkJCWVyciA9IFBUUl9FUlIodHNrKTsKKwkJCWJyZWFrOworCQl9CisKKwkJ Z2V0X3Rhc2tfc3RydWN0KHRzayk7CisJCXRocmVhZFtpXS50c2sgPSB0c2s7CisJfQorCisJbXNs ZWVwKDEwKTsgLyogc3RhcnQgYWxsIHRocmVhZHMgYmVmb3JlIHdlIGt0aHJlYWRfc3RvcCgpICov CisKKwlmb3IgKGkgPSAwOyBpIDwgbl9jcHVzOyArK2kpIHsKKwkJc3RydWN0IHRhc2tfc3RydWN0 ICp0c2sgPSB0aHJlYWRbaV0udHNrOworCQlpbnQgc3RhdHVzOworCisJCWlmIChJU19FUlJfT1Jf TlVMTCh0c2spKQorCQkJY29udGludWU7CisKKwkJc3RhdHVzID0ga3RocmVhZF9zdG9wKHRzayk7 CisJCWlmIChzdGF0dXMgJiYgIWVycikKKwkJCWVyciA9IHN0YXR1czsKKworCQlwdXRfdGFza19z dHJ1Y3QodHNrKTsKKwl9CisKKwlrZnJlZSh0aHJlYWQpOworCXJldHVybiBlcnI7Cit9CisKK3N0 YXRpYyBpbnQgX190aHJlYWRfbWlncmF0ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgdGhy ZWFkZWRfbWlncmF0ZSAqdG0gPSBhcmc7CisKKwlyZXR1cm4gbWlncmF0ZV9jb3B5KHRtLT5taWdy YXRlLCAyICogQ0hVTktfU1osICZ0bS0+cHJuZyk7Cit9CisKK3N0YXRpYyBpbnQgdGhyZWFkX21p Z3JhdGVfY29weSh2b2lkICphcmcpCit7CisJcmV0dXJuIHRocmVhZGVkX21pZ3JhdGUoYXJnLCBf X3RocmVhZF9taWdyYXRlX2NvcHksIDApOworfQorCitzdGF0aWMgaW50IF9fdGhyZWFkX2dsb2Jh bF9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgdGhyZWFkZWRfbWlncmF0ZSAqdG0gPSBhcmc7 CisKKwlyZXR1cm4gZ2xvYmFsX2NvcHkodG0tPm1pZ3JhdGUsIDIgKiBDSFVOS19TWiwgJnRtLT5w cm5nKTsKK30KKworc3RhdGljIGludCB0aHJlYWRfZ2xvYmFsX2NvcHkodm9pZCAqYXJnKQorewor CXJldHVybiB0aHJlYWRlZF9taWdyYXRlKGFyZywgX190aHJlYWRfZ2xvYmFsX2NvcHksIDApOwor fQorCitpbnQgaW50ZWxfbWlncmF0ZV9saXZlX3NlbGZ0ZXN0cyhzdHJ1Y3QgZHJtX2k5MTVfcHJp dmF0ZSAqaTkxNSkKK3sKKwlzdGF0aWMgY29uc3Qgc3RydWN0IGk5MTVfc3VidGVzdCB0ZXN0c1td ID0geworCQlTVUJURVNUKGxpdmVfbWlncmF0ZV9jb3B5KSwKKwkJU1VCVEVTVCh0aHJlYWRfbWln cmF0ZV9jb3B5KSwKKwkJU1VCVEVTVCh0aHJlYWRfZ2xvYmFsX2NvcHkpLAorCX07CisJc3RydWN0 IGludGVsX21pZ3JhdGUgbTsKKwlpbnQgZXJyOworCisJaWYgKGludGVsX21pZ3JhdGVfaW5pdCgm bSwgJmk5MTUtPmd0KSkKKwkJcmV0dXJuIDA7CisKKwllcnIgPSBpOTE1X3N1YnRlc3RzKHRlc3Rz LCAmbSk7CisJaW50ZWxfbWlncmF0ZV9maW5pKCZtKTsKKworCXJldHVybiBlcnI7Cit9CmRpZmYg LS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0 cy5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfbGl2ZV9zZWxmdGVzdHMu aAppbmRleCBhOTJjMGU5YjdlNmIuLmJlNWUwMTkxZWFlYSAxMDA2NDQKLS0tIGEvZHJpdmVycy9n cHUvZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfbGl2ZV9zZWxmdGVzdHMuaAorKysgYi9kcml2ZXJz L2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oCkBAIC0yNiw2ICsy Niw3IEBAIHNlbGZ0ZXN0KGd0X21vY3MsIGludGVsX21vY3NfbGl2ZV9zZWxmdGVzdHMpCiBzZWxm dGVzdChndF9wbSwgaW50ZWxfZ3RfcG1fbGl2ZV9zZWxmdGVzdHMpCiBzZWxmdGVzdChndF9oZWFy dGJlYXQsIGludGVsX2hlYXJ0YmVhdF9saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KHJlcXVlc3Rz LCBpOTE1X3JlcXVlc3RfbGl2ZV9zZWxmdGVzdHMpCitzZWxmdGVzdChtaWdyYXRlLCBpbnRlbF9t aWdyYXRlX2xpdmVfc2VsZnRlc3RzKQogc2VsZnRlc3QoYWN0aXZlLCBpOTE1X2FjdGl2ZV9saXZl X3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KG9iamVjdHMsIGk5MTVfZ2VtX29iamVjdF9saXZlX3NlbGZ0 ZXN0cykKIHNlbGZ0ZXN0KG1tYW4sIGk5MTVfZ2VtX21tYW5fbGl2ZV9zZWxmdGVzdHMpCi0tIAoy LjMxLjEKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCklu dGVsLWdmeCBtYWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRw czovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdmeAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D14C49EA2 for ; Thu, 17 Jun 2021 06:31:03 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E5470613E9 for ; Thu, 17 Jun 2021 06:31:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5470613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8CB0C6E881; Thu, 17 Jun 2021 06:30:53 +0000 (UTC) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by gabe.freedesktop.org (Postfix) with ESMTPS id 821436E87B; Thu, 17 Jun 2021 06:30:46 +0000 (UTC) IronPort-SDR: wX5uFlaYLbnLMQSh+zwnrRnfhQeL7JwmSYcDMc1ZevUGaTc0tVKg9WXicb5Jfx9bVhqInBeER5 k13/mPIsARMA== X-IronPort-AV: E=McAfee;i="6200,9189,10017"; a="203287834" X-IronPort-AV: E=Sophos;i="5.83,278,1616482800"; d="scan'208";a="203287834" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2021 23:30:46 -0700 IronPort-SDR: B07bbOGu03TVQCtYXyw/IipnBxK7MZ4FrJoRd3uwa0lkM4DPD1+2ENTJvDumVasu+5sRps/EvL DzYNvChAswZQ== X-IronPort-AV: E=Sophos;i="5.83,278,1616482800"; d="scan'208";a="554302720" Received: from vanderss-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.193]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2021 23:30:43 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v5 07/12] drm/i915/gt: Pipelined page migration Date: Thu, 17 Jun 2021 08:30:13 +0200 Message-Id: <20210617063018.92802-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210617063018.92802-1-thomas.hellstrom@linux.intel.com> References: <20210617063018.92802-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Chris Wilson If we pipeline the PTE updates and then do the copy of those pages within a single unpreemptible command packet, we can submit the copies and leave them to be scheduled without having to synchronously wait under a global lock. In order to manage migration, we need to preallocate the page tables (and keep them pinned and available for use at any time), causing a bottleneck for migrations as all clients must contend on the limited resources. By inlining the ppGTT updates and performing the blit atomically, each client only owns the PTE while in use, and so we can reschedule individual operations however we see fit. And most importantly, we do not need to take a global lock on the shared vm, and wait until the operation is complete before releasing the lock for others to claim the PTE for themselves. Signed-off-by: Chris Wilson Co-developed-by: Thomas Hellström Signed-off-by: Thomas Hellström Reviewed-by: Matthew Auld --- v2: - Add a TODO for huge LMEM ptes (Pointed out by Matthew Auld) - Use intel_engine_destroy_pinned_context() to properly take the pinned context timeline off the engine list. (CI warning). v3: - Remove an obsolete GEM_BUG_ON (Pointed out by Matthew Auld) - Fix the size argument in allocate_va_range() to not include the base (Pointed out by Matthew Auld) --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_gpu_commands.h | 2 + drivers/gpu/drm/i915/gt/intel_migrate.c | 542 ++++++++++++++++++ drivers/gpu/drm/i915/gt/intel_migrate.h | 45 ++ drivers/gpu/drm/i915/gt/intel_migrate_types.h | 15 + drivers/gpu/drm/i915/gt/intel_ring.h | 1 + drivers/gpu/drm/i915/gt/selftest_migrate.c | 291 ++++++++++ .../drm/i915/selftests/i915_live_selftests.h | 1 + 9 files changed, 899 insertions(+) create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.c create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.h create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate_types.h create mode 100644 drivers/gpu/drm/i915/gt/selftest_migrate.c diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index dde698f3bff4..5e10e0628c56 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -108,6 +108,7 @@ gt-y += \ gt/intel_gtt.o \ gt/intel_llc.o \ gt/intel_lrc.o \ + gt/intel_migrate.o \ gt/intel_mocs.o \ gt/intel_ppgtt.o \ gt/intel_rc6.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 36ea9eb52bb5..62f7440bc111 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -188,6 +188,7 @@ intel_write_status_page(struct intel_engine_cs *engine, int reg, u32 value) #define I915_GEM_HWS_PREEMPT_ADDR (I915_GEM_HWS_PREEMPT * sizeof(u32)) #define I915_GEM_HWS_SEQNO 0x40 #define I915_GEM_HWS_SEQNO_ADDR (I915_GEM_HWS_SEQNO * sizeof(u32)) +#define I915_GEM_HWS_MIGRATE (0x42 * sizeof(u32)) #define I915_GEM_HWS_SCRATCH 0x80 #define I915_HWS_CSB_BUF0_INDEX 0x10 diff --git a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h index 2694dbb9967e..1c3af0fc0456 100644 --- a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h +++ b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h @@ -123,8 +123,10 @@ #define MI_SEMAPHORE_SAD_NEQ_SDD (5 << 12) #define MI_SEMAPHORE_TOKEN_MASK REG_GENMASK(9, 5) #define MI_SEMAPHORE_TOKEN_SHIFT 5 +#define MI_STORE_DATA_IMM MI_INSTR(0x20, 0) #define MI_STORE_DWORD_IMM MI_INSTR(0x20, 1) #define MI_STORE_DWORD_IMM_GEN4 MI_INSTR(0x20, 2) +#define MI_STORE_QWORD_IMM_GEN8 (MI_INSTR(0x20, 3) | REG_BIT(21)) #define MI_MEM_VIRTUAL (1 << 22) /* 945,g33,965 */ #define MI_USE_GGTT (1 << 22) /* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1) diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.c b/drivers/gpu/drm/i915/gt/intel_migrate.c new file mode 100644 index 000000000000..e2e860063e7b --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.c @@ -0,0 +1,542 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "i915_drv.h" +#include "intel_context.h" +#include "intel_gpu_commands.h" +#include "intel_gt.h" +#include "intel_gtt.h" +#include "intel_migrate.h" +#include "intel_ring.h" + +struct insert_pte_data { + u64 offset; + bool is_lmem; +}; + +#define CHUNK_SZ SZ_8M /* ~1ms at 8GiB/s preemption delay */ + +static bool engine_supports_migration(struct intel_engine_cs *engine) +{ + if (!engine) + return false; + + /* + * We need the ability to prevent aribtration (MI_ARB_ON_OFF), + * the ability to write PTE using inline data (MI_STORE_DATA) + * and of course the ability to do the block transfer (blits). + */ + GEM_BUG_ON(engine->class != COPY_ENGINE_CLASS); + + return true; +} + +static void insert_pte(struct i915_address_space *vm, + struct i915_page_table *pt, + void *data) +{ + struct insert_pte_data *d = data; + + vm->insert_page(vm, px_dma(pt), d->offset, I915_CACHE_NONE, + d->is_lmem ? PTE_LM : 0); + d->offset += PAGE_SIZE; +} + +static struct i915_address_space *migrate_vm(struct intel_gt *gt) +{ + struct i915_vm_pt_stash stash = {}; + struct i915_ppgtt *vm; + int err; + int i; + + /* + * We construct a very special VM for use by all migration contexts, + * it is kept pinned so that it can be used at any time. As we need + * to pre-allocate the page directories for the migration VM, this + * limits us to only using a small number of prepared vma. + * + * To be able to pipeline and reschedule migration operations while + * avoiding unnecessary contention on the vm itself, the PTE updates + * are inline with the blits. All the blits use the same fixed + * addresses, with the backing store redirection being updated on the + * fly. Only 2 implicit vma are used for all migration operations. + * + * We lay the ppGTT out as: + * + * [0, CHUNK_SZ) -> first object + * [CHUNK_SZ, 2 * CHUNK_SZ) -> second object + * [2 * CHUNK_SZ, 2 * CHUNK_SZ + 2 * CHUNK_SZ >> 9] -> PTE + * + * By exposing the dma addresses of the page directories themselves + * within the ppGTT, we are then able to rewrite the PTE prior to use. + * But the PTE update and subsequent migration operation must be atomic, + * i.e. within the same non-preemptible window so that we do not switch + * to another migration context that overwrites the PTE. + * + * TODO: Add support for huge LMEM PTEs + */ + + vm = i915_ppgtt_create(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + if (!vm->vm.allocate_va_range || !vm->vm.foreach) { + err = -ENODEV; + goto err_vm; + } + + /* + * Each engine instance is assigned its own chunk in the VM, so + * that we can run multiple instances concurrently + */ + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + struct intel_engine_cs *engine; + u64 base = (u64)i << 32; + struct insert_pte_data d = {}; + struct i915_gem_ww_ctx ww; + u64 sz; + + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (!engine_supports_migration(engine)) + continue; + + /* + * We copy in 8MiB chunks. Each PDE covers 2MiB, so we need + * 4x2 page directories for source/destination. + */ + sz = 2 * CHUNK_SZ; + d.offset = base + sz; + + /* + * We need another page directory setup so that we can write + * the 8x512 PTE in each chunk. + */ + sz += (sz >> 12) * sizeof(u64); + + err = i915_vm_alloc_pt_stash(&vm->vm, &stash, sz); + if (err) + goto err_vm; + + for_i915_gem_ww(&ww, err, true) { + err = i915_vm_lock_objects(&vm->vm, &ww); + if (err) + continue; + err = i915_vm_map_pt_stash(&vm->vm, &stash); + if (err) + continue; + + vm->vm.allocate_va_range(&vm->vm, &stash, base, sz); + } + i915_vm_free_pt_stash(&vm->vm, &stash); + if (err) + goto err_vm; + + /* Now allow the GPU to rewrite the PTE via its own ppGTT */ + d.is_lmem = i915_gem_object_is_lmem(vm->vm.scratch[0]); + vm->vm.foreach(&vm->vm, base, base + sz, insert_pte, &d); + } + + return &vm->vm; + +err_vm: + i915_vm_put(&vm->vm); + return ERR_PTR(err); +} + +static struct intel_engine_cs *first_copy_engine(struct intel_gt *gt) +{ + struct intel_engine_cs *engine; + int i; + + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + return engine; + } + + return NULL; +} + +static struct intel_context *pinned_context(struct intel_gt *gt) +{ + static struct lock_class_key key; + struct intel_engine_cs *engine; + struct i915_address_space *vm; + struct intel_context *ce; + + engine = first_copy_engine(gt); + if (!engine) + return ERR_PTR(-ENODEV); + + vm = migrate_vm(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + ce = intel_engine_create_pinned_context(engine, vm, SZ_512K, + I915_GEM_HWS_MIGRATE, + &key, "migrate"); + i915_vm_put(ce->vm); + return ce; +} + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt) +{ + struct intel_context *ce; + + memset(m, 0, sizeof(*m)); + + ce = pinned_context(gt); + if (IS_ERR(ce)) + return PTR_ERR(ce); + + m->context = ce; + return 0; +} + +static int random_index(unsigned int max) +{ + return upper_32_bits(mul_u32_u32(get_random_u32(), max)); +} + +static struct intel_context *__migrate_engines(struct intel_gt *gt) +{ + struct intel_engine_cs *engines[MAX_ENGINE_INSTANCE]; + struct intel_engine_cs *engine; + unsigned int count, i; + + count = 0; + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + engines[count++] = engine; + } + + return intel_context_create(engines[random_index(count)]); +} + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m) +{ + struct intel_context *ce; + + /* + * We randomly distribute contexts across the engines upon constrction, + * as they all share the same pinned vm, and so in order to allow + * multiple blits to run in parallel, we must construct each blit + * to use a different range of the vm for its GTT. This has to be + * known at construction, so we can not use the late greedy load + * balancing of the virtual-engine. + */ + ce = __migrate_engines(m->context->engine->gt); + if (IS_ERR(ce)) + return ce; + + ce->ring = __intel_context_ring_size(SZ_256K); + + i915_vm_put(ce->vm); + ce->vm = i915_vm_get(m->context->vm); + + return ce; +} + +static inline struct sgt_dma sg_sgt(struct scatterlist *sg) +{ + dma_addr_t addr = sg_dma_address(sg); + + return (struct sgt_dma){ sg, addr, addr + sg_dma_len(sg) }; +} + +static int emit_no_arbitration(struct i915_request *rq) +{ + u32 *cs; + + cs = intel_ring_begin(rq, 2); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Explicitly disable preemption for this request. */ + *cs++ = MI_ARB_ON_OFF; + *cs++ = MI_NOOP; + intel_ring_advance(rq, cs); + + return 0; +} + +static int emit_pte(struct i915_request *rq, + struct sgt_dma *it, + enum i915_cache_level cache_level, + bool is_lmem, + u64 offset, + int length) +{ + const u64 encode = rq->context->vm->pte_encode(0, cache_level, + is_lmem ? PTE_LM : 0); + struct intel_ring *ring = rq->ring; + int total = 0; + u32 *hdr, *cs; + int pkt; + + GEM_BUG_ON(INTEL_GEN(rq->engine->i915) < 8); + + /* Compute the page directory offset for the target address range */ + offset += (u64)rq->engine->instance << 32; + offset >>= 12; + offset *= sizeof(u64); + offset += 2 * CHUNK_SZ; + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Pack as many PTE updates as possible into a single MI command */ + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); /* as qword elements */ + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + + do { + if (cs - hdr >= pkt) { + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + } + + *cs++ = lower_32_bits(encode | it->dma); + *cs++ = upper_32_bits(encode | it->dma); + + offset += 8; + total += I915_GTT_PAGE_SIZE; + + it->dma += I915_GTT_PAGE_SIZE; + if (it->dma >= it->max) { + it->sg = __sg_next(it->sg); + if (!it->sg || sg_dma_len(it->sg) == 0) + break; + + it->dma = sg_dma_address(it->sg); + it->max = it->dma + sg_dma_len(it->sg); + } + } while (total < length); + + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + return total; +} + +static bool wa_1209644611_applies(int gen, u32 size) +{ + u32 height = size >> PAGE_SHIFT; + + if (gen != 11) + return false; + + return height % 4 == 3 && height <= 8; +} + +static int emit_copy(struct i915_request *rq, int size) +{ + const int gen = INTEL_GEN(rq->engine->i915); + u32 instance = rq->engine->instance; + u32 *cs; + + cs = intel_ring_begin(rq, gen >= 8 ? 10 : 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + if (gen >= 9 && !wa_1209644611_applies(gen, size)) { + *cs++ = GEN9_XY_FAST_COPY_BLT_CMD | (10 - 2); + *cs++ = BLT_DEPTH_32 | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else if (gen >= 8) { + *cs++ = XY_SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (10 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else { + GEM_BUG_ON(instance); + *cs++ = SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (6 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + } + + intel_ring_advance(rq, cs); + return 0; +} + +int +intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct sgt_dma it_src = sg_sgt(src), it_dst = sg_sgt(dst); + struct i915_request *rq; + int err; + + *out = NULL; + + GEM_BUG_ON(ce->ring->size < SZ_64K); + + do { + int len; + + rq = i915_request_create(ce); + if (IS_ERR(rq)) { + err = PTR_ERR(rq); + goto out_ce; + } + + if (await) { + err = i915_request_await_dma_fence(rq, await); + if (err) + goto out_rq; + + if (rq->engine->emit_init_breadcrumb) { + err = rq->engine->emit_init_breadcrumb(rq); + if (err) + goto out_rq; + } + + await = NULL; + } + + /* The PTE updates + copy must not be interrupted. */ + err = emit_no_arbitration(rq); + if (err) + goto out_rq; + + len = emit_pte(rq, &it_src, src_cache_level, src_is_lmem, 0, + CHUNK_SZ); + if (len <= 0) { + err = len; + goto out_rq; + } + + err = emit_pte(rq, &it_dst, dst_cache_level, dst_is_lmem, + CHUNK_SZ, len); + if (err < 0) + goto out_rq; + if (err < len) { + err = -EINVAL; + goto out_rq; + } + + err = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (err) + goto out_rq; + + err = emit_copy(rq, len); + + /* Arbitration is re-enabled between requests. */ +out_rq: + if (*out) + i915_request_put(*out); + *out = i915_request_get(rq); + i915_request_add(rq); + if (err || !it_src.sg || !sg_dma_len(it_src.sg)) + break; + + cond_resched(); + } while (1); + +out_ce: + return err; +} + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct intel_context *ce; + int err; + + *out = NULL; + if (!m->context) + return -ENODEV; + + ce = intel_migrate_create_context(m); + if (IS_ERR(ce)) + ce = intel_context_get(m->context); + GEM_BUG_ON(IS_ERR(ce)); + + err = intel_context_pin_ww(ce, ww); + if (err) + goto out; + + err = intel_context_migrate_copy(ce, await, + src, src_cache_level, src_is_lmem, + dst, dst_cache_level, dst_is_lmem, + out); + + intel_context_unpin(ce); +out: + intel_context_put(ce); + return err; +} + +void intel_migrate_fini(struct intel_migrate *m) +{ + struct intel_context *ce; + + ce = fetch_and_zero(&m->context); + if (!ce) + return; + + intel_engine_destroy_pinned_context(ce); +} + +#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) +#include "selftest_migrate.c" +#endif diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.h b/drivers/gpu/drm/i915/gt/intel_migrate.h new file mode 100644 index 000000000000..32c61190ed73 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE__ +#define __INTEL_MIGRATE__ + +#include "intel_migrate_types.h" + +struct dma_fence; +struct i915_request; +struct i915_gem_ww_ctx; +struct intel_gt; +struct scatterlist; +enum i915_cache_level; + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt); + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m); + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +int intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +void intel_migrate_fini(struct intel_migrate *m); + +#endif /* __INTEL_MIGRATE__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_migrate_types.h b/drivers/gpu/drm/i915/gt/intel_migrate_types.h new file mode 100644 index 000000000000..d98230597f42 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate_types.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE_TYPES__ +#define __INTEL_MIGRATE_TYPES__ + +struct intel_context; + +struct intel_migrate { + struct intel_context *context; +}; + +#endif /* __INTEL_MIGRATE_TYPES__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_ring.h b/drivers/gpu/drm/i915/gt/intel_ring.h index dbf5f14a136f..1b32dadfb8c3 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.h +++ b/drivers/gpu/drm/i915/gt/intel_ring.h @@ -49,6 +49,7 @@ static inline void intel_ring_advance(struct i915_request *rq, u32 *cs) * intel_ring_begin()). */ GEM_BUG_ON((rq->ring->vaddr + rq->ring->emit) != cs); + GEM_BUG_ON(!IS_ALIGNED(rq->ring->emit, 8)); /* RING_TAIL qword align */ } static inline u32 intel_ring_wrap(const struct intel_ring *ring, u32 pos) diff --git a/drivers/gpu/drm/i915/gt/selftest_migrate.c b/drivers/gpu/drm/i915/gt/selftest_migrate.c new file mode 100644 index 000000000000..9784d149ebf1 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/selftest_migrate.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "selftests/i915_random.h" + +static const unsigned int sizes[] = { + SZ_4K, + SZ_64K, + SZ_2M, + CHUNK_SZ - SZ_4K, + CHUNK_SZ, + CHUNK_SZ + SZ_4K, + SZ_64M, +}; + +static struct drm_i915_gem_object * +create_lmem_or_internal(struct drm_i915_private *i915, size_t size) +{ + if (HAS_LMEM(i915)) { + struct drm_i915_gem_object *obj; + + obj = i915_gem_object_create_lmem(i915, size, 0); + if (!IS_ERR(obj)) + return obj; + } + + return i915_gem_object_create_internal(i915, size); +} + +static int copy(struct intel_migrate *migrate, + int (*fn)(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out), + u32 sz, struct rnd_state *prng) +{ + struct drm_i915_private *i915 = migrate->context->engine->i915; + struct drm_i915_gem_object *src, *dst; + struct i915_request *rq; + struct i915_gem_ww_ctx ww; + u32 *vaddr; + int err = 0; + int i; + + src = create_lmem_or_internal(i915, sz); + if (IS_ERR(src)) + return 0; + + dst = i915_gem_object_create_internal(i915, sz); + if (IS_ERR(dst)) + goto err_free_src; + + for_i915_gem_ww(&ww, err, true) { + err = i915_gem_object_lock(src, &ww); + if (err) + continue; + + err = i915_gem_object_lock(dst, &ww); + if (err) + continue; + + vaddr = i915_gem_object_pin_map(src, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + continue; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = i; + i915_gem_object_flush_map(src); + + vaddr = i915_gem_object_pin_map(dst, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto unpin_src; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = ~i; + i915_gem_object_flush_map(dst); + + err = fn(migrate, &ww, src, dst, &rq); + if (!err) + continue; + + if (err != -EDEADLK && err != -EINTR && err != -ERESTARTSYS) + pr_err("%ps failed, size: %u\n", fn, sz); + if (rq) { + i915_request_wait(rq, 0, HZ); + i915_request_put(rq); + } + i915_gem_object_unpin_map(dst); +unpin_src: + i915_gem_object_unpin_map(src); + } + if (err) + goto err_out; + + if (rq) { + if (i915_request_wait(rq, 0, HZ) < 0) { + pr_err("%ps timed out, size: %u\n", fn, sz); + err = -ETIME; + } + i915_request_put(rq); + } + + for (i = 0; !err && i < sz / PAGE_SIZE; i++) { + int x = i * 1024 + i915_prandom_u32_max_state(1024, prng); + + if (vaddr[x] != x) { + pr_err("%ps failed, size: %u, offset: %zu\n", + fn, sz, x * sizeof(u32)); + igt_hexdump(vaddr + i * 1024, 4096); + err = -EINVAL; + } + } + + i915_gem_object_unpin_map(dst); + i915_gem_object_unpin_map(src); + +err_out: + i915_gem_object_put(dst); +err_free_src: + i915_gem_object_put(src); + + return err; +} + +static int __migrate_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_migrate_copy(migrate, ww, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int __global_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_context_migrate_copy(migrate->context, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int +migrate_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __migrate_copy, sz, prng); +} + +static int +global_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __global_copy, sz, prng); +} + +static int live_migrate_copy(void *arg) +{ + struct intel_migrate *migrate = arg; + struct drm_i915_private *i915 = migrate->context->engine->i915; + I915_RND_STATE(prng); + int i; + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + int err; + + err = migrate_copy(migrate, sizes[i], &prng); + if (err == 0) + err = global_copy(migrate, sizes[i], &prng); + i915_gem_drain_freed_objects(i915); + if (err) + return err; + } + + return 0; +} + +struct threaded_migrate { + struct intel_migrate *migrate; + struct task_struct *tsk; + struct rnd_state prng; +}; + +static int threaded_migrate(struct intel_migrate *migrate, + int (*fn)(void *arg), + unsigned int flags) +{ + const unsigned int n_cpus = num_online_cpus() + 1; + struct threaded_migrate *thread; + I915_RND_STATE(prng); + unsigned int i; + int err = 0; + + thread = kcalloc(n_cpus, sizeof(*thread), GFP_KERNEL); + if (!thread) + return 0; + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk; + + thread[i].migrate = migrate; + thread[i].prng = + I915_RND_STATE_INITIALIZER(prandom_u32_state(&prng)); + + tsk = kthread_run(fn, &thread[i], "igt-%d", i); + if (IS_ERR(tsk)) { + err = PTR_ERR(tsk); + break; + } + + get_task_struct(tsk); + thread[i].tsk = tsk; + } + + msleep(10); /* start all threads before we kthread_stop() */ + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk = thread[i].tsk; + int status; + + if (IS_ERR_OR_NULL(tsk)) + continue; + + status = kthread_stop(tsk); + if (status && !err) + err = status; + + put_task_struct(tsk); + } + + kfree(thread); + return err; +} + +static int __thread_migrate_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return migrate_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_migrate_copy(void *arg) +{ + return threaded_migrate(arg, __thread_migrate_copy, 0); +} + +static int __thread_global_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return global_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_global_copy(void *arg) +{ + return threaded_migrate(arg, __thread_global_copy, 0); +} + +int intel_migrate_live_selftests(struct drm_i915_private *i915) +{ + static const struct i915_subtest tests[] = { + SUBTEST(live_migrate_copy), + SUBTEST(thread_migrate_copy), + SUBTEST(thread_global_copy), + }; + struct intel_migrate m; + int err; + + if (intel_migrate_init(&m, &i915->gt)) + return 0; + + err = i915_subtests(tests, &m); + intel_migrate_fini(&m); + + return err; +} diff --git a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h index a92c0e9b7e6b..be5e0191eaea 100644 --- a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h +++ b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h @@ -26,6 +26,7 @@ selftest(gt_mocs, intel_mocs_live_selftests) selftest(gt_pm, intel_gt_pm_live_selftests) selftest(gt_heartbeat, intel_heartbeat_live_selftests) selftest(requests, i915_request_live_selftests) +selftest(migrate, intel_migrate_live_selftests) selftest(active, i915_active_live_selftests) selftest(objects, i915_gem_object_live_selftests) selftest(mman, i915_gem_mman_live_selftests) -- 2.31.1