From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34DD8C48BDF for ; Tue, 8 Jun 2021 09:29:24 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 044806127A for ; Tue, 8 Jun 2021 09:29:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 044806127A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5FEDB6EB6F; Tue, 8 Jun 2021 09:29:18 +0000 (UTC) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by gabe.freedesktop.org (Postfix) with ESMTPS id 20F246EB71; Tue, 8 Jun 2021 09:29:17 +0000 (UTC) IronPort-SDR: ygWIbymT9awPFdITUiAw+JzMpRGXMA9AxIsSEk4pfUJln54egLCH14LPsVK4eYlJ/do7enZ01f Wyv/bSY39p9Q== X-IronPort-AV: E=McAfee;i="6200,9189,10008"; a="226150968" X-IronPort-AV: E=Sophos;i="5.83,257,1616482800"; d="scan'208";a="226150968" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 02:29:16 -0700 IronPort-SDR: FUVfUZBy/YMyQAWhfRn+6Xz06MrO4O5kwIPuJU4esCJCrsBdf+boFhDGPU8qT/tCcaRWZTYj1I wgn1D1oZ9LWw== X-IronPort-AV: E=Sophos;i="5.83,257,1616482800"; d="scan'208";a="551544449" Received: from mkayyal-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.115]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 02:29:14 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Tue, 8 Jun 2021 11:28:44 +0200 Message-Id: <20210608092846.64198-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608092846.64198-1-thomas.hellstrom@linux.intel.com> References: <20210608092846.64198-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 7/9] drm/i915/gt: Pipelined page migration X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" RnJvbTogQ2hyaXMgV2lsc29uIDxjaHJpc0BjaHJpcy13aWxzb24uY28udWs+CgpJZiB3ZSBwaXBl bGluZSB0aGUgUFRFIHVwZGF0ZXMgYW5kIHRoZW4gZG8gdGhlIGNvcHkgb2YgdGhvc2UgcGFnZXMK d2l0aGluIGEgc2luZ2xlIHVucHJlZW1wdGlibGUgY29tbWFuZCBwYWNrZXQsIHdlIGNhbiBzdWJt aXQgdGhlIGNvcGllcwphbmQgbGVhdmUgdGhlbSB0byBiZSBzY2hlZHVsZWQgd2l0aG91dCBoYXZp bmcgdG8gc3luY2hyb25vdXNseSB3YWl0CnVuZGVyIGEgZ2xvYmFsIGxvY2suIEluIG9yZGVyIHRv IG1hbmFnZSBtaWdyYXRpb24sIHdlIG5lZWQgdG8KcHJlYWxsb2NhdGUgdGhlIHBhZ2UgdGFibGVz IChhbmQga2VlcCB0aGVtIHBpbm5lZCBhbmQgYXZhaWxhYmxlIGZvciB1c2UKYXQgYW55IHRpbWUp LCBjYXVzaW5nIGEgYm90dGxlbmVjayBmb3IgbWlncmF0aW9ucyBhcyBhbGwgY2xpZW50cyBtdXN0 CmNvbnRlbmQgb24gdGhlIGxpbWl0ZWQgcmVzb3VyY2VzLiBCeSBpbmxpbmluZyB0aGUgcHBHVFQg dXBkYXRlcyBhbmQKcGVyZm9ybWluZyB0aGUgYmxpdCBhdG9taWNhbGx5LCBlYWNoIGNsaWVudCBv bmx5IG93bnMgdGhlIFBURSB3aGlsZSBpbgp1c2UsIGFuZCBzbyB3ZSBjYW4gcmVzY2hlZHVsZSBp bmRpdmlkdWFsIG9wZXJhdGlvbnMgaG93ZXZlciB3ZSBzZWUgZml0LgpBbmQgbW9zdCBpbXBvcnRh bnRseSwgd2UgZG8gbm90IG5lZWQgdG8gdGFrZSBhIGdsb2JhbCBsb2NrIG9uIHRoZSBzaGFyZWQK dm0sIGFuZCB3YWl0IHVudGlsIHRoZSBvcGVyYXRpb24gaXMgY29tcGxldGUgYmVmb3JlIHJlbGVh c2luZyB0aGUgbG9jawpmb3Igb3RoZXJzIHRvIGNsYWltIHRoZSBQVEUgZm9yIHRoZW1zZWx2ZXMu CgpTaWduZWQtb2ZmLWJ5OiBDaHJpcyBXaWxzb24gPGNocmlzQGNocmlzLXdpbHNvbi5jby51az4K Q28tZGV2ZWxvcGVkLWJ5OiBUaG9tYXMgSGVsbHN0csO2bSA8dGhvbWFzLmhlbGxzdHJvbUBsaW51 eC5pbnRlbC5jb20+ClNpZ25lZC1vZmYtYnk6IFRob21hcyBIZWxsc3Ryw7ZtIDx0aG9tYXMuaGVs bHN0cm9tQGxpbnV4LmludGVsLmNvbT4KLS0tCiBkcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmls ZSAgICAgICAgICAgICAgICAgfCAgIDEgKwogZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxf ZW5naW5lLmggICAgICAgIHwgICAxICsKIGRyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2dw dV9jb21tYW5kcy5oICB8ICAgMiArCiBkcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9taWdy YXRlLmMgICAgICAgfCA1NDMgKysrKysrKysrKysrKysrKysrCiBkcml2ZXJzL2dwdS9kcm0vaTkx NS9ndC9pbnRlbF9taWdyYXRlLmggICAgICAgfCAgNDUgKysKIGRyaXZlcnMvZ3B1L2RybS9pOTE1 L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaCB8ICAxNSArCiBkcml2ZXJzL2dwdS9kcm0vaTkxNS9n dC9pbnRlbF9yaW5nLmggICAgICAgICAgfCAgIDEgKwogZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3Qv c2VsZnRlc3RfbWlncmF0ZS5jICAgIHwgMjkxICsrKysrKysrKysKIC4uLi9kcm0vaTkxNS9zZWxm dGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oICB8ICAgMSArCiA5IGZpbGVzIGNoYW5nZWQsIDkw MCBpbnNlcnRpb25zKCspCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL2k5MTUv Z3QvaW50ZWxfbWlncmF0ZS5jCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL2k5 MTUvZ3QvaW50ZWxfbWlncmF0ZS5oCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJt L2k5MTUvZ3QvaW50ZWxfbWlncmF0ZV90eXBlcy5oCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVy cy9ncHUvZHJtL2k5MTUvZ3Qvc2VsZnRlc3RfbWlncmF0ZS5jCgpkaWZmIC0tZ2l0IGEvZHJpdmVy cy9ncHUvZHJtL2k5MTUvTWFrZWZpbGUgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZQpp bmRleCBlYThlZTRiM2UwMTguLjlmMTg5MDJiZTYyNiAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUv ZHJtL2k5MTUvTWFrZWZpbGUKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvTWFrZWZpbGUKQEAg LTEwOSw2ICsxMDksNyBAQCBndC15ICs9IFwKIAlndC9pbnRlbF9ndHQubyBcCiAJZ3QvaW50ZWxf bGxjLm8gXAogCWd0L2ludGVsX2xyYy5vIFwKKwlndC9pbnRlbF9taWdyYXRlLm8gXAogCWd0L2lu dGVsX21vY3MubyBcCiAJZ3QvaW50ZWxfcHBndHQubyBcCiAJZ3QvaW50ZWxfcmM2Lm8gXApkaWZm IC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZW5naW5lLmggYi9kcml2ZXJz L2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9lbmdpbmUuaAppbmRleCAwODYyYzQyYjRjYWMuLjk0OTk2 NTY4MGMzNyAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZW5naW5l LmgKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfZW5naW5lLmgKQEAgLTE4OCw2 ICsxODgsNyBAQCBpbnRlbF93cml0ZV9zdGF0dXNfcGFnZShzdHJ1Y3QgaW50ZWxfZW5naW5lX2Nz ICplbmdpbmUsIGludCByZWcsIHUzMiB2YWx1ZSkKICNkZWZpbmUgSTkxNV9HRU1fSFdTX1BSRUVN UFRfQUREUgkoSTkxNV9HRU1fSFdTX1BSRUVNUFQgKiBzaXplb2YodTMyKSkKICNkZWZpbmUgSTkx NV9HRU1fSFdTX1NFUU5PCQkweDQwCiAjZGVmaW5lIEk5MTVfR0VNX0hXU19TRVFOT19BRERSCQko STkxNV9HRU1fSFdTX1NFUU5PICogc2l6ZW9mKHUzMikpCisjZGVmaW5lIEk5MTVfR0VNX0hXU19N SUdSQVRFCQkoMHg0MiAqIHNpemVvZih1MzIpKQogI2RlZmluZSBJOTE1X0dFTV9IV1NfU0NSQVRD SAkJMHg4MAogCiAjZGVmaW5lIEk5MTVfSFdTX0NTQl9CVUYwX0lOREVYCQkweDEwCmRpZmYgLS1n aXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9ncHVfY29tbWFuZHMuaCBiL2RyaXZl cnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX2dwdV9jb21tYW5kcy5oCmluZGV4IDI2OTRkYmI5OTY3 ZS4uMWMzYWYwZmMwNDU2IDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRl bF9ncHVfY29tbWFuZHMuaAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9pbnRlbF9ncHVf Y29tbWFuZHMuaApAQCAtMTIzLDggKzEyMywxMCBAQAogI2RlZmluZSAgIE1JX1NFTUFQSE9SRV9T QURfTkVRX1NERAkoNSA8PCAxMikKICNkZWZpbmUgICBNSV9TRU1BUEhPUkVfVE9LRU5fTUFTSwlS RUdfR0VOTUFTSyg5LCA1KQogI2RlZmluZSAgIE1JX1NFTUFQSE9SRV9UT0tFTl9TSElGVAk1Cisj ZGVmaW5lIE1JX1NUT1JFX0RBVEFfSU1NCU1JX0lOU1RSKDB4MjAsIDApCiAjZGVmaW5lIE1JX1NU T1JFX0RXT1JEX0lNTQlNSV9JTlNUUigweDIwLCAxKQogI2RlZmluZSBNSV9TVE9SRV9EV09SRF9J TU1fR0VONAlNSV9JTlNUUigweDIwLCAyKQorI2RlZmluZSBNSV9TVE9SRV9RV09SRF9JTU1fR0VO OCAoTUlfSU5TVFIoMHgyMCwgMykgfCBSRUdfQklUKDIxKSkKICNkZWZpbmUgICBNSV9NRU1fVklS VFVBTAkoMSA8PCAyMikgLyogOTQ1LGczMyw5NjUgKi8KICNkZWZpbmUgICBNSV9VU0VfR0dUVAkJ KDEgPDwgMjIpIC8qIGc0eCsgKi8KICNkZWZpbmUgTUlfU1RPUkVfRFdPUkRfSU5ERVgJTUlfSU5T VFIoMHgyMSwgMSkKZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21p Z3JhdGUuYyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuYwpuZXcgZmls ZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLjFmNjBmOGVlMzZmOAotLS0gL2Rldi9u dWxsCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGUuYwpAQCAtMCww ICsxLDU0MyBAQAorLy8gU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6IE1JVAorLyoKKyAqIENvcHly aWdodCDCqSAyMDIwIEludGVsIENvcnBvcmF0aW9uCisgKi8KKworI2luY2x1ZGUgImk5MTVfZHJ2 LmgiCisjaW5jbHVkZSAiaW50ZWxfY29udGV4dC5oIgorI2luY2x1ZGUgImludGVsX2dwdV9jb21t YW5kcy5oIgorI2luY2x1ZGUgImludGVsX2d0LmgiCisjaW5jbHVkZSAiaW50ZWxfZ3R0LmgiCisj aW5jbHVkZSAiaW50ZWxfbWlncmF0ZS5oIgorI2luY2x1ZGUgImludGVsX3JpbmcuaCIKKworc3Ry dWN0IGluc2VydF9wdGVfZGF0YSB7CisJdTY0IG9mZnNldDsKKwlib29sIGlzX2xtZW07Cit9Owor CisjZGVmaW5lIENIVU5LX1NaIFNaXzhNIC8qIH4xbXMgYXQgOEdpQi9zIHByZWVtcHRpb24gZGVs YXkgKi8KKworc3RhdGljIGJvb2wgZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihzdHJ1Y3QgaW50 ZWxfZW5naW5lX2NzICplbmdpbmUpCit7CisJaWYgKCFlbmdpbmUpCisJCXJldHVybiBmYWxzZTsK KworCS8qCisJICogV2UgbmVlZCB0aGUgYWJpbGl0eSB0byBwcmV2ZW50IGFyaWJ0cmF0aW9uIChN SV9BUkJfT05fT0ZGKSwKKwkgKiB0aGUgYWJpbGl0eSB0byB3cml0ZSBQVEUgdXNpbmcgaW5saW5l IGRhdGEgKE1JX1NUT1JFX0RBVEEpCisJICogYW5kIG9mIGNvdXJzZSB0aGUgYWJpbGl0eSB0byBk byB0aGUgYmxvY2sgdHJhbnNmZXIgKGJsaXRzKS4KKwkgKi8KKwlHRU1fQlVHX09OKGVuZ2luZS0+ Y2xhc3MgIT0gQ09QWV9FTkdJTkVfQ0xBU1MpOworCisJcmV0dXJuIHRydWU7Cit9CisKK3N0YXRp YyB2b2lkIGluc2VydF9wdGUoc3RydWN0IGk5MTVfYWRkcmVzc19zcGFjZSAqdm0sCisJCSAgICAg ICBzdHJ1Y3QgaTkxNV9wYWdlX3RhYmxlICpwdCwKKwkJICAgICAgIHZvaWQgKmRhdGEpCit7CisJ c3RydWN0IGluc2VydF9wdGVfZGF0YSAqZCA9IGRhdGE7CisKKwl2bS0+aW5zZXJ0X3BhZ2Uodm0s IHB4X2RtYShwdCksIGQtPm9mZnNldCwgSTkxNV9DQUNIRV9OT05FLAorCQkJZC0+aXNfbG1lbSA/ IFBURV9MTSA6IDApOworCWQtPm9mZnNldCArPSBQQUdFX1NJWkU7Cit9CisKK3N0YXRpYyBzdHJ1 Y3QgaTkxNV9hZGRyZXNzX3NwYWNlICptaWdyYXRlX3ZtKHN0cnVjdCBpbnRlbF9ndCAqZ3QpCit7 CisJc3RydWN0IGk5MTVfdm1fcHRfc3Rhc2ggc3Rhc2ggPSB7fTsKKwlzdHJ1Y3QgaTkxNV9wcGd0 dCAqdm07CisJaW50IGVycjsKKwlpbnQgaTsKKworCS8qCisJICogV2UgY29uc3RydWN0IGEgdmVy eSBzcGVjaWFsIFZNIGZvciB1c2UgYnkgYWxsIG1pZ3JhdGlvbiBjb250ZXh0cywKKwkgKiBpdCBp cyBrZXB0IHBpbm5lZCBzbyB0aGF0IGl0IGNhbiBiZSB1c2VkIGF0IGFueSB0aW1lLiBBcyB3ZSBu ZWVkCisJICogdG8gcHJlLWFsbG9jYXRlIHRoZSBwYWdlIGRpcmVjdG9yaWVzIGZvciB0aGUgbWln cmF0aW9uIFZNLCB0aGlzCisJICogbGltaXRzIHVzIHRvIG9ubHkgdXNpbmcgYSBzbWFsbCBudW1i ZXIgb2YgcHJlcGFyZWQgdm1hLgorCSAqCisJICogVG8gYmUgYWJsZSB0byBwaXBlbGluZSBhbmQg cmVzY2hlZHVsZSBtaWdyYXRpb24gb3BlcmF0aW9ucyB3aGlsZQorCSAqIGF2b2lkaW5nIHVubmVj ZXNzYXJ5IGNvbnRlbnRpb24gb24gdGhlIHZtIGl0c2VsZiwgdGhlIFBURSB1cGRhdGVzCisJICog YXJlIGlubGluZSB3aXRoIHRoZSBibGl0cy4gQWxsIHRoZSBibGl0cyB1c2UgdGhlIHNhbWUgZml4 ZWQKKwkgKiBhZGRyZXNzZXMsIHdpdGggdGhlIGJhY2tpbmcgc3RvcmUgcmVkaXJlY3Rpb24gYmVp bmcgdXBkYXRlZCBvbiB0aGUKKwkgKiBmbHkuIE9ubHkgMiBpbXBsaWNpdCB2bWEgYXJlIHVzZWQg Zm9yIGFsbCBtaWdyYXRpb24gb3BlcmF0aW9ucy4KKwkgKgorCSAqIFdlIGxheSB0aGUgcHBHVFQg b3V0IGFzOgorCSAqCisJICoJWzAsIENIVU5LX1NaKSAtPiBmaXJzdCBvYmplY3QKKwkgKglbQ0hV TktfU1osIDIgKiBDSFVOS19TWikgLT4gc2Vjb25kIG9iamVjdAorCSAqCVsyICogQ0hVTktfU1os IDIgKiBDSFVOS19TWiArIDIgKiBDSFVOS19TWiA+PiA5XSAtPiBQVEUKKwkgKgorCSAqIEJ5IGV4 cG9zaW5nIHRoZSBkbWEgYWRkcmVzc2VzIG9mIHRoZSBwYWdlIGRpcmVjdG9yaWVzIHRoZW1zZWx2 ZXMKKwkgKiB3aXRoaW4gdGhlIHBwR1RULCB3ZSBhcmUgdGhlbiBhYmxlIHRvIHJld3JpdGUgdGhl IFBURSBwcmlvciB0byB1c2UuCisJICogQnV0IHRoZSBQVEUgdXBkYXRlIGFuZCBzdWJzZXF1ZW50 IG1pZ3JhdGlvbiBvcGVyYXRpb24gbXVzdCBiZSBhdG9taWMsCisJICogaS5lLiB3aXRoaW4gdGhl IHNhbWUgbm9uLXByZWVtcHRpYmxlIHdpbmRvdyBzbyB0aGF0IHdlIGRvIG5vdCBzd2l0Y2gKKwkg KiB0byBhbm90aGVyIG1pZ3JhdGlvbiBjb250ZXh0IHRoYXQgb3ZlcndyaXRlcyB0aGUgUFRFLgor CSAqLworCisJdm0gPSBpOTE1X3BwZ3R0X2NyZWF0ZShndCk7CisJaWYgKElTX0VSUih2bSkpCisJ CXJldHVybiBFUlJfQ0FTVCh2bSk7CisKKwlpZiAoIXZtLT52bS5hbGxvY2F0ZV92YV9yYW5nZSB8 fCAhdm0tPnZtLmZvcmVhY2gpIHsKKwkJZXJyID0gLUVOT0RFVjsKKwkJZ290byBlcnJfdm07CisJ fQorCisJLyoKKwkgKiBFYWNoIGVuZ2luZSBpbnN0YW5jZSBpcyBhc3NpZ25lZCBpdHMgb3duIGNo dW5rIGluIHRoZSBWTSwgc28KKwkgKiB0aGF0IHdlIGNhbiBydW4gbXVsdGlwbGUgaW5zdGFuY2Vz IGNvbmN1cnJlbnRseQorCSAqLworCWZvciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKGd0LT5lbmdp bmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xBU1NdKTsgaSsrKSB7CisJCXN0cnVjdCBpbnRlbF9lbmdp bmVfY3MgKmVuZ2luZTsKKwkJdTY0IGJhc2UgPSAodTY0KWkgPDwgMzI7CisJCXN0cnVjdCBpbnNl cnRfcHRlX2RhdGEgZCA9IHt9OworCQlzdHJ1Y3QgaTkxNV9nZW1fd3dfY3R4IHd3OworCQl1NjQg c3o7CisKKwkJZW5naW5lID0gZ3QtPmVuZ2luZV9jbGFzc1tDT1BZX0VOR0lORV9DTEFTU11baV07 CisJCWlmICghZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihlbmdpbmUpKQorCQkJY29udGludWU7 CisKKwkJLyoKKwkJICogV2UgY29weSBpbiA4TWlCIGNodW5rcy4gRWFjaCBQREUgY292ZXJzIDJN aUIsIHNvIHdlIG5lZWQKKwkJICogNHgyIHBhZ2UgZGlyZWN0b3JpZXMgZm9yIHNvdXJjZS9kZXN0 aW5hdGlvbi4KKwkJICovCisJCXN6ID0gMiAqIENIVU5LX1NaOworCQlkLm9mZnNldCA9IGJhc2Ug KyBzejsKKworCQkvKgorCQkgKiBXZSBuZWVkIGFub3RoZXIgcGFnZSBkaXJlY3Rvcnkgc2V0dXAg c28gdGhhdCB3ZSBjYW4gd3JpdGUKKwkJICogdGhlIDh4NTEyIFBURSBpbiBlYWNoIGNodW5rLgor CQkgKi8KKwkJc3ogKz0gKHN6ID4+IDEyKSAqIHNpemVvZih1NjQpOworCisJCWVyciA9IGk5MTVf dm1fYWxsb2NfcHRfc3Rhc2goJnZtLT52bSwgJnN0YXNoLCBzeik7CisJCWlmIChlcnIpCisJCQln b3RvIGVycl92bTsKKworCQlmb3JfaTkxNV9nZW1fd3coJnd3LCBlcnIsIHRydWUpIHsKKwkJCWVy ciA9IGk5MTVfdm1fbG9ja19vYmplY3RzKCZ2bS0+dm0sICZ3dyk7CisJCQlpZiAoZXJyKQorCQkJ CWNvbnRpbnVlOworCQkJZXJyID0gaTkxNV92bV9tYXBfcHRfc3Rhc2goJnZtLT52bSwgJnN0YXNo KTsKKwkJCWlmIChlcnIpCisJCQkJY29udGludWU7CisKKwkJCXZtLT52bS5hbGxvY2F0ZV92YV9y YW5nZSgmdm0tPnZtLCAmc3Rhc2gsIGJhc2UsIGJhc2UgKyBzeik7CisJCX0KKwkJaTkxNV92bV9m cmVlX3B0X3N0YXNoKCZ2bS0+dm0sICZzdGFzaCk7CisJCWlmIChlcnIpCisJCQlnb3RvIGVycl92 bTsKKworCQkvKiBOb3cgYWxsb3cgdGhlIEdQVSB0byByZXdyaXRlIHRoZSBQVEUgdmlhIGl0cyBv d24gcHBHVFQgKi8KKwkJZC5pc19sbWVtID0gaTkxNV9nZW1fb2JqZWN0X2lzX2xtZW0odm0tPnZt LnNjcmF0Y2hbMF0pOworCQl2bS0+dm0uZm9yZWFjaCgmdm0tPnZtLCBiYXNlLCBiYXNlICsgc3os IGluc2VydF9wdGUsICZkKTsKKwl9CisKKwlyZXR1cm4gJnZtLT52bTsKKworZXJyX3ZtOgorCWk5 MTVfdm1fcHV0KCZ2bS0+dm0pOworCXJldHVybiBFUlJfUFRSKGVycik7Cit9CisKK3N0YXRpYyBz dHJ1Y3QgaW50ZWxfZW5naW5lX2NzICpmaXJzdF9jb3B5X2VuZ2luZShzdHJ1Y3QgaW50ZWxfZ3Qg Kmd0KQoreworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwlpbnQgaTsKKworCWZv ciAoaSA9IDA7IGkgPCBBUlJBWV9TSVpFKGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xB U1NdKTsgaSsrKSB7CisJCWVuZ2luZSA9IGd0LT5lbmdpbmVfY2xhc3NbQ09QWV9FTkdJTkVfQ0xB U1NdW2ldOworCQlpZiAoZW5naW5lX3N1cHBvcnRzX21pZ3JhdGlvbihlbmdpbmUpKQorCQkJcmV0 dXJuIGVuZ2luZTsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworc3RhdGljIHN0cnVjdCBpbnRl bF9jb250ZXh0ICpwaW5uZWRfY29udGV4dChzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0YXRp YyBzdHJ1Y3QgbG9ja19jbGFzc19rZXkga2V5OworCXN0cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVu Z2luZTsKKwlzdHJ1Y3QgaTkxNV9hZGRyZXNzX3NwYWNlICp2bTsKKwlzdHJ1Y3QgaW50ZWxfY29u dGV4dCAqY2U7CisKKwllbmdpbmUgPSBmaXJzdF9jb3B5X2VuZ2luZShndCk7CisJaWYgKCFlbmdp bmUpCisJCXJldHVybiBFUlJfUFRSKC1FTk9ERVYpOworCisJdm0gPSBtaWdyYXRlX3ZtKGd0KTsK KwlpZiAoSVNfRVJSKHZtKSkKKwkJcmV0dXJuIEVSUl9DQVNUKHZtKTsKKworCWNlID0gaW50ZWxf ZW5naW5lX2NyZWF0ZV9waW5uZWRfY29udGV4dChlbmdpbmUsIHZtLCBTWl81MTJLLAorCQkJCQkJ STkxNV9HRU1fSFdTX01JR1JBVEUsCisJCQkJCQkma2V5LCAibWlncmF0ZSIpOworCWk5MTVfdm1f cHV0KGNlLT52bSk7CisJcmV0dXJuIGNlOworfQorCitpbnQgaW50ZWxfbWlncmF0ZV9pbml0KHN0 cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0KQoreworCXN0cnVjdCBp bnRlbF9jb250ZXh0ICpjZTsKKworCW1lbXNldChtLCAwLCBzaXplb2YoKm0pKTsKKworCWNlID0g cGlubmVkX2NvbnRleHQoZ3QpOworCWlmIChJU19FUlIoY2UpKQorCQlyZXR1cm4gUFRSX0VSUihj ZSk7CisKKwltLT5jb250ZXh0ID0gY2U7CisJcmV0dXJuIDA7Cit9CisKK3N0YXRpYyBpbnQgcmFu ZG9tX2luZGV4KHVuc2lnbmVkIGludCBtYXgpCit7CisJcmV0dXJuIHVwcGVyXzMyX2JpdHMobXVs X3UzMl91MzIoZ2V0X3JhbmRvbV91MzIoKSwgbWF4KSk7Cit9CisKK3N0YXRpYyBzdHJ1Y3QgaW50 ZWxfY29udGV4dCAqX19taWdyYXRlX2VuZ2luZXMoc3RydWN0IGludGVsX2d0ICpndCkKK3sKKwlz dHJ1Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmVzW01BWF9FTkdJTkVfSU5TVEFOQ0VdOworCXN0 cnVjdCBpbnRlbF9lbmdpbmVfY3MgKmVuZ2luZTsKKwl1bnNpZ25lZCBpbnQgY291bnQsIGk7CisK Kwljb3VudCA9IDA7CisJZm9yIChpID0gMDsgaSA8IEFSUkFZX1NJWkUoZ3QtPmVuZ2luZV9jbGFz c1tDT1BZX0VOR0lORV9DTEFTU10pOyBpKyspIHsKKwkJZW5naW5lID0gZ3QtPmVuZ2luZV9jbGFz c1tDT1BZX0VOR0lORV9DTEFTU11baV07CisJCWlmIChlbmdpbmVfc3VwcG9ydHNfbWlncmF0aW9u KGVuZ2luZSkpCisJCQllbmdpbmVzW2NvdW50KytdID0gZW5naW5lOworCX0KKworCXJldHVybiBp bnRlbF9jb250ZXh0X2NyZWF0ZShlbmdpbmVzW3JhbmRvbV9pbmRleChjb3VudCldKTsKK30KKwor c3RydWN0IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQoc3RydWN0 IGludGVsX21pZ3JhdGUgKm0pCit7CisJc3RydWN0IGludGVsX2NvbnRleHQgKmNlOworCisJLyoK KwkgKiBXZSByYW5kb21seSBkaXN0cmlidXRlIGNvbnRleHRzIGFjcm9zcyB0aGUgZW5naW5lcyB1 cG9uIGNvbnN0cmN0aW9uLAorCSAqIGFzIHRoZXkgYWxsIHNoYXJlIHRoZSBzYW1lIHBpbm5lZCB2 bSwgYW5kIHNvIGluIG9yZGVyIHRvIGFsbG93CisJICogbXVsdGlwbGUgYmxpdHMgdG8gcnVuIGlu IHBhcmFsbGVsLCB3ZSBtdXN0IGNvbnN0cnVjdCBlYWNoIGJsaXQKKwkgKiB0byB1c2UgYSBkaWZm ZXJlbnQgcmFuZ2Ugb2YgdGhlIHZtIGZvciBpdHMgR1RULiBUaGlzIGhhcyB0byBiZQorCSAqIGtu b3duIGF0IGNvbnN0cnVjdGlvbiwgc28gd2UgY2FuIG5vdCB1c2UgdGhlIGxhdGUgZ3JlZWR5IGxv YWQKKwkgKiBiYWxhbmNpbmcgb2YgdGhlIHZpcnR1YWwtZW5naW5lLgorCSAqLworCWNlID0gX19t aWdyYXRlX2VuZ2luZXMobS0+Y29udGV4dC0+ZW5naW5lLT5ndCk7CisJaWYgKElTX0VSUihjZSkp CisJCXJldHVybiBjZTsKKworCWNlLT5yaW5nID0gX19pbnRlbF9jb250ZXh0X3Jpbmdfc2l6ZShT Wl8yNTZLKTsKKworCWk5MTVfdm1fcHV0KGNlLT52bSk7CisJY2UtPnZtID0gaTkxNV92bV9nZXQo bS0+Y29udGV4dC0+dm0pOworCisJcmV0dXJuIGNlOworfQorCitzdGF0aWMgaW5saW5lIHN0cnVj dCBzZ3RfZG1hIHNnX3NndChzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnKQoreworCWRtYV9hZGRyX3Qg YWRkciA9IHNnX2RtYV9hZGRyZXNzKHNnKTsKKworCXJldHVybiAoc3RydWN0IHNndF9kbWEpeyBz ZywgYWRkciwgYWRkciArIHNnX2RtYV9sZW4oc2cpIH07Cit9CisKK3N0YXRpYyBpbnQgZW1pdF9u b19hcmJpdHJhdGlvbihzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycSkKK3sKKwl1MzIgKmNzOworCisJ Y3MgPSBpbnRlbF9yaW5nX2JlZ2luKHJxLCAyKTsKKwlpZiAoSVNfRVJSKGNzKSkKKwkJcmV0dXJu IFBUUl9FUlIoY3MpOworCisJLyogRXhwbGljaXRseSBkaXNhYmxlIHByZWVtcHRpb24gZm9yIHRo aXMgcmVxdWVzdC4gKi8KKwkqY3MrKyA9IE1JX0FSQl9PTl9PRkY7CisJKmNzKysgPSBNSV9OT09Q OworCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCisJcmV0dXJuIDA7Cit9CisKK3N0YXRp YyBpbnQgZW1pdF9wdGUoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsCisJCSAgICBzdHJ1Y3Qgc2d0 X2RtYSAqaXQsCisJCSAgICBlbnVtIGk5MTVfY2FjaGVfbGV2ZWwgY2FjaGVfbGV2ZWwsCisJCSAg ICBib29sIGlzX2xtZW0sCisJCSAgICB1NjQgb2Zmc2V0LAorCQkgICAgaW50IGxlbmd0aCkKK3sK Kwljb25zdCB1NjQgZW5jb2RlID0gcnEtPmNvbnRleHQtPnZtLT5wdGVfZW5jb2RlKDAsIGNhY2hl X2xldmVsLAorCQkJCQkJICAgICAgIGlzX2xtZW0gPyBQVEVfTE0gOiAwKTsKKwlzdHJ1Y3QgaW50 ZWxfcmluZyAqcmluZyA9IHJxLT5yaW5nOworCWludCB0b3RhbCA9IDA7CisJdTMyICpoZHIsICpj czsKKwlpbnQgcGt0OworCisJR0VNX0JVR19PTihJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSkg PCA4KTsKKworCS8qIENvbXB1dGUgdGhlIHBhZ2UgZGlyZWN0b3J5IG9mZnNldCBmb3IgdGhlIHRh cmdldCBhZGRyZXNzIHJhbmdlICovCisJb2Zmc2V0ICs9ICh1NjQpcnEtPmVuZ2luZS0+aW5zdGFu Y2UgPDwgMzI7CisJb2Zmc2V0ID4+PSAxMjsKKwlvZmZzZXQgKj0gc2l6ZW9mKHU2NCk7CisJb2Zm c2V0ICs9IDIgKiBDSFVOS19TWjsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgNik7CisJ aWYgKElTX0VSUihjcykpCisJCXJldHVybiBQVFJfRVJSKGNzKTsKKworCS8qIFBhY2sgYXMgbWFu eSBQVEUgdXBkYXRlcyBhcyBwb3NzaWJsZSBpbnRvIGEgc2luZ2xlIE1JIGNvbW1hbmQgKi8KKwlw a3QgPSBtaW5fdChpbnQsIDB4NDAwLCByaW5nLT5zcGFjZSAvIHNpemVvZih1MzIpICsgNSk7CisJ cGt0ID0gbWluX3QoaW50LCBwa3QsIChyaW5nLT5zaXplIC0gcmluZy0+ZW1pdCkgLyBzaXplb2Yo dTMyKSArIDUpOworCisJaGRyID0gY3M7CisJKmNzKysgPSBNSV9TVE9SRV9EQVRBX0lNTSB8IFJF R19CSVQoMjEpOyAvKiBhcyBxd29yZCBlbGVtZW50cyAqLworCSpjcysrID0gbG93ZXJfMzJfYml0 cyhvZmZzZXQpOworCSpjcysrID0gdXBwZXJfMzJfYml0cyhvZmZzZXQpOworCisJZG8geworCQlp ZiAoY3MgLSBoZHIgPj0gcGt0KSB7CisJCQkqaGRyICs9IGNzIC0gaGRyIC0gMjsKKwkJCSpjcysr ID0gTUlfTk9PUDsKKworCQkJcmluZy0+ZW1pdCA9ICh2b2lkICopY3MgLSByaW5nLT52YWRkcjsK KwkJCWludGVsX3JpbmdfYWR2YW5jZShycSwgY3MpOworCQkJaW50ZWxfcmluZ191cGRhdGVfc3Bh Y2UocmluZyk7CisKKwkJCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgNik7CisJCQlpZiAoSVNf RVJSKGNzKSkKKwkJCQlyZXR1cm4gUFRSX0VSUihjcyk7CisKKwkJCXBrdCA9IG1pbl90KGludCwg MHg0MDAsIHJpbmctPnNwYWNlIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwkJCXBrdCA9IG1pbl90KGlu dCwgcGt0LCAocmluZy0+c2l6ZSAtIHJpbmctPmVtaXQpIC8gc2l6ZW9mKHUzMikgKyA1KTsKKwor CQkJaGRyID0gY3M7CisJCQkqY3MrKyA9IE1JX1NUT1JFX0RBVEFfSU1NIHwgUkVHX0JJVCgyMSk7 CisJCQkqY3MrKyA9IGxvd2VyXzMyX2JpdHMob2Zmc2V0KTsKKwkJCSpjcysrID0gdXBwZXJfMzJf Yml0cyhvZmZzZXQpOworCQl9CisKKwkJKmNzKysgPSBsb3dlcl8zMl9iaXRzKGVuY29kZSB8IGl0 LT5kbWEpOworCQkqY3MrKyA9IHVwcGVyXzMyX2JpdHMoZW5jb2RlIHwgaXQtPmRtYSk7CisKKwkJ b2Zmc2V0ICs9IDg7CisJCXRvdGFsICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKworCQlpdC0+ZG1h ICs9IEk5MTVfR1RUX1BBR0VfU0laRTsKKwkJaWYgKGl0LT5kbWEgPj0gaXQtPm1heCkgeworCQkJ aXQtPnNnID0gX19zZ19uZXh0KGl0LT5zZyk7CisJCQlpZiAoIWl0LT5zZyB8fCBzZ19kbWFfbGVu KGl0LT5zZykgPT0gMCkKKwkJCQlicmVhazsKKworCQkJaXQtPmRtYSA9IHNnX2RtYV9hZGRyZXNz KGl0LT5zZyk7CisJCQlpdC0+bWF4ID0gaXQtPmRtYSArIHNnX2RtYV9sZW4oaXQtPnNnKTsKKwkJ fQorCX0gd2hpbGUgKHRvdGFsIDwgbGVuZ3RoKTsKKworCSpoZHIgKz0gY3MgLSBoZHIgLSAyOwor CSpjcysrID0gTUlfTk9PUDsKKworCXJpbmctPmVtaXQgPSAodm9pZCAqKWNzIC0gcmluZy0+dmFk ZHI7CisJaW50ZWxfcmluZ19hZHZhbmNlKHJxLCBjcyk7CisJaW50ZWxfcmluZ191cGRhdGVfc3Bh Y2UocmluZyk7CisKKwlyZXR1cm4gdG90YWw7Cit9CisKK3N0YXRpYyBib29sIHdhXzEyMDk2NDQ2 MTFfYXBwbGllcyhpbnQgZ2VuLCB1MzIgc2l6ZSkKK3sKKwl1MzIgaGVpZ2h0ID0gc2l6ZSA+PiBQ QUdFX1NISUZUOworCisJaWYgKGdlbiAhPSAxMSkKKwkJcmV0dXJuIGZhbHNlOworCisJcmV0dXJu IGhlaWdodCAlIDQgPT0gMyAmJiBoZWlnaHQgPD0gODsKK30KKworc3RhdGljIGludCBlbWl0X2Nv cHkoc3RydWN0IGk5MTVfcmVxdWVzdCAqcnEsIGludCBzaXplKQoreworCWNvbnN0IGludCBnZW4g PSBJTlRFTF9HRU4ocnEtPmVuZ2luZS0+aTkxNSk7CisJdTMyIGluc3RhbmNlID0gcnEtPmVuZ2lu ZS0+aW5zdGFuY2U7CisJdTMyICpjczsKKworCWNzID0gaW50ZWxfcmluZ19iZWdpbihycSwgZ2Vu ID49IDggPyAxMCA6IDYpOworCWlmIChJU19FUlIoY3MpKQorCQlyZXR1cm4gUFRSX0VSUihjcyk7 CisKKwlpZiAoZ2VuID49IDkgJiYgIXdhXzEyMDk2NDQ2MTFfYXBwbGllcyhnZW4sIHNpemUpKSB7 CisJCSpjcysrID0gR0VOOV9YWV9GQVNUX0NPUFlfQkxUX0NNRCB8ICgxMCAtIDIpOworCQkqY3Mr KyA9IEJMVF9ERVBUSF8zMiB8IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOworCQkqY3MrKyA9IHNp emUgPj4gUEFHRV9TSElGVCA8PCAxNiB8IFBBR0VfU0laRSAvIDQ7CisJCSpjcysrID0gQ0hVTktf U1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBpbnN0YW5jZTsKKwkJKmNzKysgPSAwOwor CQkqY3MrKyA9IFBBR0VfU0laRTsKKwkJKmNzKysgPSAwOyAvKiBzcmMgb2Zmc2V0ICovCisJCSpj cysrID0gaW5zdGFuY2U7CisJfSBlbHNlIGlmIChnZW4gPj0gOCkgeworCQkqY3MrKyA9IFhZX1NS Q19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICgxMCAtIDIpOworCQkqY3MrKyA9IEJM VF9ERVBUSF8zMiB8IEJMVF9ST1BfU1JDX0NPUFkgfCBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsK KwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJWkUgLyA0OworCQkq Y3MrKyA9IENIVU5LX1NaOyAvKiBkc3Qgb2Zmc2V0ICovCisJCSpjcysrID0gaW5zdGFuY2U7CisJ CSpjcysrID0gMDsKKwkJKmNzKysgPSBQQUdFX1NJWkU7CisJCSpjcysrID0gMDsgLyogc3JjIG9m ZnNldCAqLworCQkqY3MrKyA9IGluc3RhbmNlOworCX0gZWxzZSB7CisJCUdFTV9CVUdfT04oaW5z dGFuY2UpOworCQkqY3MrKyA9IFNSQ19DT1BZX0JMVF9DTUQgfCBCTFRfV1JJVEVfUkdCQSB8ICg2 IC0gMik7CisJCSpjcysrID0gQkxUX0RFUFRIXzMyIHwgQkxUX1JPUF9TUkNfQ09QWSB8IFBBR0Vf U0laRTsKKwkJKmNzKysgPSBzaXplID4+IFBBR0VfU0hJRlQgPDwgMTYgfCBQQUdFX1NJWkU7CisJ CSpjcysrID0gQ0hVTktfU1o7IC8qIGRzdCBvZmZzZXQgKi8KKwkJKmNzKysgPSBQQUdFX1NJWkU7 CisJCSpjcysrID0gMDsgLyogc3JjIG9mZnNldCAqLworCX0KKworCWludGVsX3JpbmdfYWR2YW5j ZShycSwgY3MpOworCXJldHVybiAwOworfQorCitpbnQKK2ludGVsX2NvbnRleHRfbWlncmF0ZV9j b3B5KHN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZSwKKwkJCSAgIHN0cnVjdCBkbWFfZmVuY2UgKmF3 YWl0LAorCQkJICAgc3RydWN0IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkgICBlbnVtIGk5MTVfY2Fj aGVfbGV2ZWwgc3JjX2NhY2hlX2xldmVsLAorCQkJICAgYm9vbCBzcmNfaXNfbG1lbSwKKwkJCSAg IHN0cnVjdCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIGRz dF9jYWNoZV9sZXZlbCwKKwkJCSAgIGJvb2wgZHN0X2lzX2xtZW0sCisJCQkgICBzdHJ1Y3QgaTkx NV9yZXF1ZXN0ICoqb3V0KQoreworCXN0cnVjdCBzZ3RfZG1hIGl0X3NyYyA9IHNnX3NndChzcmMp LCBpdF9kc3QgPSBzZ19zZ3QoZHN0KTsKKwlzdHJ1Y3QgaTkxNV9yZXF1ZXN0ICpycTsKKwlpbnQg ZXJyOworCisJKm91dCA9IE5VTEw7CisKKwkvKiBHRU1fQlVHX09OKGNlLT52bSAhPSBtaWdyYXRl X3ZtKTsgKi8KKworCUdFTV9CVUdfT04oY2UtPnJpbmctPnNpemUgPCBTWl82NEspOworCisJZG8g eworCQlpbnQgbGVuOworCisJCXJxID0gaTkxNV9yZXF1ZXN0X2NyZWF0ZShjZSk7CisJCWlmIChJ U19FUlIocnEpKSB7CisJCQllcnIgPSBQVFJfRVJSKHJxKTsKKwkJCWdvdG8gb3V0X2NlOworCQl9 CisKKwkJaWYgKGF3YWl0KSB7CisJCQllcnIgPSBpOTE1X3JlcXVlc3RfYXdhaXRfZG1hX2ZlbmNl KHJxLCBhd2FpdCk7CisJCQlpZiAoZXJyKQorCQkJCWdvdG8gb3V0X3JxOworCisJCQlpZiAocnEt PmVuZ2luZS0+ZW1pdF9pbml0X2JyZWFkY3J1bWIpIHsKKwkJCQllcnIgPSBycS0+ZW5naW5lLT5l bWl0X2luaXRfYnJlYWRjcnVtYihycSk7CisJCQkJaWYgKGVycikKKwkJCQkJZ290byBvdXRfcnE7 CisJCQl9CisKKwkJCWF3YWl0ID0gTlVMTDsKKwkJfQorCisJCS8qIFRoZSBQVEUgdXBkYXRlcyAr IGNvcHkgbXVzdCBub3QgYmUgaW50ZXJydXB0ZWQuICovCisJCWVyciA9IGVtaXRfbm9fYXJiaXRy YXRpb24ocnEpOworCQlpZiAoZXJyKQorCQkJZ290byBvdXRfcnE7CisKKwkJbGVuID0gZW1pdF9w dGUocnEsICZpdF9zcmMsIHNyY19jYWNoZV9sZXZlbCwgc3JjX2lzX2xtZW0sIDAsCisJCQkgICAg ICAgQ0hVTktfU1opOworCQlpZiAobGVuIDw9IDApIHsKKwkJCWVyciA9IGxlbjsKKwkJCWdvdG8g b3V0X3JxOworCQl9CisKKwkJZXJyID0gZW1pdF9wdGUocnEsICZpdF9kc3QsIGRzdF9jYWNoZV9s ZXZlbCwgZHN0X2lzX2xtZW0sCisJCQkgICAgICAgQ0hVTktfU1osIGxlbik7CisJCWlmIChlcnIg PCAwKQorCQkJZ290byBvdXRfcnE7CisJCWlmIChlcnIgPCBsZW4pIHsKKwkJCWVyciA9IC1FSU5W QUw7CisJCQlnb3RvIG91dF9ycTsKKwkJfQorCisJCWVyciA9IHJxLT5lbmdpbmUtPmVtaXRfZmx1 c2gocnEsIEVNSVRfSU5WQUxJREFURSk7CisJCWlmIChlcnIpCisJCQlnb3RvIG91dF9ycTsKKwor CQllcnIgPSBlbWl0X2NvcHkocnEsIGxlbik7CisKKwkJLyogQXJiaXRyYXRpb24gaXMgcmUtZW5h YmxlZCBiZXR3ZWVuIHJlcXVlc3RzLiAqLworb3V0X3JxOgorCQlpZiAoKm91dCkKKwkJCWk5MTVf cmVxdWVzdF9wdXQoKm91dCk7CisJCSpvdXQgPSBpOTE1X3JlcXVlc3RfZ2V0KHJxKTsKKwkJaTkx NV9yZXF1ZXN0X2FkZChycSk7CisJCWlmIChlcnIgfHwgIWl0X3NyYy5zZyB8fCAhc2dfZG1hX2xl bihpdF9zcmMuc2cpKQorCQkJYnJlYWs7CisKKwkJY29uZF9yZXNjaGVkKCk7CisJfSB3aGlsZSAo MSk7CisKK291dF9jZToKKwlyZXR1cm4gZXJyOworfQorCitpbnQgaW50ZWxfbWlncmF0ZV9jb3B5 KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLAorCQkgICAgICAgc3RydWN0IGk5MTVfZ2VtX3d3X2N0 eCAqd3csCisJCSAgICAgICBzdHJ1Y3QgZG1hX2ZlbmNlICphd2FpdCwKKwkJICAgICAgIHN0cnVj dCBzY2F0dGVybGlzdCAqc3JjLAorCQkgICAgICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIHNyY19j YWNoZV9sZXZlbCwKKwkJICAgICAgIGJvb2wgc3JjX2lzX2xtZW0sCisJCSAgICAgICBzdHJ1Y3Qg c2NhdHRlcmxpc3QgKmRzdCwKKwkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBkc3RfY2Fj aGVfbGV2ZWwsCisJCSAgICAgICBib29sIGRzdF9pc19sbWVtLAorCQkgICAgICAgc3RydWN0IGk5 MTVfcmVxdWVzdCAqKm91dCkKK3sKKwlzdHJ1Y3QgaW50ZWxfY29udGV4dCAqY2U7CisJaW50IGVy cjsKKworCSpvdXQgPSBOVUxMOworCWlmICghbS0+Y29udGV4dCkKKwkJcmV0dXJuIC1FTk9ERVY7 CisKKwljZSA9IGludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQobSk7CisJaWYgKElTX0VSUihj ZSkpCisJCWNlID0gaW50ZWxfY29udGV4dF9nZXQobS0+Y29udGV4dCk7CisJR0VNX0JVR19PTihJ U19FUlIoY2UpKTsKKworCWVyciA9IGludGVsX2NvbnRleHRfcGluX3d3KGNlLCB3dyk7CisJaWYg KGVycikKKwkJZ290byBvdXQ7CisKKwllcnIgPSBpbnRlbF9jb250ZXh0X21pZ3JhdGVfY29weShj ZSwgYXdhaXQsCisJCQkJCSBzcmMsIHNyY19jYWNoZV9sZXZlbCwgc3JjX2lzX2xtZW0sCisJCQkJ CSBkc3QsIGRzdF9jYWNoZV9sZXZlbCwgZHN0X2lzX2xtZW0sCisJCQkJCSBvdXQpOworCisJaW50 ZWxfY29udGV4dF91bnBpbihjZSk7CitvdXQ6CisJaW50ZWxfY29udGV4dF9wdXQoY2UpOworCXJl dHVybiBlcnI7Cit9CisKK3ZvaWQgaW50ZWxfbWlncmF0ZV9maW5pKHN0cnVjdCBpbnRlbF9taWdy YXRlICptKQoreworCXN0cnVjdCBpbnRlbF9jb250ZXh0ICpjZTsKKworCWNlID0gZmV0Y2hfYW5k X3plcm8oJm0tPmNvbnRleHQpOworCWlmICghY2UpCisJCXJldHVybjsKKworCWludGVsX2NvbnRl eHRfdW5waW4oY2UpOworCWludGVsX2NvbnRleHRfcHV0KGNlKTsKK30KKworI2lmIElTX0VOQUJM RUQoQ09ORklHX0RSTV9JOTE1X1NFTEZURVNUKQorI2luY2x1ZGUgInNlbGZ0ZXN0X21pZ3JhdGUu YyIKKyNlbmRpZgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWln cmF0ZS5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5oCm5ldyBmaWxl IG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwMC4uMzJjNjExOTBlZDczCi0tLSAvZGV2L251 bGwKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50ZWxfbWlncmF0ZS5oCkBAIC0wLDAg KzEsNDUgQEAKKy8qIFNQRFgtTGljZW5zZS1JZGVudGlmaWVyOiBNSVQgKi8KKy8qCisgKiBDb3B5 cmlnaHQgwqkgMjAyMCBJbnRlbCBDb3Jwb3JhdGlvbgorICovCisKKyNpZm5kZWYgX19JTlRFTF9N SUdSQVRFX18KKyNkZWZpbmUgX19JTlRFTF9NSUdSQVRFX18KKworI2luY2x1ZGUgImludGVsX21p Z3JhdGVfdHlwZXMuaCIKKworc3RydWN0IGRtYV9mZW5jZTsKK3N0cnVjdCBpOTE1X3JlcXVlc3Q7 CitzdHJ1Y3QgaTkxNV9nZW1fd3dfY3R4Oworc3RydWN0IGludGVsX2d0Oworc3RydWN0IHNjYXR0 ZXJsaXN0OworZW51bSBpOTE1X2NhY2hlX2xldmVsOworCitpbnQgaW50ZWxfbWlncmF0ZV9pbml0 KHN0cnVjdCBpbnRlbF9taWdyYXRlICptLCBzdHJ1Y3QgaW50ZWxfZ3QgKmd0KTsKKworc3RydWN0 IGludGVsX2NvbnRleHQgKmludGVsX21pZ3JhdGVfY3JlYXRlX2NvbnRleHQoc3RydWN0IGludGVs X21pZ3JhdGUgKm0pOworCitpbnQgaW50ZWxfbWlncmF0ZV9jb3B5KHN0cnVjdCBpbnRlbF9taWdy YXRlICptLAorCQkgICAgICAgc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCSAgICAgICBz dHJ1Y3QgZG1hX2ZlbmNlICphd2FpdCwKKwkJICAgICAgIHN0cnVjdCBzY2F0dGVybGlzdCAqc3Jj LAorCQkgICAgICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIHNyY19jYWNoZV9sZXZlbCwKKwkJICAg ICAgIGJvb2wgc3JjX2lzX2xtZW0sCisJCSAgICAgICBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKmRzdCwK KwkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBkc3RfY2FjaGVfbGV2ZWwsCisJCSAgICAg ICBib29sIGRzdF9pc19sbWVtLAorCQkgICAgICAgc3RydWN0IGk5MTVfcmVxdWVzdCAqKm91dCk7 CisKK2ludCBpbnRlbF9jb250ZXh0X21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfY29udGV4dCAq Y2UsCisJCQkgICAgICAgc3RydWN0IGRtYV9mZW5jZSAqYXdhaXQsCisJCQkgICAgICAgc3RydWN0 IHNjYXR0ZXJsaXN0ICpzcmMsCisJCQkgICAgICAgZW51bSBpOTE1X2NhY2hlX2xldmVsIHNyY19j YWNoZV9sZXZlbCwKKwkJCSAgICAgICBib29sIHNyY19pc19sbWVtLAorCQkJICAgICAgIHN0cnVj dCBzY2F0dGVybGlzdCAqZHN0LAorCQkJICAgICAgIGVudW0gaTkxNV9jYWNoZV9sZXZlbCBkc3Rf Y2FjaGVfbGV2ZWwsCisJCQkgICAgICAgYm9vbCBkc3RfaXNfbG1lbSwKKwkJCSAgICAgICBzdHJ1 Y3QgaTkxNV9yZXF1ZXN0ICoqb3V0KTsKKwordm9pZCBpbnRlbF9taWdyYXRlX2Zpbmkoc3RydWN0 IGludGVsX21pZ3JhdGUgKm0pOworCisjZW5kaWYgLyogX19JTlRFTF9NSUdSQVRFX18gKi8KZGlm ZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaCBi L2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaApuZXcgZmlsZSBt b2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLmQ5ODIzMDU5N2Y0MgotLS0gL2Rldi9udWxs CisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX21pZ3JhdGVfdHlwZXMuaApAQCAt MCwwICsxLDE1IEBACisvKiBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUICovCisvKgorICog Q29weXJpZ2h0IMKpIDIwMjAgSW50ZWwgQ29ycG9yYXRpb24KKyAqLworCisjaWZuZGVmIF9fSU5U RUxfTUlHUkFURV9UWVBFU19fCisjZGVmaW5lIF9fSU5URUxfTUlHUkFURV9UWVBFU19fCisKK3N0 cnVjdCBpbnRlbF9jb250ZXh0OworCitzdHJ1Y3QgaW50ZWxfbWlncmF0ZSB7CisJc3RydWN0IGlu dGVsX2NvbnRleHQgKmNvbnRleHQ7Cit9OworCisjZW5kaWYgLyogX19JTlRFTF9NSUdSQVRFX1RZ UEVTX18gKi8KZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3Jpbmcu aCBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3JpbmcuaAppbmRleCBkYmY1ZjE0YTEz NmYuLjFiMzJkYWRmYjhjMyAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ3QvaW50 ZWxfcmluZy5oCisrKyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2d0L2ludGVsX3JpbmcuaApAQCAt NDksNiArNDksNyBAQCBzdGF0aWMgaW5saW5lIHZvaWQgaW50ZWxfcmluZ19hZHZhbmNlKHN0cnVj dCBpOTE1X3JlcXVlc3QgKnJxLCB1MzIgKmNzKQogCSAqIGludGVsX3JpbmdfYmVnaW4oKSkuCiAJ ICovCiAJR0VNX0JVR19PTigocnEtPnJpbmctPnZhZGRyICsgcnEtPnJpbmctPmVtaXQpICE9IGNz KTsKKwlHRU1fQlVHX09OKCFJU19BTElHTkVEKHJxLT5yaW5nLT5lbWl0LCA4KSk7IC8qIFJJTkdf VEFJTCBxd29yZCBhbGlnbiAqLwogfQogCiBzdGF0aWMgaW5saW5lIHUzMiBpbnRlbF9yaW5nX3dy YXAoY29uc3Qgc3RydWN0IGludGVsX3JpbmcgKnJpbmcsIHUzMiBwb3MpCmRpZmYgLS1naXQgYS9k cml2ZXJzL2dwdS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMgYi9kcml2ZXJzL2dwdS9k cm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXgg MDAwMDAwMDAwMDAwLi45Nzg0ZDE0OWViZjEKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL2dw dS9kcm0vaTkxNS9ndC9zZWxmdGVzdF9taWdyYXRlLmMKQEAgLTAsMCArMSwyOTEgQEAKKy8vIFNQ RFgtTGljZW5zZS1JZGVudGlmaWVyOiBNSVQKKy8qCisgKiBDb3B5cmlnaHQgwqkgMjAyMCBJbnRl bCBDb3Jwb3JhdGlvbgorICovCisKKyNpbmNsdWRlICJzZWxmdGVzdHMvaTkxNV9yYW5kb20uaCIK Kworc3RhdGljIGNvbnN0IHVuc2lnbmVkIGludCBzaXplc1tdID0geworCVNaXzRLLAorCVNaXzY0 SywKKwlTWl8yTSwKKwlDSFVOS19TWiAtIFNaXzRLLAorCUNIVU5LX1NaLAorCUNIVU5LX1NaICsg U1pfNEssCisJU1pfNjRNLAorfTsKKworc3RhdGljIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0 ICoKK2NyZWF0ZV9sbWVtX29yX2ludGVybmFsKHN0cnVjdCBkcm1faTkxNV9wcml2YXRlICppOTE1 LCBzaXplX3Qgc2l6ZSkKK3sKKwlpZiAoSEFTX0xNRU0oaTkxNSkpIHsKKwkJc3RydWN0IGRybV9p OTE1X2dlbV9vYmplY3QgKm9iajsKKworCQlvYmogPSBpOTE1X2dlbV9vYmplY3RfY3JlYXRlX2xt ZW0oaTkxNSwgc2l6ZSwgMCk7CisJCWlmICghSVNfRVJSKG9iaikpCisJCQlyZXR1cm4gb2JqOwor CX0KKworCXJldHVybiBpOTE1X2dlbV9vYmplY3RfY3JlYXRlX2ludGVybmFsKGk5MTUsIHNpemUp OworfQorCitzdGF0aWMgaW50IGNvcHkoc3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGUsCisJ CWludCAoKmZuKShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwKKwkJCSAgc3RydWN0IGk5 MTVfZ2VtX3d3X2N0eCAqd3csCisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpzcmMs CisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpkc3QsCisJCQkgIHN0cnVjdCBpOTE1 X3JlcXVlc3QgKipvdXQpLAorCQl1MzIgc3osIHN0cnVjdCBybmRfc3RhdGUgKnBybmcpCit7CisJ c3RydWN0IGRybV9pOTE1X3ByaXZhdGUgKmk5MTUgPSBtaWdyYXRlLT5jb250ZXh0LT5lbmdpbmUt Pmk5MTU7CisJc3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKnNyYywgKmRzdDsKKwlzdHJ1Y3Qg aTkxNV9yZXF1ZXN0ICpycTsKKwlzdHJ1Y3QgaTkxNV9nZW1fd3dfY3R4IHd3OworCXUzMiAqdmFk ZHI7CisJaW50IGVyciA9IDA7CisJaW50IGk7CisKKwlzcmMgPSBjcmVhdGVfbG1lbV9vcl9pbnRl cm5hbChpOTE1LCBzeik7CisJaWYgKElTX0VSUihzcmMpKQorCQlyZXR1cm4gMDsKKworCWRzdCA9 IGk5MTVfZ2VtX29iamVjdF9jcmVhdGVfaW50ZXJuYWwoaTkxNSwgc3opOworCWlmIChJU19FUlIo ZHN0KSkKKwkJZ290byBlcnJfZnJlZV9zcmM7CisKKwlmb3JfaTkxNV9nZW1fd3coJnd3LCBlcnIs IHRydWUpIHsKKwkJZXJyID0gaTkxNV9nZW1fb2JqZWN0X2xvY2soc3JjLCAmd3cpOworCQlpZiAo ZXJyKQorCQkJY29udGludWU7CisKKwkJZXJyID0gaTkxNV9nZW1fb2JqZWN0X2xvY2soZHN0LCAm d3cpOworCQlpZiAoZXJyKQorCQkJY29udGludWU7CisKKwkJdmFkZHIgPSBpOTE1X2dlbV9vYmpl Y3RfcGluX21hcChzcmMsIEk5MTVfTUFQX1dDKTsKKwkJaWYgKElTX0VSUih2YWRkcikpIHsKKwkJ CWVyciA9IFBUUl9FUlIodmFkZHIpOworCQkJY29udGludWU7CisJCX0KKworCQlmb3IgKGkgPSAw OyBpIDwgc3ogLyBzaXplb2YodTMyKTsgaSsrKQorCQkJdmFkZHJbaV0gPSBpOworCQlpOTE1X2dl bV9vYmplY3RfZmx1c2hfbWFwKHNyYyk7CisKKwkJdmFkZHIgPSBpOTE1X2dlbV9vYmplY3RfcGlu X21hcChkc3QsIEk5MTVfTUFQX1dDKTsKKwkJaWYgKElTX0VSUih2YWRkcikpIHsKKwkJCWVyciA9 IFBUUl9FUlIodmFkZHIpOworCQkJZ290byB1bnBpbl9zcmM7CisJCX0KKworCQlmb3IgKGkgPSAw OyBpIDwgc3ogLyBzaXplb2YodTMyKTsgaSsrKQorCQkJdmFkZHJbaV0gPSB+aTsKKwkJaTkxNV9n ZW1fb2JqZWN0X2ZsdXNoX21hcChkc3QpOworCisJCWVyciA9IGZuKG1pZ3JhdGUsICZ3dywgc3Jj LCBkc3QsICZycSk7CisJCWlmICghZXJyKQorCQkJY29udGludWU7CisKKwkJaWYgKGVyciAhPSAt RURFQURMSyAmJiBlcnIgIT0gLUVJTlRSICYmIGVyciAhPSAtRVJFU1RBUlRTWVMpCisJCQlwcl9l cnIoIiVwcyBmYWlsZWQsIHNpemU6ICV1XG4iLCBmbiwgc3opOworCQlpZiAocnEpIHsKKwkJCWk5 MTVfcmVxdWVzdF93YWl0KHJxLCAwLCBIWik7CisJCQlpOTE1X3JlcXVlc3RfcHV0KHJxKTsKKwkJ fQorCQlpOTE1X2dlbV9vYmplY3RfdW5waW5fbWFwKGRzdCk7Cit1bnBpbl9zcmM6CisJCWk5MTVf Z2VtX29iamVjdF91bnBpbl9tYXAoc3JjKTsKKwl9CisJaWYgKGVycikKKwkJZ290byBlcnJfb3V0 OworCisJaWYgKHJxKSB7CisJCWlmIChpOTE1X3JlcXVlc3Rfd2FpdChycSwgMCwgSFopIDwgMCkg eworCQkJcHJfZXJyKCIlcHMgdGltZWQgb3V0LCBzaXplOiAldVxuIiwgZm4sIHN6KTsKKwkJCWVy ciA9IC1FVElNRTsKKwkJfQorCQlpOTE1X3JlcXVlc3RfcHV0KHJxKTsKKwl9CisKKwlmb3IgKGkg PSAwOyAhZXJyICYmIGkgPCBzeiAvIFBBR0VfU0laRTsgaSsrKSB7CisJCWludCB4ID0gaSAqIDEw MjQgKyBpOTE1X3ByYW5kb21fdTMyX21heF9zdGF0ZSgxMDI0LCBwcm5nKTsKKworCQlpZiAodmFk ZHJbeF0gIT0geCkgeworCQkJcHJfZXJyKCIlcHMgZmFpbGVkLCBzaXplOiAldSwgb2Zmc2V0OiAl enVcbiIsCisJCQkgICAgICAgZm4sIHN6LCB4ICogc2l6ZW9mKHUzMikpOworCQkJaWd0X2hleGR1 bXAodmFkZHIgKyBpICogMTAyNCwgNDA5Nik7CisJCQllcnIgPSAtRUlOVkFMOworCQl9CisJfQor CisJaTkxNV9nZW1fb2JqZWN0X3VucGluX21hcChkc3QpOworCWk5MTVfZ2VtX29iamVjdF91bnBp bl9tYXAoc3JjKTsKKworZXJyX291dDoKKwlpOTE1X2dlbV9vYmplY3RfcHV0KGRzdCk7CitlcnJf ZnJlZV9zcmM6CisJaTkxNV9nZW1fb2JqZWN0X3B1dChzcmMpOworCisJcmV0dXJuIGVycjsKK30K Kworc3RhdGljIGludCBfX21pZ3JhdGVfY29weShzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0 ZSwKKwkJCSAgc3RydWN0IGk5MTVfZ2VtX3d3X2N0eCAqd3csCisJCQkgIHN0cnVjdCBkcm1faTkx NV9nZW1fb2JqZWN0ICpzcmMsCisJCQkgIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpkc3Qs CisJCQkgIHN0cnVjdCBpOTE1X3JlcXVlc3QgKipvdXQpCit7CisJcmV0dXJuIGludGVsX21pZ3Jh dGVfY29weShtaWdyYXRlLCB3dywgTlVMTCwKKwkJCQkgIHNyYy0+bW0ucGFnZXMtPnNnbCwgc3Jj LT5jYWNoZV9sZXZlbCwKKwkJCQkgIGk5MTVfZ2VtX29iamVjdF9pc19sbWVtKHNyYyksCisJCQkJ ICBkc3QtPm1tLnBhZ2VzLT5zZ2wsIGRzdC0+Y2FjaGVfbGV2ZWwsCisJCQkJICBpOTE1X2dlbV9v YmplY3RfaXNfbG1lbShkc3QpLAorCQkJCSAgb3V0KTsKK30KKworc3RhdGljIGludCBfX2dsb2Jh bF9jb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlLAorCQkJIHN0cnVjdCBpOTE1X2dl bV93d19jdHggKnd3LAorCQkJIHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpzcmMsCisJCQkg c3RydWN0IGRybV9pOTE1X2dlbV9vYmplY3QgKmRzdCwKKwkJCSBzdHJ1Y3QgaTkxNV9yZXF1ZXN0 ICoqb3V0KQoreworCXJldHVybiBpbnRlbF9jb250ZXh0X21pZ3JhdGVfY29weShtaWdyYXRlLT5j b250ZXh0LCBOVUxMLAorCQkJCQkgIHNyYy0+bW0ucGFnZXMtPnNnbCwgc3JjLT5jYWNoZV9sZXZl bCwKKwkJCQkJICBpOTE1X2dlbV9vYmplY3RfaXNfbG1lbShzcmMpLAorCQkJCQkgIGRzdC0+bW0u cGFnZXMtPnNnbCwgZHN0LT5jYWNoZV9sZXZlbCwKKwkJCQkJICBpOTE1X2dlbV9vYmplY3RfaXNf bG1lbShkc3QpLAorCQkJCQkgIG91dCk7Cit9CisKK3N0YXRpYyBpbnQKK21pZ3JhdGVfY29weShz dHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSwgdTMyIHN6LCBzdHJ1Y3Qgcm5kX3N0YXRlICpw cm5nKQoreworCXJldHVybiBjb3B5KG1pZ3JhdGUsIF9fbWlncmF0ZV9jb3B5LCBzeiwgcHJuZyk7 Cit9CisKK3N0YXRpYyBpbnQKK2dsb2JhbF9jb3B5KHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdy YXRlLCB1MzIgc3osIHN0cnVjdCBybmRfc3RhdGUgKnBybmcpCit7CisJcmV0dXJuIGNvcHkobWln cmF0ZSwgX19nbG9iYWxfY29weSwgc3osIHBybmcpOworfQorCitzdGF0aWMgaW50IGxpdmVfbWln cmF0ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgaW50ZWxfbWlncmF0ZSAqbWlncmF0ZSA9 IGFyZzsKKwlzdHJ1Y3QgZHJtX2k5MTVfcHJpdmF0ZSAqaTkxNSA9IG1pZ3JhdGUtPmNvbnRleHQt PmVuZ2luZS0+aTkxNTsKKwlJOTE1X1JORF9TVEFURShwcm5nKTsKKwlpbnQgaTsKKworCWZvciAo aSA9IDA7IGkgPCBBUlJBWV9TSVpFKHNpemVzKTsgaSsrKSB7CisJCWludCBlcnI7CisKKwkJZXJy ID0gbWlncmF0ZV9jb3B5KG1pZ3JhdGUsIHNpemVzW2ldLCAmcHJuZyk7CisJCWlmIChlcnIgPT0g MCkKKwkJCWVyciA9IGdsb2JhbF9jb3B5KG1pZ3JhdGUsIHNpemVzW2ldLCAmcHJuZyk7CisJCWk5 MTVfZ2VtX2RyYWluX2ZyZWVkX29iamVjdHMoaTkxNSk7CisJCWlmIChlcnIpCisJCQlyZXR1cm4g ZXJyOworCX0KKworCXJldHVybiAwOworfQorCitzdHJ1Y3QgdGhyZWFkZWRfbWlncmF0ZSB7CisJ c3RydWN0IGludGVsX21pZ3JhdGUgKm1pZ3JhdGU7CisJc3RydWN0IHRhc2tfc3RydWN0ICp0c2s7 CisJc3RydWN0IHJuZF9zdGF0ZSBwcm5nOworfTsKKworc3RhdGljIGludCB0aHJlYWRlZF9taWdy YXRlKHN0cnVjdCBpbnRlbF9taWdyYXRlICptaWdyYXRlLAorCQkJICAgIGludCAoKmZuKSh2b2lk ICphcmcpLAorCQkJICAgIHVuc2lnbmVkIGludCBmbGFncykKK3sKKwljb25zdCB1bnNpZ25lZCBp bnQgbl9jcHVzID0gbnVtX29ubGluZV9jcHVzKCkgKyAxOworCXN0cnVjdCB0aHJlYWRlZF9taWdy YXRlICp0aHJlYWQ7CisJSTkxNV9STkRfU1RBVEUocHJuZyk7CisJdW5zaWduZWQgaW50IGk7CisJ aW50IGVyciA9IDA7CisKKwl0aHJlYWQgPSBrY2FsbG9jKG5fY3B1cywgc2l6ZW9mKCp0aHJlYWQp LCBHRlBfS0VSTkVMKTsKKwlpZiAoIXRocmVhZCkKKwkJcmV0dXJuIDA7CisKKwlmb3IgKGkgPSAw OyBpIDwgbl9jcHVzOyArK2kpIHsKKwkJc3RydWN0IHRhc2tfc3RydWN0ICp0c2s7CisKKwkJdGhy ZWFkW2ldLm1pZ3JhdGUgPSBtaWdyYXRlOworCQl0aHJlYWRbaV0ucHJuZyA9CisJCQlJOTE1X1JO RF9TVEFURV9JTklUSUFMSVpFUihwcmFuZG9tX3UzMl9zdGF0ZSgmcHJuZykpOworCisJCXRzayA9 IGt0aHJlYWRfcnVuKGZuLCAmdGhyZWFkW2ldLCAiaWd0LSVkIiwgaSk7CisJCWlmIChJU19FUlIo dHNrKSkgeworCQkJZXJyID0gUFRSX0VSUih0c2spOworCQkJYnJlYWs7CisJCX0KKworCQlnZXRf dGFza19zdHJ1Y3QodHNrKTsKKwkJdGhyZWFkW2ldLnRzayA9IHRzazsKKwl9CisKKwltc2xlZXAo MTApOyAvKiBzdGFydCBhbGwgdGhyZWFkcyBiZWZvcmUgd2Uga3RocmVhZF9zdG9wKCkgKi8KKwor CWZvciAoaSA9IDA7IGkgPCBuX2NwdXM7ICsraSkgeworCQlzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRz ayA9IHRocmVhZFtpXS50c2s7CisJCWludCBzdGF0dXM7CisKKwkJaWYgKElTX0VSUl9PUl9OVUxM KHRzaykpCisJCQljb250aW51ZTsKKworCQlzdGF0dXMgPSBrdGhyZWFkX3N0b3AodHNrKTsKKwkJ aWYgKHN0YXR1cyAmJiAhZXJyKQorCQkJZXJyID0gc3RhdHVzOworCisJCXB1dF90YXNrX3N0cnVj dCh0c2spOworCX0KKworCWtmcmVlKHRocmVhZCk7CisJcmV0dXJuIGVycjsKK30KKworc3RhdGlj IGludCBfX3RocmVhZF9taWdyYXRlX2NvcHkodm9pZCAqYXJnKQoreworCXN0cnVjdCB0aHJlYWRl ZF9taWdyYXRlICp0bSA9IGFyZzsKKworCXJldHVybiBtaWdyYXRlX2NvcHkodG0tPm1pZ3JhdGUs IDIgKiBDSFVOS19TWiwgJnRtLT5wcm5nKTsKK30KKworc3RhdGljIGludCB0aHJlYWRfbWlncmF0 ZV9jb3B5KHZvaWQgKmFyZykKK3sKKwlyZXR1cm4gdGhyZWFkZWRfbWlncmF0ZShhcmcsIF9fdGhy ZWFkX21pZ3JhdGVfY29weSwgMCk7Cit9CisKK3N0YXRpYyBpbnQgX190aHJlYWRfZ2xvYmFsX2Nv cHkodm9pZCAqYXJnKQoreworCXN0cnVjdCB0aHJlYWRlZF9taWdyYXRlICp0bSA9IGFyZzsKKwor CXJldHVybiBnbG9iYWxfY29weSh0bS0+bWlncmF0ZSwgMiAqIENIVU5LX1NaLCAmdG0tPnBybmcp OworfQorCitzdGF0aWMgaW50IHRocmVhZF9nbG9iYWxfY29weSh2b2lkICphcmcpCit7CisJcmV0 dXJuIHRocmVhZGVkX21pZ3JhdGUoYXJnLCBfX3RocmVhZF9nbG9iYWxfY29weSwgMCk7Cit9CisK K2ludCBpbnRlbF9taWdyYXRlX2xpdmVfc2VsZnRlc3RzKHN0cnVjdCBkcm1faTkxNV9wcml2YXRl ICppOTE1KQoreworCXN0YXRpYyBjb25zdCBzdHJ1Y3QgaTkxNV9zdWJ0ZXN0IHRlc3RzW10gPSB7 CisJCVNVQlRFU1QobGl2ZV9taWdyYXRlX2NvcHkpLAorCQlTVUJURVNUKHRocmVhZF9taWdyYXRl X2NvcHkpLAorCQlTVUJURVNUKHRocmVhZF9nbG9iYWxfY29weSksCisJfTsKKwlzdHJ1Y3QgaW50 ZWxfbWlncmF0ZSBtOworCWludCBlcnI7CisKKwlpZiAoaW50ZWxfbWlncmF0ZV9pbml0KCZtLCAm aTkxNS0+Z3QpKQorCQlyZXR1cm4gMDsKKworCWVyciA9IGk5MTVfc3VidGVzdHModGVzdHMsICZt KTsKKwlpbnRlbF9taWdyYXRlX2ZpbmkoJm0pOworCisJcmV0dXJuIGVycjsKK30KZGlmZiAtLWdp dCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRlc3RzLmgg Yi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oCmlu ZGV4IGE5MmMwZTliN2U2Yi4uYmU1ZTAxOTFlYWVhIDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9k cm0vaTkxNS9zZWxmdGVzdHMvaTkxNV9saXZlX3NlbGZ0ZXN0cy5oCisrKyBiL2RyaXZlcnMvZ3B1 L2RybS9pOTE1L3NlbGZ0ZXN0cy9pOTE1X2xpdmVfc2VsZnRlc3RzLmgKQEAgLTI2LDYgKzI2LDcg QEAgc2VsZnRlc3QoZ3RfbW9jcywgaW50ZWxfbW9jc19saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0 KGd0X3BtLCBpbnRlbF9ndF9wbV9saXZlX3NlbGZ0ZXN0cykKIHNlbGZ0ZXN0KGd0X2hlYXJ0YmVh dCwgaW50ZWxfaGVhcnRiZWF0X2xpdmVfc2VsZnRlc3RzKQogc2VsZnRlc3QocmVxdWVzdHMsIGk5 MTVfcmVxdWVzdF9saXZlX3NlbGZ0ZXN0cykKK3NlbGZ0ZXN0KG1pZ3JhdGUsIGludGVsX21pZ3Jh dGVfbGl2ZV9zZWxmdGVzdHMpCiBzZWxmdGVzdChhY3RpdmUsIGk5MTVfYWN0aXZlX2xpdmVfc2Vs ZnRlc3RzKQogc2VsZnRlc3Qob2JqZWN0cywgaTkxNV9nZW1fb2JqZWN0X2xpdmVfc2VsZnRlc3Rz KQogc2VsZnRlc3QobW1hbiwgaTkxNV9nZW1fbW1hbl9saXZlX3NlbGZ0ZXN0cykKLS0gCjIuMzEu MQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwt Z2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8v bGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D41FEC48BE6 for ; Tue, 8 Jun 2021 09:29:24 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AAF1C6127A for ; Tue, 8 Jun 2021 09:29:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAF1C6127A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 10BBA6EB71; Tue, 8 Jun 2021 09:29:20 +0000 (UTC) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by gabe.freedesktop.org (Postfix) with ESMTPS id 20F246EB71; Tue, 8 Jun 2021 09:29:17 +0000 (UTC) IronPort-SDR: ygWIbymT9awPFdITUiAw+JzMpRGXMA9AxIsSEk4pfUJln54egLCH14LPsVK4eYlJ/do7enZ01f Wyv/bSY39p9Q== X-IronPort-AV: E=McAfee;i="6200,9189,10008"; a="226150968" X-IronPort-AV: E=Sophos;i="5.83,257,1616482800"; d="scan'208";a="226150968" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 02:29:16 -0700 IronPort-SDR: FUVfUZBy/YMyQAWhfRn+6Xz06MrO4O5kwIPuJU4esCJCrsBdf+boFhDGPU8qT/tCcaRWZTYj1I wgn1D1oZ9LWw== X-IronPort-AV: E=Sophos;i="5.83,257,1616482800"; d="scan'208";a="551544449" Received: from mkayyal-mobl.ger.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.115]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2021 02:29:14 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 7/9] drm/i915/gt: Pipelined page migration Date: Tue, 8 Jun 2021 11:28:44 +0200 Message-Id: <20210608092846.64198-8-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210608092846.64198-1-thomas.hellstrom@linux.intel.com> References: <20210608092846.64198-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , matthew.auld@intel.com, Chris Wilson Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Chris Wilson If we pipeline the PTE updates and then do the copy of those pages within a single unpreemptible command packet, we can submit the copies and leave them to be scheduled without having to synchronously wait under a global lock. In order to manage migration, we need to preallocate the page tables (and keep them pinned and available for use at any time), causing a bottleneck for migrations as all clients must contend on the limited resources. By inlining the ppGTT updates and performing the blit atomically, each client only owns the PTE while in use, and so we can reschedule individual operations however we see fit. And most importantly, we do not need to take a global lock on the shared vm, and wait until the operation is complete before releasing the lock for others to claim the PTE for themselves. Signed-off-by: Chris Wilson Co-developed-by: Thomas Hellström Signed-off-by: Thomas Hellström --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_gpu_commands.h | 2 + drivers/gpu/drm/i915/gt/intel_migrate.c | 543 ++++++++++++++++++ drivers/gpu/drm/i915/gt/intel_migrate.h | 45 ++ drivers/gpu/drm/i915/gt/intel_migrate_types.h | 15 + drivers/gpu/drm/i915/gt/intel_ring.h | 1 + drivers/gpu/drm/i915/gt/selftest_migrate.c | 291 ++++++++++ .../drm/i915/selftests/i915_live_selftests.h | 1 + 9 files changed, 900 insertions(+) create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.c create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate.h create mode 100644 drivers/gpu/drm/i915/gt/intel_migrate_types.h create mode 100644 drivers/gpu/drm/i915/gt/selftest_migrate.c diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index ea8ee4b3e018..9f18902be626 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -109,6 +109,7 @@ gt-y += \ gt/intel_gtt.o \ gt/intel_llc.o \ gt/intel_lrc.o \ + gt/intel_migrate.o \ gt/intel_mocs.o \ gt/intel_ppgtt.o \ gt/intel_rc6.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 0862c42b4cac..949965680c37 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -188,6 +188,7 @@ intel_write_status_page(struct intel_engine_cs *engine, int reg, u32 value) #define I915_GEM_HWS_PREEMPT_ADDR (I915_GEM_HWS_PREEMPT * sizeof(u32)) #define I915_GEM_HWS_SEQNO 0x40 #define I915_GEM_HWS_SEQNO_ADDR (I915_GEM_HWS_SEQNO * sizeof(u32)) +#define I915_GEM_HWS_MIGRATE (0x42 * sizeof(u32)) #define I915_GEM_HWS_SCRATCH 0x80 #define I915_HWS_CSB_BUF0_INDEX 0x10 diff --git a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h index 2694dbb9967e..1c3af0fc0456 100644 --- a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h +++ b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h @@ -123,8 +123,10 @@ #define MI_SEMAPHORE_SAD_NEQ_SDD (5 << 12) #define MI_SEMAPHORE_TOKEN_MASK REG_GENMASK(9, 5) #define MI_SEMAPHORE_TOKEN_SHIFT 5 +#define MI_STORE_DATA_IMM MI_INSTR(0x20, 0) #define MI_STORE_DWORD_IMM MI_INSTR(0x20, 1) #define MI_STORE_DWORD_IMM_GEN4 MI_INSTR(0x20, 2) +#define MI_STORE_QWORD_IMM_GEN8 (MI_INSTR(0x20, 3) | REG_BIT(21)) #define MI_MEM_VIRTUAL (1 << 22) /* 945,g33,965 */ #define MI_USE_GGTT (1 << 22) /* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1) diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.c b/drivers/gpu/drm/i915/gt/intel_migrate.c new file mode 100644 index 000000000000..1f60f8ee36f8 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.c @@ -0,0 +1,543 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "i915_drv.h" +#include "intel_context.h" +#include "intel_gpu_commands.h" +#include "intel_gt.h" +#include "intel_gtt.h" +#include "intel_migrate.h" +#include "intel_ring.h" + +struct insert_pte_data { + u64 offset; + bool is_lmem; +}; + +#define CHUNK_SZ SZ_8M /* ~1ms at 8GiB/s preemption delay */ + +static bool engine_supports_migration(struct intel_engine_cs *engine) +{ + if (!engine) + return false; + + /* + * We need the ability to prevent aribtration (MI_ARB_ON_OFF), + * the ability to write PTE using inline data (MI_STORE_DATA) + * and of course the ability to do the block transfer (blits). + */ + GEM_BUG_ON(engine->class != COPY_ENGINE_CLASS); + + return true; +} + +static void insert_pte(struct i915_address_space *vm, + struct i915_page_table *pt, + void *data) +{ + struct insert_pte_data *d = data; + + vm->insert_page(vm, px_dma(pt), d->offset, I915_CACHE_NONE, + d->is_lmem ? PTE_LM : 0); + d->offset += PAGE_SIZE; +} + +static struct i915_address_space *migrate_vm(struct intel_gt *gt) +{ + struct i915_vm_pt_stash stash = {}; + struct i915_ppgtt *vm; + int err; + int i; + + /* + * We construct a very special VM for use by all migration contexts, + * it is kept pinned so that it can be used at any time. As we need + * to pre-allocate the page directories for the migration VM, this + * limits us to only using a small number of prepared vma. + * + * To be able to pipeline and reschedule migration operations while + * avoiding unnecessary contention on the vm itself, the PTE updates + * are inline with the blits. All the blits use the same fixed + * addresses, with the backing store redirection being updated on the + * fly. Only 2 implicit vma are used for all migration operations. + * + * We lay the ppGTT out as: + * + * [0, CHUNK_SZ) -> first object + * [CHUNK_SZ, 2 * CHUNK_SZ) -> second object + * [2 * CHUNK_SZ, 2 * CHUNK_SZ + 2 * CHUNK_SZ >> 9] -> PTE + * + * By exposing the dma addresses of the page directories themselves + * within the ppGTT, we are then able to rewrite the PTE prior to use. + * But the PTE update and subsequent migration operation must be atomic, + * i.e. within the same non-preemptible window so that we do not switch + * to another migration context that overwrites the PTE. + */ + + vm = i915_ppgtt_create(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + if (!vm->vm.allocate_va_range || !vm->vm.foreach) { + err = -ENODEV; + goto err_vm; + } + + /* + * Each engine instance is assigned its own chunk in the VM, so + * that we can run multiple instances concurrently + */ + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + struct intel_engine_cs *engine; + u64 base = (u64)i << 32; + struct insert_pte_data d = {}; + struct i915_gem_ww_ctx ww; + u64 sz; + + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (!engine_supports_migration(engine)) + continue; + + /* + * We copy in 8MiB chunks. Each PDE covers 2MiB, so we need + * 4x2 page directories for source/destination. + */ + sz = 2 * CHUNK_SZ; + d.offset = base + sz; + + /* + * We need another page directory setup so that we can write + * the 8x512 PTE in each chunk. + */ + sz += (sz >> 12) * sizeof(u64); + + err = i915_vm_alloc_pt_stash(&vm->vm, &stash, sz); + if (err) + goto err_vm; + + for_i915_gem_ww(&ww, err, true) { + err = i915_vm_lock_objects(&vm->vm, &ww); + if (err) + continue; + err = i915_vm_map_pt_stash(&vm->vm, &stash); + if (err) + continue; + + vm->vm.allocate_va_range(&vm->vm, &stash, base, base + sz); + } + i915_vm_free_pt_stash(&vm->vm, &stash); + if (err) + goto err_vm; + + /* Now allow the GPU to rewrite the PTE via its own ppGTT */ + d.is_lmem = i915_gem_object_is_lmem(vm->vm.scratch[0]); + vm->vm.foreach(&vm->vm, base, base + sz, insert_pte, &d); + } + + return &vm->vm; + +err_vm: + i915_vm_put(&vm->vm); + return ERR_PTR(err); +} + +static struct intel_engine_cs *first_copy_engine(struct intel_gt *gt) +{ + struct intel_engine_cs *engine; + int i; + + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + return engine; + } + + return NULL; +} + +static struct intel_context *pinned_context(struct intel_gt *gt) +{ + static struct lock_class_key key; + struct intel_engine_cs *engine; + struct i915_address_space *vm; + struct intel_context *ce; + + engine = first_copy_engine(gt); + if (!engine) + return ERR_PTR(-ENODEV); + + vm = migrate_vm(gt); + if (IS_ERR(vm)) + return ERR_CAST(vm); + + ce = intel_engine_create_pinned_context(engine, vm, SZ_512K, + I915_GEM_HWS_MIGRATE, + &key, "migrate"); + i915_vm_put(ce->vm); + return ce; +} + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt) +{ + struct intel_context *ce; + + memset(m, 0, sizeof(*m)); + + ce = pinned_context(gt); + if (IS_ERR(ce)) + return PTR_ERR(ce); + + m->context = ce; + return 0; +} + +static int random_index(unsigned int max) +{ + return upper_32_bits(mul_u32_u32(get_random_u32(), max)); +} + +static struct intel_context *__migrate_engines(struct intel_gt *gt) +{ + struct intel_engine_cs *engines[MAX_ENGINE_INSTANCE]; + struct intel_engine_cs *engine; + unsigned int count, i; + + count = 0; + for (i = 0; i < ARRAY_SIZE(gt->engine_class[COPY_ENGINE_CLASS]); i++) { + engine = gt->engine_class[COPY_ENGINE_CLASS][i]; + if (engine_supports_migration(engine)) + engines[count++] = engine; + } + + return intel_context_create(engines[random_index(count)]); +} + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m) +{ + struct intel_context *ce; + + /* + * We randomly distribute contexts across the engines upon constrction, + * as they all share the same pinned vm, and so in order to allow + * multiple blits to run in parallel, we must construct each blit + * to use a different range of the vm for its GTT. This has to be + * known at construction, so we can not use the late greedy load + * balancing of the virtual-engine. + */ + ce = __migrate_engines(m->context->engine->gt); + if (IS_ERR(ce)) + return ce; + + ce->ring = __intel_context_ring_size(SZ_256K); + + i915_vm_put(ce->vm); + ce->vm = i915_vm_get(m->context->vm); + + return ce; +} + +static inline struct sgt_dma sg_sgt(struct scatterlist *sg) +{ + dma_addr_t addr = sg_dma_address(sg); + + return (struct sgt_dma){ sg, addr, addr + sg_dma_len(sg) }; +} + +static int emit_no_arbitration(struct i915_request *rq) +{ + u32 *cs; + + cs = intel_ring_begin(rq, 2); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Explicitly disable preemption for this request. */ + *cs++ = MI_ARB_ON_OFF; + *cs++ = MI_NOOP; + intel_ring_advance(rq, cs); + + return 0; +} + +static int emit_pte(struct i915_request *rq, + struct sgt_dma *it, + enum i915_cache_level cache_level, + bool is_lmem, + u64 offset, + int length) +{ + const u64 encode = rq->context->vm->pte_encode(0, cache_level, + is_lmem ? PTE_LM : 0); + struct intel_ring *ring = rq->ring; + int total = 0; + u32 *hdr, *cs; + int pkt; + + GEM_BUG_ON(INTEL_GEN(rq->engine->i915) < 8); + + /* Compute the page directory offset for the target address range */ + offset += (u64)rq->engine->instance << 32; + offset >>= 12; + offset *= sizeof(u64); + offset += 2 * CHUNK_SZ; + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + /* Pack as many PTE updates as possible into a single MI command */ + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); /* as qword elements */ + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + + do { + if (cs - hdr >= pkt) { + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + cs = intel_ring_begin(rq, 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + pkt = min_t(int, 0x400, ring->space / sizeof(u32) + 5); + pkt = min_t(int, pkt, (ring->size - ring->emit) / sizeof(u32) + 5); + + hdr = cs; + *cs++ = MI_STORE_DATA_IMM | REG_BIT(21); + *cs++ = lower_32_bits(offset); + *cs++ = upper_32_bits(offset); + } + + *cs++ = lower_32_bits(encode | it->dma); + *cs++ = upper_32_bits(encode | it->dma); + + offset += 8; + total += I915_GTT_PAGE_SIZE; + + it->dma += I915_GTT_PAGE_SIZE; + if (it->dma >= it->max) { + it->sg = __sg_next(it->sg); + if (!it->sg || sg_dma_len(it->sg) == 0) + break; + + it->dma = sg_dma_address(it->sg); + it->max = it->dma + sg_dma_len(it->sg); + } + } while (total < length); + + *hdr += cs - hdr - 2; + *cs++ = MI_NOOP; + + ring->emit = (void *)cs - ring->vaddr; + intel_ring_advance(rq, cs); + intel_ring_update_space(ring); + + return total; +} + +static bool wa_1209644611_applies(int gen, u32 size) +{ + u32 height = size >> PAGE_SHIFT; + + if (gen != 11) + return false; + + return height % 4 == 3 && height <= 8; +} + +static int emit_copy(struct i915_request *rq, int size) +{ + const int gen = INTEL_GEN(rq->engine->i915); + u32 instance = rq->engine->instance; + u32 *cs; + + cs = intel_ring_begin(rq, gen >= 8 ? 10 : 6); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + if (gen >= 9 && !wa_1209644611_applies(gen, size)) { + *cs++ = GEN9_XY_FAST_COPY_BLT_CMD | (10 - 2); + *cs++ = BLT_DEPTH_32 | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else if (gen >= 8) { + *cs++ = XY_SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (10 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = 0; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE / 4; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = instance; + *cs++ = 0; + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + *cs++ = instance; + } else { + GEM_BUG_ON(instance); + *cs++ = SRC_COPY_BLT_CMD | BLT_WRITE_RGBA | (6 - 2); + *cs++ = BLT_DEPTH_32 | BLT_ROP_SRC_COPY | PAGE_SIZE; + *cs++ = size >> PAGE_SHIFT << 16 | PAGE_SIZE; + *cs++ = CHUNK_SZ; /* dst offset */ + *cs++ = PAGE_SIZE; + *cs++ = 0; /* src offset */ + } + + intel_ring_advance(rq, cs); + return 0; +} + +int +intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct sgt_dma it_src = sg_sgt(src), it_dst = sg_sgt(dst); + struct i915_request *rq; + int err; + + *out = NULL; + + /* GEM_BUG_ON(ce->vm != migrate_vm); */ + + GEM_BUG_ON(ce->ring->size < SZ_64K); + + do { + int len; + + rq = i915_request_create(ce); + if (IS_ERR(rq)) { + err = PTR_ERR(rq); + goto out_ce; + } + + if (await) { + err = i915_request_await_dma_fence(rq, await); + if (err) + goto out_rq; + + if (rq->engine->emit_init_breadcrumb) { + err = rq->engine->emit_init_breadcrumb(rq); + if (err) + goto out_rq; + } + + await = NULL; + } + + /* The PTE updates + copy must not be interrupted. */ + err = emit_no_arbitration(rq); + if (err) + goto out_rq; + + len = emit_pte(rq, &it_src, src_cache_level, src_is_lmem, 0, + CHUNK_SZ); + if (len <= 0) { + err = len; + goto out_rq; + } + + err = emit_pte(rq, &it_dst, dst_cache_level, dst_is_lmem, + CHUNK_SZ, len); + if (err < 0) + goto out_rq; + if (err < len) { + err = -EINVAL; + goto out_rq; + } + + err = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (err) + goto out_rq; + + err = emit_copy(rq, len); + + /* Arbitration is re-enabled between requests. */ +out_rq: + if (*out) + i915_request_put(*out); + *out = i915_request_get(rq); + i915_request_add(rq); + if (err || !it_src.sg || !sg_dma_len(it_src.sg)) + break; + + cond_resched(); + } while (1); + +out_ce: + return err; +} + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out) +{ + struct intel_context *ce; + int err; + + *out = NULL; + if (!m->context) + return -ENODEV; + + ce = intel_migrate_create_context(m); + if (IS_ERR(ce)) + ce = intel_context_get(m->context); + GEM_BUG_ON(IS_ERR(ce)); + + err = intel_context_pin_ww(ce, ww); + if (err) + goto out; + + err = intel_context_migrate_copy(ce, await, + src, src_cache_level, src_is_lmem, + dst, dst_cache_level, dst_is_lmem, + out); + + intel_context_unpin(ce); +out: + intel_context_put(ce); + return err; +} + +void intel_migrate_fini(struct intel_migrate *m) +{ + struct intel_context *ce; + + ce = fetch_and_zero(&m->context); + if (!ce) + return; + + intel_context_unpin(ce); + intel_context_put(ce); +} + +#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) +#include "selftest_migrate.c" +#endif diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.h b/drivers/gpu/drm/i915/gt/intel_migrate.h new file mode 100644 index 000000000000..32c61190ed73 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE__ +#define __INTEL_MIGRATE__ + +#include "intel_migrate_types.h" + +struct dma_fence; +struct i915_request; +struct i915_gem_ww_ctx; +struct intel_gt; +struct scatterlist; +enum i915_cache_level; + +int intel_migrate_init(struct intel_migrate *m, struct intel_gt *gt); + +struct intel_context *intel_migrate_create_context(struct intel_migrate *m); + +int intel_migrate_copy(struct intel_migrate *m, + struct i915_gem_ww_ctx *ww, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +int intel_context_migrate_copy(struct intel_context *ce, + struct dma_fence *await, + struct scatterlist *src, + enum i915_cache_level src_cache_level, + bool src_is_lmem, + struct scatterlist *dst, + enum i915_cache_level dst_cache_level, + bool dst_is_lmem, + struct i915_request **out); + +void intel_migrate_fini(struct intel_migrate *m); + +#endif /* __INTEL_MIGRATE__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_migrate_types.h b/drivers/gpu/drm/i915/gt/intel_migrate_types.h new file mode 100644 index 000000000000..d98230597f42 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_migrate_types.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_MIGRATE_TYPES__ +#define __INTEL_MIGRATE_TYPES__ + +struct intel_context; + +struct intel_migrate { + struct intel_context *context; +}; + +#endif /* __INTEL_MIGRATE_TYPES__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_ring.h b/drivers/gpu/drm/i915/gt/intel_ring.h index dbf5f14a136f..1b32dadfb8c3 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.h +++ b/drivers/gpu/drm/i915/gt/intel_ring.h @@ -49,6 +49,7 @@ static inline void intel_ring_advance(struct i915_request *rq, u32 *cs) * intel_ring_begin()). */ GEM_BUG_ON((rq->ring->vaddr + rq->ring->emit) != cs); + GEM_BUG_ON(!IS_ALIGNED(rq->ring->emit, 8)); /* RING_TAIL qword align */ } static inline u32 intel_ring_wrap(const struct intel_ring *ring, u32 pos) diff --git a/drivers/gpu/drm/i915/gt/selftest_migrate.c b/drivers/gpu/drm/i915/gt/selftest_migrate.c new file mode 100644 index 000000000000..9784d149ebf1 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/selftest_migrate.c @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include "selftests/i915_random.h" + +static const unsigned int sizes[] = { + SZ_4K, + SZ_64K, + SZ_2M, + CHUNK_SZ - SZ_4K, + CHUNK_SZ, + CHUNK_SZ + SZ_4K, + SZ_64M, +}; + +static struct drm_i915_gem_object * +create_lmem_or_internal(struct drm_i915_private *i915, size_t size) +{ + if (HAS_LMEM(i915)) { + struct drm_i915_gem_object *obj; + + obj = i915_gem_object_create_lmem(i915, size, 0); + if (!IS_ERR(obj)) + return obj; + } + + return i915_gem_object_create_internal(i915, size); +} + +static int copy(struct intel_migrate *migrate, + int (*fn)(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out), + u32 sz, struct rnd_state *prng) +{ + struct drm_i915_private *i915 = migrate->context->engine->i915; + struct drm_i915_gem_object *src, *dst; + struct i915_request *rq; + struct i915_gem_ww_ctx ww; + u32 *vaddr; + int err = 0; + int i; + + src = create_lmem_or_internal(i915, sz); + if (IS_ERR(src)) + return 0; + + dst = i915_gem_object_create_internal(i915, sz); + if (IS_ERR(dst)) + goto err_free_src; + + for_i915_gem_ww(&ww, err, true) { + err = i915_gem_object_lock(src, &ww); + if (err) + continue; + + err = i915_gem_object_lock(dst, &ww); + if (err) + continue; + + vaddr = i915_gem_object_pin_map(src, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + continue; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = i; + i915_gem_object_flush_map(src); + + vaddr = i915_gem_object_pin_map(dst, I915_MAP_WC); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto unpin_src; + } + + for (i = 0; i < sz / sizeof(u32); i++) + vaddr[i] = ~i; + i915_gem_object_flush_map(dst); + + err = fn(migrate, &ww, src, dst, &rq); + if (!err) + continue; + + if (err != -EDEADLK && err != -EINTR && err != -ERESTARTSYS) + pr_err("%ps failed, size: %u\n", fn, sz); + if (rq) { + i915_request_wait(rq, 0, HZ); + i915_request_put(rq); + } + i915_gem_object_unpin_map(dst); +unpin_src: + i915_gem_object_unpin_map(src); + } + if (err) + goto err_out; + + if (rq) { + if (i915_request_wait(rq, 0, HZ) < 0) { + pr_err("%ps timed out, size: %u\n", fn, sz); + err = -ETIME; + } + i915_request_put(rq); + } + + for (i = 0; !err && i < sz / PAGE_SIZE; i++) { + int x = i * 1024 + i915_prandom_u32_max_state(1024, prng); + + if (vaddr[x] != x) { + pr_err("%ps failed, size: %u, offset: %zu\n", + fn, sz, x * sizeof(u32)); + igt_hexdump(vaddr + i * 1024, 4096); + err = -EINVAL; + } + } + + i915_gem_object_unpin_map(dst); + i915_gem_object_unpin_map(src); + +err_out: + i915_gem_object_put(dst); +err_free_src: + i915_gem_object_put(src); + + return err; +} + +static int __migrate_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_migrate_copy(migrate, ww, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int __global_copy(struct intel_migrate *migrate, + struct i915_gem_ww_ctx *ww, + struct drm_i915_gem_object *src, + struct drm_i915_gem_object *dst, + struct i915_request **out) +{ + return intel_context_migrate_copy(migrate->context, NULL, + src->mm.pages->sgl, src->cache_level, + i915_gem_object_is_lmem(src), + dst->mm.pages->sgl, dst->cache_level, + i915_gem_object_is_lmem(dst), + out); +} + +static int +migrate_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __migrate_copy, sz, prng); +} + +static int +global_copy(struct intel_migrate *migrate, u32 sz, struct rnd_state *prng) +{ + return copy(migrate, __global_copy, sz, prng); +} + +static int live_migrate_copy(void *arg) +{ + struct intel_migrate *migrate = arg; + struct drm_i915_private *i915 = migrate->context->engine->i915; + I915_RND_STATE(prng); + int i; + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + int err; + + err = migrate_copy(migrate, sizes[i], &prng); + if (err == 0) + err = global_copy(migrate, sizes[i], &prng); + i915_gem_drain_freed_objects(i915); + if (err) + return err; + } + + return 0; +} + +struct threaded_migrate { + struct intel_migrate *migrate; + struct task_struct *tsk; + struct rnd_state prng; +}; + +static int threaded_migrate(struct intel_migrate *migrate, + int (*fn)(void *arg), + unsigned int flags) +{ + const unsigned int n_cpus = num_online_cpus() + 1; + struct threaded_migrate *thread; + I915_RND_STATE(prng); + unsigned int i; + int err = 0; + + thread = kcalloc(n_cpus, sizeof(*thread), GFP_KERNEL); + if (!thread) + return 0; + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk; + + thread[i].migrate = migrate; + thread[i].prng = + I915_RND_STATE_INITIALIZER(prandom_u32_state(&prng)); + + tsk = kthread_run(fn, &thread[i], "igt-%d", i); + if (IS_ERR(tsk)) { + err = PTR_ERR(tsk); + break; + } + + get_task_struct(tsk); + thread[i].tsk = tsk; + } + + msleep(10); /* start all threads before we kthread_stop() */ + + for (i = 0; i < n_cpus; ++i) { + struct task_struct *tsk = thread[i].tsk; + int status; + + if (IS_ERR_OR_NULL(tsk)) + continue; + + status = kthread_stop(tsk); + if (status && !err) + err = status; + + put_task_struct(tsk); + } + + kfree(thread); + return err; +} + +static int __thread_migrate_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return migrate_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_migrate_copy(void *arg) +{ + return threaded_migrate(arg, __thread_migrate_copy, 0); +} + +static int __thread_global_copy(void *arg) +{ + struct threaded_migrate *tm = arg; + + return global_copy(tm->migrate, 2 * CHUNK_SZ, &tm->prng); +} + +static int thread_global_copy(void *arg) +{ + return threaded_migrate(arg, __thread_global_copy, 0); +} + +int intel_migrate_live_selftests(struct drm_i915_private *i915) +{ + static const struct i915_subtest tests[] = { + SUBTEST(live_migrate_copy), + SUBTEST(thread_migrate_copy), + SUBTEST(thread_global_copy), + }; + struct intel_migrate m; + int err; + + if (intel_migrate_init(&m, &i915->gt)) + return 0; + + err = i915_subtests(tests, &m); + intel_migrate_fini(&m); + + return err; +} diff --git a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h index a92c0e9b7e6b..be5e0191eaea 100644 --- a/drivers/gpu/drm/i915/selftests/i915_live_selftests.h +++ b/drivers/gpu/drm/i915/selftests/i915_live_selftests.h @@ -26,6 +26,7 @@ selftest(gt_mocs, intel_mocs_live_selftests) selftest(gt_pm, intel_gt_pm_live_selftests) selftest(gt_heartbeat, intel_heartbeat_live_selftests) selftest(requests, i915_request_live_selftests) +selftest(migrate, intel_migrate_live_selftests) selftest(active, i915_active_live_selftests) selftest(objects, i915_gem_object_live_selftests) selftest(mman, i915_gem_mman_live_selftests) -- 2.31.1