From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA40C433B4 for ; Tue, 18 May 2021 08:27:59 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 16BED610E9 for ; Tue, 18 May 2021 08:27:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16BED610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1AB6B6EAB9; Tue, 18 May 2021 08:27:58 +0000 (UTC) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by gabe.freedesktop.org (Postfix) with ESMTPS id B366A6EAB9; Tue, 18 May 2021 08:27:56 +0000 (UTC) IronPort-SDR: tTcv+ZTpTZJT116339QxihTBsApd2Nk3v9dPvX0Ru8jdP6M7v6fCDrsbl7zgowyF97X3/XYupq MxW8gJFSz3Tg== X-IronPort-AV: E=McAfee;i="6200,9189,9987"; a="180937242" X-IronPort-AV: E=Sophos;i="5.82,309,1613462400"; d="scan'208";a="180937242" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2021 01:27:56 -0700 IronPort-SDR: 3hF2OHjKbRzaR6ubktPmv2Tp1Boc2B2nbDVFrA47keONPNY4jerpcxHPPFKIb6hGN5x03JthH7 fXNeqvvgSEfw== X-IronPort-AV: E=Sophos;i="5.82,309,1613462400"; d="scan'208";a="611892361" Received: from cmutgix-mobl.gar.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.195]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2021 01:27:40 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Tue, 18 May 2021 10:26:54 +0200 Message-Id: <20210518082701.997251-9-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210518082701.997251-1-thomas.hellstrom@linux.intel.com> References: <20210518082701.997251-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 08/15] drm/i915/ttm Add a generic TTM memcpy move for page-based iomem X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" VGhlIGludGVybmFsIHR0bV9ib191dGlsIG1lbWNweSB1c2VzIHZtYXAgZnVuY3Rpb25hbGl0eSwg YW5kIHdoaWxlIGl0CnByb2JhYmx5IG1pZ2h0IGJlIHBvc3NpYmxlIHRvIHVzZSBpdCBmb3IgY29w eWluZyBpbi0gYW5kIG91dCBvZgpzZ2xpc3QgcmVwcmVzZW50ZWQgaW8gbWVtb3J5LCB1c2luZyBp b19tZW1fcmVzZXJ2ZSgpIC8gaW9fbWVtX2ZyZWUoKQpjYWxsYmFja3MsIHRoYXQgd291bGQgY2F1 c2UgcHJvYmxlbXMgd2l0aCBmYXVsdCgpLgpJbnN0ZWFkLCBpbXBsZW1lbnQgYSBtZXRob2QgbWFw cGluZyBwYWdlLWJ5LXBhZ2UgdXNpbmcga21hcF9sb2NhbCgpCnNlbWFudGljcy4gQXMgYW4gYWRk aXRpb25hbCBiZW5lZml0IHdlIHRoZW4gYXZvaWQgdGhlIG9jY2FzaW9uYWwgZ2xvYmFsClRMQiBm bHVzaGVzIG9mIHZtYXAoKSBhbmQgY29uc3VtaW5nIHZtYXAgc3BhY2UsIGVsaW1pbmF0aW9uIG9m IGEgY3JpdGljYWwKcG9pbnQgb2YgZmFpbHVyZSBhbmQgd2l0aCBhIHNsaWdodCBjaGFuZ2Ugb2Yg c2VtYW50aWNzIHdlIGNvdWxkIGFsc28gcHVzaAp0aGUgbWVtY3B5IG91dCBhc3luYyBmb3IgdGVz dGluZyBhbmQgYXN5bmMgZHJpdmVyIGRldmVscG1lbnQgcHVycG9zZXMuClB1c2hpbmcgb3V0IGFz eW5jIGNhbiBiZSBkb25lIHNpbmNlIHRoZXJlIGlzIG5vIG1lbW9yeSBhbGxvY2F0aW9uIGdvaW5n IG9uCnRoYXQgY291bGQgdmlvbGF0ZSB0aGUgZG1hX2ZlbmNlIGxvY2tkZXAgcnVsZXMuCgpGb3Ig Y29waWVzIGZyb20gaW9tZW0sIHVzZSB0aGUgV0MgcHJlZmV0Y2hpbmcgbWVtY3B5IHZhcmlhbnQg Zm9yCmFkZGl0aW9uYWwgc3BlZWQuCgpOb3RlIHRoYXQgZHJpdmVycyB0aGF0IGRvbid0IHdhbnQg dG8gdXNlIHN0cnVjdCBpb19tYXBwaW5nIGJ1dCByZWxpZXMgb24KbWVtcmVtYXAgZnVuY3Rpb25h bGl0eSwgYW5kIHRoYXQgZG9uJ3Qgd2FudCB0byB1c2Ugc2NhdHRlcmxpc3RzIGZvcgpWUkFNIG1h eSB3ZWxsIGRlZmluZSBzcGVjaWFsaXplZCAoaG9wZWZ1bGx5IHJldXNhYmxlKSBpdGVyYXRvcnMg Zm9yIHRoZWlyCnBhcnRpY3VsYXIgZW52aXJvbm1lbnQuCgpDYzogQ2hyaXN0aWFuIEvDtm5pZyA8 Y2hyaXN0aWFuLmtvZW5pZ0BhbWQuY29tPgpTaWduZWQtb2ZmLWJ5OiBUaG9tYXMgSGVsbHN0csO2 bSA8dGhvbWFzLmhlbGxzdHJvbUBsaW51eC5pbnRlbC5jb20+Ci0tLQp2MjoKLSBNb3ZlIG5ldyBU VE0gZXhwb3J0cyB0byBhIHNlcGFyYXRlIGNvbW1pdC4gKFJlcG9ydGVkIGJ5IENocmlzdGlhbiBL w7ZuaWcpCi0gQXZvaWQgaGF2aW5nIHRoZSBpdGVyYXRvciBpbml0IGZ1bmN0aW9ucyBpbmxpbmUu IChSZXBvcnRlZCBieSBKYW5pIE5pa3VsYSkKLSBSZW1vdmUgYSBzdHJheSBjb21tZW50LgotLS0K IGRyaXZlcnMvZ3B1L2RybS9pOTE1L01ha2VmaWxlICAgICAgICAgICAgICAgICB8ICAgMSArCiAu Li4vZ3B1L2RybS9pOTE1L2dlbS9pOTE1X2dlbV90dG1fYm9fdXRpbC5jICAgfCAxOTQgKysrKysr KysrKysrKysrKysrCiAuLi4vZ3B1L2RybS9pOTE1L2dlbS9pOTE1X2dlbV90dG1fYm9fdXRpbC5o ICAgfCAxMDcgKysrKysrKysrKwogMyBmaWxlcyBjaGFuZ2VkLCAzMDIgaW5zZXJ0aW9ucygrKQog Y3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9pOTE1L2dlbS9pOTE1X2dlbV90dG1f Ym9fdXRpbC5jCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL2k5MTUvZ2VtL2k5 MTVfZ2VtX3R0bV9ib191dGlsLmgKCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9N YWtlZmlsZSBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L01ha2VmaWxlCmluZGV4IGNiODgyMzU3MDk5 Ni4uOTU4Y2NjMWVkZmVkIDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmls ZQorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9NYWtlZmlsZQpAQCAtMTU1LDYgKzE1NSw3IEBA IGdlbS15ICs9IFwKIAlnZW0vaTkxNV9nZW1fc3RvbGVuLm8gXAogCWdlbS9pOTE1X2dlbV90aHJv dHRsZS5vIFwKIAlnZW0vaTkxNV9nZW1fdGlsaW5nLm8gXAorCWdlbS9pOTE1X2dlbV90dG1fYm9f dXRpbC5vIFwKIAlnZW0vaTkxNV9nZW1fdXNlcnB0ci5vIFwKIAlnZW0vaTkxNV9nZW1fd2FpdC5v IFwKIAlnZW0vaTkxNV9nZW1mcy5vCmRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vaTkxNS9n ZW0vaTkxNV9nZW1fdHRtX2JvX3V0aWwuYyBiL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2dlbS9pOTE1 X2dlbV90dG1fYm9fdXRpbC5jCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAw MC4uNWYzNDdhODViZjQ0Ci0tLSAvZGV2L251bGwKKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUv Z2VtL2k5MTVfZ2VtX3R0bV9ib191dGlsLmMKQEAgLTAsMCArMSwxOTQgQEAKKy8vIFNQRFgtTGlj ZW5zZS1JZGVudGlmaWVyOiBNSVQKKy8qCisgKiBDb3B5cmlnaHQgwqkgMjAyMSBJbnRlbCBDb3Jw b3JhdGlvbgorICovCisKKy8qKgorICogRE9DOiBVc2FnZSBhbmQgaW50ZW50aW9ucy4KKyAqCisg KiBUaGlzIGZpbGUgY29udGFpbnMgZnVuY3Rpb25hbGl0eSB0aGF0IHdlIG1pZ2h0IHdhbnQgdG8g bW92ZSBpbnRvCisgKiB0dG1fYm9fdXRpbC5jIGlmIHRoZXJlIGlzIGEgY29tbW9uIGludGVyZXN0 LgorICogQ3VycmVudGx5IGEga21hcF9sb2NhbCBvbmx5IG1lbWNweSB3aXRoIHN1cHBvcnQgZm9y IHBhZ2UtYmFzZWQgaW9tZW0gcmVnaW9ucywKKyAqIGFuZCBmYXN0IG1lbWNweSBmcm9tIHdyaXRl LWNvbWJpbmVkIG1lbW9yeS4KKyAqLworCisjaW5jbHVkZSA8bGludXgvZG1hLWJ1Zi1tYXAuaD4K KyNpbmNsdWRlIDxsaW51eC9oaWdobWVtLmg+CisjaW5jbHVkZSA8bGludXgvaW8tbWFwcGluZy5o PgorI2luY2x1ZGUgPGxpbnV4L3NjYXR0ZXJsaXN0Lmg+CisKKyNpbmNsdWRlICJpOTE1X21lbWNw eS5oIgorCisjaW5jbHVkZSAiZ2VtL2k5MTVfZ2VtX3R0bV9ib191dGlsLmgiCisKK3N0YXRpYyB2 b2lkIGk5MTVfdHRtX2ttYXBfaXRlcl90dF9rbWFwX2xvY2FsKHN0cnVjdCBpOTE1X3R0bV9rbWFw X2l0ZXIgKml0ZXIsCisJCQkJCSAgICAgc3RydWN0IGRtYV9idWZfbWFwICpkbWFwLAorCQkJCQkg ICAgIHBnb2ZmX3QgaSkKK3sKKwlzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX3R0ICppdGVyX3R0 ID0KKwkJY29udGFpbmVyX29mKGl0ZXIsIHR5cGVvZigqaXRlcl90dCksIGJhc2UpOworCisJZG1h X2J1Zl9tYXBfc2V0X3ZhZGRyKGRtYXAsIGttYXBfbG9jYWxfcGFnZShpdGVyX3R0LT50dC0+cGFn ZXNbaV0pKTsKK30KKworc3RhdGljIHZvaWQgaTkxNV90dG1fa21hcF9pdGVyX2lvbWFwX2ttYXBf bG9jYWwoc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAqaXRlciwKKwkJCQkJCXN0cnVjdCBkbWFf YnVmX21hcCAqZG1hcCwKKwkJCQkJCXBnb2ZmX3QgaSkKK3sKKwlzdHJ1Y3QgaTkxNV90dG1fa21h cF9pdGVyX2lvbWFwICppdGVyX2lvID0KKwkJY29udGFpbmVyX29mKGl0ZXIsIHR5cGVvZigqaXRl cl9pbyksIGJhc2UpOworCXZvaWQgX19pb21lbSAqYWRkcjsKKworcmV0cnk6CisJd2hpbGUgKGkg Pj0gaXRlcl9pby0+Y2FjaGUuZW5kKSB7CisJCWl0ZXJfaW8tPmNhY2hlLnNnID0gaXRlcl9pby0+ Y2FjaGUuc2cgPworCQkJc2dfbmV4dChpdGVyX2lvLT5jYWNoZS5zZykgOiBpdGVyX2lvLT5zdC0+ c2dsOworCQlpdGVyX2lvLT5jYWNoZS5pID0gaXRlcl9pby0+Y2FjaGUuZW5kOworCQlpdGVyX2lv LT5jYWNoZS5lbmQgKz0gc2dfZG1hX2xlbihpdGVyX2lvLT5jYWNoZS5zZykgPj4KKwkJCVBBR0Vf U0hJRlQ7CisJCWl0ZXJfaW8tPmNhY2hlLm9mZnMgPSBzZ19kbWFfYWRkcmVzcyhpdGVyX2lvLT5j YWNoZS5zZykgLQorCQkJaXRlcl9pby0+c3RhcnQ7CisJfQorCisJaWYgKGkgPCBpdGVyX2lvLT5j YWNoZS5pKSB7CisJCWl0ZXJfaW8tPmNhY2hlLmVuZCA9IDA7CisJCWl0ZXJfaW8tPmNhY2hlLnNn ID0gTlVMTDsKKwkJZ290byByZXRyeTsKKwl9CisKKwlhZGRyID0gaW9fbWFwcGluZ19tYXBfbG9j YWxfd2MoaXRlcl9pby0+aW9tYXAsIGl0ZXJfaW8tPmNhY2hlLm9mZnMgKworCQkJCSAgICAgICAo KChyZXNvdXJjZV9zaXplX3QpaSAtIGl0ZXJfaW8tPmNhY2hlLmkpCisJCQkJCTw8IFBBR0VfU0hJ RlQpKTsKKwlkbWFfYnVmX21hcF9zZXRfdmFkZHJfaW9tZW0oZG1hcCwgYWRkcik7Cit9CisKK3N0 YXRpYyBjb25zdCBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX29wcyBpOTE1X3R0bV9rbWFwX2l0 ZXJfdHRfb3BzID0geworCS5rbWFwX2xvY2FsID0gaTkxNV90dG1fa21hcF9pdGVyX3R0X2ttYXBf bG9jYWwKK307CisKK3N0YXRpYyBjb25zdCBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX29wcyBp OTE1X3R0bV9rbWFwX2l0ZXJfaW9fb3BzID0geworCS5rbWFwX2xvY2FsID0gIGk5MTVfdHRtX2tt YXBfaXRlcl9pb21hcF9rbWFwX2xvY2FsCit9OworCitzdGF0aWMgdm9pZCBrdW5tYXBfbG9jYWxf ZG1hX2J1Zl9tYXAoc3RydWN0IGRtYV9idWZfbWFwICptYXApCit7CisJaWYgKG1hcC0+aXNfaW9t ZW0pCisJCWlvX21hcHBpbmdfdW5tYXBfbG9jYWwobWFwLT52YWRkcl9pb21lbSk7CisJZWxzZQor CQlrdW5tYXBfbG9jYWwobWFwLT52YWRkcik7Cit9CisKKy8qKgorICogaTkxNV90dG1fbW92ZV9t ZW1jcHkgLSBIZWxwZXIgdG8gcGVyZm9ybSBhIG1lbWNweSB0dG0gbW92ZSBvcGVyYXRpb24uCisg KiBAYm86IFRoZSBzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QuCisgKiBAbmV3X21lbTogVGhlIHN0 cnVjdCB0dG1fcmVzb3VyY2Ugd2UncmUgbW92aW5nIHRvIChjb3B5IGRlc3RpbmF0aW9uKS4KKyAq IEBuZXdfa21hcDogQSBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyIHJlcHJlc2VudGluZyB0aGUg ZGVzdGluYXRpb24gcmVzb3VyY2UuCisgKiBAb2xkX2ttYXA6IEEgc3RydWN0IGk5MTVfdHRtX2tt YXBfaXRlciByZXByZXNlbnRpbmcgdGhlIHNvdXJjZSByZXNvdXJjZS4KKyAqLwordm9pZCBpOTE1 X3R0bV9tb3ZlX21lbWNweShzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QgKmJvLAorCQkJICBzdHJ1 Y3QgdHRtX3Jlc291cmNlICpuZXdfbWVtLAorCQkJICBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVy ICpuZXdfa21hcCwKKwkJCSAgc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAqb2xkX2ttYXApCit7 CisJc3RydWN0IHR0bV9kZXZpY2UgKmJkZXYgPSBiby0+YmRldjsKKwlzdHJ1Y3QgdHRtX3Jlc291 cmNlX21hbmFnZXIgKm1hbiA9IHR0bV9tYW5hZ2VyX3R5cGUoYmRldiwgbmV3X21lbS0+bWVtX3R5 cGUpOworCXN0cnVjdCB0dG1fdHQgKnR0bSA9IGJvLT50dG07CisJc3RydWN0IHR0bV9yZXNvdXJj ZSAqb2xkX21lbSA9ICZiby0+bWVtOworCXN0cnVjdCB0dG1fcmVzb3VyY2Ugb2xkX2NvcHkgPSAq b2xkX21lbTsKKwlzdHJ1Y3QgdHRtX3Jlc291cmNlX21hbmFnZXIgKm9sZF9tYW4gPSB0dG1fbWFu YWdlcl90eXBlKGJkZXYsIG9sZF9tZW0tPm1lbV90eXBlKTsKKwlzdHJ1Y3QgZG1hX2J1Zl9tYXAg b2xkX21hcCwgbmV3X21hcDsKKwlwZ29mZl90IGk7CisKKwkvKiBTaW5nbGUgVFRNIG1vdmUuIE5P UCAqLworCWlmIChvbGRfbWFuLT51c2VfdHQgJiYgbWFuLT51c2VfdHQpCisJCWdvdG8gZG9uZTsK KworCS8qIERvbid0IG1vdmUgbm9uZXhpc3RlbnQgZGF0YS4gQ2xlYXIgZGVzdGluYXRpb24gaW5z dGVhZC4gKi8KKwlpZiAob2xkX21hbi0+dXNlX3R0ICYmICFtYW4tPnVzZV90dCAmJgorCSAgICAo IXR0bSB8fCAhdHRtX3R0X2lzX3BvcHVsYXRlZCh0dG0pKSkgeworCQlpZiAodHRtICYmICEodHRt LT5wYWdlX2ZsYWdzICYgVFRNX1BBR0VfRkxBR19aRVJPX0FMTE9DKSkKKwkJCWdvdG8gZG9uZTsK KworCQlmb3IgKGkgPSAwOyBpIDwgbmV3X21lbS0+bnVtX3BhZ2VzOyArK2kpIHsKKwkJCW5ld19r bWFwLT5vcHMtPmttYXBfbG9jYWwobmV3X2ttYXAsICZuZXdfbWFwLCBpKTsKKwkJCW1lbXNldF9p byhuZXdfbWFwLnZhZGRyX2lvbWVtLCAwLCBQQUdFX1NJWkUpOworCQkJa3VubWFwX2xvY2FsX2Rt YV9idWZfbWFwKCZuZXdfbWFwKTsKKwkJfQorCQlnb3RvIGRvbmU7CisJfQorCisJZm9yIChpID0g MDsgaSA8IG5ld19tZW0tPm51bV9wYWdlczsgKytpKSB7CisJCW5ld19rbWFwLT5vcHMtPmttYXBf bG9jYWwobmV3X2ttYXAsICZuZXdfbWFwLCBpKTsKKwkJb2xkX2ttYXAtPm9wcy0+a21hcF9sb2Nh bChvbGRfa21hcCwgJm9sZF9tYXAsIGkpOworCQlpZiAoIW9sZF9tYXAuaXNfaW9tZW0gfHwKKwkJ ICAgICFpOTE1X21lbWNweV9mcm9tX3djKG5ld19tYXAudmFkZHIsIG9sZF9tYXAudmFkZHIsIFBB R0VfU0laRSkpIHsKKwkJCWlmICghb2xkX21hcC5pc19pb21lbSkgeworCQkJCWRtYV9idWZfbWFw X21lbWNweV90bygmbmV3X21hcCwgb2xkX21hcC52YWRkciwKKwkJCQkJCSAgICAgIFBBR0VfU0la RSk7CisJCQl9IGVsc2UgaWYgKCFuZXdfbWFwLmlzX2lvbWVtKSB7CisJCQkJbWVtY3B5X2Zyb21p byhuZXdfbWFwLnZhZGRyLCBvbGRfbWFwLnZhZGRyX2lvbWVtLAorCQkJCQkgICAgICBQQUdFX1NJ WkUpOworCQkJfSBlbHNlIHsKKwkJCQlwZ29mZl90IGo7CisJCQkJdTMyIF9faW9tZW0gKnNyYyA9 IG9sZF9tYXAudmFkZHJfaW9tZW07CisJCQkJdTMyIF9faW9tZW0gKmRzdCA9IG5ld19tYXAudmFk ZHJfaW9tZW07CisKKwkJCQlmb3IgKGogPSAwOyBqIDwgKFBBR0VfU0laRSA+PiAyKTsgKytqKQor CQkJCQlpb3dyaXRlMzIoaW9yZWFkMzIoc3JjKyspLCBkc3QrKyk7CisJCQl9CisJCX0KKwkJa3Vu bWFwX2xvY2FsX2RtYV9idWZfbWFwKCZvbGRfbWFwKTsKKwkJa3VubWFwX2xvY2FsX2RtYV9idWZf bWFwKCZuZXdfbWFwKTsKKwl9CisKK2RvbmU6CisJb2xkX2NvcHkgPSAqb2xkX21lbTsKKworCXR0 bV9ib19hc3NpZ25fbWVtKGJvLCBuZXdfbWVtKTsKKworCWlmICghbWFuLT51c2VfdHQpCisJCXR0 bV9ib190dF9kZXN0cm95KGJvKTsKKworCXR0bV9yZXNvdXJjZV9mcmVlKGJvLCAmb2xkX2NvcHkp OworfQorCisvKioKKyAqIGk5MTVfdHRtX2ttYXBfaXRlcl9pb21hcF9pbml0IC0gSW5pdGlhbGl6 ZSBhIHN0cnVjdCBpOTE1X3R0bV9rbWFwX2l0ZXJfaW9tYXAKKyAqIEBpdGVyX2lvOiBUaGUgc3Ry dWN0IGk5MTVfdHRtX2ttYXBfaXRlcl9pb21hcCB0byBpbml0aWFsaXplLgorICogQGlvbWFwOiBU aGUgc3RydWN0IGlvX21hcHBpbmcgcmVwcmVzZW50aW5nIHRoZSB1bmRlcmx5aW5nIGxpbmVhciBp b19tZW1vcnkuCisgKiBAc3Q6IHNnX3RhYmxlIGludG8gQGlvbWFwLCByZXByZXNlbnRpbmcgdGhl IG1lbW9yeSBvZiB0aGUgc3RydWN0CisgKiB0dG1fcmVzb3VyY2UuCisgKiBAc3RhcnQ6IE9mZnNl dCB0aGF0IG5lZWRzIHRvIGJlIHN1YnRyYWN0ZWQgZnJvbSBAc3QgdG8gbWFrZQorICogc2dfZG1h X2FkZHJlc3Moc3QtPnNnbCkgLSBAc3RhcnQgPT0gMCBmb3IgQGlvbWFwIHN0YXJ0LgorICoKKyAq IFJldHVybjogUG9pbnRlciB0byB0aGUgZW1iZWRkZWQgc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRl ci4KKyAqLworc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAqCitpOTE1X3R0bV9rbWFwX2l0ZXJf aW9tYXBfaW5pdChzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX2lvbWFwICppdGVyX2lvLAorCQkJ ICAgICAgc3RydWN0IGlvX21hcHBpbmcgKmlvbWFwLAorCQkJICAgICAgc3RydWN0IHNnX3RhYmxl ICpzdCwKKwkJCSAgICAgIHJlc291cmNlX3NpemVfdCBzdGFydCkKK3sKKwlpdGVyX2lvLT5iYXNl Lm9wcyA9ICZpOTE1X3R0bV9rbWFwX2l0ZXJfaW9fb3BzOworCWl0ZXJfaW8tPmlvbWFwID0gaW9t YXA7CisJaXRlcl9pby0+c3QgPSBzdDsKKwlpdGVyX2lvLT5zdGFydCA9IHN0YXJ0OworCW1lbXNl dCgmaXRlcl9pby0+Y2FjaGUsIDAsIHNpemVvZihpdGVyX2lvLT5jYWNoZSkpOworCXJldHVybiAm aXRlcl9pby0+YmFzZTsKK30KKworLyoqCisgKiB0dG1fa21hcF9pdGVyX3R0X2luaXQgLSBJbml0 aWFsaXplIGEgc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlcl90dAorICogQGl0ZXJfdHQ6IFRoZSBz dHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX3R0IHRvIGluaXRpYWxpemUuCisgKiBAdHQ6IFN0cnVj dCB0dG1fdHQgaG9sZGluZyBwYWdlIHBvaW50ZXJzIG9mIHRoZSBzdHJ1Y3QgdHRtX3Jlc291cmNl LgorICoKKyAqIFJldHVybjogUG9pbnRlciB0byB0aGUgZW1iZWRkZWQgc3RydWN0IGk5MTVfdHRt X2ttYXBfaXRlci4KKyAqLworc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAqCitpOTE1X3R0bV9r bWFwX2l0ZXJfdHRfaW5pdChzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX3R0ICppdGVyX3R0LAor CQkJICAgc3RydWN0IHR0bV90dCAqdHQpCit7CisJaXRlcl90dC0+YmFzZS5vcHMgPSAmaTkxNV90 dG1fa21hcF9pdGVyX3R0X29wczsKKwlpdGVyX3R0LT50dCA9IHR0OworCXJldHVybiAmaXRlcl90 dC0+YmFzZTsKK30KZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2dlbS9pOTE1X2dl bV90dG1fYm9fdXRpbC5oIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvZ2VtL2k5MTVfZ2VtX3R0bV9i b191dGlsLmgKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMDAwMDAwLi5mMWM4Yzdh ZTdhNDIKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL2dwdS9kcm0vaTkxNS9nZW0vaTkxNV9n ZW1fdHRtX2JvX3V0aWwuaApAQCAtMCwwICsxLDEwNyBAQAorLyogU1BEWC1MaWNlbnNlLUlkZW50 aWZpZXI6IE1JVCAqLworLyoKKyAqIENvcHlyaWdodCDCqSAyMDIxIEludGVsIENvcnBvcmF0aW9u CisgKi8KKworLyoKKyAqIFRoaXMgZmlsZXMgY29udGFpbnMgZnVuY3Rpb25hbGl0eSB0aGF0IHdl IG1pZ2h0IHdhbnQgdG8gbW92ZSBpbnRvCisgKiB0dG1fYm9fdXRpbC5jIGlmIHRoZXJlIGlzIGEg Y29tbW9uIGludGVyZXN0LgorICovCisjaWZuZGVmIF9JOTE1X0dFTV9UVE1fQk9fVVRJTF9IXwor I2RlZmluZSBfSTkxNV9HRU1fVFRNX0JPX1VUSUxfSF8KKworI2luY2x1ZGUgPGRybS90dG0vdHRt X2JvX2RyaXZlci5oPgorc3RydWN0IGRtYV9idWZfbWFwOworc3RydWN0IGlvX21hcHBpbmc7Citz dHJ1Y3Qgc2dfdGFibGU7CitzdHJ1Y3Qgc2NhdHRlcmxpc3Q7CisKK3N0cnVjdCB0dG1fdHQ7Citz dHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyOworCisvKioKKyAqIHN0cnVjdCBpOTE1X3R0bV9rbWFw X2l0ZXJfb3BzIC0gT3BzIHN0cnVjdHVyZSBmb3IgYSBzdHJ1Y3QKKyAqIGk5MTVfdHRtX2ttYXBf aXRlci4KKyAqLworc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlcl9vcHMgeworCS8qKgorCSAqIGtt YXBfbG9jYWwgLSBNYXAgYSBQQUdFX1NJWkUgcGFydCBvZiB0aGUgcmVzb3VyY2UgdXNpbmcKKwkg KiBrbWFwX2xvY2FsIHNlbWFudGljcy4KKwkgKiBAcmVzX2ttYXA6IFBvaW50ZXIgdG8gdGhlIHN0 cnVjdCBpOTE1X3R0bV9rbWFwX2l0ZXIgcmVwcmVzZW50aW5nCisJICogdGhlIHJlc291cmNlLgor CSAqIEBkbWFwOiBUaGUgc3RydWN0IGRtYV9idWZfbWFwIGhvbGRpbmcgdGhlIHZpcnR1YWwgYWRk cmVzcyBhZnRlcgorCSAqIHRoZSBvcGVyYXRpb24uCisJICogQGk6IFRoZSBsb2NhdGlvbiB3aXRo aW4gdGhlIHJlc291cmNlIHRvIG1hcC4gUEFHRV9TSVpFIGdyYW51bGFyaXR5LgorCSAqLworCXZv aWQgKCprbWFwX2xvY2FsKShzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyICpyZXNfa21hcCwKKwkJ CSAgIHN0cnVjdCBkbWFfYnVmX21hcCAqZG1hcCwgcGdvZmZfdCBpKTsKK307CisKKy8qKgorICog c3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAtIEl0ZXJhdG9yIGZvciBrbWFwX2xvY2FsIHR5cGUg b3BlcmF0aW9ucyBvbiBhCisgKiByZXNvdXJjZS4KKyAqIEBvcHM6IFBvaW50ZXIgdG8gdGhlIG9w ZXJhdGlvbnMgc3RydWN0LgorICoKKyAqIFRoaXMgc3RydWN0IGlzIGludGVuZGVkIHRvIGJlIGVt YmVkZGVkIGluIGEgcmVzb3VyY2Utc3BlY2lmaWMgc3BlY2lhbGl6YXRpb24KKyAqIGltcGxlbWVu dGluZyBvcGVyYXRpb25zIGZvciB0aGUgcmVzb3VyY2UuCisgKgorICogTm90aGluZyBzdG9wcyB1 cyBmcm9tIGV4dGVuZGluZyB0aGUgb3BlcmF0aW9ucyB0byB2bWFwLCB2bWFwX3BmbiBldGMsCisg KiByZXBsYWNpbmcgc29tZSBvciBwYXJ0cyBvZiB0aGUgdHRtX2JvX3V0aWwuIGNwdS1tYXAgZnVu Y3Rpb25hbGl0eS4KKyAqLworc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciB7CisJY29uc3Qgc3Ry dWN0IGk5MTVfdHRtX2ttYXBfaXRlcl9vcHMgKm9wczsKK307CisKKy8qKgorICogc3RydWN0IGk5 MTVfdHRtX2ttYXBfaXRlcl90dCAtIFNwZWNpYWxpemF0aW9uIGZvciBhIHR0IChwYWdlKSBiYWNr ZWQgc3RydWN0CisgKiB0dG1fcmVzb3VyY2UuCisgKiBAYmFzZTogRW1iZWRkZWQgc3RydWN0IGk5 MTVfdHRtX2ttYXBfaXRlciBwcm92aWRpbmcgdGhlIHVzYWdlIGludGVyZmFjZQorICogQHR0OiBD YWNoZWQgc3RydWN0IHR0bV90dC4KKyAqLworc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlcl90dCB7 CisJc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciBiYXNlOworCXN0cnVjdCB0dG1fdHQgKnR0Owor fTsKKworLyoqCisgKiBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX2lvbWFwIC0gU3BlY2lhbGl6 YXRpb24gZm9yIGEgc3RydWN0IGlvX21hcHBpbmcgKworICogc3RydWN0IHNnX3RhYmxlIGJhY2tl ZCBzdHJ1Y3QgdHRtX3Jlc291cmNlLgorICogQGJhc2U6IEVtYmVkZGVkIHN0cnVjdCBpOTE1X3R0 bV9rbWFwX2l0ZXIgcHJvdmlkaW5nIHRoZSB1c2FnZSBpbnRlcmZhY2UuCisgKiBAaW9tYXA6IHN0 cnVjdCBpb19tYXBwaW5nIHJlcHJlc2VudGluZyB0aGUgdW5kZXJseWluZyBsaW5lYXIgaW9fbWVt b3J5LgorICogQHN0OiBzZ190YWJsZSBpbnRvIEBpb21hcCwgcmVwcmVzZW50aW5nIHRoZSBtZW1v cnkgb2YgdGhlIHN0cnVjdCB0dG1fcmVzb3VyY2UuCisgKiBAc3RhcnQ6IE9mZnNldCB0aGF0IG5l ZWRzIHRvIGJlIHN1YnRyYWN0ZWQgZnJvbSBAc3QgdG8gbWFrZQorICogc2dfZG1hX2FkZHJlc3Mo c3QtPnNnbCkgLSBAc3RhcnQgPT0gMCBmb3IgQGlvbWFwIHN0YXJ0LgorICogQGNhY2hlOiBTY2F0 dGVybGlzdCB0cmF2ZXJzYWwgY2FjaGUgZm9yIGZhc3QgbG9va3Vwcy4KKyAqIEBjYWNoZS5zZzog UG9pbnRlciB0byB0aGUgY3VycmVudGx5IGNhY2hlZCBzY2F0dGVybGlzdCBzZWdtZW50LgorICog QGNhY2hlLmk6IEZpcnN0IGluZGV4IG9mIEBzZy4gUEFHRV9TSVpFIGdyYW51bGFyaXR5LgorICog QGNhY2hlLmVuZDogTGFzdCBpbmRleCArIDEgb2YgQHNnLiBQQUdFX1NJWkUgZ3JhbnVsYXJpdHku CisgKiBAY2FjaGUub2ZmczogRmlyc3Qgb2Zmc2V0IGludG8gQGlvbWFwIG9mIEBzZy4gUEFHRV9T SVpFIGdyYW51bGFyaXR5LgorICovCitzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyX2lvbWFwIHsK KwlzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyIGJhc2U7CisJc3RydWN0IGlvX21hcHBpbmcgKmlv bWFwOworCXN0cnVjdCBzZ190YWJsZSAqc3Q7CisJcmVzb3VyY2Vfc2l6ZV90IHN0YXJ0OworCXN0 cnVjdCB7CisJCXN0cnVjdCBzY2F0dGVybGlzdCAqc2c7CisJCXBnb2ZmX3QgaTsKKwkJcGdvZmZf dCBlbmQ7CisJCXBnb2ZmX3Qgb2ZmczsKKwl9IGNhY2hlOworfTsKKwordm9pZCBpOTE1X3R0bV9t b3ZlX21lbWNweShzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QgKmJvLAorCQkJICBzdHJ1Y3QgdHRt X3Jlc291cmNlICpuZXdfbWVtLAorCQkJICBzdHJ1Y3QgaTkxNV90dG1fa21hcF9pdGVyICpuZXdf aXRlciwKKwkJCSAgc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlciAqb2xkX2l0ZXIpOworCitzdHJ1 Y3QgaTkxNV90dG1fa21hcF9pdGVyICoKK2k5MTVfdHRtX2ttYXBfaXRlcl90dF9pbml0KHN0cnVj dCBpOTE1X3R0bV9rbWFwX2l0ZXJfdHQgKml0ZXJfdHQsCisJCQkgICBzdHJ1Y3QgdHRtX3R0ICp0 dCk7CisKK3N0cnVjdCBpOTE1X3R0bV9rbWFwX2l0ZXIgKgoraTkxNV90dG1fa21hcF9pdGVyX2lv bWFwX2luaXQoc3RydWN0IGk5MTVfdHRtX2ttYXBfaXRlcl9pb21hcCAqaXRlcl9pbywKKwkJCSAg ICAgIHN0cnVjdCBpb19tYXBwaW5nICppb21hcCwKKwkJCSAgICAgIHN0cnVjdCBzZ190YWJsZSAq c3QsCisJCQkgICAgICByZXNvdXJjZV9zaXplX3Qgc3RhcnQpOworI2VuZGlmCi0tIAoyLjMxLjEK Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkludGVsLWdm eCBtYWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xp c3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdmeAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B984C43460 for ; Tue, 18 May 2021 08:28:05 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1691A6113C for ; Tue, 18 May 2021 08:28:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1691A6113C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B450B6EACD; Tue, 18 May 2021 08:28:01 +0000 (UTC) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by gabe.freedesktop.org (Postfix) with ESMTPS id B366A6EAB9; Tue, 18 May 2021 08:27:56 +0000 (UTC) IronPort-SDR: tTcv+ZTpTZJT116339QxihTBsApd2Nk3v9dPvX0Ru8jdP6M7v6fCDrsbl7zgowyF97X3/XYupq MxW8gJFSz3Tg== X-IronPort-AV: E=McAfee;i="6200,9189,9987"; a="180937242" X-IronPort-AV: E=Sophos;i="5.82,309,1613462400"; d="scan'208";a="180937242" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2021 01:27:56 -0700 IronPort-SDR: 3hF2OHjKbRzaR6ubktPmv2Tp1Boc2B2nbDVFrA47keONPNY4jerpcxHPPFKIb6hGN5x03JthH7 fXNeqvvgSEfw== X-IronPort-AV: E=Sophos;i="5.82,309,1613462400"; d="scan'208";a="611892361" Received: from cmutgix-mobl.gar.corp.intel.com (HELO thellst-mobl1.intel.com) ([10.249.254.195]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2021 01:27:40 -0700 From: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= To: intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 08/15] drm/i915/ttm Add a generic TTM memcpy move for page-based iomem Date: Tue, 18 May 2021 10:26:54 +0200 Message-Id: <20210518082701.997251-9-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210518082701.997251-1-thomas.hellstrom@linux.intel.com> References: <20210518082701.997251-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , =?UTF-8?q?Christian=20K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The internal ttm_bo_util memcpy uses vmap functionality, and while it probably might be possible to use it for copying in- and out of sglist represented io memory, using io_mem_reserve() / io_mem_free() callbacks, that would cause problems with fault(). Instead, implement a method mapping page-by-page using kmap_local() semantics. As an additional benefit we then avoid the occasional global TLB flushes of vmap() and consuming vmap space, elimination of a critical point of failure and with a slight change of semantics we could also push the memcpy out async for testing and async driver develpment purposes. Pushing out async can be done since there is no memory allocation going on that could violate the dma_fence lockdep rules. For copies from iomem, use the WC prefetching memcpy variant for additional speed. Note that drivers that don't want to use struct io_mapping but relies on memremap functionality, and that don't want to use scatterlists for VRAM may well define specialized (hopefully reusable) iterators for their particular environment. Cc: Christian König Signed-off-by: Thomas Hellström --- v2: - Move new TTM exports to a separate commit. (Reported by Christian König) - Avoid having the iterator init functions inline. (Reported by Jani Nikula) - Remove a stray comment. --- drivers/gpu/drm/i915/Makefile | 1 + .../gpu/drm/i915/gem/i915_gem_ttm_bo_util.c | 194 ++++++++++++++++++ .../gpu/drm/i915/gem/i915_gem_ttm_bo_util.h | 107 ++++++++++ 3 files changed, 302 insertions(+) create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.c create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.h diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index cb8823570996..958ccc1edfed 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -155,6 +155,7 @@ gem-y += \ gem/i915_gem_stolen.o \ gem/i915_gem_throttle.o \ gem/i915_gem_tiling.o \ + gem/i915_gem_ttm_bo_util.o \ gem/i915_gem_userptr.o \ gem/i915_gem_wait.o \ gem/i915_gemfs.o diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.c new file mode 100644 index 000000000000..5f347a85bf44 --- /dev/null +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.c @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2021 Intel Corporation + */ + +/** + * DOC: Usage and intentions. + * + * This file contains functionality that we might want to move into + * ttm_bo_util.c if there is a common interest. + * Currently a kmap_local only memcpy with support for page-based iomem regions, + * and fast memcpy from write-combined memory. + */ + +#include +#include +#include +#include + +#include "i915_memcpy.h" + +#include "gem/i915_gem_ttm_bo_util.h" + +static void i915_ttm_kmap_iter_tt_kmap_local(struct i915_ttm_kmap_iter *iter, + struct dma_buf_map *dmap, + pgoff_t i) +{ + struct i915_ttm_kmap_iter_tt *iter_tt = + container_of(iter, typeof(*iter_tt), base); + + dma_buf_map_set_vaddr(dmap, kmap_local_page(iter_tt->tt->pages[i])); +} + +static void i915_ttm_kmap_iter_iomap_kmap_local(struct i915_ttm_kmap_iter *iter, + struct dma_buf_map *dmap, + pgoff_t i) +{ + struct i915_ttm_kmap_iter_iomap *iter_io = + container_of(iter, typeof(*iter_io), base); + void __iomem *addr; + +retry: + while (i >= iter_io->cache.end) { + iter_io->cache.sg = iter_io->cache.sg ? + sg_next(iter_io->cache.sg) : iter_io->st->sgl; + iter_io->cache.i = iter_io->cache.end; + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >> + PAGE_SHIFT; + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) - + iter_io->start; + } + + if (i < iter_io->cache.i) { + iter_io->cache.end = 0; + iter_io->cache.sg = NULL; + goto retry; + } + + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs + + (((resource_size_t)i - iter_io->cache.i) + << PAGE_SHIFT)); + dma_buf_map_set_vaddr_iomem(dmap, addr); +} + +static const struct i915_ttm_kmap_iter_ops i915_ttm_kmap_iter_tt_ops = { + .kmap_local = i915_ttm_kmap_iter_tt_kmap_local +}; + +static const struct i915_ttm_kmap_iter_ops i915_ttm_kmap_iter_io_ops = { + .kmap_local = i915_ttm_kmap_iter_iomap_kmap_local +}; + +static void kunmap_local_dma_buf_map(struct dma_buf_map *map) +{ + if (map->is_iomem) + io_mapping_unmap_local(map->vaddr_iomem); + else + kunmap_local(map->vaddr); +} + +/** + * i915_ttm_move_memcpy - Helper to perform a memcpy ttm move operation. + * @bo: The struct ttm_buffer_object. + * @new_mem: The struct ttm_resource we're moving to (copy destination). + * @new_kmap: A struct i915_ttm_kmap_iter representing the destination resource. + * @old_kmap: A struct i915_ttm_kmap_iter representing the source resource. + */ +void i915_ttm_move_memcpy(struct ttm_buffer_object *bo, + struct ttm_resource *new_mem, + struct i915_ttm_kmap_iter *new_kmap, + struct i915_ttm_kmap_iter *old_kmap) +{ + struct ttm_device *bdev = bo->bdev; + struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type); + struct ttm_tt *ttm = bo->ttm; + struct ttm_resource *old_mem = &bo->mem; + struct ttm_resource old_copy = *old_mem; + struct ttm_resource_manager *old_man = ttm_manager_type(bdev, old_mem->mem_type); + struct dma_buf_map old_map, new_map; + pgoff_t i; + + /* Single TTM move. NOP */ + if (old_man->use_tt && man->use_tt) + goto done; + + /* Don't move nonexistent data. Clear destination instead. */ + if (old_man->use_tt && !man->use_tt && + (!ttm || !ttm_tt_is_populated(ttm))) { + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)) + goto done; + + for (i = 0; i < new_mem->num_pages; ++i) { + new_kmap->ops->kmap_local(new_kmap, &new_map, i); + memset_io(new_map.vaddr_iomem, 0, PAGE_SIZE); + kunmap_local_dma_buf_map(&new_map); + } + goto done; + } + + for (i = 0; i < new_mem->num_pages; ++i) { + new_kmap->ops->kmap_local(new_kmap, &new_map, i); + old_kmap->ops->kmap_local(old_kmap, &old_map, i); + if (!old_map.is_iomem || + !i915_memcpy_from_wc(new_map.vaddr, old_map.vaddr, PAGE_SIZE)) { + if (!old_map.is_iomem) { + dma_buf_map_memcpy_to(&new_map, old_map.vaddr, + PAGE_SIZE); + } else if (!new_map.is_iomem) { + memcpy_fromio(new_map.vaddr, old_map.vaddr_iomem, + PAGE_SIZE); + } else { + pgoff_t j; + u32 __iomem *src = old_map.vaddr_iomem; + u32 __iomem *dst = new_map.vaddr_iomem; + + for (j = 0; j < (PAGE_SIZE >> 2); ++j) + iowrite32(ioread32(src++), dst++); + } + } + kunmap_local_dma_buf_map(&old_map); + kunmap_local_dma_buf_map(&new_map); + } + +done: + old_copy = *old_mem; + + ttm_bo_assign_mem(bo, new_mem); + + if (!man->use_tt) + ttm_bo_tt_destroy(bo); + + ttm_resource_free(bo, &old_copy); +} + +/** + * i915_ttm_kmap_iter_iomap_init - Initialize a struct i915_ttm_kmap_iter_iomap + * @iter_io: The struct i915_ttm_kmap_iter_iomap to initialize. + * @iomap: The struct io_mapping representing the underlying linear io_memory. + * @st: sg_table into @iomap, representing the memory of the struct + * ttm_resource. + * @start: Offset that needs to be subtracted from @st to make + * sg_dma_address(st->sgl) - @start == 0 for @iomap start. + * + * Return: Pointer to the embedded struct i915_ttm_kmap_iter. + */ +struct i915_ttm_kmap_iter * +i915_ttm_kmap_iter_iomap_init(struct i915_ttm_kmap_iter_iomap *iter_io, + struct io_mapping *iomap, + struct sg_table *st, + resource_size_t start) +{ + iter_io->base.ops = &i915_ttm_kmap_iter_io_ops; + iter_io->iomap = iomap; + iter_io->st = st; + iter_io->start = start; + memset(&iter_io->cache, 0, sizeof(iter_io->cache)); + return &iter_io->base; +} + +/** + * ttm_kmap_iter_tt_init - Initialize a struct i915_ttm_kmap_iter_tt + * @iter_tt: The struct i915_ttm_kmap_iter_tt to initialize. + * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource. + * + * Return: Pointer to the embedded struct i915_ttm_kmap_iter. + */ +struct i915_ttm_kmap_iter * +i915_ttm_kmap_iter_tt_init(struct i915_ttm_kmap_iter_tt *iter_tt, + struct ttm_tt *tt) +{ + iter_tt->base.ops = &i915_ttm_kmap_iter_tt_ops; + iter_tt->tt = tt; + return &iter_tt->base; +} diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.h new file mode 100644 index 000000000000..f1c8c7ae7a42 --- /dev/null +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm_bo_util.h @@ -0,0 +1,107 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2021 Intel Corporation + */ + +/* + * This files contains functionality that we might want to move into + * ttm_bo_util.c if there is a common interest. + */ +#ifndef _I915_GEM_TTM_BO_UTIL_H_ +#define _I915_GEM_TTM_BO_UTIL_H_ + +#include +struct dma_buf_map; +struct io_mapping; +struct sg_table; +struct scatterlist; + +struct ttm_tt; +struct i915_ttm_kmap_iter; + +/** + * struct i915_ttm_kmap_iter_ops - Ops structure for a struct + * i915_ttm_kmap_iter. + */ +struct i915_ttm_kmap_iter_ops { + /** + * kmap_local - Map a PAGE_SIZE part of the resource using + * kmap_local semantics. + * @res_kmap: Pointer to the struct i915_ttm_kmap_iter representing + * the resource. + * @dmap: The struct dma_buf_map holding the virtual address after + * the operation. + * @i: The location within the resource to map. PAGE_SIZE granularity. + */ + void (*kmap_local)(struct i915_ttm_kmap_iter *res_kmap, + struct dma_buf_map *dmap, pgoff_t i); +}; + +/** + * struct i915_ttm_kmap_iter - Iterator for kmap_local type operations on a + * resource. + * @ops: Pointer to the operations struct. + * + * This struct is intended to be embedded in a resource-specific specialization + * implementing operations for the resource. + * + * Nothing stops us from extending the operations to vmap, vmap_pfn etc, + * replacing some or parts of the ttm_bo_util. cpu-map functionality. + */ +struct i915_ttm_kmap_iter { + const struct i915_ttm_kmap_iter_ops *ops; +}; + +/** + * struct i915_ttm_kmap_iter_tt - Specialization for a tt (page) backed struct + * ttm_resource. + * @base: Embedded struct i915_ttm_kmap_iter providing the usage interface + * @tt: Cached struct ttm_tt. + */ +struct i915_ttm_kmap_iter_tt { + struct i915_ttm_kmap_iter base; + struct ttm_tt *tt; +}; + +/** + * struct i915_ttm_kmap_iter_iomap - Specialization for a struct io_mapping + + * struct sg_table backed struct ttm_resource. + * @base: Embedded struct i915_ttm_kmap_iter providing the usage interface. + * @iomap: struct io_mapping representing the underlying linear io_memory. + * @st: sg_table into @iomap, representing the memory of the struct ttm_resource. + * @start: Offset that needs to be subtracted from @st to make + * sg_dma_address(st->sgl) - @start == 0 for @iomap start. + * @cache: Scatterlist traversal cache for fast lookups. + * @cache.sg: Pointer to the currently cached scatterlist segment. + * @cache.i: First index of @sg. PAGE_SIZE granularity. + * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity. + * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity. + */ +struct i915_ttm_kmap_iter_iomap { + struct i915_ttm_kmap_iter base; + struct io_mapping *iomap; + struct sg_table *st; + resource_size_t start; + struct { + struct scatterlist *sg; + pgoff_t i; + pgoff_t end; + pgoff_t offs; + } cache; +}; + +void i915_ttm_move_memcpy(struct ttm_buffer_object *bo, + struct ttm_resource *new_mem, + struct i915_ttm_kmap_iter *new_iter, + struct i915_ttm_kmap_iter *old_iter); + +struct i915_ttm_kmap_iter * +i915_ttm_kmap_iter_tt_init(struct i915_ttm_kmap_iter_tt *iter_tt, + struct ttm_tt *tt); + +struct i915_ttm_kmap_iter * +i915_ttm_kmap_iter_iomap_init(struct i915_ttm_kmap_iter_iomap *iter_io, + struct io_mapping *iomap, + struct sg_table *st, + resource_size_t start); +#endif -- 2.31.1