From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joerg Roedel Subject: [PATCH 2/3] mmu_notifier: Call mmu_notifier_invalidate_range() from VMM Date: Tue, 28 Oct 2014 18:13:59 +0100 Message-ID: <1414516440-910-3-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1414516440-910-1-git-send-email-joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jay.Cornwall-5C7GfCeVMHo@public.gmane.org, John.Bridgman-5C7GfCeVMHo@public.gmane.org, jroedel-l3A5Bk7waGM@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, Jerome Glisse , iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org, Jesse Barnes , David Woodhouse , ben.sander-5C7GfCeVMHo@public.gmane.org List-Id: iommu@lists.linux-foundation.org RnJvbTogSm9lcmcgUm9lZGVsIDxqcm9lZGVsQHN1c2UuZGU+CgpBZGQgY2FsbHMgdG8gdGhlIG5l dyBtbXVfbm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZSgpCmZ1bmN0aW9uIHRvIGFsbCBwbGFjZXMg aW4gdGhlIFZNTSB0aGF0IG5lZWQgaXQuCgpSZXZpZXdlZC1ieTogQW5kcmVhIEFyY2FuZ2VsaSA8 YWFyY2FuZ2VAcmVkaGF0LmNvbT4KUmV2aWV3ZWQtYnk6IErDqXLDtG1lIEdsaXNzZSA8amdsaXNz ZUByZWRoYXQuY29tPgpTaWduZWQtb2ZmLWJ5OiBKb2VyZyBSb2VkZWwgPGpyb2VkZWxAc3VzZS5k ZT4KLS0tCiBpbmNsdWRlL2xpbnV4L21tdV9ub3RpZmllci5oIHwgNDEgKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysKIGtlcm5lbC9ldmVudHMvdXByb2Jlcy5jICAgICAg fCAgMiArLQogbW0vZnJlbWFwLmMgICAgICAgICAgICAgICAgICB8ICAyICstCiBtbS9odWdlX21l bW9yeS5jICAgICAgICAgICAgIHwgIDkgKysrKystLS0tCiBtbS9odWdldGxiLmMgICAgICAgICAg ICAgICAgIHwgIDcgKysrKysrLQogbW0va3NtLmMgICAgICAgICAgICAgICAgICAgICB8ICA0ICsr LS0KIG1tL21lbW9yeS5jICAgICAgICAgICAgICAgICAgfCAgMyArKy0KIG1tL21pZ3JhdGUuYyAg ICAgICAgICAgICAgICAgfCAgMyArKy0KIG1tL3JtYXAuYyAgICAgICAgICAgICAgICAgICAgfCAg MiArLQogOSBmaWxlcyBjaGFuZ2VkLCA2MSBpbnNlcnRpb25zKCspLCAxMiBkZWxldGlvbnMoLSkK CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21tdV9ub3RpZmllci5oIGIvaW5jbHVkZS9saW51 eC9tbXVfbm90aWZpZXIuaAppbmRleCAxNzkwNzkwLi45NjZkYTJiIDEwMDY0NAotLS0gYS9pbmNs dWRlL2xpbnV4L21tdV9ub3RpZmllci5oCisrKyBiL2luY2x1ZGUvbGludXgvbW11X25vdGlmaWVy LmgKQEAgLTI4NCw2ICsyODQsNDQgQEAgc3RhdGljIGlubGluZSB2b2lkIG1tdV9ub3RpZmllcl9t bV9kZXN0cm95KHN0cnVjdCBtbV9zdHJ1Y3QgKm1tKQogCV9feW91bmc7CQkJCQkJCVwKIH0pCiAK KyNkZWZpbmUJcHRlcF9jbGVhcl9mbHVzaF9ub3RpZnkoX192bWEsIF9fYWRkcmVzcywgX19wdGVw KQkJXAorKHsJCQkJCQkJCQlcCisJdW5zaWduZWQgbG9uZyBfX19hZGRyID0gX19hZGRyZXNzICYg UEFHRV9NQVNLOwkJCVwKKwlzdHJ1Y3QgbW1fc3RydWN0ICpfX19tbSA9IChfX3ZtYSktPnZtX21t OwkJCVwKKwlwdGVfdCBfX19wdGU7CQkJCQkJCVwKKwkJCQkJCQkJCVwKKwlfX19wdGUgPSBwdGVw X2NsZWFyX2ZsdXNoKF9fdm1hLCBfX2FkZHJlc3MsIF9fcHRlcCk7CQlcCisJbW11X25vdGlmaWVy X2ludmFsaWRhdGVfcmFuZ2UoX19fbW0sIF9fX2FkZHIsCQkJXAorCQkJCQlfX19hZGRyICsgUEFH RV9TSVpFKTsJCVwKKwkJCQkJCQkJCVwKKwlfX19wdGU7CQkJCQkJCQlcCit9KQorCisjZGVmaW5l IHBtZHBfY2xlYXJfZmx1c2hfbm90aWZ5KF9fdm1hLCBfX2hhZGRyLCBfX3BtZCkJCQlcCisoewkJ CQkJCQkJCVwKKwl1bnNpZ25lZCBsb25nIF9fX2hhZGRyID0gX19oYWRkciAmIEhQQUdFX1BNRF9N QVNLOwkJXAorCXN0cnVjdCBtbV9zdHJ1Y3QgKl9fX21tID0gKF9fdm1hKS0+dm1fbW07CQkJXAor CXBtZF90IF9fX3BtZDsJCQkJCQkJXAorCQkJCQkJCQkJXAorCV9fX3BtZCA9IHBtZHBfY2xlYXJf Zmx1c2goX192bWEsIF9faGFkZHIsIF9fcG1kKTsJCVwKKwltbXVfbm90aWZpZXJfaW52YWxpZGF0 ZV9yYW5nZShfX19tbSwgX19faGFkZHIsCQkJXAorCQkJCSAgICAgIF9fX2hhZGRyICsgSFBBR0Vf UE1EX1NJWkUpOwlcCisJCQkJCQkJCQlcCisJX19fcG1kOwkJCQkJCQkJXAorfSkKKworI2RlZmlu ZSBwbWRwX2dldF9hbmRfY2xlYXJfbm90aWZ5KF9fbW0sIF9faGFkZHIsIF9fcG1kKQkJCVwKKyh7 CQkJCQkJCQkJXAorCXVuc2lnbmVkIGxvbmcgX19faGFkZHIgPSBfX2hhZGRyICYgSFBBR0VfUE1E X01BU0s7CQlcCisJcG1kX3QgX19fcG1kOwkJCQkJCQlcCisJCQkJCQkJCQlcCisJX19fcG1kID0g cG1kcF9nZXRfYW5kX2NsZWFyKF9fbW0sIF9faGFkZHIsIF9fcG1kKTsJCVwKKwltbXVfbm90aWZp ZXJfaW52YWxpZGF0ZV9yYW5nZShfX21tLCBfX19oYWRkciwJCQlcCisJCQkJICAgICAgX19faGFk ZHIgKyBIUEFHRV9QTURfU0laRSk7CVwKKwkJCQkJCQkJCVwKKwlfX19wbWQ7CQkJCQkJCQlcCit9 KQorCiAvKgogICogc2V0X3B0ZV9hdF9ub3RpZnkoKSBzZXRzIHRoZSBwdGUgX2FmdGVyXyBydW5u aW5nIHRoZSBub3RpZmllci4KICAqIFRoaXMgaXMgc2FmZSB0byBzdGFydCBieSB1cGRhdGluZyB0 aGUgc2Vjb25kYXJ5IE1NVXMsIGJlY2F1c2UgdGhlIHByaW1hcnkgTU1VCkBAIC0zNjIsNiArNDAw LDkgQEAgc3RhdGljIGlubGluZSB2b2lkIG1tdV9ub3RpZmllcl9tbV9kZXN0cm95KHN0cnVjdCBt bV9zdHJ1Y3QgKm1tKQogCiAjZGVmaW5lIHB0ZXBfY2xlYXJfZmx1c2hfeW91bmdfbm90aWZ5IHB0 ZXBfY2xlYXJfZmx1c2hfeW91bmcKICNkZWZpbmUgcG1kcF9jbGVhcl9mbHVzaF95b3VuZ19ub3Rp ZnkgcG1kcF9jbGVhcl9mbHVzaF95b3VuZworI2RlZmluZQlwdGVwX2NsZWFyX2ZsdXNoX25vdGlm eSBwdGVwX2NsZWFyX2ZsdXNoCisjZGVmaW5lIHBtZHBfY2xlYXJfZmx1c2hfbm90aWZ5IHBtZHBf Y2xlYXJfZmx1c2gKKyNkZWZpbmUgcG1kcF9nZXRfYW5kX2NsZWFyX25vdGlmeSBwbWRwX2dldF9h bmRfY2xlYXIKICNkZWZpbmUgc2V0X3B0ZV9hdF9ub3RpZnkgc2V0X3B0ZV9hdAogCiAjZW5kaWYg LyogQ09ORklHX01NVV9OT1RJRklFUiAqLwpkaWZmIC0tZ2l0IGEva2VybmVsL2V2ZW50cy91cHJv YmVzLmMgYi9rZXJuZWwvZXZlbnRzL3Vwcm9iZXMuYwppbmRleCAxZDBhZjhhLi5iYzE0M2NmIDEw MDY0NAotLS0gYS9rZXJuZWwvZXZlbnRzL3Vwcm9iZXMuYworKysgYi9rZXJuZWwvZXZlbnRzL3Vw cm9iZXMuYwpAQCAtMTkzLDcgKzE5Myw3IEBAIHN0YXRpYyBpbnQgX19yZXBsYWNlX3BhZ2Uoc3Ry dWN0IHZtX2FyZWFfc3RydWN0ICp2bWEsIHVuc2lnbmVkIGxvbmcgYWRkciwKIAl9CiAKIAlmbHVz aF9jYWNoZV9wYWdlKHZtYSwgYWRkciwgcHRlX3BmbigqcHRlcCkpOwotCXB0ZXBfY2xlYXJfZmx1 c2godm1hLCBhZGRyLCBwdGVwKTsKKwlwdGVwX2NsZWFyX2ZsdXNoX25vdGlmeSh2bWEsIGFkZHIs IHB0ZXApOwogCXNldF9wdGVfYXRfbm90aWZ5KG1tLCBhZGRyLCBwdGVwLCBta19wdGUoa3BhZ2Us IHZtYS0+dm1fcGFnZV9wcm90KSk7CiAKIAlwYWdlX3JlbW92ZV9ybWFwKHBhZ2UpOwpkaWZmIC0t Z2l0IGEvbW0vZnJlbWFwLmMgYi9tbS9mcmVtYXAuYwppbmRleCA3MmI4ZmEzLi45MTI5MDEzIDEw MDY0NAotLS0gYS9tbS9mcmVtYXAuYworKysgYi9tbS9mcmVtYXAuYwpAQCAtMzcsNyArMzcsNyBA QCBzdGF0aWMgdm9pZCB6YXBfcHRlKHN0cnVjdCBtbV9zdHJ1Y3QgKm1tLCBzdHJ1Y3Qgdm1fYXJl YV9zdHJ1Y3QgKnZtYSwKIAogCWlmIChwdGVfcHJlc2VudChwdGUpKSB7CiAJCWZsdXNoX2NhY2hl X3BhZ2Uodm1hLCBhZGRyLCBwdGVfcGZuKHB0ZSkpOwotCQlwdGUgPSBwdGVwX2NsZWFyX2ZsdXNo KHZtYSwgYWRkciwgcHRlcCk7CisJCXB0ZSA9IHB0ZXBfY2xlYXJfZmx1c2hfbm90aWZ5KHZtYSwg YWRkciwgcHRlcCk7CiAJCXBhZ2UgPSB2bV9ub3JtYWxfcGFnZSh2bWEsIGFkZHIsIHB0ZSk7CiAJ CWlmIChwYWdlKSB7CiAJCQlpZiAocHRlX2RpcnR5KHB0ZSkpCmRpZmYgLS1naXQgYS9tbS9odWdl X21lbW9yeS5jIGIvbW0vaHVnZV9tZW1vcnkuYwppbmRleCA3NGM3OGFhLi5lZjMyMGFmIDEwMDY0 NAotLS0gYS9tbS9odWdlX21lbW9yeS5jCisrKyBiL21tL2h1Z2VfbWVtb3J5LmMKQEAgLTEwMzYs NyArMTAzNiw3IEBAIHN0YXRpYyBpbnQgZG9faHVnZV9wbWRfd3BfcGFnZV9mYWxsYmFjayhzdHJ1 Y3QgbW1fc3RydWN0ICptbSwKIAkJZ290byBvdXRfZnJlZV9wYWdlczsKIAlWTV9CVUdfT05fUEFH RSghUGFnZUhlYWQocGFnZSksIHBhZ2UpOwogCi0JcG1kcF9jbGVhcl9mbHVzaCh2bWEsIGhhZGRy LCBwbWQpOworCXBtZHBfY2xlYXJfZmx1c2hfbm90aWZ5KHZtYSwgaGFkZHIsIHBtZCk7CiAJLyog bGVhdmUgcG1kIGVtcHR5IHVudGlsIHB0ZSBpcyBmaWxsZWQgKi8KIAogCXBndGFibGUgPSBwZ3Rh YmxlX3RyYW5zX2h1Z2Vfd2l0aGRyYXcobW0sIHBtZCk7CkBAIC0xMTc5LDcgKzExNzksNyBAQCBh bGxvYzoKIAkJcG1kX3QgZW50cnk7CiAJCWVudHJ5ID0gbWtfaHVnZV9wbWQobmV3X3BhZ2UsIHZt YS0+dm1fcGFnZV9wcm90KTsKIAkJZW50cnkgPSBtYXliZV9wbWRfbWt3cml0ZShwbWRfbWtkaXJ0 eShlbnRyeSksIHZtYSk7Ci0JCXBtZHBfY2xlYXJfZmx1c2godm1hLCBoYWRkciwgcG1kKTsKKwkJ cG1kcF9jbGVhcl9mbHVzaF9ub3RpZnkodm1hLCBoYWRkciwgcG1kKTsKIAkJcGFnZV9hZGRfbmV3 X2Fub25fcm1hcChuZXdfcGFnZSwgdm1hLCBoYWRkcik7CiAJCW1lbV9jZ3JvdXBfY29tbWl0X2No YXJnZShuZXdfcGFnZSwgbWVtY2csIGZhbHNlKTsKIAkJbHJ1X2NhY2hlX2FkZF9hY3RpdmVfb3Jf dW5ldmljdGFibGUobmV3X3BhZ2UsIHZtYSk7CkBAIC0xNTEyLDcgKzE1MTIsNyBAQCBpbnQgY2hh bmdlX2h1Z2VfcG1kKHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqdm1hLCBwbWRfdCAqcG1kLAogCQlw bWRfdCBlbnRyeTsKIAkJcmV0ID0gMTsKIAkJaWYgKCFwcm90X251bWEpIHsKLQkJCWVudHJ5ID0g cG1kcF9nZXRfYW5kX2NsZWFyKG1tLCBhZGRyLCBwbWQpOworCQkJZW50cnkgPSBwbWRwX2dldF9h bmRfY2xlYXJfbm90aWZ5KG1tLCBhZGRyLCBwbWQpOwogCQkJaWYgKHBtZF9udW1hKGVudHJ5KSkK IAkJCQllbnRyeSA9IHBtZF9ta25vbm51bWEoZW50cnkpOwogCQkJZW50cnkgPSBwbWRfbW9kaWZ5 KGVudHJ5LCBuZXdwcm90KTsKQEAgLTE2NDQsNiArMTY0NCw3IEBAIHN0YXRpYyBpbnQgX19zcGxp dF9odWdlX3BhZ2Vfc3BsaXR0aW5nKHN0cnVjdCBwYWdlICpwYWdlLAogCQkgKiBzZXJpYWxpemUg YWdhaW5zdCBzcGxpdF9odWdlX3BhZ2UqLgogCQkgKi8KIAkJcG1kcF9zcGxpdHRpbmdfZmx1c2go dm1hLCBhZGRyZXNzLCBwbWQpOworCiAJCXJldCA9IDE7CiAJCXNwaW5fdW5sb2NrKHB0bCk7CiAJ fQpAQCAtMjgzMyw3ICsyODM0LDcgQEAgc3RhdGljIHZvaWQgX19zcGxpdF9odWdlX3plcm9fcGFn ZV9wbWQoc3RydWN0IHZtX2FyZWFfc3RydWN0ICp2bWEsCiAJcG1kX3QgX3BtZDsKIAlpbnQgaTsK IAotCXBtZHBfY2xlYXJfZmx1c2godm1hLCBoYWRkciwgcG1kKTsKKwlwbWRwX2NsZWFyX2ZsdXNo X25vdGlmeSh2bWEsIGhhZGRyLCBwbWQpOwogCS8qIGxlYXZlIHBtZCBlbXB0eSB1bnRpbCBwdGUg aXMgZmlsbGVkICovCiAKIAlwZ3RhYmxlID0gcGd0YWJsZV90cmFuc19odWdlX3dpdGhkcmF3KG1t LCBwbWQpOwpkaWZmIC0tZ2l0IGEvbW0vaHVnZXRsYi5jIGIvbW0vaHVnZXRsYi5jCmluZGV4IDlm ZDcyMjcuLjJlNmFkZDAgMTAwNjQ0Ci0tLSBhL21tL2h1Z2V0bGIuYworKysgYi9tbS9odWdldGxi LmMKQEAgLTI1OTgsOCArMjU5OCwxMSBAQCBpbnQgY29weV9odWdldGxiX3BhZ2VfcmFuZ2Uoc3Ry dWN0IG1tX3N0cnVjdCAqZHN0LCBzdHJ1Y3QgbW1fc3RydWN0ICpzcmMsCiAJCQl9CiAJCQlzZXRf aHVnZV9wdGVfYXQoZHN0LCBhZGRyLCBkc3RfcHRlLCBlbnRyeSk7CiAJCX0gZWxzZSB7Ci0JCQlp ZiAoY293KQorCQkJaWYgKGNvdykgewogCQkJCWh1Z2VfcHRlcF9zZXRfd3Jwcm90ZWN0KHNyYywg YWRkciwgc3JjX3B0ZSk7CisJCQkJbW11X25vdGlmaWVyX2ludmFsaWRhdGVfcmFuZ2Uoc3JjLCBt bXVuX3N0YXJ0LAorCQkJCQkJCQkgICBtbXVuX2VuZCk7CisJCQl9CiAJCQllbnRyeSA9IGh1Z2Vf cHRlcF9nZXQoc3JjX3B0ZSk7CiAJCQlwdGVwYWdlID0gcHRlX3BhZ2UoZW50cnkpOwogCQkJZ2V0 X3BhZ2UocHRlcGFnZSk7CkBAIC0yODk5LDYgKzI5MDIsNyBAQCByZXRyeV9hdm9pZGNvcHk6CiAK IAkJLyogQnJlYWsgQ09XICovCiAJCWh1Z2VfcHRlcF9jbGVhcl9mbHVzaCh2bWEsIGFkZHJlc3Ms IHB0ZXApOworCQltbXVfbm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZShtbSwgbW11bl9zdGFydCwg bW11bl9lbmQpOwogCQlzZXRfaHVnZV9wdGVfYXQobW0sIGFkZHJlc3MsIHB0ZXAsCiAJCQkJbWFr ZV9odWdlX3B0ZSh2bWEsIG5ld19wYWdlLCAxKSk7CiAJCXBhZ2VfcmVtb3ZlX3JtYXAob2xkX3Bh Z2UpOwpAQCAtMzM3NCw2ICszMzc4LDcgQEAgdW5zaWduZWQgbG9uZyBodWdldGxiX2NoYW5nZV9w cm90ZWN0aW9uKHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqdm1hLAogCSAqIGFuZCB0aGF0IHBhZ2Ug dGFibGUgYmUgcmV1c2VkIGFuZCBmaWxsZWQgd2l0aCBqdW5rLgogCSAqLwogCWZsdXNoX3RsYl9y YW5nZSh2bWEsIHN0YXJ0LCBlbmQpOworCW1tdV9ub3RpZmllcl9pbnZhbGlkYXRlX3JhbmdlKG1t LCBzdGFydCwgZW5kKTsKIAltdXRleF91bmxvY2soJnZtYS0+dm1fZmlsZS0+Zl9tYXBwaW5nLT5p X21tYXBfbXV0ZXgpOwogCW1tdV9ub3RpZmllcl9pbnZhbGlkYXRlX3JhbmdlX2VuZChtbSwgc3Rh cnQsIGVuZCk7CiAKZGlmZiAtLWdpdCBhL21tL2tzbS5jIGIvbW0va3NtLmMKaW5kZXggNmIyZTMz Ny4uZDI0N2VmYSAxMDA2NDQKLS0tIGEvbW0va3NtLmMKKysrIGIvbW0va3NtLmMKQEAgLTg5Miw3 ICs4OTIsNyBAQCBzdGF0aWMgaW50IHdyaXRlX3Byb3RlY3RfcGFnZShzdHJ1Y3Qgdm1fYXJlYV9z dHJ1Y3QgKnZtYSwgc3RydWN0IHBhZ2UgKnBhZ2UsCiAJCSAqIHRoaXMgYXNzdXJlIHVzIHRoYXQg bm8gT19ESVJFQ1QgY2FuIGhhcHBlbiBhZnRlciB0aGUgY2hlY2sKIAkJICogb3IgaW4gdGhlIG1p ZGRsZSBvZiB0aGUgY2hlY2suCiAJCSAqLwotCQllbnRyeSA9IHB0ZXBfY2xlYXJfZmx1c2godm1h LCBhZGRyLCBwdGVwKTsKKwkJZW50cnkgPSBwdGVwX2NsZWFyX2ZsdXNoX25vdGlmeSh2bWEsIGFk ZHIsIHB0ZXApOwogCQkvKgogCQkgKiBDaGVjayB0aGF0IG5vIE9fRElSRUNUIG9yIHNpbWlsYXIg SS9PIGlzIGluIHByb2dyZXNzIG9uIHRoZQogCQkgKiBwYWdlCkBAIC05NjAsNyArOTYwLDcgQEAg c3RhdGljIGludCByZXBsYWNlX3BhZ2Uoc3RydWN0IHZtX2FyZWFfc3RydWN0ICp2bWEsIHN0cnVj dCBwYWdlICpwYWdlLAogCXBhZ2VfYWRkX2Fub25fcm1hcChrcGFnZSwgdm1hLCBhZGRyKTsKIAog CWZsdXNoX2NhY2hlX3BhZ2Uodm1hLCBhZGRyLCBwdGVfcGZuKCpwdGVwKSk7Ci0JcHRlcF9jbGVh cl9mbHVzaCh2bWEsIGFkZHIsIHB0ZXApOworCXB0ZXBfY2xlYXJfZmx1c2hfbm90aWZ5KHZtYSwg YWRkciwgcHRlcCk7CiAJc2V0X3B0ZV9hdF9ub3RpZnkobW0sIGFkZHIsIHB0ZXAsIG1rX3B0ZShr cGFnZSwgdm1hLT52bV9wYWdlX3Byb3QpKTsKIAogCXBhZ2VfcmVtb3ZlX3JtYXAocGFnZSk7CmRp ZmYgLS1naXQgYS9tbS9tZW1vcnkuYyBiL21tL21lbW9yeS5jCmluZGV4IDFjYzZiZmIuLmMyODdk NGMgMTAwNjQ0Ci0tLSBhL21tL21lbW9yeS5jCisrKyBiL21tL21lbW9yeS5jCkBAIC0yMzgsNiAr MjM4LDcgQEAgc3RhdGljIHZvaWQgdGxiX2ZsdXNoX21tdV90bGJvbmx5KHN0cnVjdCBtbXVfZ2F0 aGVyICp0bGIpCiB7CiAJdGxiLT5uZWVkX2ZsdXNoID0gMDsKIAl0bGJfZmx1c2godGxiKTsKKwlt bXVfbm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZSh0bGItPm1tLCB0bGItPnN0YXJ0LCB0bGItPmVu ZCk7CiAjaWZkZWYgQ09ORklHX0hBVkVfUkNVX1RBQkxFX0ZSRUUKIAl0bGJfdGFibGVfZmx1c2go dGxiKTsKICNlbmRpZgpAQCAtMjIzMyw3ICsyMjM0LDcgQEAgZ290dGVuOgogCQkgKiBzZWVuIGlu IHRoZSBwcmVzZW5jZSBvZiBvbmUgdGhyZWFkIGRvaW5nIFNNQyBhbmQgYW5vdGhlcgogCQkgKiB0 aHJlYWQgZG9pbmcgQ09XLgogCQkgKi8KLQkJcHRlcF9jbGVhcl9mbHVzaCh2bWEsIGFkZHJlc3Ms IHBhZ2VfdGFibGUpOworCQlwdGVwX2NsZWFyX2ZsdXNoX25vdGlmeSh2bWEsIGFkZHJlc3MsIHBh Z2VfdGFibGUpOwogCQlwYWdlX2FkZF9uZXdfYW5vbl9ybWFwKG5ld19wYWdlLCB2bWEsIGFkZHJl c3MpOwogCQltZW1fY2dyb3VwX2NvbW1pdF9jaGFyZ2UobmV3X3BhZ2UsIG1lbWNnLCBmYWxzZSk7 CiAJCWxydV9jYWNoZV9hZGRfYWN0aXZlX29yX3VuZXZpY3RhYmxlKG5ld19wYWdlLCB2bWEpOwpk aWZmIC0tZ2l0IGEvbW0vbWlncmF0ZS5jIGIvbW0vbWlncmF0ZS5jCmluZGV4IDAxNDM5OTUuLjQx OTQ1Y2IgMTAwNjQ0Ci0tLSBhL21tL21pZ3JhdGUuYworKysgYi9tbS9taWdyYXRlLmMKQEAgLTE4 NTQsNyArMTg1NCw3IEBAIGZhaWxfcHV0YmFjazoKIAkgKi8KIAlmbHVzaF9jYWNoZV9yYW5nZSh2 bWEsIG1tdW5fc3RhcnQsIG1tdW5fZW5kKTsKIAlwYWdlX2FkZF9hbm9uX3JtYXAobmV3X3BhZ2Us IHZtYSwgbW11bl9zdGFydCk7Ci0JcG1kcF9jbGVhcl9mbHVzaCh2bWEsIG1tdW5fc3RhcnQsIHBt ZCk7CisJcG1kcF9jbGVhcl9mbHVzaF9ub3RpZnkodm1hLCBtbXVuX3N0YXJ0LCBwbWQpOwogCXNl dF9wbWRfYXQobW0sIG1tdW5fc3RhcnQsIHBtZCwgZW50cnkpOwogCWZsdXNoX3RsYl9yYW5nZSh2 bWEsIG1tdW5fc3RhcnQsIG1tdW5fZW5kKTsKIAl1cGRhdGVfbW11X2NhY2hlX3BtZCh2bWEsIGFk ZHJlc3MsICZlbnRyeSk7CkBAIC0xODYyLDYgKzE4NjIsNyBAQCBmYWlsX3B1dGJhY2s6CiAJaWYg KHBhZ2VfY291bnQocGFnZSkgIT0gMikgewogCQlzZXRfcG1kX2F0KG1tLCBtbXVuX3N0YXJ0LCBw bWQsIG9yaWdfZW50cnkpOwogCQlmbHVzaF90bGJfcmFuZ2Uodm1hLCBtbXVuX3N0YXJ0LCBtbXVu X2VuZCk7CisJCW1tdV9ub3RpZmllcl9pbnZhbGlkYXRlX3JhbmdlKG1tLCBtbXVuX3N0YXJ0LCBt bXVuX2VuZCk7CiAJCXVwZGF0ZV9tbXVfY2FjaGVfcG1kKHZtYSwgYWRkcmVzcywgJmVudHJ5KTsK IAkJcGFnZV9yZW1vdmVfcm1hcChuZXdfcGFnZSk7CiAJCWdvdG8gZmFpbF9wdXRiYWNrOwpkaWZm IC0tZ2l0IGEvbW0vcm1hcC5jIGIvbW0vcm1hcC5jCmluZGV4IDExNmE1MDUuLmZkYjgwNTUgMTAw NjQ0Ci0tLSBhL21tL3JtYXAuYworKysgYi9tbS9ybWFwLmMKQEAgLTEzNjQsNyArMTM2NCw3IEBA IHN0YXRpYyBpbnQgdHJ5X3RvX3VubWFwX2NsdXN0ZXIodW5zaWduZWQgbG9uZyBjdXJzb3IsIHVu c2lnbmVkIGludCAqbWFwY291bnQsCiAKIAkJLyogTnVrZSB0aGUgcGFnZSB0YWJsZSBlbnRyeS4g Ki8KIAkJZmx1c2hfY2FjaGVfcGFnZSh2bWEsIGFkZHJlc3MsIHB0ZV9wZm4oKnB0ZSkpOwotCQlw dGV2YWwgPSBwdGVwX2NsZWFyX2ZsdXNoKHZtYSwgYWRkcmVzcywgcHRlKTsKKwkJcHRldmFsID0g cHRlcF9jbGVhcl9mbHVzaF9ub3RpZnkodm1hLCBhZGRyZXNzLCBwdGUpOwogCiAJCS8qIElmIG5v bmxpbmVhciwgc3RvcmUgdGhlIGZpbGUgcGFnZSBvZmZzZXQgaW4gdGhlIHB0ZS4gKi8KIAkJaWYg KHBhZ2UtPmluZGV4ICE9IGxpbmVhcl9wYWdlX2luZGV4KHZtYSwgYWRkcmVzcykpIHsKLS0gCjEu OC40LjUKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmlv bW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczov L2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby9pb21tdQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f170.google.com (mail-lb0-f170.google.com [209.85.217.170]) by kanga.kvack.org (Postfix) with ESMTP id BD59A900021 for ; Tue, 28 Oct 2014 13:14:25 -0400 (EDT) Received: by mail-lb0-f170.google.com with SMTP id 10so1027516lbg.29 for ; Tue, 28 Oct 2014 10:14:24 -0700 (PDT) Received: from theia.8bytes.org (8bytes.org. [81.169.241.247]) by mx.google.com with ESMTPS id q5si3452798lbp.93.2014.10.28.10.14.20 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Oct 2014 10:14:21 -0700 (PDT) From: Joerg Roedel Subject: [PATCH 2/3] mmu_notifier: Call mmu_notifier_invalidate_range() from VMM Date: Tue, 28 Oct 2014 18:13:59 +0100 Message-Id: <1414516440-910-3-git-send-email-joro@8bytes.org> In-Reply-To: <1414516440-910-1-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jerome Glisse , Jay.Cornwall@amd.com, Oded.Gabbay@amd.com, John.Bridgman@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Jesse Barnes , David Woodhouse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, jroedel@suse.de, joro@8bytes.org From: Joerg Roedel Add calls to the new mmu_notifier_invalidate_range() function to all places in the VMM that need it. Reviewed-by: Andrea Arcangeli Reviewed-by: JA(C)rA'me Glisse Signed-off-by: Joerg Roedel --- include/linux/mmu_notifier.h | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/events/uprobes.c | 2 +- mm/fremap.c | 2 +- mm/huge_memory.c | 9 +++++---- mm/hugetlb.c | 7 ++++++- mm/ksm.c | 4 ++-- mm/memory.c | 3 ++- mm/migrate.c | 3 ++- mm/rmap.c | 2 +- 9 files changed, 61 insertions(+), 12 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 1790790..966da2b 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -284,6 +284,44 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) __young; \ }) +#define ptep_clear_flush_notify(__vma, __address, __ptep) \ +({ \ + unsigned long ___addr = __address & PAGE_MASK; \ + struct mm_struct *___mm = (__vma)->vm_mm; \ + pte_t ___pte; \ + \ + ___pte = ptep_clear_flush(__vma, __address, __ptep); \ + mmu_notifier_invalidate_range(___mm, ___addr, \ + ___addr + PAGE_SIZE); \ + \ + ___pte; \ +}) + +#define pmdp_clear_flush_notify(__vma, __haddr, __pmd) \ +({ \ + unsigned long ___haddr = __haddr & HPAGE_PMD_MASK; \ + struct mm_struct *___mm = (__vma)->vm_mm; \ + pmd_t ___pmd; \ + \ + ___pmd = pmdp_clear_flush(__vma, __haddr, __pmd); \ + mmu_notifier_invalidate_range(___mm, ___haddr, \ + ___haddr + HPAGE_PMD_SIZE); \ + \ + ___pmd; \ +}) + +#define pmdp_get_and_clear_notify(__mm, __haddr, __pmd) \ +({ \ + unsigned long ___haddr = __haddr & HPAGE_PMD_MASK; \ + pmd_t ___pmd; \ + \ + ___pmd = pmdp_get_and_clear(__mm, __haddr, __pmd); \ + mmu_notifier_invalidate_range(__mm, ___haddr, \ + ___haddr + HPAGE_PMD_SIZE); \ + \ + ___pmd; \ +}) + /* * set_pte_at_notify() sets the pte _after_ running the notifier. * This is safe to start by updating the secondary MMUs, because the primary MMU @@ -362,6 +400,9 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) #define ptep_clear_flush_young_notify ptep_clear_flush_young #define pmdp_clear_flush_young_notify pmdp_clear_flush_young +#define ptep_clear_flush_notify ptep_clear_flush +#define pmdp_clear_flush_notify pmdp_clear_flush +#define pmdp_get_and_clear_notify pmdp_get_and_clear #define set_pte_at_notify set_pte_at #endif /* CONFIG_MMU_NOTIFIER */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 1d0af8a..bc143cf 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -193,7 +193,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, } flush_cache_page(vma, addr, pte_pfn(*ptep)); - ptep_clear_flush(vma, addr, ptep); + ptep_clear_flush_notify(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, mk_pte(kpage, vma->vm_page_prot)); page_remove_rmap(page); diff --git a/mm/fremap.c b/mm/fremap.c index 72b8fa3..9129013 100644 --- a/mm/fremap.c +++ b/mm/fremap.c @@ -37,7 +37,7 @@ static void zap_pte(struct mm_struct *mm, struct vm_area_struct *vma, if (pte_present(pte)) { flush_cache_page(vma, addr, pte_pfn(pte)); - pte = ptep_clear_flush(vma, addr, ptep); + pte = ptep_clear_flush_notify(vma, addr, ptep); page = vm_normal_page(vma, addr, pte); if (page) { if (pte_dirty(pte)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 74c78aa..ef320af 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1036,7 +1036,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm, goto out_free_pages; VM_BUG_ON_PAGE(!PageHead(page), page); - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); /* leave pmd empty until pte is filled */ pgtable = pgtable_trans_huge_withdraw(mm, pmd); @@ -1179,7 +1179,7 @@ alloc: pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); page_add_new_anon_rmap(new_page, vma, haddr); mem_cgroup_commit_charge(new_page, memcg, false); lru_cache_add_active_or_unevictable(new_page, vma); @@ -1512,7 +1512,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, pmd_t entry; ret = 1; if (!prot_numa) { - entry = pmdp_get_and_clear(mm, addr, pmd); + entry = pmdp_get_and_clear_notify(mm, addr, pmd); if (pmd_numa(entry)) entry = pmd_mknonnuma(entry); entry = pmd_modify(entry, newprot); @@ -1644,6 +1644,7 @@ static int __split_huge_page_splitting(struct page *page, * serialize against split_huge_page*. */ pmdp_splitting_flush(vma, address, pmd); + ret = 1; spin_unlock(ptl); } @@ -2833,7 +2834,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, pmd_t _pmd; int i; - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); /* leave pmd empty until pte is filled */ pgtable = pgtable_trans_huge_withdraw(mm, pmd); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9fd7227..2e6add0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2598,8 +2598,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_pte_at(dst, addr, dst_pte, entry); } else { - if (cow) + if (cow) { huge_ptep_set_wrprotect(src, addr, src_pte); + mmu_notifier_invalidate_range(src, mmun_start, + mmun_end); + } entry = huge_ptep_get(src_pte); ptepage = pte_page(entry); get_page(ptepage); @@ -2899,6 +2902,7 @@ retry_avoidcopy: /* Break COW */ huge_ptep_clear_flush(vma, address, ptep); + mmu_notifier_invalidate_range(mm, mmun_start, mmun_end); set_huge_pte_at(mm, address, ptep, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page); @@ -3374,6 +3378,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, * and that page table be reused and filled with junk. */ flush_tlb_range(vma, start, end); + mmu_notifier_invalidate_range(mm, start, end); mutex_unlock(&vma->vm_file->f_mapping->i_mmap_mutex); mmu_notifier_invalidate_range_end(mm, start, end); diff --git a/mm/ksm.c b/mm/ksm.c index 6b2e337..d247efa 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -892,7 +892,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, * this assure us that no O_DIRECT can happen after the check * or in the middle of the check. */ - entry = ptep_clear_flush(vma, addr, ptep); + entry = ptep_clear_flush_notify(vma, addr, ptep); /* * Check that no O_DIRECT or similar I/O is in progress on the * page @@ -960,7 +960,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, page_add_anon_rmap(kpage, vma, addr); flush_cache_page(vma, addr, pte_pfn(*ptep)); - ptep_clear_flush(vma, addr, ptep); + ptep_clear_flush_notify(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, mk_pte(kpage, vma->vm_page_prot)); page_remove_rmap(page); diff --git a/mm/memory.c b/mm/memory.c index 1cc6bfb..c287d4c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -238,6 +238,7 @@ static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { tlb->need_flush = 0; tlb_flush(tlb); + mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); #ifdef CONFIG_HAVE_RCU_TABLE_FREE tlb_table_flush(tlb); #endif @@ -2233,7 +2234,7 @@ gotten: * seen in the presence of one thread doing SMC and another * thread doing COW. */ - ptep_clear_flush(vma, address, page_table); + ptep_clear_flush_notify(vma, address, page_table); page_add_new_anon_rmap(new_page, vma, address); mem_cgroup_commit_charge(new_page, memcg, false); lru_cache_add_active_or_unevictable(new_page, vma); diff --git a/mm/migrate.c b/mm/migrate.c index 0143995..41945cb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1854,7 +1854,7 @@ fail_putback: */ flush_cache_range(vma, mmun_start, mmun_end); page_add_anon_rmap(new_page, vma, mmun_start); - pmdp_clear_flush(vma, mmun_start, pmd); + pmdp_clear_flush_notify(vma, mmun_start, pmd); set_pmd_at(mm, mmun_start, pmd, entry); flush_tlb_range(vma, mmun_start, mmun_end); update_mmu_cache_pmd(vma, address, &entry); @@ -1862,6 +1862,7 @@ fail_putback: if (page_count(page) != 2) { set_pmd_at(mm, mmun_start, pmd, orig_entry); flush_tlb_range(vma, mmun_start, mmun_end); + mmu_notifier_invalidate_range(mm, mmun_start, mmun_end); update_mmu_cache_pmd(vma, address, &entry); page_remove_rmap(new_page); goto fail_putback; diff --git a/mm/rmap.c b/mm/rmap.c index 116a505..fdb8055 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1364,7 +1364,7 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount, /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* If nonlinear, store the file page offset in the pte. */ if (page->index != linear_page_index(vma, address)) { -- 1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754031AbaJ1ROw (ORCPT ); Tue, 28 Oct 2014 13:14:52 -0400 Received: from 8bytes.org ([81.169.241.247]:46033 "EHLO theia.8bytes.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753642AbaJ1ROW (ORCPT ); Tue, 28 Oct 2014 13:14:22 -0400 From: Joerg Roedel To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jerome Glisse , Jay.Cornwall@amd.com, Oded.Gabbay@amd.com, John.Bridgman@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Jesse Barnes , David Woodhouse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, jroedel@suse.de, joro@8bytes.org Subject: [PATCH 2/3] mmu_notifier: Call mmu_notifier_invalidate_range() from VMM Date: Tue, 28 Oct 2014 18:13:59 +0100 Message-Id: <1414516440-910-3-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1414516440-910-1-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Joerg Roedel Add calls to the new mmu_notifier_invalidate_range() function to all places in the VMM that need it. Reviewed-by: Andrea Arcangeli Reviewed-by: Jérôme Glisse Signed-off-by: Joerg Roedel --- include/linux/mmu_notifier.h | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/events/uprobes.c | 2 +- mm/fremap.c | 2 +- mm/huge_memory.c | 9 +++++---- mm/hugetlb.c | 7 ++++++- mm/ksm.c | 4 ++-- mm/memory.c | 3 ++- mm/migrate.c | 3 ++- mm/rmap.c | 2 +- 9 files changed, 61 insertions(+), 12 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 1790790..966da2b 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -284,6 +284,44 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) __young; \ }) +#define ptep_clear_flush_notify(__vma, __address, __ptep) \ +({ \ + unsigned long ___addr = __address & PAGE_MASK; \ + struct mm_struct *___mm = (__vma)->vm_mm; \ + pte_t ___pte; \ + \ + ___pte = ptep_clear_flush(__vma, __address, __ptep); \ + mmu_notifier_invalidate_range(___mm, ___addr, \ + ___addr + PAGE_SIZE); \ + \ + ___pte; \ +}) + +#define pmdp_clear_flush_notify(__vma, __haddr, __pmd) \ +({ \ + unsigned long ___haddr = __haddr & HPAGE_PMD_MASK; \ + struct mm_struct *___mm = (__vma)->vm_mm; \ + pmd_t ___pmd; \ + \ + ___pmd = pmdp_clear_flush(__vma, __haddr, __pmd); \ + mmu_notifier_invalidate_range(___mm, ___haddr, \ + ___haddr + HPAGE_PMD_SIZE); \ + \ + ___pmd; \ +}) + +#define pmdp_get_and_clear_notify(__mm, __haddr, __pmd) \ +({ \ + unsigned long ___haddr = __haddr & HPAGE_PMD_MASK; \ + pmd_t ___pmd; \ + \ + ___pmd = pmdp_get_and_clear(__mm, __haddr, __pmd); \ + mmu_notifier_invalidate_range(__mm, ___haddr, \ + ___haddr + HPAGE_PMD_SIZE); \ + \ + ___pmd; \ +}) + /* * set_pte_at_notify() sets the pte _after_ running the notifier. * This is safe to start by updating the secondary MMUs, because the primary MMU @@ -362,6 +400,9 @@ static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) #define ptep_clear_flush_young_notify ptep_clear_flush_young #define pmdp_clear_flush_young_notify pmdp_clear_flush_young +#define ptep_clear_flush_notify ptep_clear_flush +#define pmdp_clear_flush_notify pmdp_clear_flush +#define pmdp_get_and_clear_notify pmdp_get_and_clear #define set_pte_at_notify set_pte_at #endif /* CONFIG_MMU_NOTIFIER */ diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 1d0af8a..bc143cf 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -193,7 +193,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, } flush_cache_page(vma, addr, pte_pfn(*ptep)); - ptep_clear_flush(vma, addr, ptep); + ptep_clear_flush_notify(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, mk_pte(kpage, vma->vm_page_prot)); page_remove_rmap(page); diff --git a/mm/fremap.c b/mm/fremap.c index 72b8fa3..9129013 100644 --- a/mm/fremap.c +++ b/mm/fremap.c @@ -37,7 +37,7 @@ static void zap_pte(struct mm_struct *mm, struct vm_area_struct *vma, if (pte_present(pte)) { flush_cache_page(vma, addr, pte_pfn(pte)); - pte = ptep_clear_flush(vma, addr, ptep); + pte = ptep_clear_flush_notify(vma, addr, ptep); page = vm_normal_page(vma, addr, pte); if (page) { if (pte_dirty(pte)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 74c78aa..ef320af 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1036,7 +1036,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm, goto out_free_pages; VM_BUG_ON_PAGE(!PageHead(page), page); - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); /* leave pmd empty until pte is filled */ pgtable = pgtable_trans_huge_withdraw(mm, pmd); @@ -1179,7 +1179,7 @@ alloc: pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); page_add_new_anon_rmap(new_page, vma, haddr); mem_cgroup_commit_charge(new_page, memcg, false); lru_cache_add_active_or_unevictable(new_page, vma); @@ -1512,7 +1512,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, pmd_t entry; ret = 1; if (!prot_numa) { - entry = pmdp_get_and_clear(mm, addr, pmd); + entry = pmdp_get_and_clear_notify(mm, addr, pmd); if (pmd_numa(entry)) entry = pmd_mknonnuma(entry); entry = pmd_modify(entry, newprot); @@ -1644,6 +1644,7 @@ static int __split_huge_page_splitting(struct page *page, * serialize against split_huge_page*. */ pmdp_splitting_flush(vma, address, pmd); + ret = 1; spin_unlock(ptl); } @@ -2833,7 +2834,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, pmd_t _pmd; int i; - pmdp_clear_flush(vma, haddr, pmd); + pmdp_clear_flush_notify(vma, haddr, pmd); /* leave pmd empty until pte is filled */ pgtable = pgtable_trans_huge_withdraw(mm, pmd); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9fd7227..2e6add0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2598,8 +2598,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_pte_at(dst, addr, dst_pte, entry); } else { - if (cow) + if (cow) { huge_ptep_set_wrprotect(src, addr, src_pte); + mmu_notifier_invalidate_range(src, mmun_start, + mmun_end); + } entry = huge_ptep_get(src_pte); ptepage = pte_page(entry); get_page(ptepage); @@ -2899,6 +2902,7 @@ retry_avoidcopy: /* Break COW */ huge_ptep_clear_flush(vma, address, ptep); + mmu_notifier_invalidate_range(mm, mmun_start, mmun_end); set_huge_pte_at(mm, address, ptep, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page); @@ -3374,6 +3378,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, * and that page table be reused and filled with junk. */ flush_tlb_range(vma, start, end); + mmu_notifier_invalidate_range(mm, start, end); mutex_unlock(&vma->vm_file->f_mapping->i_mmap_mutex); mmu_notifier_invalidate_range_end(mm, start, end); diff --git a/mm/ksm.c b/mm/ksm.c index 6b2e337..d247efa 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -892,7 +892,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, * this assure us that no O_DIRECT can happen after the check * or in the middle of the check. */ - entry = ptep_clear_flush(vma, addr, ptep); + entry = ptep_clear_flush_notify(vma, addr, ptep); /* * Check that no O_DIRECT or similar I/O is in progress on the * page @@ -960,7 +960,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, page_add_anon_rmap(kpage, vma, addr); flush_cache_page(vma, addr, pte_pfn(*ptep)); - ptep_clear_flush(vma, addr, ptep); + ptep_clear_flush_notify(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, mk_pte(kpage, vma->vm_page_prot)); page_remove_rmap(page); diff --git a/mm/memory.c b/mm/memory.c index 1cc6bfb..c287d4c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -238,6 +238,7 @@ static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { tlb->need_flush = 0; tlb_flush(tlb); + mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); #ifdef CONFIG_HAVE_RCU_TABLE_FREE tlb_table_flush(tlb); #endif @@ -2233,7 +2234,7 @@ gotten: * seen in the presence of one thread doing SMC and another * thread doing COW. */ - ptep_clear_flush(vma, address, page_table); + ptep_clear_flush_notify(vma, address, page_table); page_add_new_anon_rmap(new_page, vma, address); mem_cgroup_commit_charge(new_page, memcg, false); lru_cache_add_active_or_unevictable(new_page, vma); diff --git a/mm/migrate.c b/mm/migrate.c index 0143995..41945cb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1854,7 +1854,7 @@ fail_putback: */ flush_cache_range(vma, mmun_start, mmun_end); page_add_anon_rmap(new_page, vma, mmun_start); - pmdp_clear_flush(vma, mmun_start, pmd); + pmdp_clear_flush_notify(vma, mmun_start, pmd); set_pmd_at(mm, mmun_start, pmd, entry); flush_tlb_range(vma, mmun_start, mmun_end); update_mmu_cache_pmd(vma, address, &entry); @@ -1862,6 +1862,7 @@ fail_putback: if (page_count(page) != 2) { set_pmd_at(mm, mmun_start, pmd, orig_entry); flush_tlb_range(vma, mmun_start, mmun_end); + mmu_notifier_invalidate_range(mm, mmun_start, mmun_end); update_mmu_cache_pmd(vma, address, &entry); page_remove_rmap(new_page); goto fail_putback; diff --git a/mm/rmap.c b/mm/rmap.c index 116a505..fdb8055 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1364,7 +1364,7 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount, /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* If nonlinear, store the file page offset in the pte. */ if (page->index != linear_page_index(vma, address)) { -- 1.8.4.5