From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joerg Roedel Subject: [PATCH 3/3] mmu_notifier: Add the call-back for mmu_notifier_invalidate_range() Date: Tue, 28 Oct 2014 18:14:00 +0100 Message-ID: <1414516440-910-4-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1414516440-910-1-git-send-email-joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jay.Cornwall-5C7GfCeVMHo@public.gmane.org, John.Bridgman-5C7GfCeVMHo@public.gmane.org, jroedel-l3A5Bk7waGM@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, Jerome Glisse , iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org, Jesse Barnes , David Woodhouse , ben.sander-5C7GfCeVMHo@public.gmane.org List-Id: iommu@lists.linux-foundation.org RnJvbTogSm9lcmcgUm9lZGVsIDxqcm9lZGVsQHN1c2UuZGU+CgpOb3cgdGhhdCB0aGUgbW11X25v dGlmaWVyX2ludmFsaWRhdGVfcmFuZ2UoKSBjYWxscyBhcmUgaW4KcGxhY2UsIGFkZCB0aGUgY2Fs bC1iYWNrIHRvIGFsbG93IHN1YnN5c3RlbXMgdG8gcmVnaXN0ZXIKYWdhaW5zdCBpdC4KClJldmll d2VkLWJ5OiBBbmRyZWEgQXJjYW5nZWxpIDxhYXJjYW5nZUByZWRoYXQuY29tPgpSZXZpZXdlZC1i eTogSsOpcsO0bWUgR2xpc3NlIDxqZ2xpc3NlQHJlZGhhdC5jb20+ClNpZ25lZC1vZmYtYnk6IEpv ZXJnIFJvZWRlbCA8anJvZWRlbEBzdXNlLmRlPgotLS0KIGluY2x1ZGUvbGludXgvbW11X25vdGlm aWVyLmggfCAzNyArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tCiBtbS9tbXVf bm90aWZpZXIuYyAgICAgICAgICAgIHwgMjUgKysrKysrKysrKysrKysrKysrKysrKysrKwogMiBm aWxlcyBjaGFuZ2VkLCA1NyBpbnNlcnRpb25zKCspLCA1IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdp dCBhL2luY2x1ZGUvbGludXgvbW11X25vdGlmaWVyLmggYi9pbmNsdWRlL2xpbnV4L21tdV9ub3Rp Zmllci5oCmluZGV4IDk2NmRhMmIuLjk0ZDE5ZjYgMTAwNjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgv bW11X25vdGlmaWVyLmgKKysrIGIvaW5jbHVkZS9saW51eC9tbXVfbm90aWZpZXIuaApAQCAtOTgs MTEgKzk4LDExIEBAIHN0cnVjdCBtbXVfbm90aWZpZXJfb3BzIHsKIAkvKgogCSAqIGludmFsaWRh dGVfcmFuZ2Vfc3RhcnQoKSBhbmQgaW52YWxpZGF0ZV9yYW5nZV9lbmQoKSBtdXN0IGJlCiAJICog cGFpcmVkIGFuZCBhcmUgY2FsbGVkIG9ubHkgd2hlbiB0aGUgbW1hcF9zZW0gYW5kL29yIHRoZQot CSAqIGxvY2tzIHByb3RlY3RpbmcgdGhlIHJldmVyc2UgbWFwcyBhcmUgaGVsZC4gVGhlIHN1YnN5 c3RlbQotCSAqIG11c3QgZ3VhcmFudGVlIHRoYXQgbm8gYWRkaXRpb25hbCByZWZlcmVuY2VzIGFy ZSB0YWtlbiB0bwotCSAqIHRoZSBwYWdlcyBpbiB0aGUgcmFuZ2UgZXN0YWJsaXNoZWQgYmV0d2Vl biB0aGUgY2FsbCB0bwotCSAqIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQoKSBhbmQgdGhlIG1hdGNo aW5nIGNhbGwgdG8KLQkgKiBpbnZhbGlkYXRlX3JhbmdlX2VuZCgpLgorCSAqIGxvY2tzIHByb3Rl Y3RpbmcgdGhlIHJldmVyc2UgbWFwcyBhcmUgaGVsZC4gSWYgdGhlIHN1YnN5c3RlbQorCSAqIGNh bid0IGd1YXJhbnRlZSB0aGF0IG5vIGFkZGl0aW9uYWwgcmVmZXJlbmNlcyBhcmUgdGFrZW4gdG8K KwkgKiB0aGUgcGFnZXMgaW4gdGhlIHJhbmdlLCBpdCBoYXMgdG8gaW1wbGVtZW50IHRoZQorCSAq IGludmFsaWRhdGVfcmFuZ2UoKSBub3RpZmllciB0byByZW1vdmUgYW55IHJlZmVyZW5jZXMgdGFr ZW4KKwkgKiBhZnRlciBpbnZhbGlkYXRlX3JhbmdlX3N0YXJ0KCkuCiAJICoKIAkgKiBJbnZhbGlk YXRpb24gb2YgbXVsdGlwbGUgY29uY3VycmVudCByYW5nZXMgbWF5IGJlCiAJICogb3B0aW9uYWxs eSBwZXJtaXR0ZWQgYnkgdGhlIGRyaXZlci4gRWl0aGVyIHdheSB0aGUKQEAgLTE0NCw2ICsxNDQs MjkgQEAgc3RydWN0IG1tdV9ub3RpZmllcl9vcHMgewogCXZvaWQgKCppbnZhbGlkYXRlX3Jhbmdl X2VuZCkoc3RydWN0IG1tdV9ub3RpZmllciAqbW4sCiAJCQkJICAgICBzdHJ1Y3QgbW1fc3RydWN0 ICptbSwKIAkJCQkgICAgIHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgZW5kKTsK KworCS8qCisJICogaW52YWxpZGF0ZV9yYW5nZSgpIGlzIGVpdGhlciBjYWxsZWQgYmV0d2Vlbgor CSAqIGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQoKSBhbmQgaW52YWxpZGF0ZV9yYW5nZV9lbmQoKSB3 aGVuIHRoZQorCSAqIFZNIGhhcyB0byBmcmVlIHBhZ2VzIHRoYXQgd2hlcmUgdW5tYXBwZWQsIGJ1 dCBiZWZvcmUgdGhlCisJICogcGFnZXMgYXJlIGFjdHVhbGx5IGZyZWVkLCBvciBvdXRzaWRlIG9m IF9zdGFydCgpL19lbmQoKSB3aGVuCisJICogYSAocmVtb3RlKSBUTEIgaXMgbmVjZXNzYXJ5Lgor CSAqCisJICogSWYgaW52YWxpZGF0ZV9yYW5nZSgpIGlzIHVzZWQgdG8gbWFuYWdlIGEgbm9uLUNQ VSBUTEIgd2l0aAorCSAqIHNoYXJlZCBwYWdlLXRhYmxlcywgaXQgbm90IG5lY2Vzc2FyeSB0byBp bXBsZW1lbnQgdGhlCisJICogaW52YWxpZGF0ZV9yYW5nZV9zdGFydCgpL2VuZCgpIG5vdGlmaWVy cywgYXMKKwkgKiBpbnZhbGlkYXRlX3JhbmdlKCkgYWxyZWFkIGNhdGNoZXMgdGhlIHBvaW50cyBp biB0aW1lIHdoZW4gYW4KKwkgKiBleHRlcm5hbCBUTEIgcmFuZ2UgbmVlZHMgdG8gYmUgZmx1c2hl ZC4KKwkgKgorCSAqIFRoZSBpbnZhbGlkYXRlX3JhbmdlKCkgZnVuY3Rpb24gaXMgY2FsbGVkIHVu ZGVyIHRoZSBwdGwKKwkgKiBzcGluLWxvY2sgYW5kIG5vdCBhbGxvd2VkIHRvIHNsZWVwLgorCSAq CisJICogTm90ZSB0aGF0IHRoaXMgZnVuY3Rpb24gbWlnaHQgYmUgY2FsbGVkIHdpdGgganVzdCBh IHN1Yi1yYW5nZQorCSAqIG9mIHdoYXQgd2FzIHBhc3NlZCB0byBpbnZhbGlkYXRlX3JhbmdlX3N0 YXJ0KCkvZW5kKCksIGlmCisJICogY2FsbGVkIGJldHdlZW4gdGhvc2UgZnVuY3Rpb25zLgorCSAq LworCXZvaWQgKCppbnZhbGlkYXRlX3JhbmdlKShzdHJ1Y3QgbW11X25vdGlmaWVyICptbiwgc3Ry dWN0IG1tX3N0cnVjdCAqbW0sCisJCQkJIHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxv bmcgZW5kKTsKIH07CiAKIC8qCkBAIC0xOTAsNiArMjEzLDggQEAgZXh0ZXJuIHZvaWQgX19tbXVf bm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZV9zdGFydChzdHJ1Y3QgbW1fc3RydWN0ICptbSwKIAkJ CQkgIHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgZW5kKTsKIGV4dGVybiB2b2lk IF9fbW11X25vdGlmaWVyX2ludmFsaWRhdGVfcmFuZ2VfZW5kKHN0cnVjdCBtbV9zdHJ1Y3QgKm1t LAogCQkJCSAgdW5zaWduZWQgbG9uZyBzdGFydCwgdW5zaWduZWQgbG9uZyBlbmQpOworZXh0ZXJu IHZvaWQgX19tbXVfbm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZShzdHJ1Y3QgbW1fc3RydWN0ICpt bSwKKwkJCQkgIHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgZW5kKTsKIAogc3Rh dGljIGlubGluZSB2b2lkIG1tdV9ub3RpZmllcl9yZWxlYXNlKHN0cnVjdCBtbV9zdHJ1Y3QgKm1t KQogewpAQCAtMjQ1LDYgKzI3MCw4IEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBtbXVfbm90aWZpZXJf aW52YWxpZGF0ZV9yYW5nZV9lbmQoc3RydWN0IG1tX3N0cnVjdCAqbW0sCiBzdGF0aWMgaW5saW5l IHZvaWQgbW11X25vdGlmaWVyX2ludmFsaWRhdGVfcmFuZ2Uoc3RydWN0IG1tX3N0cnVjdCAqbW0s CiAJCQkJICB1bnNpZ25lZCBsb25nIHN0YXJ0LCB1bnNpZ25lZCBsb25nIGVuZCkKIHsKKwlpZiAo bW1faGFzX25vdGlmaWVycyhtbSkpCisJCV9fbW11X25vdGlmaWVyX2ludmFsaWRhdGVfcmFuZ2Uo bW0sIHN0YXJ0LCBlbmQpOwogfQogCiBzdGF0aWMgaW5saW5lIHZvaWQgbW11X25vdGlmaWVyX21t X2luaXQoc3RydWN0IG1tX3N0cnVjdCAqbW0pCmRpZmYgLS1naXQgYS9tbS9tbXVfbm90aWZpZXIu YyBiL21tL21tdV9ub3RpZmllci5jCmluZGV4IDJjOGRhOTguLjNiOWIzZDAgMTAwNjQ0Ci0tLSBh L21tL21tdV9ub3RpZmllci5jCisrKyBiL21tL21tdV9ub3RpZmllci5jCkBAIC0xOTMsNiArMTkz LDE2IEBAIHZvaWQgX19tbXVfbm90aWZpZXJfaW52YWxpZGF0ZV9yYW5nZV9lbmQoc3RydWN0IG1t X3N0cnVjdCAqbW0sCiAKIAlpZCA9IHNyY3VfcmVhZF9sb2NrKCZzcmN1KTsKIAlobGlzdF9mb3Jf ZWFjaF9lbnRyeV9yY3UobW4sICZtbS0+bW11X25vdGlmaWVyX21tLT5saXN0LCBobGlzdCkgewor CQkvKgorCQkgKiBDYWxsIGludmFsaWRhdGVfcmFuZ2UgaGVyZSB0b28gdG8gYXZvaWQgdGhlIG5l ZWQgZm9yIHRoZQorCQkgKiBzdWJzeXN0ZW0gb2YgaGF2aW5nIHRvIHJlZ2lzdGVyIGFuIGludmFs aWRhdGVfcmFuZ2VfZW5kCisJCSAqIGNhbGwtYmFjayB3aGVuIHRoZXJlIGlzIGludmFsaWRhdGVf cmFuZ2UgYWxyZWFkeS4gVXN1YWxseSBhCisJCSAqIHN1YnN5c3RlbSByZWdpc3RlcnMgZWl0aGVy IGludmFsaWRhdGVfcmFuZ2Vfc3RhcnQoKS9lbmQoKSBvcgorCQkgKiBpbnZhbGlkYXRlX3Jhbmdl KCksIHNvIHRoaXMgd2lsbCBiZSBubyBhZGRpdGlvbmFsIG92ZXJoZWFkCisJCSAqIChiZXNpZGVz IHRoZSBwb2ludGVyIGNoZWNrKS4KKwkJICovCisJCWlmIChtbi0+b3BzLT5pbnZhbGlkYXRlX3Jh bmdlKQorCQkJbW4tPm9wcy0+aW52YWxpZGF0ZV9yYW5nZShtbiwgbW0sIHN0YXJ0LCBlbmQpOwog CQlpZiAobW4tPm9wcy0+aW52YWxpZGF0ZV9yYW5nZV9lbmQpCiAJCQltbi0+b3BzLT5pbnZhbGlk YXRlX3JhbmdlX2VuZChtbiwgbW0sIHN0YXJ0LCBlbmQpOwogCX0KQEAgLTIwMCw2ICsyMTAsMjEg QEAgdm9pZCBfX21tdV9ub3RpZmllcl9pbnZhbGlkYXRlX3JhbmdlX2VuZChzdHJ1Y3QgbW1fc3Ry dWN0ICptbSwKIH0KIEVYUE9SVF9TWU1CT0xfR1BMKF9fbW11X25vdGlmaWVyX2ludmFsaWRhdGVf cmFuZ2VfZW5kKTsKIAordm9pZCBfX21tdV9ub3RpZmllcl9pbnZhbGlkYXRlX3JhbmdlKHN0cnVj dCBtbV9zdHJ1Y3QgKm1tLAorCQkJCSAgdW5zaWduZWQgbG9uZyBzdGFydCwgdW5zaWduZWQgbG9u ZyBlbmQpCit7CisJc3RydWN0IG1tdV9ub3RpZmllciAqbW47CisJaW50IGlkOworCisJaWQgPSBz cmN1X3JlYWRfbG9jaygmc3JjdSk7CisJaGxpc3RfZm9yX2VhY2hfZW50cnlfcmN1KG1uLCAmbW0t Pm1tdV9ub3RpZmllcl9tbS0+bGlzdCwgaGxpc3QpIHsKKwkJaWYgKG1uLT5vcHMtPmludmFsaWRh dGVfcmFuZ2UpCisJCQltbi0+b3BzLT5pbnZhbGlkYXRlX3JhbmdlKG1uLCBtbSwgc3RhcnQsIGVu ZCk7CisJfQorCXNyY3VfcmVhZF91bmxvY2soJnNyY3UsIGlkKTsKK30KK0VYUE9SVF9TWU1CT0xf R1BMKF9fbW11X25vdGlmaWVyX2ludmFsaWRhdGVfcmFuZ2UpOworCiBzdGF0aWMgaW50IGRvX21t dV9ub3RpZmllcl9yZWdpc3RlcihzdHJ1Y3QgbW11X25vdGlmaWVyICptbiwKIAkJCQkgICAgc3Ry dWN0IG1tX3N0cnVjdCAqbW0sCiAJCQkJICAgIGludCB0YWtlX21tYXBfc2VtKQotLSAKMS44LjQu NQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KaW9tbXUg bWFpbGluZyBsaXN0CmlvbW11QGxpc3RzLmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBzOi8vbGlz dHMubGludXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2lvbW11 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f45.google.com (mail-la0-f45.google.com [209.85.215.45]) by kanga.kvack.org (Postfix) with ESMTP id 2A44C900021 for ; Tue, 28 Oct 2014 13:14:27 -0400 (EDT) Received: by mail-la0-f45.google.com with SMTP id gm9so1037651lab.4 for ; Tue, 28 Oct 2014 10:14:25 -0700 (PDT) Received: from theia.8bytes.org (8bytes.org. [2a01:238:4383:600:38bc:a715:4b6d:a889]) by mx.google.com with ESMTPS id b9si3499895lbp.50.2014.10.28.10.14.20 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Oct 2014 10:14:21 -0700 (PDT) From: Joerg Roedel Subject: [PATCH 3/3] mmu_notifier: Add the call-back for mmu_notifier_invalidate_range() Date: Tue, 28 Oct 2014 18:14:00 +0100 Message-Id: <1414516440-910-4-git-send-email-joro@8bytes.org> In-Reply-To: <1414516440-910-1-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jerome Glisse , Jay.Cornwall@amd.com, Oded.Gabbay@amd.com, John.Bridgman@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Jesse Barnes , David Woodhouse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, jroedel@suse.de, joro@8bytes.org From: Joerg Roedel Now that the mmu_notifier_invalidate_range() calls are in place, add the call-back to allow subsystems to register against it. Reviewed-by: Andrea Arcangeli Reviewed-by: JA(C)rA'me Glisse Signed-off-by: Joerg Roedel --- include/linux/mmu_notifier.h | 37 ++++++++++++++++++++++++++++++++----- mm/mmu_notifier.c | 25 +++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 966da2b..94d19f6 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -98,11 +98,11 @@ struct mmu_notifier_ops { /* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_sem and/or the - * locks protecting the reverse maps are held. The subsystem - * must guarantee that no additional references are taken to - * the pages in the range established between the call to - * invalidate_range_start() and the matching call to - * invalidate_range_end(). + * locks protecting the reverse maps are held. If the subsystem + * can't guarantee that no additional references are taken to + * the pages in the range, it has to implement the + * invalidate_range() notifier to remove any references taken + * after invalidate_range_start(). * * Invalidation of multiple concurrent ranges may be * optionally permitted by the driver. Either way the @@ -144,6 +144,29 @@ struct mmu_notifier_ops { void (*invalidate_range_end)(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end); + + /* + * invalidate_range() is either called between + * invalidate_range_start() and invalidate_range_end() when the + * VM has to free pages that where unmapped, but before the + * pages are actually freed, or outside of _start()/_end() when + * a (remote) TLB is necessary. + * + * If invalidate_range() is used to manage a non-CPU TLB with + * shared page-tables, it not necessary to implement the + * invalidate_range_start()/end() notifiers, as + * invalidate_range() alread catches the points in time when an + * external TLB range needs to be flushed. + * + * The invalidate_range() function is called under the ptl + * spin-lock and not allowed to sleep. + * + * Note that this function might be called with just a sub-range + * of what was passed to invalidate_range_start()/end(), if + * called between those functions. + */ + void (*invalidate_range)(struct mmu_notifier *mn, struct mm_struct *mm, + unsigned long start, unsigned long end); }; /* @@ -190,6 +213,8 @@ extern void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, unsigned long start, unsigned long end); extern void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, unsigned long start, unsigned long end); +extern void __mmu_notifier_invalidate_range(struct mm_struct *mm, + unsigned long start, unsigned long end); static inline void mmu_notifier_release(struct mm_struct *mm) { @@ -245,6 +270,8 @@ static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, unsigned long start, unsigned long end) { + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range(mm, start, end); } static inline void mmu_notifier_mm_init(struct mm_struct *mm) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 2c8da98..3b9b3d0 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -193,6 +193,16 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(mn, &mm->mmu_notifier_mm->list, hlist) { + /* + * Call invalidate_range here too to avoid the need for the + * subsystem of having to register an invalidate_range_end + * call-back when there is invalidate_range already. Usually a + * subsystem registers either invalidate_range_start()/end() or + * invalidate_range(), so this will be no additional overhead + * (besides the pointer check). + */ + if (mn->ops->invalidate_range) + mn->ops->invalidate_range(mn, mm, start, end); if (mn->ops->invalidate_range_end) mn->ops->invalidate_range_end(mn, mm, start, end); } @@ -200,6 +210,21 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_end); +void __mmu_notifier_invalidate_range(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + int id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(mn, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range) + mn->ops->invalidate_range(mn, mm, start, end); + } + srcu_read_unlock(&srcu, id); +} +EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range); + static int do_mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm, int take_mmap_sem) -- 1.8.4.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753962AbaJ1ROv (ORCPT ); Tue, 28 Oct 2014 13:14:51 -0400 Received: from 8bytes.org ([81.169.241.247]:46034 "EHLO theia.8bytes.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753655AbaJ1ROW (ORCPT ); Tue, 28 Oct 2014 13:14:22 -0400 From: Joerg Roedel To: Andrew Morton , Andrea Arcangeli , Peter Zijlstra , Rik van Riel , Hugh Dickins , Mel Gorman , Johannes Weiner Cc: Jerome Glisse , Jay.Cornwall@amd.com, Oded.Gabbay@amd.com, John.Bridgman@amd.com, Suravee.Suthikulpanit@amd.com, ben.sander@amd.com, Jesse Barnes , David Woodhouse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, jroedel@suse.de, joro@8bytes.org Subject: [PATCH 3/3] mmu_notifier: Add the call-back for mmu_notifier_invalidate_range() Date: Tue, 28 Oct 2014 18:14:00 +0100 Message-Id: <1414516440-910-4-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1414516440-910-1-git-send-email-joro@8bytes.org> References: <1414516440-910-1-git-send-email-joro@8bytes.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Joerg Roedel Now that the mmu_notifier_invalidate_range() calls are in place, add the call-back to allow subsystems to register against it. Reviewed-by: Andrea Arcangeli Reviewed-by: Jérôme Glisse Signed-off-by: Joerg Roedel --- include/linux/mmu_notifier.h | 37 ++++++++++++++++++++++++++++++++----- mm/mmu_notifier.c | 25 +++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 966da2b..94d19f6 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -98,11 +98,11 @@ struct mmu_notifier_ops { /* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_sem and/or the - * locks protecting the reverse maps are held. The subsystem - * must guarantee that no additional references are taken to - * the pages in the range established between the call to - * invalidate_range_start() and the matching call to - * invalidate_range_end(). + * locks protecting the reverse maps are held. If the subsystem + * can't guarantee that no additional references are taken to + * the pages in the range, it has to implement the + * invalidate_range() notifier to remove any references taken + * after invalidate_range_start(). * * Invalidation of multiple concurrent ranges may be * optionally permitted by the driver. Either way the @@ -144,6 +144,29 @@ struct mmu_notifier_ops { void (*invalidate_range_end)(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end); + + /* + * invalidate_range() is either called between + * invalidate_range_start() and invalidate_range_end() when the + * VM has to free pages that where unmapped, but before the + * pages are actually freed, or outside of _start()/_end() when + * a (remote) TLB is necessary. + * + * If invalidate_range() is used to manage a non-CPU TLB with + * shared page-tables, it not necessary to implement the + * invalidate_range_start()/end() notifiers, as + * invalidate_range() alread catches the points in time when an + * external TLB range needs to be flushed. + * + * The invalidate_range() function is called under the ptl + * spin-lock and not allowed to sleep. + * + * Note that this function might be called with just a sub-range + * of what was passed to invalidate_range_start()/end(), if + * called between those functions. + */ + void (*invalidate_range)(struct mmu_notifier *mn, struct mm_struct *mm, + unsigned long start, unsigned long end); }; /* @@ -190,6 +213,8 @@ extern void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, unsigned long start, unsigned long end); extern void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, unsigned long start, unsigned long end); +extern void __mmu_notifier_invalidate_range(struct mm_struct *mm, + unsigned long start, unsigned long end); static inline void mmu_notifier_release(struct mm_struct *mm) { @@ -245,6 +270,8 @@ static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, unsigned long start, unsigned long end) { + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range(mm, start, end); } static inline void mmu_notifier_mm_init(struct mm_struct *mm) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 2c8da98..3b9b3d0 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -193,6 +193,16 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(mn, &mm->mmu_notifier_mm->list, hlist) { + /* + * Call invalidate_range here too to avoid the need for the + * subsystem of having to register an invalidate_range_end + * call-back when there is invalidate_range already. Usually a + * subsystem registers either invalidate_range_start()/end() or + * invalidate_range(), so this will be no additional overhead + * (besides the pointer check). + */ + if (mn->ops->invalidate_range) + mn->ops->invalidate_range(mn, mm, start, end); if (mn->ops->invalidate_range_end) mn->ops->invalidate_range_end(mn, mm, start, end); } @@ -200,6 +210,21 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_end); +void __mmu_notifier_invalidate_range(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + int id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(mn, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range) + mn->ops->invalidate_range(mn, mm, start, end); + } + srcu_read_unlock(&srcu, id); +} +EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range); + static int do_mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm, int take_mmap_sem) -- 1.8.4.5