From mboxrd@z Thu Jan 1 00:00:00 1970 From: "ira.weiny--- via dri-devel" Subject: [PATCH V2 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM Date: Wed, 13 Feb 2019 15:04:49 -0800 Message-ID: <20190213230455.5605-2-ira.weiny@intel.com> References: <20190211201643.7599-1-ira.weiny@intel.com> <20190213230455.5605-1-ira.weiny@intel.com> Reply-To: ira.weiny@intel.com Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20190213230455.5605-1-ira.weiny@intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: linux-mips@vger.kernel.org, linux-kernel@vger.kernel.org, kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org, linux-fpga@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-rdma@vger.kernel.org, linux-media@vger.kernel.org, linux-scsi@vger.kernel.org, devel@driverdev.osuosl.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-fbdev@vger.kernel.org, xen-devel@lists.xenproject.org, devel@lists.orangefs.org, linux-mm@kvack.org, ceph-devel@vger.kernel.org, rds-devel@oss.oracle.com Cc: Martin Brandenburg , Rich Felker , David Hildenbrand , David Airlie , Jason Wang , =?UTF-8?q?Kai=20M=C3=A4kisara?= , Michal Hocko , Ira Weiny , Ben Chan , Rob Springer , Todd Poynor , Yoshinori Sato , "Michael S. Tsirkin" , Jason Gunthorpe , Ingo Molnar , Matt Porter , Wu Hao , Alan Tull , John Hubbard , "James E.J. Bottomley" , Alex Williamson , Stefano Stabellini , Borislav Petkov , Alexander Viro List-Id: ceph-devel.vger.kernel.org RnJvbTogSXJhIFdlaW55IDxpcmEud2VpbnlAaW50ZWwuY29tPgoKUmF0aGVyIHRoYW4gaGF2ZSBh IHNlcGFyYXRlIGdldF91c2VyX3BhZ2VzX2xvbmd0ZXJtKCkgY2FsbCwKaW50cm9kdWNlIEZPTExf TE9OR1RFUk0gYW5kIGNoYW5nZSB0aGUgbG9uZ3Rlcm0gY2FsbGVycyB0byB1c2UKaXQuCgpUaGlz IHBhdGNoIGRvZXMgbm90IGNoYW5nZSBhbnkgZnVuY3Rpb25hbGl0eS4KCkZPTExfTE9OR1RFUk0g Y2FuIG9ubHkgYmUgc3VwcG9ydGVkIHdpdGggZ2V0X3VzZXJfcGFnZXMoKSBhcyBpdApyZXF1aXJl cyB2bWFzIHRvIGRldGVybWluZSBpZiBEQVggaXMgaW4gdXNlLgoKU2lnbmVkLW9mZi1ieTogSXJh IFdlaW55IDxpcmEud2VpbnlAaW50ZWwuY29tPgotLS0KIGRyaXZlcnMvaW5maW5pYmFuZC9jb3Jl L3VtZW0uYyAgICAgICAgICAgICB8ICAgNSArLQogZHJpdmVycy9pbmZpbmliYW5kL2h3L3FpYi9x aWJfdXNlcl9wYWdlcy5jIHwgICA4ICstCiBkcml2ZXJzL2luZmluaWJhbmQvaHcvdXNuaWMvdXNu aWNfdWlvbS5jICAgfCAgIDkgKy0KIGRyaXZlcnMvbWVkaWEvdjRsMi1jb3JlL3ZpZGVvYnVmLWRt YS1zZy5jICB8ICAgNiArLQogZHJpdmVycy92ZmlvL3ZmaW9faW9tbXVfdHlwZTEuYyAgICAgICAg ICAgIHwgICAzICstCiBpbmNsdWRlL2xpbnV4L21tLmggICAgICAgICAgICAgICAgICAgICAgICAg fCAgMTMgKy0KIG1tL2d1cC5jICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IDEz OCArKysrKysrKysrKystLS0tLS0tLS0KIG1tL2d1cF9iZW5jaG1hcmsuYyAgICAgICAgICAgICAg ICAgICAgICAgICB8ICAgNSArLQogOCBmaWxlcyBjaGFuZ2VkLCAxMDEgaW5zZXJ0aW9ucygrKSwg ODYgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9pbmZpbmliYW5kL2NvcmUvdW1l bS5jIGIvZHJpdmVycy9pbmZpbmliYW5kL2NvcmUvdW1lbS5jCmluZGV4IGI2OWQzZWZhODcxMi4u MTIwYTQwZGY5MWI0IDEwMDY0NAotLS0gYS9kcml2ZXJzL2luZmluaWJhbmQvY29yZS91bWVtLmMK KysrIGIvZHJpdmVycy9pbmZpbmliYW5kL2NvcmUvdW1lbS5jCkBAIC0xODUsMTAgKzE4NSwxMSBA QCBzdHJ1Y3QgaWJfdW1lbSAqaWJfdW1lbV9nZXQoc3RydWN0IGliX3VkYXRhICp1ZGF0YSwgdW5z aWduZWQgbG9uZyBhZGRyLAogCiAJd2hpbGUgKG5wYWdlcykgewogCQlkb3duX3JlYWQoJm1tLT5t bWFwX3NlbSk7Ci0JCXJldCA9IGdldF91c2VyX3BhZ2VzX2xvbmd0ZXJtKGN1cl9iYXNlLAorCQly ZXQgPSBnZXRfdXNlcl9wYWdlcyhjdXJfYmFzZSwKIAkJCQkgICAgIG1pbl90KHVuc2lnbmVkIGxv bmcsIG5wYWdlcywKIAkJCQkJICAgUEFHRV9TSVpFIC8gc2l6ZW9mIChzdHJ1Y3QgcGFnZSAqKSks Ci0JCQkJICAgICBndXBfZmxhZ3MsIHBhZ2VfbGlzdCwgdm1hX2xpc3QpOworCQkJCSAgICAgZ3Vw X2ZsYWdzIHwgRk9MTF9MT05HVEVSTSwKKwkJCQkgICAgIHBhZ2VfbGlzdCwgdm1hX2xpc3QpOwog CQlpZiAocmV0IDwgMCkgewogCQkJdXBfcmVhZCgmbW0tPm1tYXBfc2VtKTsKIAkJCWdvdG8gdW1l bV9yZWxlYXNlOwpkaWZmIC0tZ2l0IGEvZHJpdmVycy9pbmZpbmliYW5kL2h3L3FpYi9xaWJfdXNl cl9wYWdlcy5jIGIvZHJpdmVycy9pbmZpbmliYW5kL2h3L3FpYi9xaWJfdXNlcl9wYWdlcy5jCmlu ZGV4IGVmOGJjZjM2NmRkYy4uMWI5MzY4MjYxMDM1IDEwMDY0NAotLS0gYS9kcml2ZXJzL2luZmlu aWJhbmQvaHcvcWliL3FpYl91c2VyX3BhZ2VzLmMKKysrIGIvZHJpdmVycy9pbmZpbmliYW5kL2h3 L3FpYi9xaWJfdXNlcl9wYWdlcy5jCkBAIC0xMTQsMTAgKzExNCwxMCBAQCBpbnQgcWliX2dldF91 c2VyX3BhZ2VzKHVuc2lnbmVkIGxvbmcgc3RhcnRfcGFnZSwgc2l6ZV90IG51bV9wYWdlcywKIAog CWRvd25fcmVhZCgmY3VycmVudC0+bW0tPm1tYXBfc2VtKTsKIAlmb3IgKGdvdCA9IDA7IGdvdCA8 IG51bV9wYWdlczsgZ290ICs9IHJldCkgewotCQlyZXQgPSBnZXRfdXNlcl9wYWdlc19sb25ndGVy bShzdGFydF9wYWdlICsgZ290ICogUEFHRV9TSVpFLAotCQkJCQkgICAgICBudW1fcGFnZXMgLSBn b3QsCi0JCQkJCSAgICAgIEZPTExfV1JJVEUgfCBGT0xMX0ZPUkNFLAotCQkJCQkgICAgICBwICsg Z290LCBOVUxMKTsKKwkJcmV0ID0gZ2V0X3VzZXJfcGFnZXMoc3RhcnRfcGFnZSArIGdvdCAqIFBB R0VfU0laRSwKKwkJCQkgICAgIG51bV9wYWdlcyAtIGdvdCwKKwkJCQkgICAgIEZPTExfTE9OR1RF Uk0gfCBGT0xMX1dSSVRFIHwgRk9MTF9GT1JDRSwKKwkJCQkgICAgIHAgKyBnb3QsIE5VTEwpOwog CQlpZiAocmV0IDwgMCkgewogCQkJdXBfcmVhZCgmY3VycmVudC0+bW0tPm1tYXBfc2VtKTsKIAkJ CWdvdG8gYmFpbF9yZWxlYXNlOwpkaWZmIC0tZ2l0IGEvZHJpdmVycy9pbmZpbmliYW5kL2h3L3Vz bmljL3VzbmljX3Vpb20uYyBiL2RyaXZlcnMvaW5maW5pYmFuZC9ody91c25pYy91c25pY191aW9t LmMKaW5kZXggMDY4NjJhNmFmMTg1Li4xZDlhMTgyYWMxNjMgMTAwNjQ0Ci0tLSBhL2RyaXZlcnMv aW5maW5pYmFuZC9ody91c25pYy91c25pY191aW9tLmMKKysrIGIvZHJpdmVycy9pbmZpbmliYW5k L2h3L3VzbmljL3VzbmljX3Vpb20uYwpAQCAtMTQzLDEwICsxNDMsMTEgQEAgc3RhdGljIGludCB1 c25pY191aW9tX2dldF9wYWdlcyh1bnNpZ25lZCBsb25nIGFkZHIsIHNpemVfdCBzaXplLCBpbnQg d3JpdGFibGUsCiAJcmV0ID0gMDsKIAogCXdoaWxlIChucGFnZXMpIHsKLQkJcmV0ID0gZ2V0X3Vz ZXJfcGFnZXNfbG9uZ3Rlcm0oY3VyX2Jhc2UsCi0JCQkJCW1pbl90KHVuc2lnbmVkIGxvbmcsIG5w YWdlcywKLQkJCQkJUEFHRV9TSVpFIC8gc2l6ZW9mKHN0cnVjdCBwYWdlICopKSwKLQkJCQkJZ3Vw X2ZsYWdzLCBwYWdlX2xpc3QsIE5VTEwpOworCQlyZXQgPSBnZXRfdXNlcl9wYWdlcyhjdXJfYmFz ZSwKKwkJCQkgICAgIG1pbl90KHVuc2lnbmVkIGxvbmcsIG5wYWdlcywKKwkJCQkgICAgIFBBR0Vf U0laRSAvIHNpemVvZihzdHJ1Y3QgcGFnZSAqKSksCisJCQkJICAgICBndXBfZmxhZ3MgfCBGT0xM X0xPTkdURVJNLAorCQkJCSAgICAgcGFnZV9saXN0LCBOVUxMKTsKIAogCQlpZiAocmV0IDwgMCkK IAkJCWdvdG8gb3V0OwpkaWZmIC0tZ2l0IGEvZHJpdmVycy9tZWRpYS92NGwyLWNvcmUvdmlkZW9i dWYtZG1hLXNnLmMgYi9kcml2ZXJzL21lZGlhL3Y0bDItY29yZS92aWRlb2J1Zi1kbWEtc2cuYwpp bmRleCAwODkyOWMwODdlMjcuLjg3MGEyYTUyNmUwYiAxMDA2NDQKLS0tIGEvZHJpdmVycy9tZWRp YS92NGwyLWNvcmUvdmlkZW9idWYtZG1hLXNnLmMKKysrIGIvZHJpdmVycy9tZWRpYS92NGwyLWNv cmUvdmlkZW9idWYtZG1hLXNnLmMKQEAgLTE4NiwxMiArMTg2LDEyIEBAIHN0YXRpYyBpbnQgdmlk ZW9idWZfZG1hX2luaXRfdXNlcl9sb2NrZWQoc3RydWN0IHZpZGVvYnVmX2RtYWJ1ZiAqZG1hLAog CWRwcmludGsoMSwgImluaXQgdXNlciBbMHglbHgrMHglbHggPT4gJWQgcGFnZXNdXG4iLAogCQlk YXRhLCBzaXplLCBkbWEtPm5yX3BhZ2VzKTsKIAotCWVyciA9IGdldF91c2VyX3BhZ2VzX2xvbmd0 ZXJtKGRhdGEgJiBQQUdFX01BU0ssIGRtYS0+bnJfcGFnZXMsCi0JCQkgICAgIGZsYWdzLCBkbWEt PnBhZ2VzLCBOVUxMKTsKKwllcnIgPSBnZXRfdXNlcl9wYWdlcyhkYXRhICYgUEFHRV9NQVNLLCBk bWEtPm5yX3BhZ2VzLAorCQkJICAgICBmbGFncyB8IEZPTExfTE9OR1RFUk0sIGRtYS0+cGFnZXMs IE5VTEwpOwogCiAJaWYgKGVyciAhPSBkbWEtPm5yX3BhZ2VzKSB7CiAJCWRtYS0+bnJfcGFnZXMg PSAoZXJyID49IDApID8gZXJyIDogMDsKLQkJZHByaW50aygxLCAiZ2V0X3VzZXJfcGFnZXNfbG9u Z3Rlcm06IGVycj0lZCBbJWRdXG4iLCBlcnIsCisJCWRwcmludGsoMSwgImdldF91c2VyX3BhZ2Vz OiBlcnI9JWQgWyVkXVxuIiwgZXJyLAogCQkJZG1hLT5ucl9wYWdlcyk7CiAJCXJldHVybiBlcnIg PCAwID8gZXJyIDogLUVJTlZBTDsKIAl9CmRpZmYgLS1naXQgYS9kcml2ZXJzL3ZmaW8vdmZpb19p b21tdV90eXBlMS5jIGIvZHJpdmVycy92ZmlvL3ZmaW9faW9tbXVfdHlwZTEuYwppbmRleCA3MzY1 MmUyMWVmZWMuLjE1MDBiZDBiYjZkYSAxMDA2NDQKLS0tIGEvZHJpdmVycy92ZmlvL3ZmaW9faW9t bXVfdHlwZTEuYworKysgYi9kcml2ZXJzL3ZmaW8vdmZpb19pb21tdV90eXBlMS5jCkBAIC0zNTEs NyArMzUxLDggQEAgc3RhdGljIGludCB2YWRkcl9nZXRfcGZuKHN0cnVjdCBtbV9zdHJ1Y3QgKm1t LCB1bnNpZ25lZCBsb25nIHZhZGRyLAogCiAJZG93bl9yZWFkKCZtbS0+bW1hcF9zZW0pOwogCWlm IChtbSA9PSBjdXJyZW50LT5tbSkgewotCQlyZXQgPSBnZXRfdXNlcl9wYWdlc19sb25ndGVybSh2 YWRkciwgMSwgZmxhZ3MsIHBhZ2UsIHZtYXMpOworCQlyZXQgPSBnZXRfdXNlcl9wYWdlcyh2YWRk ciwgMSwgZmxhZ3MgfCBGT0xMX0xPTkdURVJNLCBwYWdlLAorCQkJCSAgICAgdm1hcyk7CiAJfSBl bHNlIHsKIAkJcmV0ID0gZ2V0X3VzZXJfcGFnZXNfcmVtb3RlKE5VTEwsIG1tLCB2YWRkciwgMSwg ZmxhZ3MsIHBhZ2UsCiAJCQkJCSAgICB2bWFzLCBOVUxMKTsKZGlmZiAtLWdpdCBhL2luY2x1ZGUv bGludXgvbW0uaCBiL2luY2x1ZGUvbGludXgvbW0uaAppbmRleCA4MGJiNjQwOGZlNzMuLjA1YTEw NWQ5ZDRjMyAxMDA2NDQKLS0tIGEvaW5jbHVkZS9saW51eC9tbS5oCisrKyBiL2luY2x1ZGUvbGlu dXgvbW0uaApAQCAtMTUzNiwxOCArMTUzNiw2IEBAIGxvbmcgZ2V0X3VzZXJfcGFnZXNfbG9ja2Vk KHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgbnJfcGFnZXMsCiAJCSAgICB1bnNp Z25lZCBpbnQgZ3VwX2ZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBhZ2VzLCBpbnQgKmxvY2tlZCk7CiBs b25nIGdldF91c2VyX3BhZ2VzX3VubG9ja2VkKHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVk IGxvbmcgbnJfcGFnZXMsCiAJCSAgICBzdHJ1Y3QgcGFnZSAqKnBhZ2VzLCB1bnNpZ25lZCBpbnQg Z3VwX2ZsYWdzKTsKLSNpZmRlZiBDT05GSUdfRlNfREFYCi1sb25nIGdldF91c2VyX3BhZ2VzX2xv bmd0ZXJtKHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgbnJfcGFnZXMsCi0JCQkg ICAgdW5zaWduZWQgaW50IGd1cF9mbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywKLQkJCSAgICBz dHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKip2bWFzKTsKLSNlbHNlCi1zdGF0aWMgaW5saW5lIGxvbmcg Z2V0X3VzZXJfcGFnZXNfbG9uZ3Rlcm0odW5zaWduZWQgbG9uZyBzdGFydCwKLQkJdW5zaWduZWQg bG9uZyBucl9wYWdlcywgdW5zaWduZWQgaW50IGd1cF9mbGFncywKLQkJc3RydWN0IHBhZ2UgKipw YWdlcywgc3RydWN0IHZtX2FyZWFfc3RydWN0ICoqdm1hcykKLXsKLQlyZXR1cm4gZ2V0X3VzZXJf cGFnZXMoc3RhcnQsIG5yX3BhZ2VzLCBndXBfZmxhZ3MsIHBhZ2VzLCB2bWFzKTsKLX0KLSNlbmRp ZiAvKiBDT05GSUdfRlNfREFYICovCiAKIGludCBnZXRfdXNlcl9wYWdlc19mYXN0KHVuc2lnbmVk IGxvbmcgc3RhcnQsIGludCBucl9wYWdlcywgaW50IHdyaXRlLAogCQkJc3RydWN0IHBhZ2UgKipw YWdlcyk7CkBAIC0yNjE1LDYgKzI2MDMsNyBAQCBzdHJ1Y3QgcGFnZSAqZm9sbG93X3BhZ2Uoc3Ry dWN0IHZtX2FyZWFfc3RydWN0ICp2bWEsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcywKICNkZWZpbmUg Rk9MTF9SRU1PVEUJMHgyMDAwCS8qIHdlIGFyZSB3b3JraW5nIG9uIG5vbi1jdXJyZW50IHRzay9t bSAqLwogI2RlZmluZSBGT0xMX0NPVwkweDQwMDAJLyogaW50ZXJuYWwgR1VQIGZsYWcgKi8KICNk ZWZpbmUgRk9MTF9BTk9OCTB4ODAwMAkvKiBkb24ndCBkbyBmaWxlIG1hcHBpbmdzICovCisjZGVm aW5lIEZPTExfTE9OR1RFUk0JMHgxMDAwMAkvKiBtYXBwaW5nIGlzIGludGVuZGVkIGZvciBhIGxv bmcgdGVybSBwaW4gKi8KIAogc3RhdGljIGlubGluZSBpbnQgdm1fZmF1bHRfdG9fZXJybm8odm1f ZmF1bHRfdCB2bV9mYXVsdCwgaW50IGZvbGxfZmxhZ3MpCiB7CmRpZmYgLS1naXQgYS9tbS9ndXAu YyBiL21tL2d1cC5jCmluZGV4IGI2M2U4OGVjYTMxYi4uZWU5NmVhZmYxMThjIDEwMDY0NAotLS0g YS9tbS9ndXAuYworKysgYi9tbS9ndXAuYwpAQCAtMTEwOSw4NyArMTEwOSwxMDkgQEAgbG9uZyBn ZXRfdXNlcl9wYWdlc19yZW1vdGUoc3RydWN0IHRhc2tfc3RydWN0ICp0c2ssIHN0cnVjdCBtbV9z dHJ1Y3QgKm1tLAogfQogRVhQT1JUX1NZTUJPTChnZXRfdXNlcl9wYWdlc19yZW1vdGUpOwogCi0v KgotICogVGhpcyBpcyB0aGUgc2FtZSBhcyBnZXRfdXNlcl9wYWdlc19yZW1vdGUoKSwganVzdCB3 aXRoIGEKLSAqIGxlc3MtZmxleGlibGUgY2FsbGluZyBjb252ZW50aW9uIHdoZXJlIHdlIGFzc3Vt ZSB0aGF0IHRoZSB0YXNrCi0gKiBhbmQgbW0gYmVpbmcgb3BlcmF0ZWQgb24gYXJlIHRoZSBjdXJy ZW50IHRhc2sncyBhbmQgZG9uJ3QgYWxsb3cKLSAqIHBhc3Npbmcgb2YgYSBsb2NrZWQgcGFyYW1l dGVyLiAgV2UgYWxzbyBvYnZpb3VzbHkgZG9uJ3QgcGFzcwotICogRk9MTF9SRU1PVEUgaW4gaGVy ZS4KLSAqLwotbG9uZyBnZXRfdXNlcl9wYWdlcyh1bnNpZ25lZCBsb25nIHN0YXJ0LCB1bnNpZ25l ZCBsb25nIG5yX3BhZ2VzLAotCQl1bnNpZ25lZCBpbnQgZ3VwX2ZsYWdzLCBzdHJ1Y3QgcGFnZSAq KnBhZ2VzLAotCQlzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKip2bWFzKQotewotCXJldHVybiBfX2dl dF91c2VyX3BhZ2VzX2xvY2tlZChjdXJyZW50LCBjdXJyZW50LT5tbSwgc3RhcnQsIG5yX3BhZ2Vz LAotCQkJCSAgICAgICBwYWdlcywgdm1hcywgTlVMTCwKLQkJCQkgICAgICAgZ3VwX2ZsYWdzIHwg Rk9MTF9UT1VDSCk7Ci19Ci1FWFBPUlRfU1lNQk9MKGdldF91c2VyX3BhZ2VzKTsKLQogI2lmZGVm IENPTkZJR19GU19EQVgKIC8qCi0gKiBUaGlzIGlzIHRoZSBzYW1lIGFzIGdldF91c2VyX3BhZ2Vz KCkgaW4gdGhhdCBpdCBhc3N1bWVzIHdlIGFyZQotICogb3BlcmF0aW5nIG9uIHRoZSBjdXJyZW50 IHRhc2sncyBtbSwgYnV0IGl0IGdvZXMgZnVydGhlciB0byB2YWxpZGF0ZQotICogdGhhdCB0aGUg dm1hcyBhc3NvY2lhdGVkIHdpdGggdGhlIGFkZHJlc3MgcmFuZ2UgYXJlIHN1aXRhYmxlIGZvcgot ICogbG9uZ3Rlcm0gZWxldmF0ZWQgcGFnZSByZWZlcmVuY2UgY291bnRzLiBGb3IgZXhhbXBsZSwg ZmlsZXN5c3RlbS1kYXgKLSAqIG1hcHBpbmdzIGFyZSBzdWJqZWN0IHRvIHRoZSBsaWZldGltZSBl bmZvcmNlZCBieSB0aGUgZmlsZXN5c3RlbSBhbmQKLSAqIHdlIG5lZWQgZ3VhcmFudGVlcyB0aGF0 IGxvbmd0ZXJtIHVzZXJzIGxpa2UgUkRNQSBhbmQgVjRMMiBvbmx5Ci0gKiBlc3RhYmxpc2ggbWFw cGluZ3MgdGhhdCBoYXZlIGEga2VybmVsIGVuZm9yY2VkIHJldm9jYXRpb24gbWVjaGFuaXNtLgor ICogX19ndXBfbG9uZ3Rlcm1fbG9ja2VkKCkgaXMgYSB3cmFwcGVyIGZvciBfX2dldF91ZXJfcGFn ZXNfbG9ja2VkIHdoaWNoCisgKiBhbGxvd3MgdXMgdG8gcHJvY2VzcyB0aGUgRk9MTF9MT05HVEVS TSBmbGFnIGlmIHByZXNlbnQuCisgKgorICogX19ndXBfbG9uZ3Rlcm1fbG9ja2VkKCkgdmFsaWRh dGVzIHRoYXQgdGhlIHZtYXMgYXNzb2NpYXRlZCB3aXRoIHRoZSBhZGRyZXNzCisgKiByYW5nZSBh cmUgc3VpdGFibGUgZm9yIGxvbmd0ZXJtIGVsZXZhdGVkIHBhZ2UgcmVmZXJlbmNlIGNvdW50cy4g Rm9yIGV4YW1wbGUsCisgKiBmaWxlc3lzdGVtLWRheCBtYXBwaW5ncyBhcmUgc3ViamVjdCB0byB0 aGUgbGlmZXRpbWUgZW5mb3JjZWQgYnkgdGhlCisgKiBmaWxlc3lzdGVtIGFuZCB3ZSBuZWVkIGd1 YXJhbnRlZXMgdGhhdCBsb25ndGVybSB1c2VycyBsaWtlIFJETUEgYW5kIFY0TDIKKyAqIG9ubHkg ZXN0YWJsaXNoIG1hcHBpbmdzIHRoYXQgaGF2ZSBhIGtlcm5lbCBlbmZvcmNlZCByZXZvY2F0aW9u IG1lY2hhbmlzbS4KICAqCiAgKiAibG9uZ3Rlcm0iID09IHVzZXJzcGFjZSBjb250cm9sbGVkIGVs ZXZhdGVkIHBhZ2UgY291bnQgbGlmZXRpbWUuCiAgKiBDb250cmFzdCB0aGlzIHRvIGlvdl9pdGVy X2dldF9wYWdlcygpIHVzYWdlcyB3aGljaCBhcmUgdHJhbnNpZW50LgogICovCi1sb25nIGdldF91 c2VyX3BhZ2VzX2xvbmd0ZXJtKHVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgbnJf cGFnZXMsCi0JCXVuc2lnbmVkIGludCBndXBfZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsCi0J CXN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqKnZtYXNfYXJnKQorc3RhdGljIF9fYWx3YXlzX2lubGlu ZSBsb25nIF9fZ3VwX2xvbmd0ZXJtX2xvY2tlZChzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRzaywKKwkJ CQkJCSAgc3RydWN0IG1tX3N0cnVjdCAqbW0sCisJCQkJCQkgIHVuc2lnbmVkIGxvbmcgc3RhcnQs CisJCQkJCQkgIHVuc2lnbmVkIGxvbmcgbnJfcGFnZXMsCisJCQkJCQkgIHN0cnVjdCBwYWdlICoq cGFnZXMsCisJCQkJCQkgIHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqKnZtYXMsCisJCQkJCQkgIHVu c2lnbmVkIGludCBmbGFncykKIHsKLQlzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKip2bWFzID0gdm1h c19hcmc7CisJc3RydWN0IHZtX2FyZWFfc3RydWN0ICoqdm1hc190bXAgPSB2bWFzOwogCXN0cnVj dCB2bV9hcmVhX3N0cnVjdCAqdm1hX3ByZXYgPSBOVUxMOwogCWxvbmcgcmMsIGk7CiAKLQlpZiAo IXBhZ2VzKQotCQlyZXR1cm4gLUVJTlZBTDsKLQotCWlmICghdm1hcykgewotCQl2bWFzID0ga2Nh bGxvYyhucl9wYWdlcywgc2l6ZW9mKHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqKSwKLQkJCSAgICAg ICBHRlBfS0VSTkVMKTsKLQkJaWYgKCF2bWFzKQotCQkJcmV0dXJuIC1FTk9NRU07CisJaWYgKGZs YWdzICYgRk9MTF9MT05HVEVSTSkgeworCQlpZiAoIXBhZ2VzKQorCQkJcmV0dXJuIC1FSU5WQUw7 CisKKwkJaWYgKCF2bWFzX3RtcCkgeworCQkJdm1hc190bXAgPSBrY2FsbG9jKG5yX3BhZ2VzLAor CQkJCQkgICBzaXplb2Yoc3RydWN0IHZtX2FyZWFfc3RydWN0ICopLAorCQkJCQkgICBHRlBfS0VS TkVMKTsKKwkJCWlmICghdm1hc190bXApCisJCQkJcmV0dXJuIC1FTk9NRU07CisJCX0KIAl9CiAK LQlyYyA9IGdldF91c2VyX3BhZ2VzKHN0YXJ0LCBucl9wYWdlcywgZ3VwX2ZsYWdzLCBwYWdlcywg dm1hcyk7CisJcmMgPSBfX2dldF91c2VyX3BhZ2VzX2xvY2tlZCh0c2ssIG1tLCBzdGFydCwgbnJf cGFnZXMsIHBhZ2VzLAorCQkJCSAgICAgdm1hc190bXAsIE5VTEwsIGZsYWdzKTsKIAotCWZvciAo aSA9IDA7IGkgPCByYzsgaSsrKSB7Ci0JCXN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqdm1hID0gdm1h c1tpXTsKKwlpZiAoZmxhZ3MgJiBGT0xMX0xPTkdURVJNKSB7CisJCWZvciAoaSA9IDA7IGkgPCBy YzsgaSsrKSB7CisJCQlzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnZtYSA9IHZtYXNfdG1wW2ldOwog Ci0JCWlmICh2bWEgPT0gdm1hX3ByZXYpCi0JCQljb250aW51ZTsKKwkJCWlmICh2bWEgPT0gdm1h X3ByZXYpCisJCQkJY29udGludWU7CiAKLQkJdm1hX3ByZXYgPSB2bWE7CisJCQl2bWFfcHJldiA9 IHZtYTsKIAotCQlpZiAodm1hX2lzX2ZzZGF4KHZtYSkpCi0JCQlicmVhazsKLQl9CisJCQlpZiAo dm1hX2lzX2ZzZGF4KHZtYSkpCisJCQkJYnJlYWs7CisJCX0KIAotCS8qCi0JICogRWl0aGVyIGdl dF91c2VyX3BhZ2VzKCkgZmFpbGVkLCBvciB0aGUgdm1hIHZhbGlkYXRpb24KLQkgKiBzdWNjZWVk ZWQsIGluIGVpdGhlciBjYXNlIHdlIGRvbid0IG5lZWQgdG8gcHV0X3BhZ2UoKSBiZWZvcmUKLQkg KiByZXR1cm5pbmcuCi0JICovCi0JaWYgKGkgPj0gcmMpCi0JCWdvdG8gb3V0OworCQkvKgorCQkg KiBFaXRoZXIgZ2V0X3VzZXJfcGFnZXMoKSBmYWlsZWQsIG9yIHRoZSB2bWEgdmFsaWRhdGlvbgor CQkgKiBzdWNjZWVkZWQsIGluIGVpdGhlciBjYXNlIHdlIGRvbid0IG5lZWQgdG8gcHV0X3BhZ2Uo KSBiZWZvcmUKKwkJICogcmV0dXJuaW5nLgorCQkgKi8KKwkJaWYgKGkgPj0gcmMpCisJCQlnb3Rv IG91dDsKIAotCWZvciAoaSA9IDA7IGkgPCByYzsgaSsrKQotCQlwdXRfcGFnZShwYWdlc1tpXSk7 Ci0JcmMgPSAtRU9QTk9UU1VQUDsKKwkJZm9yIChpID0gMDsgaSA8IHJjOyBpKyspCisJCQlwdXRf cGFnZShwYWdlc1tpXSk7CisJCXJjID0gLUVPUE5PVFNVUFA7CiBvdXQ6Ci0JaWYgKHZtYXMgIT0g dm1hc19hcmcpCi0JCWtmcmVlKHZtYXMpOworCQlpZiAodm1hc190bXAgIT0gdm1hcykKKwkJCWtm cmVlKHZtYXNfdG1wKTsKKwl9CisKIAlyZXR1cm4gcmM7CiB9Ci1FWFBPUlRfU1lNQk9MKGdldF91 c2VyX3BhZ2VzX2xvbmd0ZXJtKTsKKyNlbHNlIC8qICFDT05GSUdfRlNfREFYICovCitzdGF0aWMg X19hbHdheXNfaW5saW5lIGxvbmcgX19ndXBfbG9uZ3Rlcm1fbG9ja2VkKHN0cnVjdCB0YXNrX3N0 cnVjdCAqdHNrLAorCQkJCQkJICBzdHJ1Y3QgbW1fc3RydWN0ICptbSwKKwkJCQkJCSAgdW5zaWdu ZWQgbG9uZyBzdGFydCwKKwkJCQkJCSAgdW5zaWduZWQgbG9uZyBucl9wYWdlcywKKwkJCQkJCSAg c3RydWN0IHBhZ2UgKipwYWdlcywKKwkJCQkJCSAgc3RydWN0IHZtX2FyZWFfc3RydWN0ICoqdm1h cywKKwkJCQkJCSAgdW5zaWduZWQgaW50IGZsYWdzKQoreworCXJldHVybiBfX2dldF91c2VyX3Bh Z2VzX2xvY2tlZCh0c2ssIG1tLCBzdGFydCwgbnJfcGFnZXMsIHBhZ2VzLCB2bWFzLAorCQkJCSAg ICAgICBOVUxMLCBmbGFncyk7Cit9CiAjZW5kaWYgLyogQ09ORklHX0ZTX0RBWCAqLwogCisvKgor ICogVGhpcyBpcyB0aGUgc2FtZSBhcyBnZXRfdXNlcl9wYWdlc19yZW1vdGUoKSwganVzdCB3aXRo IGEKKyAqIGxlc3MtZmxleGlibGUgY2FsbGluZyBjb252ZW50aW9uIHdoZXJlIHdlIGFzc3VtZSB0 aGF0IHRoZSB0YXNrCisgKiBhbmQgbW0gYmVpbmcgb3BlcmF0ZWQgb24gYXJlIHRoZSBjdXJyZW50 IHRhc2sncyBhbmQgZG9uJ3QgYWxsb3cKKyAqIHBhc3Npbmcgb2YgYSBsb2NrZWQgcGFyYW1ldGVy LiAgV2UgYWxzbyBvYnZpb3VzbHkgZG9uJ3QgcGFzcworICogRk9MTF9SRU1PVEUgaW4gaGVyZS4K KyAqLworbG9uZyBnZXRfdXNlcl9wYWdlcyh1bnNpZ25lZCBsb25nIHN0YXJ0LCB1bnNpZ25lZCBs b25nIG5yX3BhZ2VzLAorCQl1bnNpZ25lZCBpbnQgZ3VwX2ZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBh Z2VzLAorCQlzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKip2bWFzKQoreworCXJldHVybiBfX2d1cF9s b25ndGVybV9sb2NrZWQoY3VycmVudCwgY3VycmVudC0+bW0sIHN0YXJ0LCBucl9wYWdlcywKKwkJ CQkgICAgIHBhZ2VzLCB2bWFzLCBndXBfZmxhZ3MgfCBGT0xMX1RPVUNIKTsKK30KK0VYUE9SVF9T WU1CT0woZ2V0X3VzZXJfcGFnZXMpOworCiAvKioKICAqIHBvcHVsYXRlX3ZtYV9wYWdlX3Jhbmdl KCkgLSAgcG9wdWxhdGUgYSByYW5nZSBvZiBwYWdlcyBpbiB0aGUgdm1hLgogICogQHZtYTogICB0 YXJnZXQgdm1hCmRpZmYgLS1naXQgYS9tbS9ndXBfYmVuY2htYXJrLmMgYi9tbS9ndXBfYmVuY2ht YXJrLmMKaW5kZXggNWI0MmQzZDRiNjBhLi5jODk4ZTJlMGQxZTQgMTAwNjQ0Ci0tLSBhL21tL2d1 cF9iZW5jaG1hcmsuYworKysgYi9tbS9ndXBfYmVuY2htYXJrLmMKQEAgLTU0LDggKzU0LDkgQEAg c3RhdGljIGludCBfX2d1cF9iZW5jaG1hcmtfaW9jdGwodW5zaWduZWQgaW50IGNtZCwKIAkJCQkJ CSBwYWdlcyArIGkpOwogCQkJYnJlYWs7CiAJCWNhc2UgR1VQX0xPTkdURVJNX0JFTkNITUFSSzoK LQkJCW5yID0gZ2V0X3VzZXJfcGFnZXNfbG9uZ3Rlcm0oYWRkciwgbnIsIGd1cC0+ZmxhZ3MgJiAx LAotCQkJCQkJICAgICBwYWdlcyArIGksIE5VTEwpOworCQkJbnIgPSBnZXRfdXNlcl9wYWdlcyhh ZGRyLCBuciwKKwkJCQkJICAgIChndXAtPmZsYWdzICYgMSkgfCBGT0xMX0xPTkdURVJNLAorCQkJ CQkgICAgcGFnZXMgKyBpLCBOVUxMKTsKIAkJCWJyZWFrOwogCQljYXNlIEdVUF9CRU5DSE1BUks6 CiAJCQluciA9IGdldF91c2VyX3BhZ2VzKGFkZHIsIG5yLCBndXAtPmZsYWdzICYgMSwgcGFnZXMg KyBpLAotLSAKMi4yMC4xCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3Rv cC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmkt ZGV2ZWw= From mboxrd@z Thu Jan 1 00:00:00 1970 From: ira.weiny@intel.com Date: Wed, 13 Feb 2019 23:04:49 +0000 Subject: [PATCH V2 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM Message-Id: <20190213230455.5605-2-ira.weiny@intel.com> List-Id: References: <20190211201643.7599-1-ira.weiny@intel.com> <20190213230455.5605-1-ira.weiny@intel.com> In-Reply-To: <20190213230455.5605-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: linux-mips@vger.kernel.org, linux-kernel@vger.kernel.org, kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org, linux-fpga@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-rdma@vger.kernel.org, linux-media@vger.kernel.org, linux-scsi@vger.kernel.org, devel@driverdev.osuosl.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-fbdev@vger.kernel.org, xen-devel@lists.xenproject.org, devel@lists.orangefs.org, linux-mm@kvack.org, ceph-devel@vger.kernel.org, rds-devel@oss.oracle.com Cc: Martin Brandenburg , Rich Felker , David Hildenbrand , David Airlie , Jason Wang , =?UTF-8?q?Kai=20M=C3=A4kisara?= , Michal Hocko , Ira Weiny , Ben Chan , Rob Springer , Todd Poynor , Yoshinori Sato , "Michael S. Tsirkin" , Jason Gunthorpe , Ingo Molnar , Matt Porter , Wu Hao , Alan Tull , John Hubbard , "James E.J. Bottomley" , Alex Williamson , Stefano Stabellini , Borislav Petkov From: Ira Weiny Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it. This patch does not change any functionality. FOLL_LONGTERM can only be supported with get_user_pages() as it requires vmas to determine if DAX is in use. Signed-off-by: Ira Weiny --- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/vfio/vfio_iommu_type1.c | 3 +- include/linux/mm.h | 13 +- mm/gup.c | 138 ++++++++++++--------- mm/gup_benchmark.c | 5 +- 8 files changed, 101 insertions(+), 86 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index b69d3efa8712..120a40df91b4 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -185,10 +185,11 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, while (npages) { down_read(&mm->mmap_sem); - ret = get_user_pages_longterm(cur_base, + ret = get_user_pages(cur_base, min_t(unsigned long, npages, PAGE_SIZE / sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags | FOLL_LONGTERM, + page_list, vma_list); if (ret < 0) { up_read(&mm->mmap_sem); goto umem_release; diff --git a/drivers/infiniband/hw/qib/qib_user_pages.c b/drivers/infiniband/hw/qib/qib_user_pages.c index ef8bcf366ddc..1b9368261035 100644 --- a/drivers/infiniband/hw/qib/qib_user_pages.c +++ b/drivers/infiniband/hw/qib/qib_user_pages.c @@ -114,10 +114,10 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages, down_read(¤t->mm->mmap_sem); for (got = 0; got < num_pages; got += ret) { - ret = get_user_pages_longterm(start_page + got * PAGE_SIZE, - num_pages - got, - FOLL_WRITE | FOLL_FORCE, - p + got, NULL); + ret = get_user_pages(start_page + got * PAGE_SIZE, + num_pages - got, + FOLL_LONGTERM | FOLL_WRITE | FOLL_FORCE, + p + got, NULL); if (ret < 0) { up_read(¤t->mm->mmap_sem); goto bail_release; diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.c b/drivers/infiniband/hw/usnic/usnic_uiom.c index 06862a6af185..1d9a182ac163 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom.c @@ -143,10 +143,11 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret = 0; while (npages) { - ret = get_user_pages_longterm(cur_base, - min_t(unsigned long, npages, - PAGE_SIZE / sizeof(struct page *)), - gup_flags, page_list, NULL); + ret = get_user_pages(cur_base, + min_t(unsigned long, npages, + PAGE_SIZE / sizeof(struct page *)), + gup_flags | FOLL_LONGTERM, + page_list, NULL); if (ret < 0) goto out; diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c index 08929c087e27..870a2a526e0b 100644 --- a/drivers/media/v4l2-core/videobuf-dma-sg.c +++ b/drivers/media/v4l2-core/videobuf-dma-sg.c @@ -186,12 +186,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma, dprintk(1, "init user [0x%lx+0x%lx => %d pages]\n", data, size, dma->nr_pages); - err = get_user_pages_longterm(data & PAGE_MASK, dma->nr_pages, - flags, dma->pages, NULL); + err = get_user_pages(data & PAGE_MASK, dma->nr_pages, + flags | FOLL_LONGTERM, dma->pages, NULL); if (err != dma->nr_pages) { dma->nr_pages = (err >= 0) ? err : 0; - dprintk(1, "get_user_pages_longterm: err=%d [%d]\n", err, + dprintk(1, "get_user_pages: err=%d [%d]\n", err, dma->nr_pages); return err < 0 ? err : -EINVAL; } diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 73652e21efec..1500bd0bb6da 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -351,7 +351,8 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, down_read(&mm->mmap_sem); if (mm = current->mm) { - ret = get_user_pages_longterm(vaddr, 1, flags, page, vmas); + ret = get_user_pages(vaddr, 1, flags | FOLL_LONGTERM, page, + vmas); } else { ret = get_user_pages_remote(NULL, mm, vaddr, 1, flags, page, vmas, NULL); diff --git a/include/linux/mm.h b/include/linux/mm.h index 80bb6408fe73..05a105d9d4c3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1536,18 +1536,6 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, int *locked); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags); -#ifdef CONFIG_FS_DAX -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas); -#else -static inline long get_user_pages_longterm(unsigned long start, - unsigned long nr_pages, unsigned int gup_flags, - struct page **pages, struct vm_area_struct **vmas) -{ - return get_user_pages(start, nr_pages, gup_flags, pages, vmas); -} -#endif /* CONFIG_FS_DAX */ int get_user_pages_fast(unsigned long start, int nr_pages, int write, struct page **pages); @@ -2615,6 +2603,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ +#define FOLL_LONGTERM 0x10000 /* mapping is intended for a long term pin */ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) { diff --git a/mm/gup.c b/mm/gup.c index b63e88eca31b..ee96eaff118c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1109,87 +1109,109 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, } EXPORT_SYMBOL(get_user_pages_remote); -/* - * This is the same as get_user_pages_remote(), just with a - * less-flexible calling convention where we assume that the task - * and mm being operated on are the current task's and don't allow - * passing of a locked parameter. We also obviously don't pass - * FOLL_REMOTE in here. - */ -long get_user_pages(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas) -{ - return __get_user_pages_locked(current, current->mm, start, nr_pages, - pages, vmas, NULL, - gup_flags | FOLL_TOUCH); -} -EXPORT_SYMBOL(get_user_pages); - #ifdef CONFIG_FS_DAX /* - * This is the same as get_user_pages() in that it assumes we are - * operating on the current task's mm, but it goes further to validate - * that the vmas associated with the address range are suitable for - * longterm elevated page reference counts. For example, filesystem-dax - * mappings are subject to the lifetime enforced by the filesystem and - * we need guarantees that longterm users like RDMA and V4L2 only - * establish mappings that have a kernel enforced revocation mechanism. + * __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which + * allows us to process the FOLL_LONGTERM flag if present. + * + * __gup_longterm_locked() validates that the vmas associated with the address + * range are suitable for longterm elevated page reference counts. For example, + * filesystem-dax mappings are subject to the lifetime enforced by the + * filesystem and we need guarantees that longterm users like RDMA and V4L2 + * only establish mappings that have a kernel enforced revocation mechanism. * * "longterm" = userspace controlled elevated page count lifetime. * Contrast this to iov_iter_get_pages() usages which are transient. */ -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas_arg) +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) { - struct vm_area_struct **vmas = vmas_arg; + struct vm_area_struct **vmas_tmp = vmas; struct vm_area_struct *vma_prev = NULL; long rc, i; - if (!pages) - return -EINVAL; - - if (!vmas) { - vmas = kcalloc(nr_pages, sizeof(struct vm_area_struct *), - GFP_KERNEL); - if (!vmas) - return -ENOMEM; + if (flags & FOLL_LONGTERM) { + if (!pages) + return -EINVAL; + + if (!vmas_tmp) { + vmas_tmp = kcalloc(nr_pages, + sizeof(struct vm_area_struct *), + GFP_KERNEL); + if (!vmas_tmp) + return -ENOMEM; + } } - rc = get_user_pages(start, nr_pages, gup_flags, pages, vmas); + rc = __get_user_pages_locked(tsk, mm, start, nr_pages, pages, + vmas_tmp, NULL, flags); - for (i = 0; i < rc; i++) { - struct vm_area_struct *vma = vmas[i]; + if (flags & FOLL_LONGTERM) { + for (i = 0; i < rc; i++) { + struct vm_area_struct *vma = vmas_tmp[i]; - if (vma = vma_prev) - continue; + if (vma = vma_prev) + continue; - vma_prev = vma; + vma_prev = vma; - if (vma_is_fsdax(vma)) - break; - } + if (vma_is_fsdax(vma)) + break; + } - /* - * Either get_user_pages() failed, or the vma validation - * succeeded, in either case we don't need to put_page() before - * returning. - */ - if (i >= rc) - goto out; + /* + * Either get_user_pages() failed, or the vma validation + * succeeded, in either case we don't need to put_page() before + * returning. + */ + if (i >= rc) + goto out; - for (i = 0; i < rc; i++) - put_page(pages[i]); - rc = -EOPNOTSUPP; + for (i = 0; i < rc; i++) + put_page(pages[i]); + rc = -EOPNOTSUPP; out: - if (vmas != vmas_arg) - kfree(vmas); + if (vmas_tmp != vmas) + kfree(vmas_tmp); + } + return rc; } -EXPORT_SYMBOL(get_user_pages_longterm); +#else /* !CONFIG_FS_DAX */ +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) +{ + return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas, + NULL, flags); +} #endif /* CONFIG_FS_DAX */ +/* + * This is the same as get_user_pages_remote(), just with a + * less-flexible calling convention where we assume that the task + * and mm being operated on are the current task's and don't allow + * passing of a locked parameter. We also obviously don't pass + * FOLL_REMOTE in here. + */ +long get_user_pages(unsigned long start, unsigned long nr_pages, + unsigned int gup_flags, struct page **pages, + struct vm_area_struct **vmas) +{ + return __gup_longterm_locked(current, current->mm, start, nr_pages, + pages, vmas, gup_flags | FOLL_TOUCH); +} +EXPORT_SYMBOL(get_user_pages); + /** * populate_vma_page_range() - populate a range of pages in the vma. * @vma: target vma diff --git a/mm/gup_benchmark.c b/mm/gup_benchmark.c index 5b42d3d4b60a..c898e2e0d1e4 100644 --- a/mm/gup_benchmark.c +++ b/mm/gup_benchmark.c @@ -54,8 +54,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, pages + i); break; case GUP_LONGTERM_BENCHMARK: - nr = get_user_pages_longterm(addr, nr, gup->flags & 1, - pages + i, NULL); + nr = get_user_pages(addr, nr, + (gup->flags & 1) | FOLL_LONGTERM, + pages + i, NULL); break; case GUP_BENCHMARK: nr = get_user_pages(addr, nr, gup->flags & 1, pages + i, -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: From: ira.weiny@intel.com Subject: [PATCH V2 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM Date: Wed, 13 Feb 2019 15:04:49 -0800 Message-Id: <20190213230455.5605-2-ira.weiny@intel.com> In-Reply-To: <20190213230455.5605-1-ira.weiny@intel.com> References: <20190211201643.7599-1-ira.weiny@intel.com> <20190213230455.5605-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit To: linux-mips@vger.kernel.org, linux-kernel@vger.kernel.org, kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org, linux-fpga@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-rdma@vger.kernel.org, linux-media@vger.kernel.org, linux-scsi@vger.kernel.org, devel@driverdev.osuosl.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-fbdev@vger.kernel.org, xen-devel@lists.xenproject.org, devel@lists.orangefs.org, linux-mm@kvack.org, ceph-devel@vger.kernel.org, rds-devel@oss.oracle.com Cc: Ira Weiny , John Hubbard , David Hildenbrand , Cornelia Huck , Yoshinori Sato , Rich Felker , "David S. Miller" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Joerg Roedel , Wu Hao , Alan Tull , Moritz Fischer , David Airlie , Daniel Vetter , Jason Gunthorpe , Dennis Dalessandro , Christian Benvenuti , Mauro Carvalho Chehab , Matt Porter , Alexandre Bounine , =?UTF-8?q?Kai=20M=C3=A4kisara?= , "James E.J. Bottomley" , "Martin K. Petersen" , Rob Springer , Todd Poynor , Ben Chan , Jens Wiklander , Alex Williamson , "Michael S. Tsirkin" , Jason Wang , Bartlomiej Zolnierkiewicz , Stefano Stabellini , Martin Brandenburg , Peter Zijlstra , Alexander Viro , Andrew Morton , Michal Hocko , "Kirill A. Shutemov" List-ID: From: Ira Weiny Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it. This patch does not change any functionality. FOLL_LONGTERM can only be supported with get_user_pages() as it requires vmas to determine if DAX is in use. Signed-off-by: Ira Weiny --- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/vfio/vfio_iommu_type1.c | 3 +- include/linux/mm.h | 13 +- mm/gup.c | 138 ++++++++++++--------- mm/gup_benchmark.c | 5 +- 8 files changed, 101 insertions(+), 86 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index b69d3efa8712..120a40df91b4 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -185,10 +185,11 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, while (npages) { down_read(&mm->mmap_sem); - ret = get_user_pages_longterm(cur_base, + ret = get_user_pages(cur_base, min_t(unsigned long, npages, PAGE_SIZE / sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags | FOLL_LONGTERM, + page_list, vma_list); if (ret < 0) { up_read(&mm->mmap_sem); goto umem_release; diff --git a/drivers/infiniband/hw/qib/qib_user_pages.c b/drivers/infiniband/hw/qib/qib_user_pages.c index ef8bcf366ddc..1b9368261035 100644 --- a/drivers/infiniband/hw/qib/qib_user_pages.c +++ b/drivers/infiniband/hw/qib/qib_user_pages.c @@ -114,10 +114,10 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages, down_read(¤t->mm->mmap_sem); for (got = 0; got < num_pages; got += ret) { - ret = get_user_pages_longterm(start_page + got * PAGE_SIZE, - num_pages - got, - FOLL_WRITE | FOLL_FORCE, - p + got, NULL); + ret = get_user_pages(start_page + got * PAGE_SIZE, + num_pages - got, + FOLL_LONGTERM | FOLL_WRITE | FOLL_FORCE, + p + got, NULL); if (ret < 0) { up_read(¤t->mm->mmap_sem); goto bail_release; diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.c b/drivers/infiniband/hw/usnic/usnic_uiom.c index 06862a6af185..1d9a182ac163 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom.c @@ -143,10 +143,11 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret = 0; while (npages) { - ret = get_user_pages_longterm(cur_base, - min_t(unsigned long, npages, - PAGE_SIZE / sizeof(struct page *)), - gup_flags, page_list, NULL); + ret = get_user_pages(cur_base, + min_t(unsigned long, npages, + PAGE_SIZE / sizeof(struct page *)), + gup_flags | FOLL_LONGTERM, + page_list, NULL); if (ret < 0) goto out; diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c index 08929c087e27..870a2a526e0b 100644 --- a/drivers/media/v4l2-core/videobuf-dma-sg.c +++ b/drivers/media/v4l2-core/videobuf-dma-sg.c @@ -186,12 +186,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma, dprintk(1, "init user [0x%lx+0x%lx => %d pages]\n", data, size, dma->nr_pages); - err = get_user_pages_longterm(data & PAGE_MASK, dma->nr_pages, - flags, dma->pages, NULL); + err = get_user_pages(data & PAGE_MASK, dma->nr_pages, + flags | FOLL_LONGTERM, dma->pages, NULL); if (err != dma->nr_pages) { dma->nr_pages = (err >= 0) ? err : 0; - dprintk(1, "get_user_pages_longterm: err=%d [%d]\n", err, + dprintk(1, "get_user_pages: err=%d [%d]\n", err, dma->nr_pages); return err < 0 ? err : -EINVAL; } diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 73652e21efec..1500bd0bb6da 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -351,7 +351,8 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, down_read(&mm->mmap_sem); if (mm == current->mm) { - ret = get_user_pages_longterm(vaddr, 1, flags, page, vmas); + ret = get_user_pages(vaddr, 1, flags | FOLL_LONGTERM, page, + vmas); } else { ret = get_user_pages_remote(NULL, mm, vaddr, 1, flags, page, vmas, NULL); diff --git a/include/linux/mm.h b/include/linux/mm.h index 80bb6408fe73..05a105d9d4c3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1536,18 +1536,6 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, int *locked); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags); -#ifdef CONFIG_FS_DAX -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas); -#else -static inline long get_user_pages_longterm(unsigned long start, - unsigned long nr_pages, unsigned int gup_flags, - struct page **pages, struct vm_area_struct **vmas) -{ - return get_user_pages(start, nr_pages, gup_flags, pages, vmas); -} -#endif /* CONFIG_FS_DAX */ int get_user_pages_fast(unsigned long start, int nr_pages, int write, struct page **pages); @@ -2615,6 +2603,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ +#define FOLL_LONGTERM 0x10000 /* mapping is intended for a long term pin */ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) { diff --git a/mm/gup.c b/mm/gup.c index b63e88eca31b..ee96eaff118c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1109,87 +1109,109 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, } EXPORT_SYMBOL(get_user_pages_remote); -/* - * This is the same as get_user_pages_remote(), just with a - * less-flexible calling convention where we assume that the task - * and mm being operated on are the current task's and don't allow - * passing of a locked parameter. We also obviously don't pass - * FOLL_REMOTE in here. - */ -long get_user_pages(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas) -{ - return __get_user_pages_locked(current, current->mm, start, nr_pages, - pages, vmas, NULL, - gup_flags | FOLL_TOUCH); -} -EXPORT_SYMBOL(get_user_pages); - #ifdef CONFIG_FS_DAX /* - * This is the same as get_user_pages() in that it assumes we are - * operating on the current task's mm, but it goes further to validate - * that the vmas associated with the address range are suitable for - * longterm elevated page reference counts. For example, filesystem-dax - * mappings are subject to the lifetime enforced by the filesystem and - * we need guarantees that longterm users like RDMA and V4L2 only - * establish mappings that have a kernel enforced revocation mechanism. + * __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which + * allows us to process the FOLL_LONGTERM flag if present. + * + * __gup_longterm_locked() validates that the vmas associated with the address + * range are suitable for longterm elevated page reference counts. For example, + * filesystem-dax mappings are subject to the lifetime enforced by the + * filesystem and we need guarantees that longterm users like RDMA and V4L2 + * only establish mappings that have a kernel enforced revocation mechanism. * * "longterm" == userspace controlled elevated page count lifetime. * Contrast this to iov_iter_get_pages() usages which are transient. */ -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas_arg) +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) { - struct vm_area_struct **vmas = vmas_arg; + struct vm_area_struct **vmas_tmp = vmas; struct vm_area_struct *vma_prev = NULL; long rc, i; - if (!pages) - return -EINVAL; - - if (!vmas) { - vmas = kcalloc(nr_pages, sizeof(struct vm_area_struct *), - GFP_KERNEL); - if (!vmas) - return -ENOMEM; + if (flags & FOLL_LONGTERM) { + if (!pages) + return -EINVAL; + + if (!vmas_tmp) { + vmas_tmp = kcalloc(nr_pages, + sizeof(struct vm_area_struct *), + GFP_KERNEL); + if (!vmas_tmp) + return -ENOMEM; + } } - rc = get_user_pages(start, nr_pages, gup_flags, pages, vmas); + rc = __get_user_pages_locked(tsk, mm, start, nr_pages, pages, + vmas_tmp, NULL, flags); - for (i = 0; i < rc; i++) { - struct vm_area_struct *vma = vmas[i]; + if (flags & FOLL_LONGTERM) { + for (i = 0; i < rc; i++) { + struct vm_area_struct *vma = vmas_tmp[i]; - if (vma == vma_prev) - continue; + if (vma == vma_prev) + continue; - vma_prev = vma; + vma_prev = vma; - if (vma_is_fsdax(vma)) - break; - } + if (vma_is_fsdax(vma)) + break; + } - /* - * Either get_user_pages() failed, or the vma validation - * succeeded, in either case we don't need to put_page() before - * returning. - */ - if (i >= rc) - goto out; + /* + * Either get_user_pages() failed, or the vma validation + * succeeded, in either case we don't need to put_page() before + * returning. + */ + if (i >= rc) + goto out; - for (i = 0; i < rc; i++) - put_page(pages[i]); - rc = -EOPNOTSUPP; + for (i = 0; i < rc; i++) + put_page(pages[i]); + rc = -EOPNOTSUPP; out: - if (vmas != vmas_arg) - kfree(vmas); + if (vmas_tmp != vmas) + kfree(vmas_tmp); + } + return rc; } -EXPORT_SYMBOL(get_user_pages_longterm); +#else /* !CONFIG_FS_DAX */ +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) +{ + return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas, + NULL, flags); +} #endif /* CONFIG_FS_DAX */ +/* + * This is the same as get_user_pages_remote(), just with a + * less-flexible calling convention where we assume that the task + * and mm being operated on are the current task's and don't allow + * passing of a locked parameter. We also obviously don't pass + * FOLL_REMOTE in here. + */ +long get_user_pages(unsigned long start, unsigned long nr_pages, + unsigned int gup_flags, struct page **pages, + struct vm_area_struct **vmas) +{ + return __gup_longterm_locked(current, current->mm, start, nr_pages, + pages, vmas, gup_flags | FOLL_TOUCH); +} +EXPORT_SYMBOL(get_user_pages); + /** * populate_vma_page_range() - populate a range of pages in the vma. * @vma: target vma diff --git a/mm/gup_benchmark.c b/mm/gup_benchmark.c index 5b42d3d4b60a..c898e2e0d1e4 100644 --- a/mm/gup_benchmark.c +++ b/mm/gup_benchmark.c @@ -54,8 +54,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, pages + i); break; case GUP_LONGTERM_BENCHMARK: - nr = get_user_pages_longterm(addr, nr, gup->flags & 1, - pages + i, NULL); + nr = get_user_pages(addr, nr, + (gup->flags & 1) | FOLL_LONGTERM, + pages + i, NULL); break; case GUP_BENCHMARK: nr = get_user_pages(addr, nr, gup->flags & 1, pages + i, -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB977C43381 for ; Wed, 13 Feb 2019 23:21:38 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 278F3222A4 for ; Wed, 13 Feb 2019 23:21:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 278F3222A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 440FsD0SnmzDqLc for ; Thu, 14 Feb 2019 10:21:36 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=intel.com (client-ip=134.134.136.65; helo=mga03.intel.com; envelope-from=ira.weiny@intel.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 440FVV759XzDqVm for ; Thu, 14 Feb 2019 10:05:19 +1100 (AEDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Feb 2019 15:05:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,366,1544515200"; d="scan'208";a="138415583" Received: from iweiny-desk2.sc.intel.com ([10.3.52.157]) by orsmga001.jf.intel.com with ESMTP; 13 Feb 2019 15:05:13 -0800 From: ira.weiny@intel.com To: linux-mips@vger.kernel.org, linux-kernel@vger.kernel.org, kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org, linux-fpga@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-rdma@vger.kernel.org, linux-media@vger.kernel.org, linux-scsi@vger.kernel.org, devel@driverdev.osuosl.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-fbdev@vger.kernel.org, xen-devel@lists.xenproject.org, devel@lists.orangefs.org, linux-mm@kvack.org, ceph-devel@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH V2 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM Date: Wed, 13 Feb 2019 15:04:49 -0800 Message-Id: <20190213230455.5605-2-ira.weiny@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213230455.5605-1-ira.weiny@intel.com> References: <20190211201643.7599-1-ira.weiny@intel.com> <20190213230455.5605-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Thu, 14 Feb 2019 10:16:37 +1100 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Martin Brandenburg , Rich Felker , David Hildenbrand , David Airlie , Jason Wang , =?UTF-8?q?Kai=20M=C3=A4kisara?= , Michal Hocko , Ira Weiny , Ben Chan , Rob Springer , Todd Poynor , Yoshinori Sato , Joerg Roedel , "Michael S. Tsirkin" , Jason Gunthorpe , Ingo Molnar , Wu Hao , Alan Tull , John Hubbard , "James E.J. Bottomley" , Alex Williamson , Stefano Stabellini , Borislav Petkov , Alexander Viro , Thomas Gleixner , Mauro Carvalho Chehab , Peter Zijlstra , "Martin K. Petersen" , Dennis Dalessandro , Cornelia Huck , Jens Wiklander , Alexandre Bounine , Bartlomiej Zolnierkiewicz , Daniel Vetter , Moritz Fischer , Andrew Morton , "David S. Miller" , "Kirill A. Shutemov" , Christian Benvenuti Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Ira Weiny Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it. This patch does not change any functionality. FOLL_LONGTERM can only be supported with get_user_pages() as it requires vmas to determine if DAX is in use. Signed-off-by: Ira Weiny --- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/vfio/vfio_iommu_type1.c | 3 +- include/linux/mm.h | 13 +- mm/gup.c | 138 ++++++++++++--------- mm/gup_benchmark.c | 5 +- 8 files changed, 101 insertions(+), 86 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index b69d3efa8712..120a40df91b4 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -185,10 +185,11 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, while (npages) { down_read(&mm->mmap_sem); - ret = get_user_pages_longterm(cur_base, + ret = get_user_pages(cur_base, min_t(unsigned long, npages, PAGE_SIZE / sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags | FOLL_LONGTERM, + page_list, vma_list); if (ret < 0) { up_read(&mm->mmap_sem); goto umem_release; diff --git a/drivers/infiniband/hw/qib/qib_user_pages.c b/drivers/infiniband/hw/qib/qib_user_pages.c index ef8bcf366ddc..1b9368261035 100644 --- a/drivers/infiniband/hw/qib/qib_user_pages.c +++ b/drivers/infiniband/hw/qib/qib_user_pages.c @@ -114,10 +114,10 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages, down_read(¤t->mm->mmap_sem); for (got = 0; got < num_pages; got += ret) { - ret = get_user_pages_longterm(start_page + got * PAGE_SIZE, - num_pages - got, - FOLL_WRITE | FOLL_FORCE, - p + got, NULL); + ret = get_user_pages(start_page + got * PAGE_SIZE, + num_pages - got, + FOLL_LONGTERM | FOLL_WRITE | FOLL_FORCE, + p + got, NULL); if (ret < 0) { up_read(¤t->mm->mmap_sem); goto bail_release; diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.c b/drivers/infiniband/hw/usnic/usnic_uiom.c index 06862a6af185..1d9a182ac163 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom.c @@ -143,10 +143,11 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret = 0; while (npages) { - ret = get_user_pages_longterm(cur_base, - min_t(unsigned long, npages, - PAGE_SIZE / sizeof(struct page *)), - gup_flags, page_list, NULL); + ret = get_user_pages(cur_base, + min_t(unsigned long, npages, + PAGE_SIZE / sizeof(struct page *)), + gup_flags | FOLL_LONGTERM, + page_list, NULL); if (ret < 0) goto out; diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c index 08929c087e27..870a2a526e0b 100644 --- a/drivers/media/v4l2-core/videobuf-dma-sg.c +++ b/drivers/media/v4l2-core/videobuf-dma-sg.c @@ -186,12 +186,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma, dprintk(1, "init user [0x%lx+0x%lx => %d pages]\n", data, size, dma->nr_pages); - err = get_user_pages_longterm(data & PAGE_MASK, dma->nr_pages, - flags, dma->pages, NULL); + err = get_user_pages(data & PAGE_MASK, dma->nr_pages, + flags | FOLL_LONGTERM, dma->pages, NULL); if (err != dma->nr_pages) { dma->nr_pages = (err >= 0) ? err : 0; - dprintk(1, "get_user_pages_longterm: err=%d [%d]\n", err, + dprintk(1, "get_user_pages: err=%d [%d]\n", err, dma->nr_pages); return err < 0 ? err : -EINVAL; } diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 73652e21efec..1500bd0bb6da 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -351,7 +351,8 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, down_read(&mm->mmap_sem); if (mm == current->mm) { - ret = get_user_pages_longterm(vaddr, 1, flags, page, vmas); + ret = get_user_pages(vaddr, 1, flags | FOLL_LONGTERM, page, + vmas); } else { ret = get_user_pages_remote(NULL, mm, vaddr, 1, flags, page, vmas, NULL); diff --git a/include/linux/mm.h b/include/linux/mm.h index 80bb6408fe73..05a105d9d4c3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1536,18 +1536,6 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, int *locked); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags); -#ifdef CONFIG_FS_DAX -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas); -#else -static inline long get_user_pages_longterm(unsigned long start, - unsigned long nr_pages, unsigned int gup_flags, - struct page **pages, struct vm_area_struct **vmas) -{ - return get_user_pages(start, nr_pages, gup_flags, pages, vmas); -} -#endif /* CONFIG_FS_DAX */ int get_user_pages_fast(unsigned long start, int nr_pages, int write, struct page **pages); @@ -2615,6 +2603,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000 /* internal GUP flag */ #define FOLL_ANON 0x8000 /* don't do file mappings */ +#define FOLL_LONGTERM 0x10000 /* mapping is intended for a long term pin */ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) { diff --git a/mm/gup.c b/mm/gup.c index b63e88eca31b..ee96eaff118c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1109,87 +1109,109 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, } EXPORT_SYMBOL(get_user_pages_remote); -/* - * This is the same as get_user_pages_remote(), just with a - * less-flexible calling convention where we assume that the task - * and mm being operated on are the current task's and don't allow - * passing of a locked parameter. We also obviously don't pass - * FOLL_REMOTE in here. - */ -long get_user_pages(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas) -{ - return __get_user_pages_locked(current, current->mm, start, nr_pages, - pages, vmas, NULL, - gup_flags | FOLL_TOUCH); -} -EXPORT_SYMBOL(get_user_pages); - #ifdef CONFIG_FS_DAX /* - * This is the same as get_user_pages() in that it assumes we are - * operating on the current task's mm, but it goes further to validate - * that the vmas associated with the address range are suitable for - * longterm elevated page reference counts. For example, filesystem-dax - * mappings are subject to the lifetime enforced by the filesystem and - * we need guarantees that longterm users like RDMA and V4L2 only - * establish mappings that have a kernel enforced revocation mechanism. + * __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which + * allows us to process the FOLL_LONGTERM flag if present. + * + * __gup_longterm_locked() validates that the vmas associated with the address + * range are suitable for longterm elevated page reference counts. For example, + * filesystem-dax mappings are subject to the lifetime enforced by the + * filesystem and we need guarantees that longterm users like RDMA and V4L2 + * only establish mappings that have a kernel enforced revocation mechanism. * * "longterm" == userspace controlled elevated page count lifetime. * Contrast this to iov_iter_get_pages() usages which are transient. */ -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages, - unsigned int gup_flags, struct page **pages, - struct vm_area_struct **vmas_arg) +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) { - struct vm_area_struct **vmas = vmas_arg; + struct vm_area_struct **vmas_tmp = vmas; struct vm_area_struct *vma_prev = NULL; long rc, i; - if (!pages) - return -EINVAL; - - if (!vmas) { - vmas = kcalloc(nr_pages, sizeof(struct vm_area_struct *), - GFP_KERNEL); - if (!vmas) - return -ENOMEM; + if (flags & FOLL_LONGTERM) { + if (!pages) + return -EINVAL; + + if (!vmas_tmp) { + vmas_tmp = kcalloc(nr_pages, + sizeof(struct vm_area_struct *), + GFP_KERNEL); + if (!vmas_tmp) + return -ENOMEM; + } } - rc = get_user_pages(start, nr_pages, gup_flags, pages, vmas); + rc = __get_user_pages_locked(tsk, mm, start, nr_pages, pages, + vmas_tmp, NULL, flags); - for (i = 0; i < rc; i++) { - struct vm_area_struct *vma = vmas[i]; + if (flags & FOLL_LONGTERM) { + for (i = 0; i < rc; i++) { + struct vm_area_struct *vma = vmas_tmp[i]; - if (vma == vma_prev) - continue; + if (vma == vma_prev) + continue; - vma_prev = vma; + vma_prev = vma; - if (vma_is_fsdax(vma)) - break; - } + if (vma_is_fsdax(vma)) + break; + } - /* - * Either get_user_pages() failed, or the vma validation - * succeeded, in either case we don't need to put_page() before - * returning. - */ - if (i >= rc) - goto out; + /* + * Either get_user_pages() failed, or the vma validation + * succeeded, in either case we don't need to put_page() before + * returning. + */ + if (i >= rc) + goto out; - for (i = 0; i < rc; i++) - put_page(pages[i]); - rc = -EOPNOTSUPP; + for (i = 0; i < rc; i++) + put_page(pages[i]); + rc = -EOPNOTSUPP; out: - if (vmas != vmas_arg) - kfree(vmas); + if (vmas_tmp != vmas) + kfree(vmas_tmp); + } + return rc; } -EXPORT_SYMBOL(get_user_pages_longterm); +#else /* !CONFIG_FS_DAX */ +static __always_inline long __gup_longterm_locked(struct task_struct *tsk, + struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int flags) +{ + return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas, + NULL, flags); +} #endif /* CONFIG_FS_DAX */ +/* + * This is the same as get_user_pages_remote(), just with a + * less-flexible calling convention where we assume that the task + * and mm being operated on are the current task's and don't allow + * passing of a locked parameter. We also obviously don't pass + * FOLL_REMOTE in here. + */ +long get_user_pages(unsigned long start, unsigned long nr_pages, + unsigned int gup_flags, struct page **pages, + struct vm_area_struct **vmas) +{ + return __gup_longterm_locked(current, current->mm, start, nr_pages, + pages, vmas, gup_flags | FOLL_TOUCH); +} +EXPORT_SYMBOL(get_user_pages); + /** * populate_vma_page_range() - populate a range of pages in the vma. * @vma: target vma diff --git a/mm/gup_benchmark.c b/mm/gup_benchmark.c index 5b42d3d4b60a..c898e2e0d1e4 100644 --- a/mm/gup_benchmark.c +++ b/mm/gup_benchmark.c @@ -54,8 +54,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, pages + i); break; case GUP_LONGTERM_BENCHMARK: - nr = get_user_pages_longterm(addr, nr, gup->flags & 1, - pages + i, NULL); + nr = get_user_pages(addr, nr, + (gup->flags & 1) | FOLL_LONGTERM, + pages + i, NULL); break; case GUP_BENCHMARK: nr = get_user_pages(addr, nr, gup->flags & 1, pages + i, -- 2.20.1