From mboxrd@z Thu Jan 1 00:00:00 1970 From: Minchan Kim Subject: [PATCH v5 02/12] mm: migrate: support non-lru movable page migration Date: Mon, 9 May 2016 11:20:23 +0900 Message-ID: <1462760433-32357-3-git-send-email-minchan@kernel.org> References: <1462760433-32357-1-git-send-email-minchan@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from lgeamrelo11.lge.com (LGEAMRELO11.lge.com [156.147.23.51]) by gabe.freedesktop.org (Postfix) with ESMTP id E6EE26E06C for ; Mon, 9 May 2016 02:20:22 +0000 (UTC) In-Reply-To: <1462760433-32357-1-git-send-email-minchan@kernel.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Andrew Morton Cc: Rik van Riel , Sergey Senozhatsky , Rafael Aquini , Minchan Kim , Jonathan Corbet , Hugh Dickins , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, virtualization@lists.linux-foundation.org, John Einar Reitan , linux-mm@kvack.org, Gioh Kim , Mel Gorman , Joonsoo Kim , Vlastimil Babka List-Id: dri-devel@lists.freedesktop.org V2UgaGF2ZSBhbGxvd2VkIG1pZ3JhdGlvbiBmb3Igb25seSBMUlUgcGFnZXMgdW50aWwgbm93IGFu ZCBpdCB3YXMKZW5vdWdoIHRvIG1ha2UgaGlnaC1vcmRlciBwYWdlcy4gQnV0IHJlY2VudGx5LCBl bWJlZGRlZCBzeXN0ZW0oZS5nLiwKd2ViT1MsIGFuZHJvaWQpIHVzZXMgbG90cyBvZiBub24tbW92 YWJsZSBwYWdlcyhlLmcuLCB6cmFtLCBHUFUgbWVtb3J5KQpzbyB3ZSBoYXZlIHNlZW4gc2V2ZXJh bCByZXBvcnRzIGFib3V0IHRyb3VibGVzIG9mIHNtYWxsIGhpZ2gtb3JkZXIKYWxsb2NhdGlvbi4g Rm9yIGZpeGluZyB0aGUgcHJvYmxlbSwgdGhlcmUgd2VyZSBzZXZlcmFsIGVmZm9ydHMKKGUsZywu IGVuaGFuY2UgY29tcGFjdGlvbiBhbGdvcml0aG0sIFNMVUIgZmFsbGJhY2sgdG8gMC1vcmRlciBw YWdlLApyZXNlcnZlZCBtZW1vcnksIHZtYWxsb2MgYW5kIHNvIG9uKSBidXQgaWYgdGhlcmUgYXJl IGxvdHMgb2YKbm9uLW1vdmFibGUgcGFnZXMgaW4gc3lzdGVtLCB0aGVpciBzb2x1dGlvbnMgYXJl IHZvaWQgaW4gdGhlIGxvbmcgcnVuLgoKU28sIHRoaXMgcGF0Y2ggaXMgdG8gc3VwcG9ydCBmYWNp bGl0eSB0byBjaGFuZ2Ugbm9uLW1vdmFibGUgcGFnZXMKd2l0aCBtb3ZhYmxlLiBGb3IgdGhlIGZl YXR1cmUsIHRoaXMgcGF0Y2ggaW50cm9kdWNlcyBmdW5jdGlvbnMgcmVsYXRlZAp0byBtaWdyYXRp b24gdG8gYWRkcmVzc19zcGFjZV9vcGVyYXRpb25zIGFzIHdlbGwgYXMgc29tZSBwYWdlIGZsYWdz LgoKSWYgYSBkcml2ZXIgd2FudCB0byBtYWtlIG93biBwYWdlcyBtb3ZhYmxlLCBpdCBzaG91bGQg ZGVmaW5lIHRocmVlIGZ1bmN0aW9ucwp3aGljaCBhcmUgZnVuY3Rpb24gcG9pbnRlcnMgb2Ygc3Ry dWN0IGFkZHJlc3Nfc3BhY2Vfb3BlcmF0aW9ucy4KCjEuIGJvb2wgKCppc29sYXRlX3BhZ2UpIChz dHJ1Y3QgcGFnZSAqcGFnZSwgaXNvbGF0ZV9tb2RlX3QgbW9kZSk7CgpXaGF0IFZNIGV4cGVjdHMg b24gaXNvbGF0ZV9wYWdlIGZ1bmN0aW9uIG9mIGRyaXZlciBpcyB0byByZXR1cm4gKnRydWUqCmlm IGRyaXZlciBpc29sYXRlcyBwYWdlIHN1Y2Nlc3NmdWxseS4gT24gcmV0dXJpbmcgdHJ1ZSwgVk0g bWFya3MgdGhlIHBhZ2UKYXMgUEdfaXNvbGF0ZWQgc28gY29uY3VycmVudCBpc29sYXRpb24gaW4g c2V2ZXJhbCBDUFVzIHNraXAgdGhlIHBhZ2UKZm9yIGlzb2xhdGlvbi4gSWYgYSBkcml2ZXIgY2Fu bm90IGlzb2xhdGUgdGhlIHBhZ2UsIGl0IHNob3VsZCByZXR1cm4gKmZhbHNlKi4KCk9uY2UgcGFn ZSBpcyBzdWNjZXNzZnVsbHkgaXNvbGF0ZWQsIFZNIHVzZXMgcGFnZS5scnUgZmllbGRzIHNvIGRy aXZlcgpzaG91bGRuJ3QgZXhwZWN0IHRvIHByZXNlcnZlIHZhbHVlcyBpbiB0aGF0IGZpZWxkcy4K CjIuIGludCAoKm1pZ3JhdGVwYWdlKSAoc3RydWN0IGFkZHJlc3Nfc3BhY2UgKm1hcHBpbmcsCgkJ c3RydWN0IHBhZ2UgKm5ld3BhZ2UsIHN0cnVjdCBwYWdlICpvbGRwYWdlLCBlbnVtIG1pZ3JhdGVf bW9kZSk7CgpBZnRlciBpc29sYXRpb24sIFZNIGNhbGxzIG1pZ3JhdGVwYWdlIG9mIGRyaXZlciB3 aXRoIGlzb2xhdGVkIHBhZ2UuClRoZSBmdW5jdGlvbiBvZiBtaWdyYXRlcGFnZSBpcyB0byBtb3Zl IGNvbnRlbnQgb2YgdGhlIG9sZCBwYWdlIHRvIG5ldyBwYWdlCmFuZCBzZXQgdXAgZmllbGRzIG9m IHN0cnVjdCBwYWdlIG5ld3BhZ2UuIEtlZXAgaW4gbWluZCB0aGF0IHlvdSBzaG91bGQKY2xlYXIg UEdfbW92YWJsZSBvZiBvbGRwYWdlIHZpYSBfX0NsZWFyUGFnZU1vdmFibGUgdW5kZXIgcGFnZV9s b2NrIGlmIHlvdQptaWdyYXRlZCB0aGUgb2xkcGFnZSBzdWNjZXNzZnVsbHkgYW5kIHJldHVybnMg TUlHUkFURVBBR0VfU1VDQ0VTUy4KSWYgZHJpdmVyIGNhbm5vdCBtaWdyYXRlIHRoZSBwYWdlIGF0 IHRoZSBtb21lbnQsIGRyaXZlciBjYW4gcmV0dXJuIC1FQUdBSU4uCk9uIC1FQUdBSU4sIFZNIHdp bGwgcmV0cnkgcGFnZSBtaWdyYXRpb24gaW4gYSBzaG9ydCB0aW1lIGJlY2F1c2UgVk0gaW50ZXJw cmV0cwotRUFHQUlOIGFzICJ0ZW1wb3JhbCBtaWdyYXRpb24gZmFpbHVyZSIuIE9uIHJldHVybmlu ZyBhbnkgZXJyb3IgZXhjZXB0IC1FQUdBSU4sClZNIHdpbGwgZ2l2ZSB1cCB0aGUgcGFnZSBtaWdy YXRpb24gd2l0aG91dCByZXRyeWluZyBpbiB0aGlzIHRpbWUuCgpEcml2ZXIgc2hvdWxkbid0IHRv dWNoIHBhZ2UubHJ1IGZpZWxkIFZNIHVzaW5nIGluIHRoZSBmdW5jdGlvbnMuCgozLiB2b2lkICgq cHV0YmFja19wYWdlKShzdHJ1Y3QgcGFnZSAqKTsKCklmIG1pZ3JhdGlvbiBmYWlscyBvbiBpc29s YXRlZCBwYWdlLCBWTSBzaG91bGQgcmV0dXJuIHRoZSBpc29sYXRlZCBwYWdlCnRvIHRoZSBkcml2 ZXIgc28gVk0gY2FsbHMgZHJpdmVyJ3MgcHV0YmFja19wYWdlIHdpdGggbWlncmF0aW9uIGZhaWxl ZCBwYWdlLgpJbiB0aGlzIGZ1bmN0aW9uLCBkcml2ZXIgc2hvdWxkIHB1dCB0aGUgaXNvbGF0ZWQg cGFnZSBiYWNrIHRvIHRoZSBvd24gZGF0YQpzdHJ1Y3R1cmUuCgo0LiBub24tbHJ1IG1vdmFibGUg cGFnZSBmbGFncwoKVGhlcmUgYXJlIHR3byBwYWdlIGZsYWdzIGZvciBzdXBwb3J0aW5nIG5vbi1s cnUgbW92YWJsZSBwYWdlLgoKKiBQR19tb3ZhYmxlCgpEcml2ZXIgc2hvdWxkIHVzZSB0aGUgYmVs b3cgZnVuY3Rpb24gdG8gbWFrZSBwYWdlIG1vdmFibGUgdW5kZXIgcGFnZV9sb2NrLgoKCXZvaWQg X19TZXRQYWdlTW92YWJsZShzdHJ1Y3QgcGFnZSAqcGFnZSwgc3RydWN0IGFkZHJlc3Nfc3BhY2Ug Km1hcHBpbmcpCgpJdCBuZWVkcyBhcmd1bWVudCBvZiBhZGRyZXNzX3NwYWNlIGZvciByZWdpc3Rl cmluZyBtaWdyYXRpb24gZmFtaWx5IGZ1bmN0aW9ucwp3aGljaCB3aWxsIGJlIGNhbGxlZCBieSBW TS4gRXhhY3RseSBzcGVha2luZywgUEdfbW92YWJsZSBpcyBub3QgYSByZWFsIGZsYWcgb2YKc3Ry dWN0IHBhZ2UuIFJhdGhlciB0aGFuLCBWTSByZXVzZXMgcGFnZS0+bWFwcGluZydzIGxvd2VyIGJp dHMgdG8gcmVwcmVzZW50IGl0LgoKCSNkZWZpbmUgUEFHRV9NQVBQSU5HX01PVkFCTEUgMHgyCglw YWdlLT5tYXBwaW5nID0gcGFnZS0+bWFwcGluZyB8IFBBR0VfTUFQUElOR19NT1ZBQkxFOwoKc28g ZHJpdmVyIHNob3VsZG4ndCBhY2Nlc3MgcGFnZS0+bWFwcGluZyBkaXJlY3RseS4gSW5zdGVhZCwg ZHJpdmVyIHNob3VsZAp1c2UgcGFnZV9tYXBwaW5nIHdoaWNoIG1hc2sgb2ZmIHRoZSBsb3cgdHdv IGJpdHMgb2YgcGFnZS0+bWFwcGluZyBzbyBpdCBjYW4gZ2V0CnJpZ2h0IHN0cnVjdCBhZGRyZXNz X3NwYWNlLgoKRm9yIHRlc3Rpbmcgb2Ygbm9uLWxydSBtb3ZhYmxlIHBhZ2UsIFZNIHN1cHBvcnRz IF9fUGFnZU1vdmFibGUgZnVuY3Rpb24uCkhvd2V2ZXIsIGl0IGRvZXNuJ3QgZ3VhcmFudGVlIHRv IGlkZW50aWZ5IG5vbi1scnUgbW92YWJsZSBwYWdlIGJlY2F1c2UKcGFnZS0+bWFwcGluZyBmaWVs ZCBpcyB1bmlmaWVkIHdpdGggb3RoZXIgdmFyaWFibGVzIGluIHN0cnVjdCBwYWdlLgpBcyB3ZWxs LCBpZiBkcml2ZXIgcmVsZWFzZXMgdGhlIHBhZ2UgYWZ0ZXIgaXNvbGF0aW9uIGJ5IFZNLCBwYWdl LT5tYXBwaW5nCmRvZXNuJ3QgaGF2ZSBzdGFibGUgdmFsdWUgYWx0aG91Z2ggaXQgaGFzIFBBR0Vf TUFQUElOR19NT1ZBQkxFCihMb29rIGF0IF9fQ2xlYXJQYWdlTW92YWJsZSkuIEJ1dCBfX1BhZ2VN b3ZhYmxlIGlzIGNoZWFwIHRvIGNhdGNoIHdoZXRoZXIKcGFnZSBpcyBMUlUgb3Igbm9uLWxydSBt b3ZhYmxlIG9uY2UgdGhlIHBhZ2UgaGFzIGJlZW4gaXNvbGF0ZWQuIEJlY2F1c2UKTFJVIHBhZ2Vz IG5ldmVyIGNhbiBoYXZlIFBBR0VfTUFQUElOR19NT1ZBQkxFIGluIHBhZ2UtPm1hcHBpbmcuIEl0 IGlzIGFsc28KZ29vZCBmb3IganVzdCBwZWVraW5nIHRvIHRlc3Qgbm9uLWxydSBtb3ZhYmxlIHBh Z2VzIGJlZm9yZSBtb3JlIGV4cGVuc2l2ZQpjaGVja2luZyB3aXRoIGxvY2tfcGFnZSBpbiBwZm4g c2Nhbm5pbmcgdG8gc2VsZWN0IHZpY3RpbS4KCkZvciBndWFyYW50ZWVpbmcgbm9uLWxydSBtb3Zh YmxlIHBhZ2UsIFZNIHByb3ZpZGVzIFBhZ2VNb3ZhYmxlIGZ1bmN0aW9uLgpVbmxpa2UgX19QYWdl TW92YWJsZSwgUGFnZU1vdmFibGUgZnVuY3Rpb25zIHZhbGlkYXRlcyBwYWdlLT5tYXBwaW5nIGFu ZAptYXBwaW5nLT5hX29wcy0+aXNvbGF0ZV9wYWdlIHVuZGVyIGxvY2tfcGFnZS4gVGhlIGxvY2tf cGFnZSBwcmV2ZW50cyBzdWRkZW4KZGVzdHJveWluZyBvZiBwYWdlLT5tYXBwaW5nLgoKRHJpdmVy IHVzaW5nIF9fU2V0UGFnZU1vdmFibGUgc2hvdWxkIGNsZWFyIHRoZSBmbGFnIHZpYSBfX0NsZWFy TW92YWJsZVBhZ2UKdW5kZXIgcGFnZV9sb2NrIGJlZm9yZSB0aGUgcmVsZWFzaW5nIHRoZSBwYWdl LgoKKiBQR19pc29sYXRlZAoKVG8gcHJldmVudCBjb25jdXJyZW50IGlzb2xhdGlvbiBhbW9uZyBz ZXZlcmFsIENQVXMsIFZNIG1hcmtzIGlzb2xhdGVkIHBhZ2UKYXMgUEdfaXNvbGF0ZWQgdW5kZXIg bG9ja19wYWdlLiBTbyBpZiBhIENQVSBlbmNvdW50ZXJzIFBHX2lzb2xhdGVkIG5vbi1scnUKbW92 YWJsZSBwYWdlLCBpdCBjYW4gc2tpcCBpdC4gRHJpdmVyIGRvZXNuJ3QgbmVlZCB0byBtYW5pcHVs YXRlIHRoZSBmbGFnCmJlY2F1c2UgVk0gd2lsbCBzZXQvY2xlYXIgaXQgYXV0b21hdGljYWxseS4g S2VlcCBpbiBtaW5kIHRoYXQgaWYgZHJpdmVyCnNlZXMgUEdfaXNvbGF0ZWQgcGFnZSwgaXQgbWVh bnMgdGhlIHBhZ2UgaGF2ZSBiZWVuIGlzb2xhdGVkIGJ5IFZNIHNvIGl0CnNob3VsZG4ndCB0b3Vj aCBwYWdlLmxydSBmaWVsZC4KUEdfaXNvbGF0ZWQgaXMgYWxpYXMgd2l0aCBQR19yZWNsYWltIGZs YWcgc28gZHJpdmVyIHNob3VsZG4ndCB1c2UgdGhlIGZsYWcKZm9yIG93biBwdXJwb3NlLgoKQ2M6 IFJpayB2YW4gUmllbCA8cmllbEByZWRoYXQuY29tPgpDYzogVmxhc3RpbWlsIEJhYmthIDx2YmFi a2FAc3VzZS5jej4KQ2M6IEpvb25zb28gS2ltIDxpYW1qb29uc29vLmtpbUBsZ2UuY29tPgpDYzog TWVsIEdvcm1hbiA8bWdvcm1hbkBzdXNlLmRlPgpDYzogSHVnaCBEaWNraW5zIDxodWdoZEBnb29n bGUuY29tPgpDYzogUmFmYWVsIEFxdWluaSA8YXF1aW5pQHJlZGhhdC5jb20+CkNjOiB2aXJ0dWFs aXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpDYzogSm9uYXRoYW4gQ29yYmV0IDxj b3JiZXRAbHduLm5ldD4KQ2M6IEpvaG4gRWluYXIgUmVpdGFuIDxqb2huLnJlaXRhbkBmb3NzLmFy bS5jb20+CkNjOiBkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCkNjOiBTZXJnZXkgU2Vu b3poYXRza3kgPHNlcmdleS5zZW5vemhhdHNreUBnbWFpbC5jb20+ClNpZ25lZC1vZmYtYnk6IEdp b2ggS2ltIDxnaS1vaC5raW1AcHJvZml0YnJpY2tzLmNvbT4KU2lnbmVkLW9mZi1ieTogTWluY2hh biBLaW0gPG1pbmNoYW5Aa2VybmVsLm9yZz4KLS0tCiBEb2N1bWVudGF0aW9uL2ZpbGVzeXN0ZW1z L0xvY2tpbmcgfCAgIDQgKwogRG9jdW1lbnRhdGlvbi9maWxlc3lzdGVtcy92ZnMudHh0IHwgIDEx ICsrCiBEb2N1bWVudGF0aW9uL3ZtL3BhZ2VfbWlncmF0aW9uICAgfCAxMDcgKysrKysrKysrKysr KysrKystCiBpbmNsdWRlL2xpbnV4L2ZzLmggICAgICAgICAgICAgICAgfCAgIDIgKwogaW5jbHVk ZS9saW51eC9rc20uaCAgICAgICAgICAgICAgIHwgICAzICstCiBpbmNsdWRlL2xpbnV4L21pZ3Jh dGUuaCAgICAgICAgICAgfCAgIDUgKwogaW5jbHVkZS9saW51eC9tbS5oICAgICAgICAgICAgICAg IHwgICAxICsKIGluY2x1ZGUvbGludXgvcGFnZS1mbGFncy5oICAgICAgICB8ICAyOSArKystLQog bW0vY29tcGFjdGlvbi5jICAgICAgICAgICAgICAgICAgIHwgIDQ3ICsrKysrLS0tCiBtbS9rc20u YyAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgIDQgKy0KIG1tL21pZ3JhdGUuYyAgICAgICAg ICAgICAgICAgICAgICB8IDIyMiArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrLS0t LQogbW0vcGFnZV9hbGxvYy5jICAgICAgICAgICAgICAgICAgIHwgICAyICstCiBtbS91dGlsLmMg ICAgICAgICAgICAgICAgICAgICAgICAgfCAgIDYgKy0KIDEzIGZpbGVzIGNoYW5nZWQsIDM5MSBp bnNlcnRpb25zKCspLCA1MiBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9Eb2N1bWVudGF0aW9u L2ZpbGVzeXN0ZW1zL0xvY2tpbmcgYi9Eb2N1bWVudGF0aW9uL2ZpbGVzeXN0ZW1zL0xvY2tpbmcK aW5kZXggNzVlZWE3Y2UzZDdjLi5kZGE2ZTNmOGUyMDMgMTAwNjQ0Ci0tLSBhL0RvY3VtZW50YXRp b24vZmlsZXN5c3RlbXMvTG9ja2luZworKysgYi9Eb2N1bWVudGF0aW9uL2ZpbGVzeXN0ZW1zL0xv Y2tpbmcKQEAgLTE5NSw3ICsxOTUsOSBAQCB1bmxvY2tzIGFuZCBkcm9wcyB0aGUgcmVmZXJlbmNl LgogCWludCAoKnJlbGVhc2VwYWdlKSAoc3RydWN0IHBhZ2UgKiwgaW50KTsKIAl2b2lkICgqZnJl ZXBhZ2UpKHN0cnVjdCBwYWdlICopOwogCWludCAoKmRpcmVjdF9JTykoc3RydWN0IGtpb2NiICos IHN0cnVjdCBpb3ZfaXRlciAqaXRlcik7CisJYm9vbCAoKmlzb2xhdGVfcGFnZSkgKHN0cnVjdCBw YWdlICosIGlzb2xhdGVfbW9kZV90KTsKIAlpbnQgKCptaWdyYXRlcGFnZSkoc3RydWN0IGFkZHJl c3Nfc3BhY2UgKiwgc3RydWN0IHBhZ2UgKiwgc3RydWN0IHBhZ2UgKik7CisJdm9pZCAoKnB1dGJh Y2tfcGFnZSkgKHN0cnVjdCBwYWdlICopOwogCWludCAoKmxhdW5kZXJfcGFnZSkoc3RydWN0IHBh Z2UgKik7CiAJaW50ICgqaXNfcGFydGlhbGx5X3VwdG9kYXRlKShzdHJ1Y3QgcGFnZSAqLCB1bnNp Z25lZCBsb25nLCB1bnNpZ25lZCBsb25nKTsKIAlpbnQgKCplcnJvcl9yZW1vdmVfcGFnZSkoc3Ry dWN0IGFkZHJlc3Nfc3BhY2UgKiwgc3RydWN0IHBhZ2UgKik7CkBAIC0yMTksNyArMjIxLDkgQEAg aW52YWxpZGF0ZXBhZ2U6CQl5ZXMKIHJlbGVhc2VwYWdlOgkJeWVzCiBmcmVlcGFnZToJCXllcwog ZGlyZWN0X0lPOgoraXNvbGF0ZV9wYWdlOgkJeWVzCiBtaWdyYXRlcGFnZToJCXllcyAoYm90aCkK K3B1dGJhY2tfcGFnZToJCXllcwogbGF1bmRlcl9wYWdlOgkJeWVzCiBpc19wYXJ0aWFsbHlfdXB0 b2RhdGU6CXllcwogZXJyb3JfcmVtb3ZlX3BhZ2U6CXllcwpkaWZmIC0tZ2l0IGEvRG9jdW1lbnRh dGlvbi9maWxlc3lzdGVtcy92ZnMudHh0IGIvRG9jdW1lbnRhdGlvbi9maWxlc3lzdGVtcy92ZnMu dHh0CmluZGV4IGM2MWEyMjNlZjNmZi4uOTAwMzYwY2JjZGFlIDEwMDY0NAotLS0gYS9Eb2N1bWVu dGF0aW9uL2ZpbGVzeXN0ZW1zL3Zmcy50eHQKKysrIGIvRG9jdW1lbnRhdGlvbi9maWxlc3lzdGVt cy92ZnMudHh0CkBAIC01OTIsOSArNTkyLDE0IEBAIHN0cnVjdCBhZGRyZXNzX3NwYWNlX29wZXJh dGlvbnMgewogCWludCAoKnJlbGVhc2VwYWdlKSAoc3RydWN0IHBhZ2UgKiwgaW50KTsKIAl2b2lk ICgqZnJlZXBhZ2UpKHN0cnVjdCBwYWdlICopOwogCXNzaXplX3QgKCpkaXJlY3RfSU8pKHN0cnVj dCBraW9jYiAqLCBzdHJ1Y3QgaW92X2l0ZXIgKml0ZXIpOworCS8qIGlzb2xhdGUgYSBwYWdlIGZv ciBtaWdyYXRpb24gKi8KKwlib29sICgqaXNvbGF0ZV9wYWdlKSAoc3RydWN0IHBhZ2UgKiwgaXNv bGF0ZV9tb2RlX3QpOwogCS8qIG1pZ3JhdGUgdGhlIGNvbnRlbnRzIG9mIGEgcGFnZSB0byB0aGUg c3BlY2lmaWVkIHRhcmdldCAqLwogCWludCAoKm1pZ3JhdGVwYWdlKSAoc3RydWN0IHBhZ2UgKiwg c3RydWN0IHBhZ2UgKik7CisJLyogcHV0IG1pZ3JhdGlvbi1mYWlsZWQgcGFnZSBiYWNrIHRvIHJp Z2h0IGxpc3QgKi8KKwl2b2lkICgqcHV0YmFja19wYWdlKSAoc3RydWN0IHBhZ2UgKik7CiAJaW50 ICgqbGF1bmRlcl9wYWdlKSAoc3RydWN0IHBhZ2UgKik7CisKIAlpbnQgKCppc19wYXJ0aWFsbHlf dXB0b2RhdGUpIChzdHJ1Y3QgcGFnZSAqLCB1bnNpZ25lZCBsb25nLAogCQkJCQl1bnNpZ25lZCBs b25nKTsKIAl2b2lkICgqaXNfZGlydHlfd3JpdGViYWNrKSAoc3RydWN0IHBhZ2UgKiwgYm9vbCAq LCBib29sICopOwpAQCAtNzQ3LDYgKzc1MiwxMCBAQCBzdHJ1Y3QgYWRkcmVzc19zcGFjZV9vcGVy YXRpb25zIHsKICAgICAgICAgYW5kIHRyYW5zZmVyIGRhdGEgZGlyZWN0bHkgYmV0d2VlbiB0aGUg c3RvcmFnZSBhbmQgdGhlCiAgICAgICAgIGFwcGxpY2F0aW9uJ3MgYWRkcmVzcyBzcGFjZS4KIAor ICBpc29sYXRlX3BhZ2U6IENhbGxlZCBieSB0aGUgVk0gd2hlbiBpc29sYXRpbmcgYSBtb3ZhYmxl IG5vbi1scnUgcGFnZS4KKwlJZiBwYWdlIGlzIHN1Y2Nlc3NmdWxseSBpc29sYXRlZCwgVk0gbWFy a3MgdGhlIHBhZ2UgYXMgUEdfaXNvbGF0ZWQKKwl2aWEgX19TZXRQYWdlSXNvbGF0ZWQuCisKICAg bWlncmF0ZV9wYWdlOiAgVGhpcyBpcyB1c2VkIHRvIGNvbXBhY3QgdGhlIHBoeXNpY2FsIG1lbW9y eSB1c2FnZS4KICAgICAgICAgSWYgdGhlIFZNIHdhbnRzIHRvIHJlbG9jYXRlIGEgcGFnZSAobWF5 YmUgb2ZmIGEgbWVtb3J5IGNhcmQKICAgICAgICAgdGhhdCBpcyBzaWduYWxsaW5nIGltbWluZW50 IGZhaWx1cmUpIGl0IHdpbGwgcGFzcyBhIG5ldyBwYWdlCkBAIC03NTQsNiArNzYzLDggQEAgc3Ry dWN0IGFkZHJlc3Nfc3BhY2Vfb3BlcmF0aW9ucyB7CiAJdHJhbnNmZXIgYW55IHByaXZhdGUgZGF0 YSBhY3Jvc3MgYW5kIHVwZGF0ZSBhbnkgcmVmZXJlbmNlcwogICAgICAgICB0aGF0IGl0IGhhcyB0 byB0aGUgcGFnZS4KIAorICBwdXRiYWNrX3BhZ2U6IENhbGxlZCBieSB0aGUgVk0gd2hlbiBpc29s YXRlZCBwYWdlJ3MgbWlncmF0aW9uIGZhaWxzLgorCiAgIGxhdW5kZXJfcGFnZTogQ2FsbGVkIGJl Zm9yZSBmcmVlaW5nIGEgcGFnZSAtIGl0IHdyaXRlcyBiYWNrIHRoZSBkaXJ0eSBwYWdlLiBUbwog ICAJcHJldmVudCByZWRpcnR5aW5nIHRoZSBwYWdlLCBpdCBpcyBrZXB0IGxvY2tlZCBkdXJpbmcg dGhlIHdob2xlCiAJb3BlcmF0aW9uLgpkaWZmIC0tZ2l0IGEvRG9jdW1lbnRhdGlvbi92bS9wYWdl X21pZ3JhdGlvbiBiL0RvY3VtZW50YXRpb24vdm0vcGFnZV9taWdyYXRpb24KaW5kZXggZmVhNWMw ODY0MTcwLi5iODlkMWRlMDI2ZGYgMTAwNjQ0Ci0tLSBhL0RvY3VtZW50YXRpb24vdm0vcGFnZV9t aWdyYXRpb24KKysrIGIvRG9jdW1lbnRhdGlvbi92bS9wYWdlX21pZ3JhdGlvbgpAQCAtMTQyLDUg KzE0MiwxMTAgQEAgaXMgaW5jcmVhc2VkIHNvIHRoYXQgdGhlIHBhZ2UgY2Fubm90IGJlIGZyZWVk IHdoaWxlIHBhZ2UgbWlncmF0aW9uIG9jY3Vycy4KIDIwLiBUaGUgbmV3IHBhZ2UgaXMgbW92ZWQg dG8gdGhlIExSVSBhbmQgY2FuIGJlIHNjYW5uZWQgYnkgdGhlIHN3YXBwZXIKICAgICBldGMgYWdh aW4uCiAKLUNocmlzdG9waCBMYW1ldGVyLCBNYXkgOCwgMjAwNi4KK0MuIE5vbi1MUlUgcGFnZSBt aWdyYXRpb24KKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KKworQWx0aG91Z2ggb3JpZ2luYWwg bWlncmF0aW9uIGFpbWVkIGZvciByZWR1Y2luZyB0aGUgbGF0ZW5jeSBvZiBtZW1vcnkgYWNjZXNz Citmb3IgTlVNQSwgY29tcGFjdGlvbiB3aG8gd2FudCB0byBjcmVhdGUgaGlnaC1vcmRlciBwYWdl IGlzIGFsc28gbWFpbiBjdXN0b21lci4KKworQ3VycmVudCBwcm9ibGVtIG9mIHRoZSBpbXBsZW1l bnRhdGlvbiBpcyB0aGF0IGl0IGlzIGRlc2lnbmVkIHRvIG1pZ3JhdGUgb25seQorKkxSVSogcGFn ZXMuIEhvd2V2ZXIsIHRoZXJlIGFyZSBwb3RlbnRpYWwgbm9uLWxydSBwYWdlcyB3aGljaCBjYW4g YmUgbWlncmF0ZWQKK2luIGRyaXZlcnMsIGZvciBleGFtcGxlLCB6c21hbGxvYywgdmlydGlvLWJh bGxvb24gcGFnZXMuCisKK0ZvciB2aXJ0aW8tYmFsbG9vbiBwYWdlcywgc29tZSBwYXJ0cyBvZiBt aWdyYXRpb24gY29kZSBwYXRoIGhhdmUgYmVlbiBob29rZWQKK3VwIGFuZCBhZGRlZCB2aXJ0aW8t YmFsbG9vbiBzcGVjaWZpYyBmdW5jdGlvbnMgdG8gaW50ZXJjZXB0IG1pZ3JhdGlvbiBsb2dpY3Mu CitJdCdzIHRvbyBzcGVjaWZpYyB0byBhIGRyaXZlciBzbyBvdGhlciBkcml2ZXJzIHdobyB3YW50 IHRvIG1ha2UgdGhlaXIgcGFnZXMKK21vdmFibGUgd291bGQgaGF2ZSB0byBhZGQgb3duIHNwZWNp ZmljIGhvb2tzIGluIG1pZ3JhdGlvbiBwYXRoLgorCitUbyBvdmVyY2xvbWUgdGhlIHByb2JsZW0s IFZNIHN1cHBvcnRzIG5vbi1MUlUgcGFnZSBtaWdyYXRpb24gd2hpY2ggcHJvdmlkZXMKK2dlbmVy aWMgZnVuY3Rpb25zIGZvciBub24tTFJVIG1vdmFibGUgcGFnZXMgd2l0aG91dCBkcml2ZXIgc3Bl Y2lmaWMgaG9va3MKK21pZ3JhdGlvbiBwYXRoLgorCitJZiBhIGRyaXZlciB3YW50IHRvIG1ha2Ug b3duIHBhZ2VzIG1vdmFibGUsIGl0IHNob3VsZCBkZWZpbmUgdGhyZWUgZnVuY3Rpb25zCit3aGlj aCBhcmUgZnVuY3Rpb24gcG9pbnRlcnMgb2Ygc3RydWN0IGFkZHJlc3Nfc3BhY2Vfb3BlcmF0aW9u cy4KKworMS4gYm9vbCAoKmlzb2xhdGVfcGFnZSkgKHN0cnVjdCBwYWdlICpwYWdlLCBpc29sYXRl X21vZGVfdCBtb2RlKTsKKworV2hhdCBWTSBleHBlY3RzIG9uIGlzb2xhdGVfcGFnZSBmdW5jdGlv biBvZiBkcml2ZXIgaXMgdG8gcmV0dXJuICp0cnVlKgoraWYgZHJpdmVyIGlzb2xhdGVzIHBhZ2Ug c3VjY2Vzc2Z1bGx5LiBPbiByZXR1cmluZyB0cnVlLCBWTSBtYXJrcyB0aGUgcGFnZQorYXMgUEdf aXNvbGF0ZWQgc28gY29uY3VycmVudCBpc29sYXRpb24gaW4gc2V2ZXJhbCBDUFVzIHNraXAgdGhl IHBhZ2UKK2ZvciBpc29sYXRpb24uIElmIGEgZHJpdmVyIGNhbm5vdCBpc29sYXRlIHRoZSBwYWdl LCBpdCBzaG91bGQgcmV0dXJuICpmYWxzZSouCisKK09uY2UgcGFnZSBpcyBzdWNjZXNzZnVsbHkg aXNvbGF0ZWQsIFZNIHVzZXMgcGFnZS5scnUgZmllbGRzIHNvIGRyaXZlcgorc2hvdWxkbid0IGV4 cGVjdCB0byBwcmVzZXJ2ZSB2YWx1ZXMgaW4gdGhhdCBmaWVsZHMuCisKKzIuIGludCAoKm1pZ3Jh dGVwYWdlKSAoc3RydWN0IGFkZHJlc3Nfc3BhY2UgKm1hcHBpbmcsCisJCXN0cnVjdCBwYWdlICpu ZXdwYWdlLCBzdHJ1Y3QgcGFnZSAqb2xkcGFnZSwgZW51bSBtaWdyYXRlX21vZGUpOworCitBZnRl ciBpc29sYXRpb24sIFZNIGNhbGxzIG1pZ3JhdGVwYWdlIG9mIGRyaXZlciB3aXRoIGlzb2xhdGVk IHBhZ2UuCitUaGUgZnVuY3Rpb24gb2YgbWlncmF0ZXBhZ2UgaXMgdG8gbW92ZSBjb250ZW50IG9m IHRoZSBvbGQgcGFnZSB0byBuZXcgcGFnZQorYW5kIHNldCB1cCBmaWVsZHMgb2Ygc3RydWN0IHBh Z2UgbmV3cGFnZS4gS2VlcCBpbiBtaW5kIHRoYXQgeW91IHNob3VsZAorY2xlYXIgUEdfbW92YWJs ZSBvZiBvbGRwYWdlIHZpYSBfX0NsZWFyUGFnZU1vdmFibGUgdW5kZXIgcGFnZV9sb2NrIGlmIHlv dQorbWlncmF0ZWQgdGhlIG9sZHBhZ2Ugc3VjY2Vzc2Z1bGx5IGFuZCByZXR1cm5zIE1JR1JBVEVQ QUdFX1NVQ0NFU1MuCitJZiBkcml2ZXIgY2Fubm90IG1pZ3JhdGUgdGhlIHBhZ2UgYXQgdGhlIG1v bWVudCwgZHJpdmVyIGNhbiByZXR1cm4gLUVBR0FJTi4KK09uIC1FQUdBSU4sIFZNIHdpbGwgcmV0 cnkgcGFnZSBtaWdyYXRpb24gaW4gYSBzaG9ydCB0aW1lIGJlY2F1c2UgVk0gaW50ZXJwcmV0cwor LUVBR0FJTiBhcyAidGVtcG9yYWwgbWlncmF0aW9uIGZhaWx1cmUiLiBPbiByZXR1cm5pbmcgYW55 IGVycm9yIGV4Y2VwdCAtRUFHQUlOLAorVk0gd2lsbCBnaXZlIHVwIHRoZSBwYWdlIG1pZ3JhdGlv biB3aXRob3V0IHJldHJ5aW5nIGluIHRoaXMgdGltZS4KKworRHJpdmVyIHNob3VsZG4ndCB0b3Vj aCBwYWdlLmxydSBmaWVsZCBWTSB1c2luZyBpbiB0aGUgZnVuY3Rpb25zLgorCiszLiB2b2lkICgq cHV0YmFja19wYWdlKShzdHJ1Y3QgcGFnZSAqKTsKKworSWYgbWlncmF0aW9uIGZhaWxzIG9uIGlz b2xhdGVkIHBhZ2UsIFZNIHNob3VsZCByZXR1cm4gdGhlIGlzb2xhdGVkIHBhZ2UKK3RvIHRoZSBk cml2ZXIgc28gVk0gY2FsbHMgZHJpdmVyJ3MgcHV0YmFja19wYWdlIHdpdGggbWlncmF0aW9uIGZh aWxlZCBwYWdlLgorSW4gdGhpcyBmdW5jdGlvbiwgZHJpdmVyIHNob3VsZCBwdXQgdGhlIGlzb2xh dGVkIHBhZ2UgYmFjayB0byB0aGUgb3duIGRhdGEKK3N0cnVjdHVyZS4KIAorNC4gbm9uLWxydSBt b3ZhYmxlIHBhZ2UgZmxhZ3MKKworVGhlcmUgYXJlIHR3byBwYWdlIGZsYWdzIGZvciBzdXBwb3J0 aW5nIG5vbi1scnUgbW92YWJsZSBwYWdlLgorCisqIFBHX21vdmFibGUKKworRHJpdmVyIHNob3Vs ZCB1c2UgdGhlIGJlbG93IGZ1bmN0aW9uIHRvIG1ha2UgcGFnZSBtb3ZhYmxlIHVuZGVyIHBhZ2Vf bG9jay4KKworCXZvaWQgX19TZXRQYWdlTW92YWJsZShzdHJ1Y3QgcGFnZSAqcGFnZSwgc3RydWN0 IGFkZHJlc3Nfc3BhY2UgKm1hcHBpbmcpCisKK0l0IG5lZWRzIGFyZ3VtZW50IG9mIGFkZHJlc3Nf c3BhY2UgZm9yIHJlZ2lzdGVyaW5nIG1pZ3JhdGlvbiBmYW1pbHkgZnVuY3Rpb25zCit3aGljaCB3 aWxsIGJlIGNhbGxlZCBieSBWTS4gRXhhY3RseSBzcGVha2luZywgUEdfbW92YWJsZSBpcyBub3Qg YSByZWFsIGZsYWcgb2YKK3N0cnVjdCBwYWdlLiBSYXRoZXIgdGhhbiwgVk0gcmV1c2VzIHBhZ2Ut Pm1hcHBpbmcncyBsb3dlciBiaXRzIHRvIHJlcHJlc2VudCBpdC4KKworCSNkZWZpbmUgUEFHRV9N QVBQSU5HX01PVkFCTEUgMHgyCisJcGFnZS0+bWFwcGluZyA9IHBhZ2UtPm1hcHBpbmcgfCBQQUdF X01BUFBJTkdfTU9WQUJMRTsKKworc28gZHJpdmVyIHNob3VsZG4ndCBhY2Nlc3MgcGFnZS0+bWFw cGluZyBkaXJlY3RseS4gSW5zdGVhZCwgZHJpdmVyIHNob3VsZAordXNlIHBhZ2VfbWFwcGluZyB3 aGljaCBtYXNrIG9mZiB0aGUgbG93IHR3byBiaXRzIG9mIHBhZ2UtPm1hcHBpbmcgc28gaXQgY2Fu IGdldAorcmlnaHQgc3RydWN0IGFkZHJlc3Nfc3BhY2UuCisKK0ZvciB0ZXN0aW5nIG9mIG5vbi1s cnUgbW92YWJsZSBwYWdlLCBWTSBzdXBwb3J0cyBfX1BhZ2VNb3ZhYmxlIGZ1bmN0aW9uLgorSG93 ZXZlciwgaXQgZG9lc24ndCBndWFyYW50ZWUgdG8gaWRlbnRpZnkgbm9uLWxydSBtb3ZhYmxlIHBh Z2UgYmVjYXVzZQorcGFnZS0+bWFwcGluZyBmaWVsZCBpcyB1bmlmaWVkIHdpdGggb3RoZXIgdmFy aWFibGVzIGluIHN0cnVjdCBwYWdlLgorQXMgd2VsbCwgaWYgZHJpdmVyIHJlbGVhc2VzIHRoZSBw YWdlIGFmdGVyIGlzb2xhdGlvbiBieSBWTSwgcGFnZS0+bWFwcGluZworZG9lc24ndCBoYXZlIHN0 YWJsZSB2YWx1ZSBhbHRob3VnaCBpdCBoYXMgUEFHRV9NQVBQSU5HX01PVkFCTEUKKyhMb29rIGF0 IF9fQ2xlYXJQYWdlTW92YWJsZSkuIEJ1dCBfX1BhZ2VNb3ZhYmxlIGlzIGNoZWFwIHRvIGNhdGNo IHdoZXRoZXIKK3BhZ2UgaXMgTFJVIG9yIG5vbi1scnUgbW92YWJsZSBvbmNlIHRoZSBwYWdlIGhh cyBiZWVuIGlzb2xhdGVkLiBCZWNhdXNlCitMUlUgcGFnZXMgbmV2ZXIgY2FuIGhhdmUgUEFHRV9N QVBQSU5HX01PVkFCTEUgaW4gcGFnZS0+bWFwcGluZy4gSXQgaXMgYWxzbworZ29vZCBmb3IganVz dCBwZWVraW5nIHRvIHRlc3Qgbm9uLWxydSBtb3ZhYmxlIHBhZ2VzIGJlZm9yZSBtb3JlIGV4cGVu c2l2ZQorY2hlY2tpbmcgd2l0aCBsb2NrX3BhZ2UgaW4gcGZuIHNjYW5uaW5nIHRvIHNlbGVjdCB2 aWN0aW0uCisKK0ZvciBndWFyYW50ZWVpbmcgbm9uLWxydSBtb3ZhYmxlIHBhZ2UsIFZNIHByb3Zp ZGVzIFBhZ2VNb3ZhYmxlIGZ1bmN0aW9uLgorVW5saWtlIF9fUGFnZU1vdmFibGUsIFBhZ2VNb3Zh YmxlIGZ1bmN0aW9ucyB2YWxpZGF0ZXMgcGFnZS0+bWFwcGluZyBhbmQKK21hcHBpbmctPmFfb3Bz LT5pc29sYXRlX3BhZ2UgdW5kZXIgbG9ja19wYWdlLiBUaGUgbG9ja19wYWdlIHByZXZlbnRzIHN1 ZGRlbgorZGVzdHJveWluZyBvZiBwYWdlLT5tYXBwaW5nLgorCitEcml2ZXIgdXNpbmcgX19TZXRQ YWdlTW92YWJsZSBzaG91bGQgY2xlYXIgdGhlIGZsYWcgdmlhIF9fQ2xlYXJNb3ZhYmxlUGFnZQor dW5kZXIgcGFnZV9sb2NrIGJlZm9yZSB0aGUgcmVsZWFzaW5nIHRoZSBwYWdlLgorCisqIFBHX2lz b2xhdGVkCisKK1RvIHByZXZlbnQgY29uY3VycmVudCBpc29sYXRpb24gYW1vbmcgc2V2ZXJhbCBD UFVzLCBWTSBtYXJrcyBpc29sYXRlZCBwYWdlCithcyBQR19pc29sYXRlZCB1bmRlciBsb2NrX3Bh Z2UuIFNvIGlmIGEgQ1BVIGVuY291bnRlcnMgUEdfaXNvbGF0ZWQgbm9uLWxydQorbW92YWJsZSBw YWdlLCBpdCBjYW4gc2tpcCBpdC4gRHJpdmVyIGRvZXNuJ3QgbmVlZCB0byBtYW5pcHVsYXRlIHRo ZSBmbGFnCitiZWNhdXNlIFZNIHdpbGwgc2V0L2NsZWFyIGl0IGF1dG9tYXRpY2FsbHkuIEtlZXAg aW4gbWluZCB0aGF0IGlmIGRyaXZlcgorc2VlcyBQR19pc29sYXRlZCBwYWdlLCBpdCBtZWFucyB0 aGUgcGFnZSBoYXZlIGJlZW4gaXNvbGF0ZWQgYnkgVk0gc28gaXQKK3Nob3VsZG4ndCB0b3VjaCBw YWdlLmxydSBmaWVsZC4KK1BHX2lzb2xhdGVkIGlzIGFsaWFzIHdpdGggUEdfcmVjbGFpbSBmbGFn IHNvIGRyaXZlciBzaG91bGRuJ3QgdXNlIHRoZSBmbGFnCitmb3Igb3duIHB1cnBvc2UuCisKK0No cmlzdG9waCBMYW1ldGVyLCBNYXkgOCwgMjAwNi4KK01pbmNoYW4gS2ltLCBNYXIgMjgsIDIwMTYu CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L2ZzLmggYi9pbmNsdWRlL2xpbnV4L2ZzLmgKaW5k ZXggZDYxY2QwODQ5ODdlLi45MjAyOTM3N2I5ZTAgMTAwNjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgv ZnMuaAorKysgYi9pbmNsdWRlL2xpbnV4L2ZzLmgKQEAgLTQwMyw2ICs0MDMsOCBAQCBzdHJ1Y3Qg YWRkcmVzc19zcGFjZV9vcGVyYXRpb25zIHsKIAkgKi8KIAlpbnQgKCptaWdyYXRlcGFnZSkgKHN0 cnVjdCBhZGRyZXNzX3NwYWNlICosCiAJCQlzdHJ1Y3QgcGFnZSAqLCBzdHJ1Y3QgcGFnZSAqLCBl bnVtIG1pZ3JhdGVfbW9kZSk7CisJYm9vbCAoKmlzb2xhdGVfcGFnZSkoc3RydWN0IHBhZ2UgKiwg aXNvbGF0ZV9tb2RlX3QpOworCXZvaWQgKCpwdXRiYWNrX3BhZ2UpKHN0cnVjdCBwYWdlICopOwog CWludCAoKmxhdW5kZXJfcGFnZSkgKHN0cnVjdCBwYWdlICopOwogCWludCAoKmlzX3BhcnRpYWxs eV91cHRvZGF0ZSkgKHN0cnVjdCBwYWdlICosIHVuc2lnbmVkIGxvbmcsCiAJCQkJCXVuc2lnbmVk IGxvbmcpOwpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9saW51eC9rc20uaCBiL2luY2x1ZGUvbGludXgv a3NtLmgKaW5kZXggN2FlMjE2YTM5YzllLi40ODFjOGM0NjI3Y2EgMTAwNjQ0Ci0tLSBhL2luY2x1 ZGUvbGludXgva3NtLmgKKysrIGIvaW5jbHVkZS9saW51eC9rc20uaApAQCAtNDMsOCArNDMsNyBA QCBzdGF0aWMgaW5saW5lIHN0cnVjdCBzdGFibGVfbm9kZSAqcGFnZV9zdGFibGVfbm9kZShzdHJ1 Y3QgcGFnZSAqcGFnZSkKIHN0YXRpYyBpbmxpbmUgdm9pZCBzZXRfcGFnZV9zdGFibGVfbm9kZShz dHJ1Y3QgcGFnZSAqcGFnZSwKIAkJCQkJc3RydWN0IHN0YWJsZV9ub2RlICpzdGFibGVfbm9kZSkK IHsKLQlwYWdlLT5tYXBwaW5nID0gKHZvaWQgKilzdGFibGVfbm9kZSArCi0JCQkJKFBBR0VfTUFQ UElOR19BTk9OIHwgUEFHRV9NQVBQSU5HX0tTTSk7CisJcGFnZS0+bWFwcGluZyA9ICh2b2lkICop KCh1bnNpZ25lZCBsb25nKXN0YWJsZV9ub2RlIHwgUEFHRV9NQVBQSU5HX0tTTSk7CiB9CiAKIC8q CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21pZ3JhdGUuaCBiL2luY2x1ZGUvbGludXgvbWln cmF0ZS5oCmluZGV4IDliNTAzMjVlNGRkZi4uMzFhNzQ4ZjQzZmE5IDEwMDY0NAotLS0gYS9pbmNs dWRlL2xpbnV4L21pZ3JhdGUuaAorKysgYi9pbmNsdWRlL2xpbnV4L21pZ3JhdGUuaApAQCAtMzIs MTEgKzMyLDE2IEBAIGV4dGVybiBjaGFyICptaWdyYXRlX3JlYXNvbl9uYW1lc1tNUl9UWVBFU107 CiAKICNpZmRlZiBDT05GSUdfTUlHUkFUSU9OCiAKK2V4dGVybiBpbnQgUGFnZU1vdmFibGUoc3Ry dWN0IHBhZ2UgKnBhZ2UpOworZXh0ZXJuIHZvaWQgX19TZXRQYWdlTW92YWJsZShzdHJ1Y3QgcGFn ZSAqcGFnZSwgc3RydWN0IGFkZHJlc3Nfc3BhY2UgKm1hcHBpbmcpOworZXh0ZXJuIHZvaWQgX19D bGVhclBhZ2VNb3ZhYmxlKHN0cnVjdCBwYWdlICpwYWdlKTsKIGV4dGVybiB2b2lkIHB1dGJhY2tf bW92YWJsZV9wYWdlcyhzdHJ1Y3QgbGlzdF9oZWFkICpsKTsKIGV4dGVybiBpbnQgbWlncmF0ZV9w YWdlKHN0cnVjdCBhZGRyZXNzX3NwYWNlICosCiAJCQlzdHJ1Y3QgcGFnZSAqLCBzdHJ1Y3QgcGFn ZSAqLCBlbnVtIG1pZ3JhdGVfbW9kZSk7CiBleHRlcm4gaW50IG1pZ3JhdGVfcGFnZXMoc3RydWN0 IGxpc3RfaGVhZCAqbCwgbmV3X3BhZ2VfdCBuZXcsIGZyZWVfcGFnZV90IGZyZWUsCiAJCXVuc2ln bmVkIGxvbmcgcHJpdmF0ZSwgZW51bSBtaWdyYXRlX21vZGUgbW9kZSwgaW50IHJlYXNvbik7Citl eHRlcm4gYm9vbCBpc29sYXRlX21vdmFibGVfcGFnZShzdHJ1Y3QgcGFnZSAqcGFnZSwgaXNvbGF0 ZV9tb2RlX3QgbW9kZSk7CitleHRlcm4gdm9pZCBwdXRiYWNrX21vdmFibGVfcGFnZShzdHJ1Y3Qg cGFnZSAqcGFnZSk7CiAKIGV4dGVybiBpbnQgbWlncmF0ZV9wcmVwKHZvaWQpOwogZXh0ZXJuIGlu dCBtaWdyYXRlX3ByZXBfbG9jYWwodm9pZCk7CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21t LmggYi9pbmNsdWRlL2xpbnV4L21tLmgKaW5kZXggN2I1Mjc1MGNhZjllLi45MWU5ZWJjNDRlMzcg MTAwNjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgvbW0uaAorKysgYi9pbmNsdWRlL2xpbnV4L21tLmgK QEAgLTEwMjEsNiArMTAyMSw3IEBAIHN0YXRpYyBpbmxpbmUgcGdvZmZfdCBwYWdlX2ZpbGVfaW5k ZXgoc3RydWN0IHBhZ2UgKnBhZ2UpCiB9CiAKIGJvb2wgcGFnZV9tYXBwZWQoc3RydWN0IHBhZ2Ug KnBhZ2UpOworc3RydWN0IGFkZHJlc3Nfc3BhY2UgKnBhZ2VfbWFwcGluZyhzdHJ1Y3QgcGFnZSAq cGFnZSk7CiAKIC8qCiAgKiBSZXR1cm4gdHJ1ZSBvbmx5IGlmIHRoZSBwYWdlIGhhcyBiZWVuIGFs bG9jYXRlZCB3aXRoCmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L3BhZ2UtZmxhZ3MuaCBiL2lu Y2x1ZGUvbGludXgvcGFnZS1mbGFncy5oCmluZGV4IGU1YTMyNDQ1ZjkzMC4uY2NiNDZiYzk3Zjgx IDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3BhZ2UtZmxhZ3MuaAorKysgYi9pbmNsdWRlL2xp bnV4L3BhZ2UtZmxhZ3MuaApAQCAtMTI5LDYgKzEyOSw5IEBAIGVudW0gcGFnZWZsYWdzIHsKIAog CS8qIENvbXBvdW5kIHBhZ2VzLiBTdG9yZWQgaW4gZmlyc3QgdGFpbCBwYWdlJ3MgZmxhZ3MgKi8K IAlQR19kb3VibGVfbWFwID0gUEdfcHJpdmF0ZV8yLAorCisJLyogbm9uLWxydSBpc29sYXRlZCBt b3ZhYmxlIHBhZ2UgKi8KKwlQR19pc29sYXRlZCA9IFBHX3JlY2xhaW0sCiB9OwogCiAjaWZuZGVm IF9fR0VORVJBVElOR19CT1VORFNfSApAQCAtMzYxLDI1ICszNjQsMzMgQEAgUEFHRUZMQUcoSWRs ZSwgaWRsZSwgUEZfQU5ZKQogICogYW5kIHRoZW4gcGFnZS0+bWFwcGluZyBwb2ludHMsIG5vdCB0 byBhbiBhbm9uX3ZtYSwgYnV0IHRvIGEgcHJpdmF0ZQogICogc3RydWN0dXJlIHdoaWNoIEtTTSBh c3NvY2lhdGVzIHdpdGggdGhhdCBtZXJnZWQgcGFnZS4gIFNlZSBrc20uaC4KICAqCi0gKiBQQUdF X01BUFBJTkdfS1NNIHdpdGhvdXQgUEFHRV9NQVBQSU5HX0FOT04gaXMgY3VycmVudGx5IG5ldmVy IHVzZWQuCisgKiBQQUdFX01BUFBJTkdfS1NNIHdpdGhvdXQgUEFHRV9NQVBQSU5HX0FOT04gaXMg dXNlZCBmb3Igbm9uLWxydSBtb3ZhYmxlCisgKiBwYWdlIGFuZCB0aGVuIHBhZ2UtPm1hcHBpbmcg cG9pbnRzIGEgc3RydWN0IGFkZHJlc3Nfc3BhY2UuCiAgKgogICogUGxlYXNlIG5vdGUgdGhhdCwg Y29uZnVzaW5nbHksICJwYWdlX21hcHBpbmciIHJlZmVycyB0byB0aGUgaW5vZGUKICAqIGFkZHJl c3Nfc3BhY2Ugd2hpY2ggbWFwcyB0aGUgcGFnZSBmcm9tIGRpc2s7IHdoZXJlYXMgInBhZ2VfbWFw cGVkIgogICogcmVmZXJzIHRvIHVzZXIgdmlydHVhbCBhZGRyZXNzIHNwYWNlIGludG8gd2hpY2gg dGhlIHBhZ2UgaXMgbWFwcGVkLgogICovCi0jZGVmaW5lIFBBR0VfTUFQUElOR19BTk9OCTEKLSNk ZWZpbmUgUEFHRV9NQVBQSU5HX0tTTQkyCi0jZGVmaW5lIFBBR0VfTUFQUElOR19GTEFHUwkoUEFH RV9NQVBQSU5HX0FOT04gfCBQQUdFX01BUFBJTkdfS1NNKQorI2RlZmluZSBQQUdFX01BUFBJTkdf QU5PTgkweDEKKyNkZWZpbmUgUEFHRV9NQVBQSU5HX01PVkFCTEUJMHgyCisjZGVmaW5lIFBBR0Vf TUFQUElOR19LU00JKFBBR0VfTUFQUElOR19BTk9OIHwgUEFHRV9NQVBQSU5HX01PVkFCTEUpCisj ZGVmaW5lIFBBR0VfTUFQUElOR19GTEFHUwkoUEFHRV9NQVBQSU5HX0FOT04gfCBQQUdFX01BUFBJ TkdfTU9WQUJMRSkKIAotc3RhdGljIF9fYWx3YXlzX2lubGluZSBpbnQgUGFnZUFub25IZWFkKHN0 cnVjdCBwYWdlICpwYWdlKQorc3RhdGljIF9fYWx3YXlzX2lubGluZSBpbnQgUGFnZU1hcHBpbmdG bGFnKHN0cnVjdCBwYWdlICpwYWdlKQogewotCXJldHVybiAoKHVuc2lnbmVkIGxvbmcpcGFnZS0+ bWFwcGluZyAmIFBBR0VfTUFQUElOR19BTk9OKSAhPSAwOworCXJldHVybiAoKHVuc2lnbmVkIGxv bmcpcGFnZS0+bWFwcGluZyAmIFBBR0VfTUFQUElOR19GTEFHUykgIT0gMDsKIH0KIAogc3RhdGlj IF9fYWx3YXlzX2lubGluZSBpbnQgUGFnZUFub24oc3RydWN0IHBhZ2UgKnBhZ2UpCiB7CiAJcGFn ZSA9IGNvbXBvdW5kX2hlYWQocGFnZSk7Ci0JcmV0dXJuIFBhZ2VBbm9uSGVhZChwYWdlKTsKKwly ZXR1cm4gKCh1bnNpZ25lZCBsb25nKXBhZ2UtPm1hcHBpbmcgJiBQQUdFX01BUFBJTkdfQU5PTikg IT0gMDsKK30KKworc3RhdGljIF9fYWx3YXlzX2lubGluZSBpbnQgX19QYWdlTW92YWJsZShzdHJ1 Y3QgcGFnZSAqcGFnZSkKK3sKKwlyZXR1cm4gKCh1bnNpZ25lZCBsb25nKXBhZ2UtPm1hcHBpbmcg JiBQQUdFX01BUFBJTkdfRkxBR1MpID09CisJCQkJUEFHRV9NQVBQSU5HX01PVkFCTEU7CiB9CiAK ICNpZmRlZiBDT05GSUdfS1NNCkBAIC0zOTMsNyArNDA0LDcgQEAgc3RhdGljIF9fYWx3YXlzX2lu bGluZSBpbnQgUGFnZUtzbShzdHJ1Y3QgcGFnZSAqcGFnZSkKIHsKIAlwYWdlID0gY29tcG91bmRf aGVhZChwYWdlKTsKIAlyZXR1cm4gKCh1bnNpZ25lZCBsb25nKXBhZ2UtPm1hcHBpbmcgJiBQQUdF X01BUFBJTkdfRkxBR1MpID09Ci0JCQkJKFBBR0VfTUFQUElOR19BTk9OIHwgUEFHRV9NQVBQSU5H X0tTTSk7CisJCQkJUEFHRV9NQVBQSU5HX0tTTTsKIH0KICNlbHNlCiBURVNUUEFHRUZMQUdfRkFM U0UoS3NtKQpAQCAtNjQxLDYgKzY1Miw4IEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBfX0NsZWFyUGFn ZUJhbGxvb24oc3RydWN0IHBhZ2UgKnBhZ2UpCiAJYXRvbWljX3NldCgmcGFnZS0+X21hcGNvdW50 LCAtMSk7CiB9CiAKK19fUEFHRUZMQUcoSXNvbGF0ZWQsIGlzb2xhdGVkLCBQRl9BTlkpOworCiAv KgogICogSWYgbmV0d29yay1iYXNlZCBzd2FwIGlzIGVuYWJsZWQsIHNsKmIgbXVzdCBrZWVwIHRy YWNrIG9mIHdoZXRoZXIgcGFnZXMKICAqIHdlcmUgYWxsb2NhdGVkIGZyb20gcGZtZW1hbGxvYyBy ZXNlcnZlcy4KZGlmZiAtLWdpdCBhL21tL2NvbXBhY3Rpb24uYyBiL21tL2NvbXBhY3Rpb24uYwpp bmRleCBkOGEyMGZjZjg2NzguLmU5ZWViOGViM2IxMCAxMDA2NDQKLS0tIGEvbW0vY29tcGFjdGlv bi5jCisrKyBiL21tL2NvbXBhY3Rpb24uYwpAQCAtNzM1LDIxICs3MzUsNiBAQCBpc29sYXRlX21p Z3JhdGVwYWdlc19ibG9jayhzdHJ1Y3QgY29tcGFjdF9jb250cm9sICpjYywgdW5zaWduZWQgbG9u ZyBsb3dfcGZuLAogCQl9CiAKIAkJLyoKLQkJICogQ2hlY2sgbWF5IGJlIGxvY2tsZXNzIGJ1dCB0 aGF0J3Mgb2sgYXMgd2UgcmVjaGVjayBsYXRlci4KLQkJICogSXQncyBwb3NzaWJsZSB0byBtaWdy YXRlIExSVSBwYWdlcyBhbmQgYmFsbG9vbiBwYWdlcwotCQkgKiBTa2lwIGFueSBvdGhlciB0eXBl IG9mIHBhZ2UKLQkJICovCi0JCWlzX2xydSA9IFBhZ2VMUlUocGFnZSk7Ci0JCWlmICghaXNfbHJ1 KSB7Ci0JCQlpZiAodW5saWtlbHkoYmFsbG9vbl9wYWdlX21vdmFibGUocGFnZSkpKSB7Ci0JCQkJ aWYgKGJhbGxvb25fcGFnZV9pc29sYXRlKHBhZ2UpKSB7Ci0JCQkJCS8qIFN1Y2Nlc3NmdWxseSBp c29sYXRlZCAqLwotCQkJCQlnb3RvIGlzb2xhdGVfc3VjY2VzczsKLQkJCQl9Ci0JCQl9Ci0JCX0K LQotCQkvKgogCQkgKiBSZWdhcmRsZXNzIG9mIGJlaW5nIG9uIExSVSwgY29tcG91bmQgcGFnZXMg c3VjaCBhcyBUSFAgYW5kCiAJCSAqIGh1Z2V0bGJmcyBhcmUgbm90IHRvIGJlIGNvbXBhY3RlZC4g V2UgY2FuIHBvdGVudGlhbGx5IHNhdmUKIAkJICogYSBsb3Qgb2YgaXRlcmF0aW9ucyBpZiB3ZSBz a2lwIHRoZW0gYXQgb25jZS4gVGhlIGNoZWNrIGlzCkBAIC03NjUsOCArNzUwLDM4IEBAIGlzb2xh dGVfbWlncmF0ZXBhZ2VzX2Jsb2NrKHN0cnVjdCBjb21wYWN0X2NvbnRyb2wgKmNjLCB1bnNpZ25l ZCBsb25nIGxvd19wZm4sCiAJCQlnb3RvIGlzb2xhdGVfZmFpbDsKIAkJfQogCi0JCWlmICghaXNf bHJ1KQorCQkvKgorCQkgKiBDaGVjayBtYXkgYmUgbG9ja2xlc3MgYnV0IHRoYXQncyBvayBhcyB3 ZSByZWNoZWNrIGxhdGVyLgorCQkgKiBJdCdzIHBvc3NpYmxlIHRvIG1pZ3JhdGUgTFJVIGFuZCBu b24tbHJ1IG1vdmFibGUgcGFnZXMuCisJCSAqIFNraXAgYW55IG90aGVyIHR5cGUgb2YgcGFnZQor CQkgKi8KKwkJaXNfbHJ1ID0gUGFnZUxSVShwYWdlKTsKKwkJaWYgKCFpc19scnUpIHsKKwkJCWlm ICh1bmxpa2VseShiYWxsb29uX3BhZ2VfbW92YWJsZShwYWdlKSkpIHsKKwkJCQlpZiAoYmFsbG9v bl9wYWdlX2lzb2xhdGUocGFnZSkpIHsKKwkJCQkJLyogU3VjY2Vzc2Z1bGx5IGlzb2xhdGVkICov CisJCQkJCWdvdG8gaXNvbGF0ZV9zdWNjZXNzOworCQkJCX0KKwkJCX0KKworCQkJLyoKKwkJCSAq IF9fUGFnZU1vdmFibGUgY2FuIHJldHVybiBmYWxzZSBwb3NpdGl2ZSBzbyB3ZSBuZWVkCisJCQkg KiB0byB2ZXJpZnkgaXQgdW5kZXIgcGFnZV9sb2NrLgorCQkJICovCisJCQlpZiAodW5saWtlbHko X19QYWdlTW92YWJsZShwYWdlKSkgJiYKKwkJCQkJIVBhZ2VJc29sYXRlZChwYWdlKSkgeworCQkJ CWlmIChsb2NrZWQpIHsKKwkJCQkJc3Bpbl91bmxvY2tfaXJxcmVzdG9yZSgmem9uZS0+bHJ1X2xv Y2ssCisJCQkJCQkJCQlmbGFncyk7CisJCQkJCWxvY2tlZCA9IGZhbHNlOworCQkJCX0KKworCQkJ CWlmIChpc29sYXRlX21vdmFibGVfcGFnZShwYWdlLCBpc29sYXRlX21vZGUpKQorCQkJCQlnb3Rv IGlzb2xhdGVfc3VjY2VzczsKKwkJCX0KKwogCQkJZ290byBpc29sYXRlX2ZhaWw7CisJCX0KIAog CQkvKgogCQkgKiBNaWdyYXRpb24gd2lsbCBmYWlsIGlmIGFuIGFub255bW91cyBwYWdlIGlzIHBp bm5lZCBpbiBtZW1vcnksCmRpZmYgLS1naXQgYS9tbS9rc20uYyBiL21tL2tzbS5jCmluZGV4IDUz YmYzYmExNzZlYy4uNjBiZDIwMmRkOWY0IDEwMDY0NAotLS0gYS9tbS9rc20uYworKysgYi9tbS9r c20uYwpAQCAtNTMyLDggKzUzMiw4IEBAIHN0YXRpYyBzdHJ1Y3QgcGFnZSAqZ2V0X2tzbV9wYWdl KHN0cnVjdCBzdGFibGVfbm9kZSAqc3RhYmxlX25vZGUsIGJvb2wgbG9ja19pdCkKIAl2b2lkICpl eHBlY3RlZF9tYXBwaW5nOwogCXVuc2lnbmVkIGxvbmcga3BmbjsKIAotCWV4cGVjdGVkX21hcHBp bmcgPSAodm9pZCAqKXN0YWJsZV9ub2RlICsKLQkJCQkoUEFHRV9NQVBQSU5HX0FOT04gfCBQQUdF X01BUFBJTkdfS1NNKTsKKwlleHBlY3RlZF9tYXBwaW5nID0gKHZvaWQgKikoKHVuc2lnbmVkIGxv bmcpc3RhYmxlX25vZGUgfAorCQkJCQlQQUdFX01BUFBJTkdfS1NNKTsKIGFnYWluOgogCWtwZm4g PSBSRUFEX09OQ0Uoc3RhYmxlX25vZGUtPmtwZm4pOwogCXBhZ2UgPSBwZm5fdG9fcGFnZShrcGZu KTsKZGlmZiAtLWdpdCBhL21tL21pZ3JhdGUuYyBiL21tL21pZ3JhdGUuYwppbmRleCBmMjkzMjQ5 OGRlZDIuLjI0MjA1ZTFmOTIwNCAxMDA2NDQKLS0tIGEvbW0vbWlncmF0ZS5jCisrKyBiL21tL21p Z3JhdGUuYwpAQCAtNDcsNiArNDcsMzggQEAKIAogI2luY2x1ZGUgImludGVybmFsLmgiCiAKK2lu dCBQYWdlTW92YWJsZShzdHJ1Y3QgcGFnZSAqcGFnZSkKK3sKKwlzdHJ1Y3QgYWRkcmVzc19zcGFj ZSAqbWFwcGluZzsKKworCVdBUk5fT04oIVBhZ2VMb2NrZWQocGFnZSkpOworCWlmICghX19QYWdl TW92YWJsZShwYWdlKSkKKwkJZ290byBvdXQ7CisKKwltYXBwaW5nID0gcGFnZV9tYXBwaW5nKHBh Z2UpOworCWlmIChtYXBwaW5nICYmIG1hcHBpbmctPmFfb3BzICYmIG1hcHBpbmctPmFfb3BzLT5p c29sYXRlX3BhZ2UpCisJCXJldHVybiAxOworb3V0OgorCXJldHVybiAwOworfQorCit2b2lkIF9f U2V0UGFnZU1vdmFibGUoc3RydWN0IHBhZ2UgKnBhZ2UsIHN0cnVjdCBhZGRyZXNzX3NwYWNlICpt YXBwaW5nKQoreworCVZNX0JVR19PTl9QQUdFKCFQYWdlTG9ja2VkKHBhZ2UpLCBwYWdlKTsKKwlW TV9CVUdfT05fUEFHRSgodW5zaWduZWQgbG9uZyltYXBwaW5nICYgUEFHRV9NQVBQSU5HX01PVkFC TEUsIHBhZ2UpOworCXBhZ2UtPm1hcHBpbmcgPSAodm9pZCAqKSgodW5zaWduZWQgbG9uZyltYXBw aW5nIHwgUEFHRV9NQVBQSU5HX01PVkFCTEUpOworfQorCit2b2lkIF9fQ2xlYXJQYWdlTW92YWJs ZShzdHJ1Y3QgcGFnZSAqcGFnZSkKK3sKKwlWTV9CVUdfT05fUEFHRSghUGFnZUxvY2tlZChwYWdl KSwgcGFnZSk7CisJVk1fQlVHX09OX1BBR0UoIVBhZ2VNb3ZhYmxlKHBhZ2UpLCBwYWdlKTsKKwlW TV9CVUdfT05fUEFHRSghKCh1bnNpZ25lZCBsb25nKXBhZ2UtPm1hcHBpbmcgJiBQQUdFX01BUFBJ TkdfTU9WQUJMRSksCisJCQkJcGFnZSk7CisJcGFnZS0+bWFwcGluZyA9ICh2b2lkICopKCh1bnNp Z25lZCBsb25nKXBhZ2UtPm1hcHBpbmcgJgorCQkJCVBBR0VfTUFQUElOR19NT1ZBQkxFKTsKK30K KwogLyoKICAqIG1pZ3JhdGVfcHJlcCgpIG5lZWRzIHRvIGJlIGNhbGxlZCBiZWZvcmUgd2Ugc3Rh cnQgY29tcGlsaW5nIGEgbGlzdCBvZiBwYWdlcwogICogdG8gYmUgbWlncmF0ZWQgdXNpbmcgaXNv bGF0ZV9scnVfcGFnZSgpLiBJZiBzY2hlZHVsaW5nIHdvcmsgb24gb3RoZXIgQ1BVcyBpcwpAQCAt NzMsNiArMTA1LDc5IEBAIGludCBtaWdyYXRlX3ByZXBfbG9jYWwodm9pZCkKIAlyZXR1cm4gMDsK IH0KIAorYm9vbCBpc29sYXRlX21vdmFibGVfcGFnZShzdHJ1Y3QgcGFnZSAqcGFnZSwgaXNvbGF0 ZV9tb2RlX3QgbW9kZSkKK3sKKwlzdHJ1Y3QgYWRkcmVzc19zcGFjZSAqbWFwcGluZzsKKworCS8q CisJICogQXZvaWQgYnVybmluZyBjeWNsZXMgd2l0aCBwYWdlcyB0aGF0IGFyZSB5ZXQgdW5kZXIg X19mcmVlX3BhZ2VzKCksCisJICogb3IganVzdCBnb3QgZnJlZWQgdW5kZXIgdXMuCisJICoKKwkg KiBJbiBjYXNlIHdlICd3aW4nIGEgcmFjZSBmb3IgYSBtb3ZhYmxlIHBhZ2UgYmVpbmcgZnJlZWQg dW5kZXIgdXMgYW5kCisJICogcmFpc2UgaXRzIHJlZmNvdW50IHByZXZlbnRpbmcgX19mcmVlX3Bh Z2VzKCkgZnJvbSBkb2luZyBpdHMgam9iCisJICogdGhlIHB1dF9wYWdlKCkgYXQgdGhlIGVuZCBv ZiB0aGlzIGJsb2NrIHdpbGwgdGFrZSBjYXJlIG9mCisJICogcmVsZWFzZSB0aGlzIHBhZ2UsIHRo dXMgYXZvaWRpbmcgYSBuYXN0eSBsZWFrYWdlLgorCSAqLworCWlmICh1bmxpa2VseSghZ2V0X3Bh Z2VfdW5sZXNzX3plcm8ocGFnZSkpKQorCQlnb3RvIG91dDsKKworCS8qCisJICogQ2hlY2sgUGFn ZU1vdmFibGUgYmVmb3JlIGhvbGRpbmcgYSBQR19sb2NrIGJlY2F1c2UgcGFnZSdzIG93bmVyCisJ ICogYXNzdW1lcyBhbnlib2R5IGRvZXNuJ3QgdG91Y2ggUEdfbG9jayBvZiBuZXdseSBhbGxvY2F0 ZWQgcGFnZQorCSAqIHNvIHVuY29uZGl0aW9uYWxseSBncmFwcGluZyB0aGUgbG9jayBydWlucyBw YWdlJ3Mgb3duZXIgc2lkZS4KKwkgKi8KKwlpZiAodW5saWtlbHkoIV9fUGFnZU1vdmFibGUocGFn ZSkpKQorCQlnb3RvIG91dF9wdXRwYWdlOworCS8qCisJICogQXMgbW92YWJsZSBwYWdlcyBhcmUg bm90IGlzb2xhdGVkIGZyb20gTFJVIGxpc3RzLCBjb25jdXJyZW50CisJICogY29tcGFjdGlvbiB0 aHJlYWRzIGNhbiByYWNlIGFnYWluc3QgcGFnZSBtaWdyYXRpb24gZnVuY3Rpb25zCisJICogYXMg d2VsbCBhcyByYWNlIGFnYWluc3QgdGhlIHJlbGVhc2luZyBhIHBhZ2UuCisJICoKKwkgKiBJbiBv cmRlciB0byBhdm9pZCBoYXZpbmcgYW4gYWxyZWFkeSBpc29sYXRlZCBtb3ZhYmxlIHBhZ2UKKwkg KiBiZWluZyAod3JvbmdseSkgcmUtaXNvbGF0ZWQgd2hpbGUgaXQgaXMgdW5kZXIgbWlncmF0aW9u LAorCSAqIG9yIHRvIGF2b2lkIGF0dGVtcHRpbmcgdG8gaXNvbGF0ZSBwYWdlcyBiZWluZyByZWxl YXNlZCwKKwkgKiBsZXRzIGJlIHN1cmUgd2UgaGF2ZSB0aGUgcGFnZSBsb2NrCisJICogYmVmb3Jl IHByb2NlZWRpbmcgd2l0aCB0aGUgbW92YWJsZSBwYWdlIGlzb2xhdGlvbiBzdGVwcy4KKwkgKi8K KwlpZiAodW5saWtlbHkoIXRyeWxvY2tfcGFnZShwYWdlKSkpCisJCWdvdG8gb3V0X3B1dHBhZ2U7 CisKKwlpZiAoIVBhZ2VNb3ZhYmxlKHBhZ2UpIHx8IFBhZ2VJc29sYXRlZChwYWdlKSkKKwkJZ290 byBvdXRfbm9faXNvbGF0ZWQ7CisKKwltYXBwaW5nID0gcGFnZV9tYXBwaW5nKHBhZ2UpOworCWlm ICghbWFwcGluZy0+YV9vcHMtPmlzb2xhdGVfcGFnZShwYWdlLCBtb2RlKSkKKwkJZ290byBvdXRf bm9faXNvbGF0ZWQ7CisKKwkvKiBEcml2ZXIgc2hvdWxkbid0IHVzZSBQR19pc29sYXRlZCBiaXQg b2YgcGFnZS0+ZmxhZ3MgKi8KKwlXQVJOX09OX09OQ0UoUGFnZUlzb2xhdGVkKHBhZ2UpKTsKKwlf X1NldFBhZ2VJc29sYXRlZChwYWdlKTsKKwl1bmxvY2tfcGFnZShwYWdlKTsKKworCXJldHVybiB0 cnVlOworCitvdXRfbm9faXNvbGF0ZWQ6CisJdW5sb2NrX3BhZ2UocGFnZSk7CitvdXRfcHV0cGFn ZToKKwlwdXRfcGFnZShwYWdlKTsKK291dDoKKwlyZXR1cm4gZmFsc2U7Cit9CisKKy8qIEl0IHNo b3VsZCBiZSBjYWxsZWQgb24gcGFnZSB3aGljaCBpcyBQR19tb3ZhYmxlICovCit2b2lkIHB1dGJh Y2tfbW92YWJsZV9wYWdlKHN0cnVjdCBwYWdlICpwYWdlKQoreworCXN0cnVjdCBhZGRyZXNzX3Nw YWNlICptYXBwaW5nOworCisJVk1fQlVHX09OX1BBR0UoIVBhZ2VMb2NrZWQocGFnZSksIHBhZ2Up OworCVZNX0JVR19PTl9QQUdFKCFQYWdlTW92YWJsZShwYWdlKSwgcGFnZSk7CisJVk1fQlVHX09O X1BBR0UoIVBhZ2VJc29sYXRlZChwYWdlKSwgcGFnZSk7CisKKwltYXBwaW5nID0gcGFnZV9tYXBw aW5nKHBhZ2UpOworCW1hcHBpbmctPmFfb3BzLT5wdXRiYWNrX3BhZ2UocGFnZSk7CisJX19DbGVh clBhZ2VJc29sYXRlZChwYWdlKTsKK30KKwogLyoKICAqIFB1dCBwcmV2aW91c2x5IGlzb2xhdGVk IHBhZ2VzIGJhY2sgb250byB0aGUgYXBwcm9wcmlhdGUgbGlzdHMKICAqIGZyb20gd2hlcmUgdGhl eSB3ZXJlIG9uY2UgdGFrZW4gb2ZmIGZvciBjb21wYWN0aW9uL21pZ3JhdGlvbi4KQEAgLTk0LDEw ICsxOTksMjUgQEAgdm9pZCBwdXRiYWNrX21vdmFibGVfcGFnZXMoc3RydWN0IGxpc3RfaGVhZCAq bCkKIAkJbGlzdF9kZWwoJnBhZ2UtPmxydSk7CiAJCWRlY196b25lX3BhZ2Vfc3RhdGUocGFnZSwg TlJfSVNPTEFURURfQU5PTiArCiAJCQkJcGFnZV9pc19maWxlX2NhY2hlKHBhZ2UpKTsKLQkJaWYg KHVubGlrZWx5KGlzb2xhdGVkX2JhbGxvb25fcGFnZShwYWdlKSkpCisJCWlmICh1bmxpa2VseShp c29sYXRlZF9iYWxsb29uX3BhZ2UocGFnZSkpKSB7CiAJCQliYWxsb29uX3BhZ2VfcHV0YmFjayhw YWdlKTsKLQkJZWxzZQorCQkvKgorCQkgKiBXZSBpc29sYXRlZCBub24tbHJ1IG1vdmFibGUgcGFn ZSBzbyBoZXJlIHdlIGNhbiB1c2UKKwkJICogX19QYWdlTW92YWJsZSBiZWNhdXNlIExSVSBwYWdl J3MgbWFwcGluZyBjYW5ub3QgaGF2ZQorCQkgKiBQQUdFX01BUFBJTkdfTU9WQUJMRS4KKwkJICov CisJCX0gZWxzZSBpZiAodW5saWtlbHkoX19QYWdlTW92YWJsZShwYWdlKSkpIHsKKwkJCVZNX0JV R19PTl9QQUdFKCFQYWdlSXNvbGF0ZWQocGFnZSksIHBhZ2UpOworCQkJbG9ja19wYWdlKHBhZ2Up OworCQkJaWYgKFBhZ2VNb3ZhYmxlKHBhZ2UpKQorCQkJCXB1dGJhY2tfbW92YWJsZV9wYWdlKHBh Z2UpOworCQkJZWxzZQorCQkJCV9fQ2xlYXJQYWdlSXNvbGF0ZWQocGFnZSk7CisJCQl1bmxvY2tf cGFnZShwYWdlKTsKKwkJCXB1dF9wYWdlKHBhZ2UpOworCQl9IGVsc2UgewogCQkJcHV0YmFja19s cnVfcGFnZShwYWdlKTsKKwkJfQogCX0KIH0KIApAQCAtNTkyLDcgKzcxMiw3IEBAIHZvaWQgbWln cmF0ZV9wYWdlX2NvcHkoc3RydWN0IHBhZ2UgKm5ld3BhZ2UsIHN0cnVjdCBwYWdlICpwYWdlKQog ICoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqLwogCiAvKgotICogQ29tbW9uIGxvZ2ljIHRvIGRpcmVjdGx5IG1pZ3JhdGUgYSBzaW5nbGUg cGFnZSBzdWl0YWJsZSBmb3IKKyAqIENvbW1vbiBsb2dpYyB0byBkaXJlY3RseSBtaWdyYXRlIGEg c2luZ2xlIExSVSBwYWdlIHN1aXRhYmxlIGZvcgogICogcGFnZXMgdGhhdCBkbyBub3QgdXNlIFBh Z2VQcml2YXRlL1BhZ2VQcml2YXRlMi4KICAqCiAgKiBQYWdlcyBhcmUgbG9ja2VkIHVwb24gZW50 cnkgYW5kIGV4aXQuCkBAIC03NTUsMzMgKzg3NSw2OSBAQCBzdGF0aWMgaW50IG1vdmVfdG9fbmV3 X3BhZ2Uoc3RydWN0IHBhZ2UgKm5ld3BhZ2UsIHN0cnVjdCBwYWdlICpwYWdlLAogCQkJCWVudW0g bWlncmF0ZV9tb2RlIG1vZGUpCiB7CiAJc3RydWN0IGFkZHJlc3Nfc3BhY2UgKm1hcHBpbmc7Ci0J aW50IHJjOworCWludCByYyA9IC1FQUdBSU47CisJYm9vbCBpc19scnUgPSAhX19QYWdlTW92YWJs ZShwYWdlKTsKIAogCVZNX0JVR19PTl9QQUdFKCFQYWdlTG9ja2VkKHBhZ2UpLCBwYWdlKTsKIAlW TV9CVUdfT05fUEFHRSghUGFnZUxvY2tlZChuZXdwYWdlKSwgbmV3cGFnZSk7CiAKIAltYXBwaW5n ID0gcGFnZV9tYXBwaW5nKHBhZ2UpOwotCWlmICghbWFwcGluZykKLQkJcmMgPSBtaWdyYXRlX3Bh Z2UobWFwcGluZywgbmV3cGFnZSwgcGFnZSwgbW9kZSk7Ci0JZWxzZSBpZiAobWFwcGluZy0+YV9v cHMtPm1pZ3JhdGVwYWdlKQotCQkvKgotCQkgKiBNb3N0IHBhZ2VzIGhhdmUgYSBtYXBwaW5nIGFu ZCBtb3N0IGZpbGVzeXN0ZW1zIHByb3ZpZGUgYQotCQkgKiBtaWdyYXRlcGFnZSBjYWxsYmFjay4g QW5vbnltb3VzIHBhZ2VzIGFyZSBwYXJ0IG9mIHN3YXAKLQkJICogc3BhY2Ugd2hpY2ggYWxzbyBo YXMgaXRzIG93biBtaWdyYXRlcGFnZSBjYWxsYmFjay4gVGhpcwotCQkgKiBpcyB0aGUgbW9zdCBj b21tb24gcGF0aCBmb3IgcGFnZSBtaWdyYXRpb24uCi0JCSAqLwotCQlyYyA9IG1hcHBpbmctPmFf b3BzLT5taWdyYXRlcGFnZShtYXBwaW5nLCBuZXdwYWdlLCBwYWdlLCBtb2RlKTsKLQllbHNlCi0J CXJjID0gZmFsbGJhY2tfbWlncmF0ZV9wYWdlKG1hcHBpbmcsIG5ld3BhZ2UsIHBhZ2UsIG1vZGUp OworCS8qCisJICogSW4gY2FzZSBvZiBub24tbHJ1IHBhZ2UsIGl0IGNvdWxkIGJlIHJlbGVhc2Vk IGFmdGVyCisJICogaXNvbGF0aW9uIHN0ZXAuIEluIHRoYXQgY2FzZSwgd2Ugc2hvdWxkbid0IHRy eQorCSAqIGZhbGxiYWNrIG1pZ3JhdGlvbiB3aGljaCBpcyBkZXNpZ25lZCBmb3IgTFJVIHBhZ2Vz LgorCSAqLworCWlmICh1bmxpa2VseSghaXNfbHJ1KSkgeworCQlWTV9CVUdfT05fUEFHRSghUGFn ZUlzb2xhdGVkKHBhZ2UpLCBwYWdlKTsKKwkJaWYgKCFQYWdlTW92YWJsZShwYWdlKSkgeworCQkJ cmMgPSBNSUdSQVRFUEFHRV9TVUNDRVNTOworCQkJX19DbGVhclBhZ2VJc29sYXRlZChwYWdlKTsK KwkJCWdvdG8gb3V0OworCQl9CisJfQorCisJaWYgKGxpa2VseShpc19scnUpKSB7CisJCWlmICgh bWFwcGluZykKKwkJCXJjID0gbWlncmF0ZV9wYWdlKG1hcHBpbmcsIG5ld3BhZ2UsIHBhZ2UsIG1v ZGUpOworCQllbHNlIGlmIChtYXBwaW5nLT5hX29wcy0+bWlncmF0ZXBhZ2UpCisJCQkvKgorCQkJ ICogTW9zdCBwYWdlcyBoYXZlIGEgbWFwcGluZyBhbmQgbW9zdCBmaWxlc3lzdGVtcworCQkJICog cHJvdmlkZSBhIG1pZ3JhdGVwYWdlIGNhbGxiYWNrLiBBbm9ueW1vdXMgcGFnZXMKKwkJCSAqIGFy ZSBwYXJ0IG9mIHN3YXAgc3BhY2Ugd2hpY2ggYWxzbyBoYXMgaXRzIG93bgorCQkJICogbWlncmF0 ZXBhZ2UgY2FsbGJhY2suIFRoaXMgaXMgdGhlIG1vc3QgY29tbW9uIHBhdGgKKwkJCSAqIGZvciBw YWdlIG1pZ3JhdGlvbi4KKwkJCSAqLworCQkJcmMgPSBtYXBwaW5nLT5hX29wcy0+bWlncmF0ZXBh Z2UobWFwcGluZywgbmV3cGFnZSwKKwkJCQkJCQlwYWdlLCBtb2RlKTsKKwkJZWxzZQorCQkJcmMg PSBmYWxsYmFja19taWdyYXRlX3BhZ2UobWFwcGluZywgbmV3cGFnZSwKKwkJCQkJCQlwYWdlLCBt b2RlKTsKKwl9IGVsc2UgeworCQlyYyA9IG1hcHBpbmctPmFfb3BzLT5taWdyYXRlcGFnZShtYXBw aW5nLCBuZXdwYWdlLAorCQkJCQkJcGFnZSwgbW9kZSk7CisJCVdBUk5fT05fT05DRShyYyA9PSBN SUdSQVRFUEFHRV9TVUNDRVNTICYmCisJCQkhUGFnZUlzb2xhdGVkKHBhZ2UpKTsKKwl9CiAKIAkv KgogCSAqIFdoZW4gc3VjY2Vzc2Z1bCwgb2xkIHBhZ2VjYWNoZSBwYWdlLT5tYXBwaW5nIG11c3Qg YmUgY2xlYXJlZCBiZWZvcmUKIAkgKiBwYWdlIGlzIGZyZWVkOyBidXQgc3RhdHMgcmVxdWlyZSB0 aGF0IFBhZ2VBbm9uIGJlIGxlZnQgYXMgUGFnZUFub24uCiAJICovCiAJaWYgKHJjID09IE1JR1JB VEVQQUdFX1NVQ0NFU1MpIHsKLQkJaWYgKCFQYWdlQW5vbihwYWdlKSkKKwkJaWYgKF9fUGFnZU1v dmFibGUocGFnZSkpIHsKKwkJCVZNX0JVR19PTl9QQUdFKCFQYWdlSXNvbGF0ZWQocGFnZSksIHBh Z2UpOworCisJCQkvKgorCQkJICogV2UgY2xlYXIgUEdfbW92YWJsZSB1bmRlciBwYWdlX2xvY2sg c28gYW55IGNvbXBhY3RvcgorCQkJICogY2Fubm90IHRyeSB0byBtaWdyYXRlIHRoaXMgcGFnZS4K KwkJCSAqLworCQkJX19DbGVhclBhZ2VJc29sYXRlZChwYWdlKTsKKwkJfQorCisJCWlmICghKCh1 bnNpZ25lZCBsb25nKXBhZ2UtPm1hcHBpbmcgJiBQQUdFX01BUFBJTkdfRkxBR1MpKQogCQkJcGFn ZS0+bWFwcGluZyA9IE5VTEw7CiAJfQorb3V0OgogCXJldHVybiByYzsKIH0KIApAQCAtNzkxLDYg Kzk0Nyw3IEBAIHN0YXRpYyBpbnQgX191bm1hcF9hbmRfbW92ZShzdHJ1Y3QgcGFnZSAqcGFnZSwg c3RydWN0IHBhZ2UgKm5ld3BhZ2UsCiAJaW50IHJjID0gLUVBR0FJTjsKIAlpbnQgcGFnZV93YXNf bWFwcGVkID0gMDsKIAlzdHJ1Y3QgYW5vbl92bWEgKmFub25fdm1hID0gTlVMTDsKKwlib29sIGlz X2xydSA9ICFfX1BhZ2VNb3ZhYmxlKHBhZ2UpOwogCiAJaWYgKCF0cnlsb2NrX3BhZ2UocGFnZSkp IHsKIAkJaWYgKCFmb3JjZSB8fCBtb2RlID09IE1JR1JBVEVfQVNZTkMpCkBAIC04NzEsNiArMTAy OCwxMSBAQCBzdGF0aWMgaW50IF9fdW5tYXBfYW5kX21vdmUoc3RydWN0IHBhZ2UgKnBhZ2UsIHN0 cnVjdCBwYWdlICpuZXdwYWdlLAogCQlnb3RvIG91dF91bmxvY2tfYm90aDsKIAl9CiAKKwlpZiAo dW5saWtlbHkoIWlzX2xydSkpIHsKKwkJcmMgPSBtb3ZlX3RvX25ld19wYWdlKG5ld3BhZ2UsIHBh Z2UsIG1vZGUpOworCQlnb3RvIG91dF91bmxvY2tfYm90aDsKKwl9CisKIAkvKgogCSAqIENvcm5l ciBjYXNlIGhhbmRsaW5nOgogCSAqIDEuIFdoZW4gYSBuZXcgc3dhcC1jYWNoZSBwYWdlIGlzIHJl YWQgaW50bywgaXQgaXMgYWRkZWQgdG8gdGhlIExSVQpAQCAtOTIwLDcgKzEwODIsOCBAQCBzdGF0 aWMgaW50IF9fdW5tYXBfYW5kX21vdmUoc3RydWN0IHBhZ2UgKnBhZ2UsIHN0cnVjdCBwYWdlICpu ZXdwYWdlLAogCSAqIGxpc3QgaW4gaGVyZS4KIAkgKi8KIAlpZiAocmMgPT0gTUlHUkFURVBBR0Vf U1VDQ0VTUykgewotCQlpZiAodW5saWtlbHkoX19pc19tb3ZhYmxlX2JhbGxvb25fcGFnZShuZXdw YWdlKSkpCisJCWlmICh1bmxpa2VseShfX2lzX21vdmFibGVfYmFsbG9vbl9wYWdlKG5ld3BhZ2Up IHx8CisJCQkJX19QYWdlTW92YWJsZShuZXdwYWdlKSkpCiAJCQlwdXRfcGFnZShuZXdwYWdlKTsK IAkJZWxzZQogCQkJcHV0YmFja19scnVfcGFnZShuZXdwYWdlKTsKQEAgLTk2MSw2ICsxMTI0LDEy IEBAIHN0YXRpYyBJQ0Vfbm9pbmxpbmUgaW50IHVubWFwX2FuZF9tb3ZlKG5ld19wYWdlX3QgZ2V0 X25ld19wYWdlLAogCQkvKiBwYWdlIHdhcyBmcmVlZCBmcm9tIHVuZGVyIHVzLiBTbyB3ZSBhcmUg ZG9uZS4gKi8KIAkJQ2xlYXJQYWdlQWN0aXZlKHBhZ2UpOwogCQlDbGVhclBhZ2VVbmV2aWN0YWJs ZShwYWdlKTsKKwkJaWYgKHVubGlrZWx5KF9fUGFnZU1vdmFibGUocGFnZSkpKSB7CisJCQlsb2Nr X3BhZ2UocGFnZSk7CisJCQlpZiAoIVBhZ2VNb3ZhYmxlKHBhZ2UpKQorCQkJCV9fQ2xlYXJQYWdl SXNvbGF0ZWQocGFnZSk7CisJCQl1bmxvY2tfcGFnZShwYWdlKTsKKwkJfQogCQlpZiAocHV0X25l d19wYWdlKQogCQkJcHV0X25ld19wYWdlKG5ld3BhZ2UsIHByaXZhdGUpOwogCQllbHNlCkBAIC0x MDEwLDggKzExNzksMjEgQEAgc3RhdGljIElDRV9ub2lubGluZSBpbnQgdW5tYXBfYW5kX21vdmUo bmV3X3BhZ2VfdCBnZXRfbmV3X3BhZ2UsCiAJCQkJbnVtX3BvaXNvbmVkX3BhZ2VzX2luYygpOwog CQl9CiAJfSBlbHNlIHsKLQkJaWYgKHJjICE9IC1FQUdBSU4pCi0JCQlwdXRiYWNrX2xydV9wYWdl KHBhZ2UpOworCQlpZiAocmMgIT0gLUVBR0FJTikgeworCQkJaWYgKGxpa2VseSghX19QYWdlTW92 YWJsZShwYWdlKSkpIHsKKwkJCQlwdXRiYWNrX2xydV9wYWdlKHBhZ2UpOworCQkJCWdvdG8gcHV0 X25ldzsKKwkJCX0KKworCQkJbG9ja19wYWdlKHBhZ2UpOworCQkJaWYgKFBhZ2VNb3ZhYmxlKHBh Z2UpKQorCQkJCXB1dGJhY2tfbW92YWJsZV9wYWdlKHBhZ2UpOworCQkJZWxzZQorCQkJCV9fQ2xl YXJQYWdlSXNvbGF0ZWQocGFnZSk7CisJCQl1bmxvY2tfcGFnZShwYWdlKTsKKwkJCXB1dF9wYWdl KHBhZ2UpOworCQl9CitwdXRfbmV3OgogCQlpZiAocHV0X25ld19wYWdlKQogCQkJcHV0X25ld19w YWdlKG5ld3BhZ2UsIHByaXZhdGUpOwogCQllbHNlCmRpZmYgLS1naXQgYS9tbS9wYWdlX2FsbG9j LmMgYi9tbS9wYWdlX2FsbG9jLmMKaW5kZXggNjY2M2QxNzM2YmNlLi5jMzc0NzJmNjY1NjYgMTAw NjQ0Ci0tLSBhL21tL3BhZ2VfYWxsb2MuYworKysgYi9tbS9wYWdlX2FsbG9jLmMKQEAgLTEwMTUs NyArMTAxNSw3IEBAIHN0YXRpYyBfX2Fsd2F5c19pbmxpbmUgYm9vbCBmcmVlX3BhZ2VzX3ByZXBh cmUoc3RydWN0IHBhZ2UgKnBhZ2UsCiAJCQkocGFnZSArIGkpLT5mbGFncyAmPSB+UEFHRV9GTEFH U19DSEVDS19BVF9QUkVQOwogCQl9CiAJfQotCWlmIChQYWdlQW5vbkhlYWQocGFnZSkpCisJaWYg KFBhZ2VNYXBwaW5nRmxhZyhwYWdlKSkKIAkJcGFnZS0+bWFwcGluZyA9IE5VTEw7CiAJaWYgKGNo ZWNrX2ZyZWUpCiAJCWJhZCArPSBmcmVlX3BhZ2VzX2NoZWNrKHBhZ2UpOwpkaWZmIC0tZ2l0IGEv bW0vdXRpbC5jIGIvbW0vdXRpbC5jCmluZGV4IDkxN2UwZTNkMGY4ZS4uYjc1NmVlMzZmN2YwIDEw MDY0NAotLS0gYS9tbS91dGlsLmMKKysrIGIvbW0vdXRpbC5jCkBAIC0zOTksMTAgKzM5OSwxMiBA QCBzdHJ1Y3QgYWRkcmVzc19zcGFjZSAqcGFnZV9tYXBwaW5nKHN0cnVjdCBwYWdlICpwYWdlKQog CX0KIAogCW1hcHBpbmcgPSBwYWdlLT5tYXBwaW5nOwotCWlmICgodW5zaWduZWQgbG9uZyltYXBw aW5nICYgUEFHRV9NQVBQSU5HX0ZMQUdTKQorCWlmICgodW5zaWduZWQgbG9uZyltYXBwaW5nICYg UEFHRV9NQVBQSU5HX0FOT04pCiAJCXJldHVybiBOVUxMOwotCXJldHVybiBtYXBwaW5nOworCisJ cmV0dXJuICh2b2lkICopKCh1bnNpZ25lZCBsb25nKW1hcHBpbmcgJiB+UEFHRV9NQVBQSU5HX0ZM QUdTKTsKIH0KK0VYUE9SVF9TWU1CT0wocGFnZV9tYXBwaW5nKTsKIAogLyogU2xvdyBwYXRoIG9m IHBhZ2VfbWFwY291bnQoKSBmb3IgY29tcG91bmQgcGFnZXMgKi8KIGludCBfX3BhZ2VfbWFwY291 bnQoc3RydWN0IHBhZ2UgKnBhZ2UpCi0tIAoxLjkuMQoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxA bGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxt YW4vbGlzdGluZm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f72.google.com (mail-pa0-f72.google.com [209.85.220.72]) by kanga.kvack.org (Postfix) with ESMTP id 4E0826B0263 for ; Sun, 8 May 2016 22:20:37 -0400 (EDT) Received: by mail-pa0-f72.google.com with SMTP id gw7so245697405pac.0 for ; Sun, 08 May 2016 19:20:37 -0700 (PDT) Received: from lgeamrelo11.lge.com (LGEAMRELO11.lge.com. [156.147.23.51]) by mx.google.com with ESMTP id i1si34474183pfb.54.2016.05.08.19.20.23 for ; Sun, 08 May 2016 19:20:24 -0700 (PDT) From: Minchan Kim Subject: [PATCH v5 02/12] mm: migrate: support non-lru movable page migration Date: Mon, 9 May 2016 11:20:23 +0900 Message-Id: <1462760433-32357-3-git-send-email-minchan@kernel.org> In-Reply-To: <1462760433-32357-1-git-send-email-minchan@kernel.org> References: <1462760433-32357-1-git-send-email-minchan@kernel.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Rik van Riel , Vlastimil Babka , Joonsoo Kim , Mel Gorman , Hugh Dickins , Rafael Aquini , virtualization@lists.linux-foundation.org, Jonathan Corbet , John Einar Reitan , dri-devel@lists.freedesktop.org, Sergey Senozhatsky , Gioh Kim We have allowed migration for only LRU pages until now and it was enough to make high-order pages. But recently, embedded system(e.g., webOS, android) uses lots of non-movable pages(e.g., zram, GPU memory) so we have seen several reports about troubles of small high-order allocation. For fixing the problem, there were several efforts (e,g,. enhance compaction algorithm, SLUB fallback to 0-order page, reserved memory, vmalloc and so on) but if there are lots of non-movable pages in system, their solutions are void in the long run. So, this patch is to support facility to change non-movable pages with movable. For the feature, this patch introduces functions related to migration to address_space_operations as well as some page flags. If a driver want to make own pages movable, it should define three functions which are function pointers of struct address_space_operations. 1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); What VM expects on isolate_page function of driver is to return *true* if driver isolates page successfully. On returing true, VM marks the page as PG_isolated so concurrent isolation in several CPUs skip the page for isolation. If a driver cannot isolate the page, it should return *false*. Once page is successfully isolated, VM uses page.lru fields so driver shouldn't expect to preserve values in that fields. 2. int (*migratepage) (struct address_space *mapping, struct page *newpage, struct page *oldpage, enum migrate_mode); After isolation, VM calls migratepage of driver with isolated page. The function of migratepage is to move content of the old page to new page and set up fields of struct page newpage. Keep in mind that you should clear PG_movable of oldpage via __ClearPageMovable under page_lock if you migrated the oldpage successfully and returns MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time because VM interprets -EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, VM will give up the page migration without retrying in this time. Driver shouldn't touch page.lru field VM using in the functions. 3. void (*putback_page)(struct page *); If migration fails on isolated page, VM should return the isolated page to the driver so VM calls driver's putback_page with migration failed page. In this function, driver should put the isolated page back to the own data structure. 4. non-lru movable page flags There are two page flags for supporting non-lru movable page. * PG_movable Driver should use the below function to make page movable under page_lock. void __SetPageMovable(struct page *page, struct address_space *mapping) It needs argument of address_space for registering migration family functions which will be called by VM. Exactly speaking, PG_movable is not a real flag of struct page. Rather than, VM reuses page->mapping's lower bits to represent it. #define PAGE_MAPPING_MOVABLE 0x2 page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; so driver shouldn't access page->mapping directly. Instead, driver should use page_mapping which mask off the low two bits of page->mapping so it can get right struct address_space. For testing of non-lru movable page, VM supports __PageMovable function. However, it doesn't guarantee to identify non-lru movable page because page->mapping field is unified with other variables in struct page. As well, if driver releases the page after isolation by VM, page->mapping doesn't have stable value although it has PAGE_MAPPING_MOVABLE (Look at __ClearPageMovable). But __PageMovable is cheap to catch whether page is LRU or non-lru movable once the page has been isolated. Because LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also good for just peeking to test non-lru movable pages before more expensive checking with lock_page in pfn scanning to select victim. For guaranteeing non-lru movable page, VM provides PageMovable function. Unlike __PageMovable, PageMovable functions validates page->mapping and mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden destroying of page->mapping. Driver using __SetPageMovable should clear the flag via __ClearMovablePage under page_lock before the releasing the page. * PG_isolated To prevent concurrent isolation among several CPUs, VM marks isolated page as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru movable page, it can skip it. Driver doesn't need to manipulate the flag because VM will set/clear it automatically. Keep in mind that if driver sees PG_isolated page, it means the page have been isolated by VM so it shouldn't touch page.lru field. PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag for own purpose. Cc: Rik van Riel Cc: Vlastimil Babka Cc: Joonsoo Kim Cc: Mel Gorman Cc: Hugh Dickins Cc: Rafael Aquini Cc: virtualization@lists.linux-foundation.org Cc: Jonathan Corbet Cc: John Einar Reitan Cc: dri-devel@lists.freedesktop.org Cc: Sergey Senozhatsky Signed-off-by: Gioh Kim Signed-off-by: Minchan Kim --- Documentation/filesystems/Locking | 4 + Documentation/filesystems/vfs.txt | 11 ++ Documentation/vm/page_migration | 107 +++++++++++++++++- include/linux/fs.h | 2 + include/linux/ksm.h | 3 +- include/linux/migrate.h | 5 + include/linux/mm.h | 1 + include/linux/page-flags.h | 29 +++-- mm/compaction.c | 47 +++++--- mm/ksm.c | 4 +- mm/migrate.c | 222 ++++++++++++++++++++++++++++++++++---- mm/page_alloc.c | 2 +- mm/util.c | 6 +- 13 files changed, 391 insertions(+), 52 deletions(-) diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking index 75eea7ce3d7c..dda6e3f8e203 100644 --- a/Documentation/filesystems/Locking +++ b/Documentation/filesystems/Locking @@ -195,7 +195,9 @@ unlocks and drops the reference. int (*releasepage) (struct page *, int); void (*freepage)(struct page *); int (*direct_IO)(struct kiocb *, struct iov_iter *iter); + bool (*isolate_page) (struct page *, isolate_mode_t); int (*migratepage)(struct address_space *, struct page *, struct page *); + void (*putback_page) (struct page *); int (*launder_page)(struct page *); int (*is_partially_uptodate)(struct page *, unsigned long, unsigned long); int (*error_remove_page)(struct address_space *, struct page *); @@ -219,7 +221,9 @@ invalidatepage: yes releasepage: yes freepage: yes direct_IO: +isolate_page: yes migratepage: yes (both) +putback_page: yes launder_page: yes is_partially_uptodate: yes error_remove_page: yes diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt index c61a223ef3ff..900360cbcdae 100644 --- a/Documentation/filesystems/vfs.txt +++ b/Documentation/filesystems/vfs.txt @@ -592,9 +592,14 @@ struct address_space_operations { int (*releasepage) (struct page *, int); void (*freepage)(struct page *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); + /* isolate a page for migration */ + bool (*isolate_page) (struct page *, isolate_mode_t); /* migrate the contents of a page to the specified target */ int (*migratepage) (struct page *, struct page *); + /* put migration-failed page back to right list */ + void (*putback_page) (struct page *); int (*launder_page) (struct page *); + int (*is_partially_uptodate) (struct page *, unsigned long, unsigned long); void (*is_dirty_writeback) (struct page *, bool *, bool *); @@ -747,6 +752,10 @@ struct address_space_operations { and transfer data directly between the storage and the application's address space. + isolate_page: Called by the VM when isolating a movable non-lru page. + If page is successfully isolated, VM marks the page as PG_isolated + via __SetPageIsolated. + migrate_page: This is used to compact the physical memory usage. If the VM wants to relocate a page (maybe off a memory card that is signalling imminent failure) it will pass a new page @@ -754,6 +763,8 @@ struct address_space_operations { transfer any private data across and update any references that it has to the page. + putback_page: Called by the VM when isolated page's migration fails. + launder_page: Called before freeing a page - it writes back the dirty page. To prevent redirtying the page, it is kept locked during the whole operation. diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration index fea5c0864170..b89d1de026df 100644 --- a/Documentation/vm/page_migration +++ b/Documentation/vm/page_migration @@ -142,5 +142,110 @@ is increased so that the page cannot be freed while page migration occurs. 20. The new page is moved to the LRU and can be scanned by the swapper etc again. -Christoph Lameter, May 8, 2006. +C. Non-LRU page migration +------------------------- + +Although original migration aimed for reducing the latency of memory access +for NUMA, compaction who want to create high-order page is also main customer. + +Current problem of the implementation is that it is designed to migrate only +*LRU* pages. However, there are potential non-lru pages which can be migrated +in drivers, for example, zsmalloc, virtio-balloon pages. + +For virtio-balloon pages, some parts of migration code path have been hooked +up and added virtio-balloon specific functions to intercept migration logics. +It's too specific to a driver so other drivers who want to make their pages +movable would have to add own specific hooks in migration path. + +To overclome the problem, VM supports non-LRU page migration which provides +generic functions for non-LRU movable pages without driver specific hooks +migration path. + +If a driver want to make own pages movable, it should define three functions +which are function pointers of struct address_space_operations. + +1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); + +What VM expects on isolate_page function of driver is to return *true* +if driver isolates page successfully. On returing true, VM marks the page +as PG_isolated so concurrent isolation in several CPUs skip the page +for isolation. If a driver cannot isolate the page, it should return *false*. + +Once page is successfully isolated, VM uses page.lru fields so driver +shouldn't expect to preserve values in that fields. + +2. int (*migratepage) (struct address_space *mapping, + struct page *newpage, struct page *oldpage, enum migrate_mode); + +After isolation, VM calls migratepage of driver with isolated page. +The function of migratepage is to move content of the old page to new page +and set up fields of struct page newpage. Keep in mind that you should +clear PG_movable of oldpage via __ClearPageMovable under page_lock if you +migrated the oldpage successfully and returns MIGRATEPAGE_SUCCESS. +If driver cannot migrate the page at the moment, driver can return -EAGAIN. +On -EAGAIN, VM will retry page migration in a short time because VM interprets +-EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, +VM will give up the page migration without retrying in this time. + +Driver shouldn't touch page.lru field VM using in the functions. + +3. void (*putback_page)(struct page *); + +If migration fails on isolated page, VM should return the isolated page +to the driver so VM calls driver's putback_page with migration failed page. +In this function, driver should put the isolated page back to the own data +structure. +4. non-lru movable page flags + +There are two page flags for supporting non-lru movable page. + +* PG_movable + +Driver should use the below function to make page movable under page_lock. + + void __SetPageMovable(struct page *page, struct address_space *mapping) + +It needs argument of address_space for registering migration family functions +which will be called by VM. Exactly speaking, PG_movable is not a real flag of +struct page. Rather than, VM reuses page->mapping's lower bits to represent it. + + #define PAGE_MAPPING_MOVABLE 0x2 + page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; + +so driver shouldn't access page->mapping directly. Instead, driver should +use page_mapping which mask off the low two bits of page->mapping so it can get +right struct address_space. + +For testing of non-lru movable page, VM supports __PageMovable function. +However, it doesn't guarantee to identify non-lru movable page because +page->mapping field is unified with other variables in struct page. +As well, if driver releases the page after isolation by VM, page->mapping +doesn't have stable value although it has PAGE_MAPPING_MOVABLE +(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether +page is LRU or non-lru movable once the page has been isolated. Because +LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also +good for just peeking to test non-lru movable pages before more expensive +checking with lock_page in pfn scanning to select victim. + +For guaranteeing non-lru movable page, VM provides PageMovable function. +Unlike __PageMovable, PageMovable functions validates page->mapping and +mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden +destroying of page->mapping. + +Driver using __SetPageMovable should clear the flag via __ClearMovablePage +under page_lock before the releasing the page. + +* PG_isolated + +To prevent concurrent isolation among several CPUs, VM marks isolated page +as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru +movable page, it can skip it. Driver doesn't need to manipulate the flag +because VM will set/clear it automatically. Keep in mind that if driver +sees PG_isolated page, it means the page have been isolated by VM so it +shouldn't touch page.lru field. +PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag +for own purpose. + +Christoph Lameter, May 8, 2006. +Minchan Kim, Mar 28, 2016. diff --git a/include/linux/fs.h b/include/linux/fs.h index d61cd084987e..92029377b9e0 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -403,6 +403,8 @@ struct address_space_operations { */ int (*migratepage) (struct address_space *, struct page *, struct page *, enum migrate_mode); + bool (*isolate_page)(struct page *, isolate_mode_t); + void (*putback_page)(struct page *); int (*launder_page) (struct page *); int (*is_partially_uptodate) (struct page *, unsigned long, unsigned long); diff --git a/include/linux/ksm.h b/include/linux/ksm.h index 7ae216a39c9e..481c8c4627ca 100644 --- a/include/linux/ksm.h +++ b/include/linux/ksm.h @@ -43,8 +43,7 @@ static inline struct stable_node *page_stable_node(struct page *page) static inline void set_page_stable_node(struct page *page, struct stable_node *stable_node) { - page->mapping = (void *)stable_node + - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + page->mapping = (void *)((unsigned long)stable_node | PAGE_MAPPING_KSM); } /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 9b50325e4ddf..31a748f43fa9 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -32,11 +32,16 @@ extern char *migrate_reason_names[MR_TYPES]; #ifdef CONFIG_MIGRATION +extern int PageMovable(struct page *page); +extern void __SetPageMovable(struct page *page, struct address_space *mapping); +extern void __ClearPageMovable(struct page *page); extern void putback_movable_pages(struct list_head *l); extern int migrate_page(struct address_space *, struct page *, struct page *, enum migrate_mode); extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free, unsigned long private, enum migrate_mode mode, int reason); +extern bool isolate_movable_page(struct page *page, isolate_mode_t mode); +extern void putback_movable_page(struct page *page); extern int migrate_prep(void); extern int migrate_prep_local(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b52750caf9e..91e9ebc44e37 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1021,6 +1021,7 @@ static inline pgoff_t page_file_index(struct page *page) } bool page_mapped(struct page *page); +struct address_space *page_mapping(struct page *page); /* * Return true only if the page has been allocated with diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index e5a32445f930..ccb46bc97f81 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -129,6 +129,9 @@ enum pageflags { /* Compound pages. Stored in first tail page's flags */ PG_double_map = PG_private_2, + + /* non-lru isolated movable page */ + PG_isolated = PG_reclaim, }; #ifndef __GENERATING_BOUNDS_H @@ -361,25 +364,33 @@ PAGEFLAG(Idle, idle, PF_ANY) * and then page->mapping points, not to an anon_vma, but to a private * structure which KSM associates with that merged page. See ksm.h. * - * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is currently never used. + * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable + * page and then page->mapping points a struct address_space. * * Please note that, confusingly, "page_mapping" refers to the inode * address_space which maps the page from disk; whereas "page_mapped" * refers to user virtual address space into which the page is mapped. */ -#define PAGE_MAPPING_ANON 1 -#define PAGE_MAPPING_KSM 2 -#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM) +#define PAGE_MAPPING_ANON 0x1 +#define PAGE_MAPPING_MOVABLE 0x2 +#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) +#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) -static __always_inline int PageAnonHead(struct page *page) +static __always_inline int PageMappingFlag(struct page *page) { - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; + return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0; } static __always_inline int PageAnon(struct page *page) { page = compound_head(page); - return PageAnonHead(page); + return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; +} + +static __always_inline int __PageMovable(struct page *page) +{ + return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == + PAGE_MAPPING_MOVABLE; } #ifdef CONFIG_KSM @@ -393,7 +404,7 @@ static __always_inline int PageKsm(struct page *page) { page = compound_head(page); return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + PAGE_MAPPING_KSM; } #else TESTPAGEFLAG_FALSE(Ksm) @@ -641,6 +652,8 @@ static inline void __ClearPageBalloon(struct page *page) atomic_set(&page->_mapcount, -1); } +__PAGEFLAG(Isolated, isolated, PF_ANY); + /* * If network-based swap is enabled, sl*b must keep track of whether pages * were allocated from pfmemalloc reserves. diff --git a/mm/compaction.c b/mm/compaction.c index d8a20fcf8678..e9eeb8eb3b10 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -735,21 +735,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } /* - * Check may be lockless but that's ok as we recheck later. - * It's possible to migrate LRU pages and balloon pages - * Skip any other type of page - */ - is_lru = PageLRU(page); - if (!is_lru) { - if (unlikely(balloon_page_movable(page))) { - if (balloon_page_isolate(page)) { - /* Successfully isolated */ - goto isolate_success; - } - } - } - - /* * Regardless of being on LRU, compound pages such as THP and * hugetlbfs are not to be compacted. We can potentially save * a lot of iterations if we skip them at once. The check is @@ -765,8 +750,38 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail; } - if (!is_lru) + /* + * Check may be lockless but that's ok as we recheck later. + * It's possible to migrate LRU and non-lru movable pages. + * Skip any other type of page + */ + is_lru = PageLRU(page); + if (!is_lru) { + if (unlikely(balloon_page_movable(page))) { + if (balloon_page_isolate(page)) { + /* Successfully isolated */ + goto isolate_success; + } + } + + /* + * __PageMovable can return false positive so we need + * to verify it under page_lock. + */ + if (unlikely(__PageMovable(page)) && + !PageIsolated(page)) { + if (locked) { + spin_unlock_irqrestore(&zone->lru_lock, + flags); + locked = false; + } + + if (isolate_movable_page(page, isolate_mode)) + goto isolate_success; + } + goto isolate_fail; + } /* * Migration will fail if an anonymous page is pinned in memory, diff --git a/mm/ksm.c b/mm/ksm.c index 53bf3ba176ec..60bd202dd9f4 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -532,8 +532,8 @@ static struct page *get_ksm_page(struct stable_node *stable_node, bool lock_it) void *expected_mapping; unsigned long kpfn; - expected_mapping = (void *)stable_node + - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + expected_mapping = (void *)((unsigned long)stable_node | + PAGE_MAPPING_KSM); again: kpfn = READ_ONCE(stable_node->kpfn); page = pfn_to_page(kpfn); diff --git a/mm/migrate.c b/mm/migrate.c index f2932498ded2..24205e1f9204 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -47,6 +47,38 @@ #include "internal.h" +int PageMovable(struct page *page) +{ + struct address_space *mapping; + + WARN_ON(!PageLocked(page)); + if (!__PageMovable(page)) + goto out; + + mapping = page_mapping(page); + if (mapping && mapping->a_ops && mapping->a_ops->isolate_page) + return 1; +out: + return 0; +} + +void __SetPageMovable(struct page *page, struct address_space *mapping) +{ + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE((unsigned long)mapping & PAGE_MAPPING_MOVABLE, page); + page->mapping = (void *)((unsigned long)mapping | PAGE_MAPPING_MOVABLE); +} + +void __ClearPageMovable(struct page *page) +{ + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(!((unsigned long)page->mapping & PAGE_MAPPING_MOVABLE), + page); + page->mapping = (void *)((unsigned long)page->mapping & + PAGE_MAPPING_MOVABLE); +} + /* * migrate_prep() needs to be called before we start compiling a list of pages * to be migrated using isolate_lru_page(). If scheduling work on other CPUs is @@ -73,6 +105,79 @@ int migrate_prep_local(void) return 0; } +bool isolate_movable_page(struct page *page, isolate_mode_t mode) +{ + struct address_space *mapping; + + /* + * Avoid burning cycles with pages that are yet under __free_pages(), + * or just got freed under us. + * + * In case we 'win' a race for a movable page being freed under us and + * raise its refcount preventing __free_pages() from doing its job + * the put_page() at the end of this block will take care of + * release this page, thus avoiding a nasty leakage. + */ + if (unlikely(!get_page_unless_zero(page))) + goto out; + + /* + * Check PageMovable before holding a PG_lock because page's owner + * assumes anybody doesn't touch PG_lock of newly allocated page + * so unconditionally grapping the lock ruins page's owner side. + */ + if (unlikely(!__PageMovable(page))) + goto out_putpage; + /* + * As movable pages are not isolated from LRU lists, concurrent + * compaction threads can race against page migration functions + * as well as race against the releasing a page. + * + * In order to avoid having an already isolated movable page + * being (wrongly) re-isolated while it is under migration, + * or to avoid attempting to isolate pages being released, + * lets be sure we have the page lock + * before proceeding with the movable page isolation steps. + */ + if (unlikely(!trylock_page(page))) + goto out_putpage; + + if (!PageMovable(page) || PageIsolated(page)) + goto out_no_isolated; + + mapping = page_mapping(page); + if (!mapping->a_ops->isolate_page(page, mode)) + goto out_no_isolated; + + /* Driver shouldn't use PG_isolated bit of page->flags */ + WARN_ON_ONCE(PageIsolated(page)); + __SetPageIsolated(page); + unlock_page(page); + + return true; + +out_no_isolated: + unlock_page(page); +out_putpage: + put_page(page); +out: + return false; +} + +/* It should be called on page which is PG_movable */ +void putback_movable_page(struct page *page) +{ + struct address_space *mapping; + + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(!PageIsolated(page), page); + + mapping = page_mapping(page); + mapping->a_ops->putback_page(page); + __ClearPageIsolated(page); +} + /* * Put previously isolated pages back onto the appropriate lists * from where they were once taken off for compaction/migration. @@ -94,10 +199,25 @@ void putback_movable_pages(struct list_head *l) list_del(&page->lru); dec_zone_page_state(page, NR_ISOLATED_ANON + page_is_file_cache(page)); - if (unlikely(isolated_balloon_page(page))) + if (unlikely(isolated_balloon_page(page))) { balloon_page_putback(page); - else + /* + * We isolated non-lru movable page so here we can use + * __PageMovable because LRU page's mapping cannot have + * PAGE_MAPPING_MOVABLE. + */ + } else if (unlikely(__PageMovable(page))) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + lock_page(page); + if (PageMovable(page)) + putback_movable_page(page); + else + __ClearPageIsolated(page); + unlock_page(page); + put_page(page); + } else { putback_lru_page(page); + } } } @@ -592,7 +712,7 @@ void migrate_page_copy(struct page *newpage, struct page *page) ***********************************************************/ /* - * Common logic to directly migrate a single page suitable for + * Common logic to directly migrate a single LRU page suitable for * pages that do not use PagePrivate/PagePrivate2. * * Pages are locked upon entry and exit. @@ -755,33 +875,69 @@ static int move_to_new_page(struct page *newpage, struct page *page, enum migrate_mode mode) { struct address_space *mapping; - int rc; + int rc = -EAGAIN; + bool is_lru = !__PageMovable(page); VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); mapping = page_mapping(page); - if (!mapping) - rc = migrate_page(mapping, newpage, page, mode); - else if (mapping->a_ops->migratepage) - /* - * Most pages have a mapping and most filesystems provide a - * migratepage callback. Anonymous pages are part of swap - * space which also has its own migratepage callback. This - * is the most common path for page migration. - */ - rc = mapping->a_ops->migratepage(mapping, newpage, page, mode); - else - rc = fallback_migrate_page(mapping, newpage, page, mode); + /* + * In case of non-lru page, it could be released after + * isolation step. In that case, we shouldn't try + * fallback migration which is designed for LRU pages. + */ + if (unlikely(!is_lru)) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + if (!PageMovable(page)) { + rc = MIGRATEPAGE_SUCCESS; + __ClearPageIsolated(page); + goto out; + } + } + + if (likely(is_lru)) { + if (!mapping) + rc = migrate_page(mapping, newpage, page, mode); + else if (mapping->a_ops->migratepage) + /* + * Most pages have a mapping and most filesystems + * provide a migratepage callback. Anonymous pages + * are part of swap space which also has its own + * migratepage callback. This is the most common path + * for page migration. + */ + rc = mapping->a_ops->migratepage(mapping, newpage, + page, mode); + else + rc = fallback_migrate_page(mapping, newpage, + page, mode); + } else { + rc = mapping->a_ops->migratepage(mapping, newpage, + page, mode); + WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && + !PageIsolated(page)); + } /* * When successful, old pagecache page->mapping must be cleared before * page is freed; but stats require that PageAnon be left as PageAnon. */ if (rc == MIGRATEPAGE_SUCCESS) { - if (!PageAnon(page)) + if (__PageMovable(page)) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + + /* + * We clear PG_movable under page_lock so any compactor + * cannot try to migrate this page. + */ + __ClearPageIsolated(page); + } + + if (!((unsigned long)page->mapping & PAGE_MAPPING_FLAGS)) page->mapping = NULL; } +out: return rc; } @@ -791,6 +947,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, int rc = -EAGAIN; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; + bool is_lru = !__PageMovable(page); if (!trylock_page(page)) { if (!force || mode == MIGRATE_ASYNC) @@ -871,6 +1028,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, goto out_unlock_both; } + if (unlikely(!is_lru)) { + rc = move_to_new_page(newpage, page, mode); + goto out_unlock_both; + } + /* * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU @@ -920,7 +1082,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * list in here. */ if (rc == MIGRATEPAGE_SUCCESS) { - if (unlikely(__is_movable_balloon_page(newpage))) + if (unlikely(__is_movable_balloon_page(newpage) || + __PageMovable(newpage))) put_page(newpage); else putback_lru_page(newpage); @@ -961,6 +1124,12 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, /* page was freed from under us. So we are done. */ ClearPageActive(page); ClearPageUnevictable(page); + if (unlikely(__PageMovable(page))) { + lock_page(page); + if (!PageMovable(page)) + __ClearPageIsolated(page); + unlock_page(page); + } if (put_new_page) put_new_page(newpage, private); else @@ -1010,8 +1179,21 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, num_poisoned_pages_inc(); } } else { - if (rc != -EAGAIN) - putback_lru_page(page); + if (rc != -EAGAIN) { + if (likely(!__PageMovable(page))) { + putback_lru_page(page); + goto put_new; + } + + lock_page(page); + if (PageMovable(page)) + putback_movable_page(page); + else + __ClearPageIsolated(page); + unlock_page(page); + put_page(page); + } +put_new: if (put_new_page) put_new_page(newpage, private); else diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6663d1736bce..c37472f66566 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1015,7 +1015,7 @@ static __always_inline bool free_pages_prepare(struct page *page, (page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; } } - if (PageAnonHead(page)) + if (PageMappingFlag(page)) page->mapping = NULL; if (check_free) bad += free_pages_check(page); diff --git a/mm/util.c b/mm/util.c index 917e0e3d0f8e..b756ee36f7f0 100644 --- a/mm/util.c +++ b/mm/util.c @@ -399,10 +399,12 @@ struct address_space *page_mapping(struct page *page) } mapping = page->mapping; - if ((unsigned long)mapping & PAGE_MAPPING_FLAGS) + if ((unsigned long)mapping & PAGE_MAPPING_ANON) return NULL; - return mapping; + + return (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS); } +EXPORT_SYMBOL(page_mapping); /* Slow path of page_mapcount() for compound pages */ int __page_mapcount(struct page *page) -- 1.9.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751443AbcEICVz (ORCPT ); Sun, 8 May 2016 22:21:55 -0400 Received: from LGEAMRELO11.lge.com ([156.147.23.51]:55652 "EHLO lgeamrelo11.lge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750935AbcEICUY (ORCPT ); Sun, 8 May 2016 22:20:24 -0400 X-Original-SENDERIP: 156.147.1.151 X-Original-MAILFROM: minchan@kernel.org X-Original-SENDERIP: 10.177.223.161 X-Original-MAILFROM: minchan@kernel.org From: Minchan Kim To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Rik van Riel , Vlastimil Babka , Joonsoo Kim , Mel Gorman , Hugh Dickins , Rafael Aquini , virtualization@lists.linux-foundation.org, Jonathan Corbet , John Einar Reitan , dri-devel@lists.freedesktop.org, Sergey Senozhatsky , Gioh Kim Subject: [PATCH v5 02/12] mm: migrate: support non-lru movable page migration Date: Mon, 9 May 2016 11:20:23 +0900 Message-Id: <1462760433-32357-3-git-send-email-minchan@kernel.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1462760433-32357-1-git-send-email-minchan@kernel.org> References: <1462760433-32357-1-git-send-email-minchan@kernel.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We have allowed migration for only LRU pages until now and it was enough to make high-order pages. But recently, embedded system(e.g., webOS, android) uses lots of non-movable pages(e.g., zram, GPU memory) so we have seen several reports about troubles of small high-order allocation. For fixing the problem, there were several efforts (e,g,. enhance compaction algorithm, SLUB fallback to 0-order page, reserved memory, vmalloc and so on) but if there are lots of non-movable pages in system, their solutions are void in the long run. So, this patch is to support facility to change non-movable pages with movable. For the feature, this patch introduces functions related to migration to address_space_operations as well as some page flags. If a driver want to make own pages movable, it should define three functions which are function pointers of struct address_space_operations. 1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); What VM expects on isolate_page function of driver is to return *true* if driver isolates page successfully. On returing true, VM marks the page as PG_isolated so concurrent isolation in several CPUs skip the page for isolation. If a driver cannot isolate the page, it should return *false*. Once page is successfully isolated, VM uses page.lru fields so driver shouldn't expect to preserve values in that fields. 2. int (*migratepage) (struct address_space *mapping, struct page *newpage, struct page *oldpage, enum migrate_mode); After isolation, VM calls migratepage of driver with isolated page. The function of migratepage is to move content of the old page to new page and set up fields of struct page newpage. Keep in mind that you should clear PG_movable of oldpage via __ClearPageMovable under page_lock if you migrated the oldpage successfully and returns MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time because VM interprets -EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, VM will give up the page migration without retrying in this time. Driver shouldn't touch page.lru field VM using in the functions. 3. void (*putback_page)(struct page *); If migration fails on isolated page, VM should return the isolated page to the driver so VM calls driver's putback_page with migration failed page. In this function, driver should put the isolated page back to the own data structure. 4. non-lru movable page flags There are two page flags for supporting non-lru movable page. * PG_movable Driver should use the below function to make page movable under page_lock. void __SetPageMovable(struct page *page, struct address_space *mapping) It needs argument of address_space for registering migration family functions which will be called by VM. Exactly speaking, PG_movable is not a real flag of struct page. Rather than, VM reuses page->mapping's lower bits to represent it. #define PAGE_MAPPING_MOVABLE 0x2 page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; so driver shouldn't access page->mapping directly. Instead, driver should use page_mapping which mask off the low two bits of page->mapping so it can get right struct address_space. For testing of non-lru movable page, VM supports __PageMovable function. However, it doesn't guarantee to identify non-lru movable page because page->mapping field is unified with other variables in struct page. As well, if driver releases the page after isolation by VM, page->mapping doesn't have stable value although it has PAGE_MAPPING_MOVABLE (Look at __ClearPageMovable). But __PageMovable is cheap to catch whether page is LRU or non-lru movable once the page has been isolated. Because LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also good for just peeking to test non-lru movable pages before more expensive checking with lock_page in pfn scanning to select victim. For guaranteeing non-lru movable page, VM provides PageMovable function. Unlike __PageMovable, PageMovable functions validates page->mapping and mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden destroying of page->mapping. Driver using __SetPageMovable should clear the flag via __ClearMovablePage under page_lock before the releasing the page. * PG_isolated To prevent concurrent isolation among several CPUs, VM marks isolated page as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru movable page, it can skip it. Driver doesn't need to manipulate the flag because VM will set/clear it automatically. Keep in mind that if driver sees PG_isolated page, it means the page have been isolated by VM so it shouldn't touch page.lru field. PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag for own purpose. Cc: Rik van Riel Cc: Vlastimil Babka Cc: Joonsoo Kim Cc: Mel Gorman Cc: Hugh Dickins Cc: Rafael Aquini Cc: virtualization@lists.linux-foundation.org Cc: Jonathan Corbet Cc: John Einar Reitan Cc: dri-devel@lists.freedesktop.org Cc: Sergey Senozhatsky Signed-off-by: Gioh Kim Signed-off-by: Minchan Kim --- Documentation/filesystems/Locking | 4 + Documentation/filesystems/vfs.txt | 11 ++ Documentation/vm/page_migration | 107 +++++++++++++++++- include/linux/fs.h | 2 + include/linux/ksm.h | 3 +- include/linux/migrate.h | 5 + include/linux/mm.h | 1 + include/linux/page-flags.h | 29 +++-- mm/compaction.c | 47 +++++--- mm/ksm.c | 4 +- mm/migrate.c | 222 ++++++++++++++++++++++++++++++++++---- mm/page_alloc.c | 2 +- mm/util.c | 6 +- 13 files changed, 391 insertions(+), 52 deletions(-) diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking index 75eea7ce3d7c..dda6e3f8e203 100644 --- a/Documentation/filesystems/Locking +++ b/Documentation/filesystems/Locking @@ -195,7 +195,9 @@ unlocks and drops the reference. int (*releasepage) (struct page *, int); void (*freepage)(struct page *); int (*direct_IO)(struct kiocb *, struct iov_iter *iter); + bool (*isolate_page) (struct page *, isolate_mode_t); int (*migratepage)(struct address_space *, struct page *, struct page *); + void (*putback_page) (struct page *); int (*launder_page)(struct page *); int (*is_partially_uptodate)(struct page *, unsigned long, unsigned long); int (*error_remove_page)(struct address_space *, struct page *); @@ -219,7 +221,9 @@ invalidatepage: yes releasepage: yes freepage: yes direct_IO: +isolate_page: yes migratepage: yes (both) +putback_page: yes launder_page: yes is_partially_uptodate: yes error_remove_page: yes diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt index c61a223ef3ff..900360cbcdae 100644 --- a/Documentation/filesystems/vfs.txt +++ b/Documentation/filesystems/vfs.txt @@ -592,9 +592,14 @@ struct address_space_operations { int (*releasepage) (struct page *, int); void (*freepage)(struct page *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); + /* isolate a page for migration */ + bool (*isolate_page) (struct page *, isolate_mode_t); /* migrate the contents of a page to the specified target */ int (*migratepage) (struct page *, struct page *); + /* put migration-failed page back to right list */ + void (*putback_page) (struct page *); int (*launder_page) (struct page *); + int (*is_partially_uptodate) (struct page *, unsigned long, unsigned long); void (*is_dirty_writeback) (struct page *, bool *, bool *); @@ -747,6 +752,10 @@ struct address_space_operations { and transfer data directly between the storage and the application's address space. + isolate_page: Called by the VM when isolating a movable non-lru page. + If page is successfully isolated, VM marks the page as PG_isolated + via __SetPageIsolated. + migrate_page: This is used to compact the physical memory usage. If the VM wants to relocate a page (maybe off a memory card that is signalling imminent failure) it will pass a new page @@ -754,6 +763,8 @@ struct address_space_operations { transfer any private data across and update any references that it has to the page. + putback_page: Called by the VM when isolated page's migration fails. + launder_page: Called before freeing a page - it writes back the dirty page. To prevent redirtying the page, it is kept locked during the whole operation. diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration index fea5c0864170..b89d1de026df 100644 --- a/Documentation/vm/page_migration +++ b/Documentation/vm/page_migration @@ -142,5 +142,110 @@ is increased so that the page cannot be freed while page migration occurs. 20. The new page is moved to the LRU and can be scanned by the swapper etc again. -Christoph Lameter, May 8, 2006. +C. Non-LRU page migration +------------------------- + +Although original migration aimed for reducing the latency of memory access +for NUMA, compaction who want to create high-order page is also main customer. + +Current problem of the implementation is that it is designed to migrate only +*LRU* pages. However, there are potential non-lru pages which can be migrated +in drivers, for example, zsmalloc, virtio-balloon pages. + +For virtio-balloon pages, some parts of migration code path have been hooked +up and added virtio-balloon specific functions to intercept migration logics. +It's too specific to a driver so other drivers who want to make their pages +movable would have to add own specific hooks in migration path. + +To overclome the problem, VM supports non-LRU page migration which provides +generic functions for non-LRU movable pages without driver specific hooks +migration path. + +If a driver want to make own pages movable, it should define three functions +which are function pointers of struct address_space_operations. + +1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); + +What VM expects on isolate_page function of driver is to return *true* +if driver isolates page successfully. On returing true, VM marks the page +as PG_isolated so concurrent isolation in several CPUs skip the page +for isolation. If a driver cannot isolate the page, it should return *false*. + +Once page is successfully isolated, VM uses page.lru fields so driver +shouldn't expect to preserve values in that fields. + +2. int (*migratepage) (struct address_space *mapping, + struct page *newpage, struct page *oldpage, enum migrate_mode); + +After isolation, VM calls migratepage of driver with isolated page. +The function of migratepage is to move content of the old page to new page +and set up fields of struct page newpage. Keep in mind that you should +clear PG_movable of oldpage via __ClearPageMovable under page_lock if you +migrated the oldpage successfully and returns MIGRATEPAGE_SUCCESS. +If driver cannot migrate the page at the moment, driver can return -EAGAIN. +On -EAGAIN, VM will retry page migration in a short time because VM interprets +-EAGAIN as "temporal migration failure". On returning any error except -EAGAIN, +VM will give up the page migration without retrying in this time. + +Driver shouldn't touch page.lru field VM using in the functions. + +3. void (*putback_page)(struct page *); + +If migration fails on isolated page, VM should return the isolated page +to the driver so VM calls driver's putback_page with migration failed page. +In this function, driver should put the isolated page back to the own data +structure. +4. non-lru movable page flags + +There are two page flags for supporting non-lru movable page. + +* PG_movable + +Driver should use the below function to make page movable under page_lock. + + void __SetPageMovable(struct page *page, struct address_space *mapping) + +It needs argument of address_space for registering migration family functions +which will be called by VM. Exactly speaking, PG_movable is not a real flag of +struct page. Rather than, VM reuses page->mapping's lower bits to represent it. + + #define PAGE_MAPPING_MOVABLE 0x2 + page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; + +so driver shouldn't access page->mapping directly. Instead, driver should +use page_mapping which mask off the low two bits of page->mapping so it can get +right struct address_space. + +For testing of non-lru movable page, VM supports __PageMovable function. +However, it doesn't guarantee to identify non-lru movable page because +page->mapping field is unified with other variables in struct page. +As well, if driver releases the page after isolation by VM, page->mapping +doesn't have stable value although it has PAGE_MAPPING_MOVABLE +(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether +page is LRU or non-lru movable once the page has been isolated. Because +LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also +good for just peeking to test non-lru movable pages before more expensive +checking with lock_page in pfn scanning to select victim. + +For guaranteeing non-lru movable page, VM provides PageMovable function. +Unlike __PageMovable, PageMovable functions validates page->mapping and +mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden +destroying of page->mapping. + +Driver using __SetPageMovable should clear the flag via __ClearMovablePage +under page_lock before the releasing the page. + +* PG_isolated + +To prevent concurrent isolation among several CPUs, VM marks isolated page +as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru +movable page, it can skip it. Driver doesn't need to manipulate the flag +because VM will set/clear it automatically. Keep in mind that if driver +sees PG_isolated page, it means the page have been isolated by VM so it +shouldn't touch page.lru field. +PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag +for own purpose. + +Christoph Lameter, May 8, 2006. +Minchan Kim, Mar 28, 2016. diff --git a/include/linux/fs.h b/include/linux/fs.h index d61cd084987e..92029377b9e0 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -403,6 +403,8 @@ struct address_space_operations { */ int (*migratepage) (struct address_space *, struct page *, struct page *, enum migrate_mode); + bool (*isolate_page)(struct page *, isolate_mode_t); + void (*putback_page)(struct page *); int (*launder_page) (struct page *); int (*is_partially_uptodate) (struct page *, unsigned long, unsigned long); diff --git a/include/linux/ksm.h b/include/linux/ksm.h index 7ae216a39c9e..481c8c4627ca 100644 --- a/include/linux/ksm.h +++ b/include/linux/ksm.h @@ -43,8 +43,7 @@ static inline struct stable_node *page_stable_node(struct page *page) static inline void set_page_stable_node(struct page *page, struct stable_node *stable_node) { - page->mapping = (void *)stable_node + - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + page->mapping = (void *)((unsigned long)stable_node | PAGE_MAPPING_KSM); } /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 9b50325e4ddf..31a748f43fa9 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -32,11 +32,16 @@ extern char *migrate_reason_names[MR_TYPES]; #ifdef CONFIG_MIGRATION +extern int PageMovable(struct page *page); +extern void __SetPageMovable(struct page *page, struct address_space *mapping); +extern void __ClearPageMovable(struct page *page); extern void putback_movable_pages(struct list_head *l); extern int migrate_page(struct address_space *, struct page *, struct page *, enum migrate_mode); extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free, unsigned long private, enum migrate_mode mode, int reason); +extern bool isolate_movable_page(struct page *page, isolate_mode_t mode); +extern void putback_movable_page(struct page *page); extern int migrate_prep(void); extern int migrate_prep_local(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b52750caf9e..91e9ebc44e37 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1021,6 +1021,7 @@ static inline pgoff_t page_file_index(struct page *page) } bool page_mapped(struct page *page); +struct address_space *page_mapping(struct page *page); /* * Return true only if the page has been allocated with diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index e5a32445f930..ccb46bc97f81 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -129,6 +129,9 @@ enum pageflags { /* Compound pages. Stored in first tail page's flags */ PG_double_map = PG_private_2, + + /* non-lru isolated movable page */ + PG_isolated = PG_reclaim, }; #ifndef __GENERATING_BOUNDS_H @@ -361,25 +364,33 @@ PAGEFLAG(Idle, idle, PF_ANY) * and then page->mapping points, not to an anon_vma, but to a private * structure which KSM associates with that merged page. See ksm.h. * - * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is currently never used. + * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable + * page and then page->mapping points a struct address_space. * * Please note that, confusingly, "page_mapping" refers to the inode * address_space which maps the page from disk; whereas "page_mapped" * refers to user virtual address space into which the page is mapped. */ -#define PAGE_MAPPING_ANON 1 -#define PAGE_MAPPING_KSM 2 -#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM) +#define PAGE_MAPPING_ANON 0x1 +#define PAGE_MAPPING_MOVABLE 0x2 +#define PAGE_MAPPING_KSM (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) +#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_MOVABLE) -static __always_inline int PageAnonHead(struct page *page) +static __always_inline int PageMappingFlag(struct page *page) { - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; + return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0; } static __always_inline int PageAnon(struct page *page) { page = compound_head(page); - return PageAnonHead(page); + return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; +} + +static __always_inline int __PageMovable(struct page *page) +{ + return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == + PAGE_MAPPING_MOVABLE; } #ifdef CONFIG_KSM @@ -393,7 +404,7 @@ static __always_inline int PageKsm(struct page *page) { page = compound_head(page); return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + PAGE_MAPPING_KSM; } #else TESTPAGEFLAG_FALSE(Ksm) @@ -641,6 +652,8 @@ static inline void __ClearPageBalloon(struct page *page) atomic_set(&page->_mapcount, -1); } +__PAGEFLAG(Isolated, isolated, PF_ANY); + /* * If network-based swap is enabled, sl*b must keep track of whether pages * were allocated from pfmemalloc reserves. diff --git a/mm/compaction.c b/mm/compaction.c index d8a20fcf8678..e9eeb8eb3b10 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -735,21 +735,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } /* - * Check may be lockless but that's ok as we recheck later. - * It's possible to migrate LRU pages and balloon pages - * Skip any other type of page - */ - is_lru = PageLRU(page); - if (!is_lru) { - if (unlikely(balloon_page_movable(page))) { - if (balloon_page_isolate(page)) { - /* Successfully isolated */ - goto isolate_success; - } - } - } - - /* * Regardless of being on LRU, compound pages such as THP and * hugetlbfs are not to be compacted. We can potentially save * a lot of iterations if we skip them at once. The check is @@ -765,8 +750,38 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail; } - if (!is_lru) + /* + * Check may be lockless but that's ok as we recheck later. + * It's possible to migrate LRU and non-lru movable pages. + * Skip any other type of page + */ + is_lru = PageLRU(page); + if (!is_lru) { + if (unlikely(balloon_page_movable(page))) { + if (balloon_page_isolate(page)) { + /* Successfully isolated */ + goto isolate_success; + } + } + + /* + * __PageMovable can return false positive so we need + * to verify it under page_lock. + */ + if (unlikely(__PageMovable(page)) && + !PageIsolated(page)) { + if (locked) { + spin_unlock_irqrestore(&zone->lru_lock, + flags); + locked = false; + } + + if (isolate_movable_page(page, isolate_mode)) + goto isolate_success; + } + goto isolate_fail; + } /* * Migration will fail if an anonymous page is pinned in memory, diff --git a/mm/ksm.c b/mm/ksm.c index 53bf3ba176ec..60bd202dd9f4 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -532,8 +532,8 @@ static struct page *get_ksm_page(struct stable_node *stable_node, bool lock_it) void *expected_mapping; unsigned long kpfn; - expected_mapping = (void *)stable_node + - (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM); + expected_mapping = (void *)((unsigned long)stable_node | + PAGE_MAPPING_KSM); again: kpfn = READ_ONCE(stable_node->kpfn); page = pfn_to_page(kpfn); diff --git a/mm/migrate.c b/mm/migrate.c index f2932498ded2..24205e1f9204 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -47,6 +47,38 @@ #include "internal.h" +int PageMovable(struct page *page) +{ + struct address_space *mapping; + + WARN_ON(!PageLocked(page)); + if (!__PageMovable(page)) + goto out; + + mapping = page_mapping(page); + if (mapping && mapping->a_ops && mapping->a_ops->isolate_page) + return 1; +out: + return 0; +} + +void __SetPageMovable(struct page *page, struct address_space *mapping) +{ + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE((unsigned long)mapping & PAGE_MAPPING_MOVABLE, page); + page->mapping = (void *)((unsigned long)mapping | PAGE_MAPPING_MOVABLE); +} + +void __ClearPageMovable(struct page *page) +{ + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(!((unsigned long)page->mapping & PAGE_MAPPING_MOVABLE), + page); + page->mapping = (void *)((unsigned long)page->mapping & + PAGE_MAPPING_MOVABLE); +} + /* * migrate_prep() needs to be called before we start compiling a list of pages * to be migrated using isolate_lru_page(). If scheduling work on other CPUs is @@ -73,6 +105,79 @@ int migrate_prep_local(void) return 0; } +bool isolate_movable_page(struct page *page, isolate_mode_t mode) +{ + struct address_space *mapping; + + /* + * Avoid burning cycles with pages that are yet under __free_pages(), + * or just got freed under us. + * + * In case we 'win' a race for a movable page being freed under us and + * raise its refcount preventing __free_pages() from doing its job + * the put_page() at the end of this block will take care of + * release this page, thus avoiding a nasty leakage. + */ + if (unlikely(!get_page_unless_zero(page))) + goto out; + + /* + * Check PageMovable before holding a PG_lock because page's owner + * assumes anybody doesn't touch PG_lock of newly allocated page + * so unconditionally grapping the lock ruins page's owner side. + */ + if (unlikely(!__PageMovable(page))) + goto out_putpage; + /* + * As movable pages are not isolated from LRU lists, concurrent + * compaction threads can race against page migration functions + * as well as race against the releasing a page. + * + * In order to avoid having an already isolated movable page + * being (wrongly) re-isolated while it is under migration, + * or to avoid attempting to isolate pages being released, + * lets be sure we have the page lock + * before proceeding with the movable page isolation steps. + */ + if (unlikely(!trylock_page(page))) + goto out_putpage; + + if (!PageMovable(page) || PageIsolated(page)) + goto out_no_isolated; + + mapping = page_mapping(page); + if (!mapping->a_ops->isolate_page(page, mode)) + goto out_no_isolated; + + /* Driver shouldn't use PG_isolated bit of page->flags */ + WARN_ON_ONCE(PageIsolated(page)); + __SetPageIsolated(page); + unlock_page(page); + + return true; + +out_no_isolated: + unlock_page(page); +out_putpage: + put_page(page); +out: + return false; +} + +/* It should be called on page which is PG_movable */ +void putback_movable_page(struct page *page) +{ + struct address_space *mapping; + + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(!PageIsolated(page), page); + + mapping = page_mapping(page); + mapping->a_ops->putback_page(page); + __ClearPageIsolated(page); +} + /* * Put previously isolated pages back onto the appropriate lists * from where they were once taken off for compaction/migration. @@ -94,10 +199,25 @@ void putback_movable_pages(struct list_head *l) list_del(&page->lru); dec_zone_page_state(page, NR_ISOLATED_ANON + page_is_file_cache(page)); - if (unlikely(isolated_balloon_page(page))) + if (unlikely(isolated_balloon_page(page))) { balloon_page_putback(page); - else + /* + * We isolated non-lru movable page so here we can use + * __PageMovable because LRU page's mapping cannot have + * PAGE_MAPPING_MOVABLE. + */ + } else if (unlikely(__PageMovable(page))) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + lock_page(page); + if (PageMovable(page)) + putback_movable_page(page); + else + __ClearPageIsolated(page); + unlock_page(page); + put_page(page); + } else { putback_lru_page(page); + } } } @@ -592,7 +712,7 @@ void migrate_page_copy(struct page *newpage, struct page *page) ***********************************************************/ /* - * Common logic to directly migrate a single page suitable for + * Common logic to directly migrate a single LRU page suitable for * pages that do not use PagePrivate/PagePrivate2. * * Pages are locked upon entry and exit. @@ -755,33 +875,69 @@ static int move_to_new_page(struct page *newpage, struct page *page, enum migrate_mode mode) { struct address_space *mapping; - int rc; + int rc = -EAGAIN; + bool is_lru = !__PageMovable(page); VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); mapping = page_mapping(page); - if (!mapping) - rc = migrate_page(mapping, newpage, page, mode); - else if (mapping->a_ops->migratepage) - /* - * Most pages have a mapping and most filesystems provide a - * migratepage callback. Anonymous pages are part of swap - * space which also has its own migratepage callback. This - * is the most common path for page migration. - */ - rc = mapping->a_ops->migratepage(mapping, newpage, page, mode); - else - rc = fallback_migrate_page(mapping, newpage, page, mode); + /* + * In case of non-lru page, it could be released after + * isolation step. In that case, we shouldn't try + * fallback migration which is designed for LRU pages. + */ + if (unlikely(!is_lru)) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + if (!PageMovable(page)) { + rc = MIGRATEPAGE_SUCCESS; + __ClearPageIsolated(page); + goto out; + } + } + + if (likely(is_lru)) { + if (!mapping) + rc = migrate_page(mapping, newpage, page, mode); + else if (mapping->a_ops->migratepage) + /* + * Most pages have a mapping and most filesystems + * provide a migratepage callback. Anonymous pages + * are part of swap space which also has its own + * migratepage callback. This is the most common path + * for page migration. + */ + rc = mapping->a_ops->migratepage(mapping, newpage, + page, mode); + else + rc = fallback_migrate_page(mapping, newpage, + page, mode); + } else { + rc = mapping->a_ops->migratepage(mapping, newpage, + page, mode); + WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && + !PageIsolated(page)); + } /* * When successful, old pagecache page->mapping must be cleared before * page is freed; but stats require that PageAnon be left as PageAnon. */ if (rc == MIGRATEPAGE_SUCCESS) { - if (!PageAnon(page)) + if (__PageMovable(page)) { + VM_BUG_ON_PAGE(!PageIsolated(page), page); + + /* + * We clear PG_movable under page_lock so any compactor + * cannot try to migrate this page. + */ + __ClearPageIsolated(page); + } + + if (!((unsigned long)page->mapping & PAGE_MAPPING_FLAGS)) page->mapping = NULL; } +out: return rc; } @@ -791,6 +947,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, int rc = -EAGAIN; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; + bool is_lru = !__PageMovable(page); if (!trylock_page(page)) { if (!force || mode == MIGRATE_ASYNC) @@ -871,6 +1028,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, goto out_unlock_both; } + if (unlikely(!is_lru)) { + rc = move_to_new_page(newpage, page, mode); + goto out_unlock_both; + } + /* * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU @@ -920,7 +1082,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * list in here. */ if (rc == MIGRATEPAGE_SUCCESS) { - if (unlikely(__is_movable_balloon_page(newpage))) + if (unlikely(__is_movable_balloon_page(newpage) || + __PageMovable(newpage))) put_page(newpage); else putback_lru_page(newpage); @@ -961,6 +1124,12 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, /* page was freed from under us. So we are done. */ ClearPageActive(page); ClearPageUnevictable(page); + if (unlikely(__PageMovable(page))) { + lock_page(page); + if (!PageMovable(page)) + __ClearPageIsolated(page); + unlock_page(page); + } if (put_new_page) put_new_page(newpage, private); else @@ -1010,8 +1179,21 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, num_poisoned_pages_inc(); } } else { - if (rc != -EAGAIN) - putback_lru_page(page); + if (rc != -EAGAIN) { + if (likely(!__PageMovable(page))) { + putback_lru_page(page); + goto put_new; + } + + lock_page(page); + if (PageMovable(page)) + putback_movable_page(page); + else + __ClearPageIsolated(page); + unlock_page(page); + put_page(page); + } +put_new: if (put_new_page) put_new_page(newpage, private); else diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6663d1736bce..c37472f66566 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1015,7 +1015,7 @@ static __always_inline bool free_pages_prepare(struct page *page, (page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; } } - if (PageAnonHead(page)) + if (PageMappingFlag(page)) page->mapping = NULL; if (check_free) bad += free_pages_check(page); diff --git a/mm/util.c b/mm/util.c index 917e0e3d0f8e..b756ee36f7f0 100644 --- a/mm/util.c +++ b/mm/util.c @@ -399,10 +399,12 @@ struct address_space *page_mapping(struct page *page) } mapping = page->mapping; - if ((unsigned long)mapping & PAGE_MAPPING_FLAGS) + if ((unsigned long)mapping & PAGE_MAPPING_ANON) return NULL; - return mapping; + + return (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS); } +EXPORT_SYMBOL(page_mapping); /* Slow path of page_mapcount() for compound pages */ int __page_mapcount(struct page *page) -- 1.9.1