From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx127.postini.com [74.125.245.127]) by kanga.kvack.org (Postfix) with SMTP id 746306B0032 for ; Tue, 23 Jul 2013 00:58:24 -0400 (EDT) From: Lisa Du Date: Mon, 22 Jul 2013 21:58:17 -0700 Subject: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_89813612683626448B837EE5A0B6A7CB3B62F8F272SCVEXCH4marve_" MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: "linux-mm@kvack.org" --_000_89813612683626448B837EE5A0B6A7CB3B62F8F272SCVEXCH4marve_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Dear Sir: Currently I met a possible deadloop in direct reclaim. After run plenty of = the application, system run into a status that system memory is very fragme= ntized. Like only order-0 and order-1 memory left. Then one process required a order-2 buffer but it enter an endless direct r= eclaim. From my trace log, I can see this loop already over 200,000 times. = Kswapd was first wake up and then go back to sleep as it cannot rebalance t= his order's memory. But zone->all_unreclaimable remains 1. Though direct_reclaim every time returns no pages, but as zone->all_unrecla= imable =3D 1, so it loop again and again. Even when zone->pages_scanned als= o becomes very large. It will block the process for long time, until some w= atchdog thread detect this and kill this process. Though it's in __alloc_pa= ges_slowpath, but it's too slow right? Maybe cost over 50 seconds or even m= ore. I think it's not as expected right? Can we also add below check in the fun= ction all_unreclaimable() to terminate this loop? @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist *zoneli= st, continue; if (!zone->all_unreclaimable) return false; + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)) + return true; } BTW: I'm using kernel3.4, I also try to search in the kernel3.9, d= idn't see a possible fix for such issue. Or is anyone also met such issue b= efore? Any comment will be welcomed, looking forward to your reply! Thanks! Best Regards Lisa Du --_000_89813612683626448B837EE5A0B6A7CB3B62F8F272SCVEXCH4marve_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Dear Sir:

Curren= tly I met a possible deadloop in direct reclaim. After run plenty of the application, system run into a status that system memory is very fragmentized. Like only order-0 and order-1 memory left.

Then o= ne process required a order-2 buffer but it enter an endless direct reclaim. From my t= race log, I can see this loop already over 200,000 times. Kswapd was first wake = up and then go back to sleep as it cannot rebalance this order’s memory.= But zone->all_unreclaimable remains 1.

Though direct_reclaim every time returns no pages, but as zone->all_unreclaimab= le =3D 1, so it loop again and again. Even when zone->pages_scanned also become= s very large. It will block the process for long time, until some watchdog th= read detect this and kill this process. Though it’s in __alloc_pages_slowp= ath, but it’s too slow right? Maybe cost over 50 seconds or even more.

I thin= k it’s not as expected right? Can we also add below check in the function al= l_unreclaimable() to terminate this loop?

&= nbsp;

@@ -23= 55,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist *zonelist,

= &nb= sp; continue;

= &nb= sp; if (!zone->all_unreclaimable)

= &nb= sp; return false;

+ = ; &n= bsp; if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone))

+ = ; &n= bsp; return true;

= }

= ; BTW: I’m using kernel3.4, I also try to search in the kernel3.9, didn̵= 7;t see a possible fix for such issue. Or is anyone also met such issue before?= Any comment will be welcomed, looking forward to your reply!<= /p>

Thanks!

Best Regards

Lisa Du

--_000_89813612683626448B837EE5A0B6A7CB3B62F8F272SCVEXCH4marve_-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx180.postini.com [74.125.245.180]) by kanga.kvack.org (Postfix) with SMTP id EE2D76B0033 for ; Tue, 23 Jul 2013 16:28:47 -0400 (EDT) Date: Tue, 23 Jul 2013 20:28:46 +0000 From: Christoph Lameter Subject: Re: Possible deadloop in direct reclaim? In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Message-ID: <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , Mel Gorman On Mon, 22 Jul 2013, Lisa Du wrote: > Currently I met a possible deadloop in direct reclaim. After run plenty of the application, system run into a status that system memory is very fragmentized. Like only order-0 and order-1 memory left. Can you verify that by doing a cat /proc/buddyinfo ? > Then one process required a order-2 buffer but it enter an endless > direct reclaim. From my trace log, I can see this loop already over > 200,000 times. Kswapd was first wake up and then go back to sleep as it > cannot rebalance this order's memory. But zone->all_unreclaimable > remains 1. Though direct_reclaim every time returns no pages, but as > zone->all_unreclaimable = 1, so it loop again and again. Even when > zone->pages_scanned also becomes very large. It will block the process > for long time, until some watchdog thread detect this and kill this > process. Though it's in __alloc_pages_slowpath, but it's too slow right? > Maybe cost over 50 seconds or even more. > I think it's not as expected right? Can we also add below check in the > function all_unreclaimable() to terminate this loop? > > @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!zone->all_unreclaimable) > return false; > + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) > + return true; > } Mel? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx180.postini.com [74.125.245.180]) by kanga.kvack.org (Postfix) with SMTP id 62F806B0033 for ; Tue, 23 Jul 2013 21:18:11 -0400 (EDT) Received: by mail-vc0-f176.google.com with SMTP id ha11so752836vcb.7 for ; Tue, 23 Jul 2013 18:18:10 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Date: Wed, 24 Jul 2013 09:18:10 +0800 Message-ID: Subject: Re: Possible deadloop in direct reclaim? From: Bob Liu Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman On Tue, Jul 23, 2013 at 12:58 PM, Lisa Du wrote: > Dear Sir: > > Currently I met a possible deadloop in direct reclaim. After run plenty o= f > the application, system run into a status that system memory is very > fragmentized. Like only order-0 and order-1 memory left. > > Then one process required a order-2 buffer but it enter an endless direct > reclaim. From my trace log, I can see this loop already over 200,000 time= s. > Kswapd was first wake up and then go back to sleep as it cannot rebalance > this order=E2=80=99s memory. But zone->all_unreclaimable remains 1. > > Though direct_reclaim every time returns no pages, but as > zone->all_unreclaimable =3D 1, so it loop again and again. Even when > zone->pages_scanned also becomes very large. It will block the process fo= r > long time, until some watchdog thread detect this and kill this process. > Though it=E2=80=99s in __alloc_pages_slowpath, but it=E2=80=99s too slow = right? Maybe cost > over 50 seconds or even more. You must be mean zone->all_unreclaimable =3D 0? > > I think it=E2=80=99s not as expected right? Can we also add below check = in the > function all_unreclaimable() to terminate this loop? > > > > @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist > *zonelist, > > continue; > > if (!zone->all_unreclaimable) > > return false; > > + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)) > > + return true; > How about replace the checking in kswapd_shrink_zone()? @@ -2824,7 +2824,7 @@ static bool kswapd_shrink_zone(struct zone *zone, /* Account for the number of pages attempted to reclaim */ *nr_attempted +=3D sc->nr_to_reclaim; - if (nr_slab =3D=3D 0 && !zone_reclaimable(zone)) + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)) zone->all_unreclaimable =3D 1; zone_clear_flag(zone, ZONE_WRITEBACK); I think the current check is wrong, reclaimed a slab doesn't mean reclaimed a page. --=20 Regards, --Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx109.postini.com [74.125.245.109]) by kanga.kvack.org (Postfix) with SMTP id 190646B0031 for ; Tue, 23 Jul 2013 21:21:46 -0400 (EDT) From: Lisa Du Date: Tue, 23 Jul 2013 18:21:29 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> In-Reply-To: <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: "linux-mm@kvack.org" , Mel Gorman , Bob Liu RGVhciBDaHJpc3RvcGgNCiAgIFRoYW5rcyBhIGxvdCBmb3IgeW91ciBjb21tZW50LiBXaGVuIHRo aXMgaXNzdWUgaGFwcGVuIEkganVzdCB0cmlnZ2VyIGEga2VybmVsIHBhbmljIGFuZCBnb3QgdGhl IGtkdW1wLg0KRnJvbSB0aGUga2R1bXAsIEkgZ290IHRoZSBnbG9iYWwgdmFyaWFibGUgcGdfZGF0 YV90IGNvbmdpdF9wYWdlX2RhdGEuIEZyb20gdGhpcyBzdHJ1Y3R1cmUsIEkgY2FuIHNlZSBpbiBu b3JtYWwgem9uZSwgb25seSBvcmRlci0wJ3MgbnJfZnJlZSA9IDE4NDQyLCBvcmRlci0xJ3MgbnJf ZnJlZSA9IDM2NywgYWxsIHRoZSBvdGhlciBvcmRlcidzIG5yX2ZyZWUgaXMgMC4NCg0KVGhhbmtz IQ0KDQpCZXN0IFJlZ2FyZHMNCkxpc2EgRHUNCg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0t LQ0KRnJvbTogQ2hyaXN0b3BoIExhbWV0ZXIgW21haWx0bzpjbEBsaW51eC5jb21dIA0KU2VudDog MjAxM8TqN9TCMjTI1SA0OjI5DQpUbzogTGlzYSBEdQ0KQ2M6IGxpbnV4LW1tQGt2YWNrLm9yZzsg TWVsIEdvcm1hbg0KU3ViamVjdDogUmU6IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNs YWltPw0KDQpPbiBNb24sIDIyIEp1bCAyMDEzLCBMaXNhIER1IHdyb3RlOg0KDQo+IEN1cnJlbnRs eSBJIG1ldCBhIHBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNsYWltLiBBZnRlciBydW4g cGxlbnR5IG9mIHRoZSBhcHBsaWNhdGlvbiwgc3lzdGVtIHJ1biBpbnRvIGEgc3RhdHVzIHRoYXQg c3lzdGVtIG1lbW9yeSBpcyB2ZXJ5IGZyYWdtZW50aXplZC4gTGlrZSBvbmx5IG9yZGVyLTAgYW5k IG9yZGVyLTEgbWVtb3J5IGxlZnQuDQoNCkNhbiB5b3UgdmVyaWZ5IHRoYXQgYnkgZG9pbmcgYQ0K DQogY2F0IC9wcm9jL2J1ZGR5aW5mbw0KDQo/DQoNCj4gVGhlbiBvbmUgcHJvY2VzcyByZXF1aXJl ZCBhIG9yZGVyLTIgYnVmZmVyIGJ1dCBpdCBlbnRlciBhbiBlbmRsZXNzDQo+IGRpcmVjdCByZWNs YWltLiBGcm9tIG15IHRyYWNlIGxvZywgSSBjYW4gc2VlIHRoaXMgbG9vcCBhbHJlYWR5IG92ZXIN Cj4gMjAwLDAwMCB0aW1lcy4gS3N3YXBkIHdhcyBmaXJzdCB3YWtlIHVwIGFuZCB0aGVuIGdvIGJh Y2sgdG8gc2xlZXAgYXMgaXQNCj4gY2Fubm90IHJlYmFsYW5jZSB0aGlzIG9yZGVyJ3MgbWVtb3J5 LiBCdXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUNCj4gcmVtYWlucyAxLiBUaG91Z2ggZGlyZWN0 X3JlY2xhaW0gZXZlcnkgdGltZSByZXR1cm5zIG5vIHBhZ2VzLCBidXQgYXMNCj4gem9uZS0+YWxs X3VucmVjbGFpbWFibGUgPSAxLCBzbyBpdCBsb29wIGFnYWluIGFuZCBhZ2Fpbi4gRXZlbiB3aGVu DQo+IHpvbmUtPnBhZ2VzX3NjYW5uZWQgYWxzbyBiZWNvbWVzIHZlcnkgbGFyZ2UuIEl0IHdpbGwg YmxvY2sgdGhlIHByb2Nlc3MNCj4gZm9yIGxvbmcgdGltZSwgdW50aWwgc29tZSB3YXRjaGRvZyB0 aHJlYWQgZGV0ZWN0IHRoaXMgYW5kIGtpbGwgdGhpcw0KPiBwcm9jZXNzLiBUaG91Z2ggaXQncyBp biBfX2FsbG9jX3BhZ2VzX3Nsb3dwYXRoLCBidXQgaXQncyB0b28gc2xvdyByaWdodD8NCj4gTWF5 YmUgY29zdCBvdmVyIDUwIHNlY29uZHMgb3IgZXZlbiBtb3JlLg0KDQo+IEkgdGhpbmsgaXQncyBu b3QgYXMgZXhwZWN0ZWQgcmlnaHQ/ICBDYW4gd2UgYWxzbyBhZGQgYmVsb3cgY2hlY2sgaW4gdGhl DQo+IGZ1bmN0aW9uIGFsbF91bnJlY2xhaW1hYmxlKCkgdG8gdGVybWluYXRlIHRoaXMgbG9vcD8N Cj4NCj4gQEAgLTIzNTUsNiArMjM1NSw4IEBAIHN0YXRpYyBib29sIGFsbF91bnJlY2xhaW1hYmxl KHN0cnVjdCB6b25lbGlzdCAqem9uZWxpc3QsDQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGNv bnRpbnVlOw0KPiAgICAgICAgICAgICAgICAgaWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkN Cj4gICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOw0KPiArICAgICAgICAgICAg ICAgaWYgKHNjLT5ucl9yZWNsYWltZWQgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6b25lKSkN Cj4gKyAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7DQo+ICAgICAgICAgfQ0KDQpN ZWw/DQoNCg== -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx106.postini.com [74.125.245.106]) by kanga.kvack.org (Postfix) with SMTP id 028CB6B0033 for ; Tue, 23 Jul 2013 21:31:37 -0400 (EDT) From: Lisa Du Date: Tue, 23 Jul 2013 18:31:02 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8F5CE@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman RGVhciBCb2INCiAgICBUaGFuayB5b3Ugc28gbXVjaCBmb3IgdGhlIGNhcmVmdWwgcmV2aWV3LCBZ ZXMsIGl0J3MgYSB0eXBvLCBJIG1lYW4gem9uZS0+YWxsX3VucmVjbGFpbWFibGUgPSAwLg0KICAg IFlvdSBtZW50aW9uZWQgYWRkIHRoZSBjaGVjayBpbiBrc3dhcGRfc2hyaW5rX3pvbmUoKSwgc29y cnkgdGhhdCBJIGRpZG4ndCBmaW5kIHRoaXMgZnVuY3Rpb24gaW4ga2VybmVsMy40IG9yIGtlcm5l bDMuOS4NCiAgICBJcyB0aGlzIGZ1bmN0aW9uIGNhbGxlZCBpbiBkaXJlY3RfcmVjbGFpbT8gDQog ICAgQXMgSSBtZW50aW9uZWQgdGhpcyBpc3N1ZSBoYXBwZW5lZCBhZnRlciBrc3dhcGQgdGhyZWFk IHNsZWVwLCBpZiBpdCBvbmx5IGNhbGxlZCBpbiBrc3dhcGQsIHRoZW4gSSB0aGluayBpdCBjYW4n dCBoZWxwLg0KDQpUaGFua3MhDQoNCkJlc3QgUmVnYXJkcw0KTGlzYSBEdQ0KDQoNCi0tLS0tT3Jp Z2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCb2IgTGl1IFttYWlsdG86bGxpdWJib0BnbWFpbC5j b21dIA0KU2VudDogMjAxM+W5tDfmnIgyNOaXpSA5OjE4DQpUbzogTGlzYSBEdQ0KQ2M6IGxpbnV4 LW1tQGt2YWNrLm9yZzsgQ2hyaXN0b3BoIExhbWV0ZXI7IE1lbCBHb3JtYW4NClN1YmplY3Q6IFJl OiBQb3NzaWJsZSBkZWFkbG9vcCBpbiBkaXJlY3QgcmVjbGFpbT8NCg0KT24gVHVlLCBKdWwgMjMs IDIwMTMgYXQgMTI6NTggUE0sIExpc2EgRHUgPGNsZHVAbWFydmVsbC5jb20+IHdyb3RlOg0KPiBE ZWFyIFNpcjoNCj4NCj4gQ3VycmVudGx5IEkgbWV0IGEgcG9zc2libGUgZGVhZGxvb3AgaW4gZGly ZWN0IHJlY2xhaW0uIEFmdGVyIHJ1biBwbGVudHkgb2YNCj4gdGhlIGFwcGxpY2F0aW9uLCBzeXN0 ZW0gcnVuIGludG8gYSBzdGF0dXMgdGhhdCBzeXN0ZW0gbWVtb3J5IGlzIHZlcnkNCj4gZnJhZ21l bnRpemVkLiBMaWtlIG9ubHkgb3JkZXItMCBhbmQgb3JkZXItMSBtZW1vcnkgbGVmdC4NCj4NCj4g VGhlbiBvbmUgcHJvY2VzcyByZXF1aXJlZCBhIG9yZGVyLTIgYnVmZmVyIGJ1dCBpdCBlbnRlciBh biBlbmRsZXNzIGRpcmVjdA0KPiByZWNsYWltLiBGcm9tIG15IHRyYWNlIGxvZywgSSBjYW4gc2Vl IHRoaXMgbG9vcCBhbHJlYWR5IG92ZXIgMjAwLDAwMCB0aW1lcy4NCj4gS3N3YXBkIHdhcyBmaXJz dCB3YWtlIHVwIGFuZCB0aGVuIGdvIGJhY2sgdG8gc2xlZXAgYXMgaXQgY2Fubm90IHJlYmFsYW5j ZQ0KPiB0aGlzIG9yZGVy4oCZcyBtZW1vcnkuIEJ1dCB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBy ZW1haW5zIDEuDQo+DQo+IFRob3VnaCBkaXJlY3RfcmVjbGFpbSBldmVyeSB0aW1lIHJldHVybnMg bm8gcGFnZXMsIGJ1dCBhcw0KPiB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSA9IDEsIHNvIGl0IGxv b3AgYWdhaW4gYW5kIGFnYWluLiBFdmVuIHdoZW4NCj4gem9uZS0+cGFnZXNfc2Nhbm5lZCBhbHNv IGJlY29tZXMgdmVyeSBsYXJnZS4gSXQgd2lsbCBibG9jayB0aGUgcHJvY2VzcyBmb3INCj4gbG9u ZyB0aW1lLCB1bnRpbCBzb21lIHdhdGNoZG9nIHRocmVhZCBkZXRlY3QgdGhpcyBhbmQga2lsbCB0 aGlzIHByb2Nlc3MuDQo+IFRob3VnaCBpdOKAmXMgaW4gX19hbGxvY19wYWdlc19zbG93cGF0aCwg YnV0IGl04oCZcyB0b28gc2xvdyByaWdodD8gTWF5YmUgY29zdA0KPiBvdmVyIDUwIHNlY29uZHMg b3IgZXZlbiBtb3JlLg0KDQpZb3UgbXVzdCBiZSBtZWFuIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxl ID0gMD8NCg0KPg0KPiBJIHRoaW5rIGl04oCZcyBub3QgYXMgZXhwZWN0ZWQgcmlnaHQ/ICBDYW4g d2UgYWxzbyBhZGQgYmVsb3cgY2hlY2sgaW4gdGhlDQo+IGZ1bmN0aW9uIGFsbF91bnJlY2xhaW1h YmxlKCkgdG8gdGVybWluYXRlIHRoaXMgbG9vcD8NCj4NCj4NCj4NCj4gQEAgLTIzNTUsNiArMjM1 NSw4IEBAIHN0YXRpYyBib29sIGFsbF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6b25lbGlzdA0KPiAq em9uZWxpc3QsDQo+DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPg0KPiAg ICAgICAgICAgICAgICAgaWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkNCj4NCj4gICAgICAg ICAgICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOw0KPg0KPiArICAgICAgICAgICAgICAgaWYg KHNjLT5ucl9yZWNsYWltZWQgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCj4NCj4g KyAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7DQo+DQoNCkhvdyBhYm91dCByZXBs YWNlIHRoZSBjaGVja2luZyBpbiBrc3dhcGRfc2hyaW5rX3pvbmUoKT8NCg0KQEAgLTI4MjQsNyAr MjgyNCw3IEBAIHN0YXRpYyBib29sIGtzd2FwZF9zaHJpbmtfem9uZShzdHJ1Y3Qgem9uZSAqem9u ZSwNCiAgICAgICAgLyogQWNjb3VudCBmb3IgdGhlIG51bWJlciBvZiBwYWdlcyBhdHRlbXB0ZWQg dG8gcmVjbGFpbSAqLw0KICAgICAgICAqbnJfYXR0ZW1wdGVkICs9IHNjLT5ucl90b19yZWNsYWlt Ow0KDQotICAgICAgIGlmIChucl9zbGFiID09IDAgJiYgIXpvbmVfcmVjbGFpbWFibGUoem9uZSkp DQorICAgICAgIGlmIChzYy0+bnJfcmVjbGFpbWVkID09IDAgJiYgIXpvbmVfcmVjbGFpbWFibGUo em9uZSkpDQogICAgICAgICAgICAgICAgem9uZS0+YWxsX3VucmVjbGFpbWFibGUgPSAxOw0KDQog ICAgICAgIHpvbmVfY2xlYXJfZmxhZyh6b25lLCBaT05FX1dSSVRFQkFDSyk7DQoNCg0KSSB0aGlu ayB0aGUgY3VycmVudCBjaGVjayBpcyB3cm9uZywgcmVjbGFpbWVkIGEgc2xhYiBkb2Vzbid0IG1l YW4NCnJlY2xhaW1lZCBhIHBhZ2UuDQoNCi0tIA0KUmVnYXJkcywNCi0tQm9iDQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx194.postini.com [74.125.245.194]) by kanga.kvack.org (Postfix) with SMTP id 91A6F6B0031 for ; Tue, 23 Jul 2013 22:23:38 -0400 (EDT) From: Lisa Du Date: Tue, 23 Jul 2013 19:23:34 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8F61A@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du , Bob Liu Cc: "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman RGVhciBCb2INCiAgIEFsc28gZnJvbSBteSBjaGVjayBiZWZvcmUga3N3YXBkIHNsZWVwLCB0aG91 Z2ggbnJfc2xhYiA9IDAgYnV0IHpvbmVfcmVjbGFpbWFibGUoem9uZSkgcmV0dXJucyB0cnVlLCBz byB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBjYW4ndCBiZSBjaGFuZ2VkIHRvIDE7IFNvIGV2ZW4g d2hlbiBjaGFuZ2UgdGhlIG5yX3NsYWIgdG8gc2MtPm5yX3JlY2xhaW1lZCwgaXQgY2FuJ3QgaGVs cC4NCg0KVGhhbmtzIQ0KDQpCZXN0IFJlZ2FyZHMNCkxpc2EgRHUNCg0KDQotLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KRnJvbTogTGlzYSBEdSANClNlbnQ6IDIwMTPlubQ35pyIMjTml6UgOToz MQ0KVG86ICdCb2IgTGl1Jw0KQ2M6IGxpbnV4LW1tQGt2YWNrLm9yZzsgQ2hyaXN0b3BoIExhbWV0 ZXI7IE1lbCBHb3JtYW4NClN1YmplY3Q6IFJFOiBQb3NzaWJsZSBkZWFkbG9vcCBpbiBkaXJlY3Qg cmVjbGFpbT8NCg0KRGVhciBCb2INCiAgICBUaGFuayB5b3Ugc28gbXVjaCBmb3IgdGhlIGNhcmVm dWwgcmV2aWV3LCBZZXMsIGl0J3MgYSB0eXBvLCBJIG1lYW4gem9uZS0+YWxsX3VucmVjbGFpbWFi bGUgPSAwLg0KICAgIFlvdSBtZW50aW9uZWQgYWRkIHRoZSBjaGVjayBpbiBrc3dhcGRfc2hyaW5r X3pvbmUoKSwgc29ycnkgdGhhdCBJIGRpZG4ndCBmaW5kIHRoaXMgZnVuY3Rpb24gaW4ga2VybmVs My40IG9yIGtlcm5lbDMuOS4NCiAgICBJcyB0aGlzIGZ1bmN0aW9uIGNhbGxlZCBpbiBkaXJlY3Rf cmVjbGFpbT8gDQogICAgQXMgSSBtZW50aW9uZWQgdGhpcyBpc3N1ZSBoYXBwZW5lZCBhZnRlciBr c3dhcGQgdGhyZWFkIHNsZWVwLCBpZiBpdCBvbmx5IGNhbGxlZCBpbiBrc3dhcGQsIHRoZW4gSSB0 aGluayBpdCBjYW4ndCBoZWxwLg0KDQpUaGFua3MhDQoNCkJlc3QgUmVnYXJkcw0KTGlzYSBEdQ0K DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCb2IgTGl1IFttYWlsdG86bGxp dWJib0BnbWFpbC5jb21dIA0KU2VudDogMjAxM+W5tDfmnIgyNOaXpSA5OjE4DQpUbzogTGlzYSBE dQ0KQ2M6IGxpbnV4LW1tQGt2YWNrLm9yZzsgQ2hyaXN0b3BoIExhbWV0ZXI7IE1lbCBHb3JtYW4N ClN1YmplY3Q6IFJlOiBQb3NzaWJsZSBkZWFkbG9vcCBpbiBkaXJlY3QgcmVjbGFpbT8NCg0KT24g VHVlLCBKdWwgMjMsIDIwMTMgYXQgMTI6NTggUE0sIExpc2EgRHUgPGNsZHVAbWFydmVsbC5jb20+ IHdyb3RlOg0KPiBEZWFyIFNpcjoNCj4NCj4gQ3VycmVudGx5IEkgbWV0IGEgcG9zc2libGUgZGVh ZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0uIEFmdGVyIHJ1biBwbGVudHkgb2YNCj4gdGhlIGFwcGxp Y2F0aW9uLCBzeXN0ZW0gcnVuIGludG8gYSBzdGF0dXMgdGhhdCBzeXN0ZW0gbWVtb3J5IGlzIHZl cnkNCj4gZnJhZ21lbnRpemVkLiBMaWtlIG9ubHkgb3JkZXItMCBhbmQgb3JkZXItMSBtZW1vcnkg bGVmdC4NCj4NCj4gVGhlbiBvbmUgcHJvY2VzcyByZXF1aXJlZCBhIG9yZGVyLTIgYnVmZmVyIGJ1 dCBpdCBlbnRlciBhbiBlbmRsZXNzIGRpcmVjdA0KPiByZWNsYWltLiBGcm9tIG15IHRyYWNlIGxv ZywgSSBjYW4gc2VlIHRoaXMgbG9vcCBhbHJlYWR5IG92ZXIgMjAwLDAwMCB0aW1lcy4NCj4gS3N3 YXBkIHdhcyBmaXJzdCB3YWtlIHVwIGFuZCB0aGVuIGdvIGJhY2sgdG8gc2xlZXAgYXMgaXQgY2Fu bm90IHJlYmFsYW5jZQ0KPiB0aGlzIG9yZGVy4oCZcyBtZW1vcnkuIEJ1dCB6b25lLT5hbGxfdW5y ZWNsYWltYWJsZSByZW1haW5zIDEuDQo+DQo+IFRob3VnaCBkaXJlY3RfcmVjbGFpbSBldmVyeSB0 aW1lIHJldHVybnMgbm8gcGFnZXMsIGJ1dCBhcw0KPiB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSA9 IDEsIHNvIGl0IGxvb3AgYWdhaW4gYW5kIGFnYWluLiBFdmVuIHdoZW4NCj4gem9uZS0+cGFnZXNf c2Nhbm5lZCBhbHNvIGJlY29tZXMgdmVyeSBsYXJnZS4gSXQgd2lsbCBibG9jayB0aGUgcHJvY2Vz cyBmb3INCj4gbG9uZyB0aW1lLCB1bnRpbCBzb21lIHdhdGNoZG9nIHRocmVhZCBkZXRlY3QgdGhp cyBhbmQga2lsbCB0aGlzIHByb2Nlc3MuDQo+IFRob3VnaCBpdOKAmXMgaW4gX19hbGxvY19wYWdl c19zbG93cGF0aCwgYnV0IGl04oCZcyB0b28gc2xvdyByaWdodD8gTWF5YmUgY29zdA0KPiBvdmVy IDUwIHNlY29uZHMgb3IgZXZlbiBtb3JlLg0KDQpZb3UgbXVzdCBiZSBtZWFuIHpvbmUtPmFsbF91 bnJlY2xhaW1hYmxlID0gMD8NCg0KPg0KPiBJIHRoaW5rIGl04oCZcyBub3QgYXMgZXhwZWN0ZWQg cmlnaHQ/ICBDYW4gd2UgYWxzbyBhZGQgYmVsb3cgY2hlY2sgaW4gdGhlDQo+IGZ1bmN0aW9uIGFs bF91bnJlY2xhaW1hYmxlKCkgdG8gdGVybWluYXRlIHRoaXMgbG9vcD8NCj4NCj4NCj4NCj4gQEAg LTIzNTUsNiArMjM1NSw4IEBAIHN0YXRpYyBib29sIGFsbF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6 b25lbGlzdA0KPiAqem9uZWxpc3QsDQo+DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRp bnVlOw0KPg0KPiAgICAgICAgICAgICAgICAgaWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkN Cj4NCj4gICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOw0KPg0KPiArICAgICAg ICAgICAgICAgaWYgKHNjLT5ucl9yZWNsYWltZWQgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6 b25lKSkNCj4NCj4gKyAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7DQo+DQoNCkhv dyBhYm91dCByZXBsYWNlIHRoZSBjaGVja2luZyBpbiBrc3dhcGRfc2hyaW5rX3pvbmUoKT8NCg0K QEAgLTI4MjQsNyArMjgyNCw3IEBAIHN0YXRpYyBib29sIGtzd2FwZF9zaHJpbmtfem9uZShzdHJ1 Y3Qgem9uZSAqem9uZSwNCiAgICAgICAgLyogQWNjb3VudCBmb3IgdGhlIG51bWJlciBvZiBwYWdl cyBhdHRlbXB0ZWQgdG8gcmVjbGFpbSAqLw0KICAgICAgICAqbnJfYXR0ZW1wdGVkICs9IHNjLT5u cl90b19yZWNsYWltOw0KDQotICAgICAgIGlmIChucl9zbGFiID09IDAgJiYgIXpvbmVfcmVjbGFp bWFibGUoem9uZSkpDQorICAgICAgIGlmIChzYy0+bnJfcmVjbGFpbWVkID09IDAgJiYgIXpvbmVf cmVjbGFpbWFibGUoem9uZSkpDQogICAgICAgICAgICAgICAgem9uZS0+YWxsX3VucmVjbGFpbWFi bGUgPSAxOw0KDQogICAgICAgIHpvbmVfY2xlYXJfZmxhZyh6b25lLCBaT05FX1dSSVRFQkFDSyk7 DQoNCg0KSSB0aGluayB0aGUgY3VycmVudCBjaGVjayBpcyB3cm9uZywgcmVjbGFpbWVkIGEgc2xh YiBkb2Vzbid0IG1lYW4NCnJlY2xhaW1lZCBhIHBhZ2UuDQoNCi0tIA0KUmVnYXJkcywNCi0tQm9i DQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx120.postini.com [74.125.245.120]) by kanga.kvack.org (Postfix) with SMTP id 57F446B0031 for ; Tue, 23 Jul 2013 23:38:43 -0400 (EDT) Received: by mail-vc0-f178.google.com with SMTP id hr11so5149811vcb.37 for ; Tue, 23 Jul 2013 20:38:42 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8F61A@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8F61A@SC-VEXCH4.marvell.com> Date: Wed, 24 Jul 2013 11:38:42 +0800 Message-ID: Subject: Re: Possible deadloop in direct reclaim? From: Bob Liu Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman On Wed, Jul 24, 2013 at 10:23 AM, Lisa Du wrote: > Dear Bob > Also from my check before kswapd sleep, though nr_slab =3D 0 but zone_= reclaimable(zone) returns true, so zone->all_unreclaimable can't be changed= to 1; So even when change the nr_slab to sc->nr_reclaimed, it can't help. > Then the other fix might be set zone->all_unreclaimable in direct reclaim path also, like: @@ -2278,6 +2278,8 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc) } shrink_zone(zone, sc); + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)) + zone->all_unreclaimable =3D 1; } > Thanks! > > Best Regards > Lisa Du > > > -----Original Message----- > From: Lisa Du > Sent: 2013=E5=B9=B47=E6=9C=8824=E6=97=A5 9:31 > To: 'Bob Liu' > Cc: linux-mm@kvack.org; Christoph Lameter; Mel Gorman > Subject: RE: Possible deadloop in direct reclaim? > > Dear Bob > Thank you so much for the careful review, Yes, it's a typo, I mean zo= ne->all_unreclaimable =3D 0. > You mentioned add the check in kswapd_shrink_zone(), sorry that I did= n't find this function in kernel3.4 or kernel3.9. > Is this function called in direct_reclaim? > As I mentioned this issue happened after kswapd thread sleep, if it o= nly called in kswapd, then I think it can't help. > > Thanks! > > Best Regards > Lisa Du > > > -----Original Message----- > From: Bob Liu [mailto:lliubbo@gmail.com] > Sent: 2013=E5=B9=B47=E6=9C=8824=E6=97=A5 9:18 > To: Lisa Du > Cc: linux-mm@kvack.org; Christoph Lameter; Mel Gorman > Subject: Re: Possible deadloop in direct reclaim? > > On Tue, Jul 23, 2013 at 12:58 PM, Lisa Du wrote: >> Dear Sir: >> >> Currently I met a possible deadloop in direct reclaim. After run plenty = of >> the application, system run into a status that system memory is very >> fragmentized. Like only order-0 and order-1 memory left. >> >> Then one process required a order-2 buffer but it enter an endless direc= t >> reclaim. From my trace log, I can see this loop already over 200,000 tim= es. >> Kswapd was first wake up and then go back to sleep as it cannot rebalanc= e >> this order=E2=80=99s memory. But zone->all_unreclaimable remains 1. >> >> Though direct_reclaim every time returns no pages, but as >> zone->all_unreclaimable =3D 1, so it loop again and again. Even when >> zone->pages_scanned also becomes very large. It will block the process f= or >> long time, until some watchdog thread detect this and kill this process. >> Though it=E2=80=99s in __alloc_pages_slowpath, but it=E2=80=99s too slow= right? Maybe cost >> over 50 seconds or even more. > > You must be mean zone->all_unreclaimable =3D 0? > >> >> I think it=E2=80=99s not as expected right? Can we also add below check= in the >> function all_unreclaimable() to terminate this loop? >> >> >> >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist >> *zonelist, >> >> continue; >> >> if (!zone->all_unreclaimable) >> >> return false; >> >> + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)= ) >> >> + return true; >> > > How about replace the checking in kswapd_shrink_zone()? > > @@ -2824,7 +2824,7 @@ static bool kswapd_shrink_zone(struct zone *zone, > /* Account for the number of pages attempted to reclaim */ > *nr_attempted +=3D sc->nr_to_reclaim; > > - if (nr_slab =3D=3D 0 && !zone_reclaimable(zone)) > + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)) > zone->all_unreclaimable =3D 1; > > zone_clear_flag(zone, ZONE_WRITEBACK); > > > I think the current check is wrong, reclaimed a slab doesn't mean > reclaimed a page. > > -- > Regards, > --Bob --=20 Regards, --Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx159.postini.com [74.125.245.159]) by kanga.kvack.org (Postfix) with SMTP id C38066B0031 for ; Wed, 24 Jul 2013 02:00:08 -0400 (EDT) From: Lisa Du Date: Tue, 23 Jul 2013 22:58:23 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8F6B0@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8F61A@SC-VEXCH4.marvell.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman , "kosaki.motohiro@jp.fujitsu.com" RGVhciBCb2INCiAgIFJlYWxseSBhcHByZWNpYXRlIGZvciB5b3VyIHJldmlldyBhbmQgc3VnZ2Vz dGlvbnMhDQogICBZZXMsIHlvdXIgc3VnZ2VzdGlvbiBjYW4gZW5kIG15IGluZmluaXRlIGxvb3Ag aW4gZGlyZWN0X3JlY2xhaW0uIFRoaXMgY2hhbmdlIEkgdGhpbmsgd2lsbCBlYXNpZXIgdGhhbiBi ZWZvcmUgdG8gbWFyayBhIHpvbmUgYXMgdW5yZWNsYWltYWJsZSByaWdodD8gIFdpbGwgaXQgaGF2 ZSBvdGhlciBzaWRlIGVmZmVjdD8NCg0KICAgSSByZXZpZXdlZCB0aGUgbWFpbmxpbmUncyBwYXRj aCBsaXN0LCBhbmQgZm91bmQgYmVsb3cgcGF0Y2ggc2hvdWxkIGJlIGEgc2ltaWxhciBjYXNlIGFz IG1pbmUsIGl0J3MgY2FzZSBpcyBrc3dhcGQgaXMgZnJvemVuLCBidXQgbXkgY2FzZSBrc3dhcGQg Z28gdG8gc2xlZXAuDQpGcm9tIGQxOTA4MzYyYWUwYjk3Mzc0ZWI4MzI4ZmJiNDcxNTc2MzMyZjlm YjEgTW9uIFNlcCAxNyAwMDowMDowMCAyMDAxDQpGcm9tOiBNaW5jaGFuIEtpbSA8bWluY2hhbi5r aW1AZ21haWwuY29tPg0KRGF0ZTogV2VkLCAyMiBTZXAgMjAxMCAxMzowNTowMSAtMDcwMA0KU3Vi amVjdDogW1BBVENIXSB2bXNjYW46IGNoZWNrIGFsbF91bnJlY2xhaW1hYmxlIGluIGRpcmVjdCBy ZWNsYWltIHBhdGgNCg0KICBCdXQgbGF0ZXIgYmVsb3cgcGF0Y2ggY2hhbmdlZCB0aGUgbG9naWMs IGFuZCBjaGVja2VkIHRoZSBmbGFnIG9vbV9raWxsZXJfZGlzYWJsZSB3aGljaCBzZWVtcyBvbmx5 IGJlIHNldCB3aGVuIGhpYmVybmF0ZSwgc28gbXkgaXNzdWUgYXBwZWFyZWQuDQoNCkZyb20gOTI5 YmVhN2M3MTQyMjBmYzc2Y2UzZjc1YmVmOTA1NjQ3N2MyOGU3NCBNb24gU2VwIDE3IDAwOjAwOjAw IDIwMDENCkZyb206IEtPU0FLSSBNb3RvaGlybyA8a29zYWtpLm1vdG9oaXJvQGpwLmZ1aml0c3Uu Y29tPg0KRGF0ZTogVGh1LCAxNCBBcHIgMjAxMSAxNToyMjoxMiAtMDcwMA0KU3ViamVjdDogW1BB VENIXSB2bXNjYW46IGFsbF91bnJlY2xhaW1hYmxlKCkgdXNlIHpvbmUtPmFsbF91bnJlY2xhaW1h YmxlIGFzIGEgbmFtZQ0KQEAgLTIwMDYsMTMgKzIwMDIsMTEgQEAgc3RhdGljIGJvb2wgYWxsX3Vu cmVjbGFpbWFibGUoc3RydWN0IHpvbmVsaXN0ICp6b25lbGlzdCwNCiAgICAgICAgICAgICAgICAg ICAgICAgIGNvbnRpbnVlOw0KICAgICAgICAgICAgICAgIGlmICghY3B1c2V0X3pvbmVfYWxsb3dl ZF9oYXJkd2FsbCh6b25lLCBHRlBfS0VSTkVMKSkNCiAgICAgICAgICAgICAgICAgICAgICAgIGNv bnRpbnVlOw0KLSAgICAgICAgICAgICAgIGlmICh6b25lX3JlY2xhaW1hYmxlKHpvbmUpKSB7DQot ICAgICAgICAgICAgICAgICAgICAgICBhbGxfdW5yZWNsYWltYWJsZSA9IGZhbHNlOw0KLSAgICAg ICAgICAgICAgICAgICAgICAgYnJlYWs7DQotICAgICAgICAgICAgICAgfQ0KKyAgICAgICAgICAg ICAgIGlmICghem9uZS0+YWxsX3VucmVjbGFpbWFibGUpDQorICAgICAgICAgICAgICAgICAgICAg ICByZXR1cm4gZmFsc2U7DQogICAgICAgIH0NCg0KLSAgICAgICByZXR1cm4gYWxsX3VucmVjbGFp bWFibGU7DQorICAgICAgIHJldHVybiB0cnVlOw0KIH0NCg0KIC8qDQpAQCAtMjEwOCw2ICsyMTAy LDE0IEBAIG91dDoNCiAgICAgICAgaWYgKHNjLT5ucl9yZWNsYWltZWQpDQogICAgICAgICAgICAg ICAgcmV0dXJuIHNjLT5ucl9yZWNsYWltZWQ7DQoNCisgICAgICAgLyoNCisgICAgICAgICogQXMg aGliZXJuYXRpb24gaXMgZ29pbmcgb24sIGtzd2FwZCBpcyBmcmVlemVkIHNvIHRoYXQgaXQgY2Fu J3QgbWFyaw0KKyAgICAgICAgKiB0aGUgem9uZSBpbnRvIGFsbF91bnJlY2xhaW1hYmxlLiBUaHVz IGJ5cGFzc2luZyBhbGxfdW5yZWNsYWltYWJsZQ0KKyAgICAgICAgKiBjaGVjay4NCisgICAgICAg ICovDQorICAgICAgIGlmIChvb21fa2lsbGVyX2Rpc2FibGVkKQ0KKyAgICAgICAgICAgICAgIHJl dHVybiAwOw0KVGhhbmtzIQ0KDQpCZXN0IFJlZ2FyZHMNCkxpc2EgRHUNCg0KDQotLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogQm9iIExpdSBbbWFpbHRvOmxsaXViYm9AZ21haWwuY29t XSANClNlbnQ6IDIwMTPlubQ35pyIMjTml6UgMTE6MzkNClRvOiBMaXNhIER1DQpDYzogbGludXgt bW1Aa3ZhY2sub3JnOyBDaHJpc3RvcGggTGFtZXRlcjsgTWVsIEdvcm1hbg0KU3ViamVjdDogUmU6 IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNsYWltPw0KDQpPbiBXZWQsIEp1bCAyNCwg MjAxMyBhdCAxMDoyMyBBTSwgTGlzYSBEdSA8Y2xkdUBtYXJ2ZWxsLmNvbT4gd3JvdGU6DQo+IERl YXIgQm9iDQo+ICAgIEFsc28gZnJvbSBteSBjaGVjayBiZWZvcmUga3N3YXBkIHNsZWVwLCB0aG91 Z2ggbnJfc2xhYiA9IDAgYnV0IHpvbmVfcmVjbGFpbWFibGUoem9uZSkgcmV0dXJucyB0cnVlLCBz byB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBjYW4ndCBiZSBjaGFuZ2VkIHRvIDE7IFNvIGV2ZW4g d2hlbiBjaGFuZ2UgdGhlIG5yX3NsYWIgdG8gc2MtPm5yX3JlY2xhaW1lZCwgaXQgY2FuJ3QgaGVs cC4NCj4NCg0KVGhlbiB0aGUgb3RoZXIgZml4IG1pZ2h0IGJlIHNldCB6b25lLT5hbGxfdW5yZWNs YWltYWJsZSBpbiBkaXJlY3QNCnJlY2xhaW0gcGF0aCBhbHNvLCBsaWtlOg0KDQpAQCAtMjI3OCw2 ICsyMjc4LDggQEAgc3RhdGljIGJvb2wgc2hyaW5rX3pvbmVzKHN0cnVjdCB6b25lbGlzdA0KKnpv bmVsaXN0LCBzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykNCiAgICAgICAgICAgICAgICB9DQoNCiAg ICAgICAgICAgICAgICBzaHJpbmtfem9uZSh6b25lLCBzYyk7DQorICAgICAgICAgICAgICAgaWYg KHNjLT5ucl9yZWNsYWltZWQgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCisgICAg ICAgICAgICAgICAgICAgICAgIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlID0gMTsNCiAgICAgICAg fQ0KDQo+IFRoYW5rcyENCj4NCj4gQmVzdCBSZWdhcmRzDQo+IExpc2EgRHUNCj4NCj4NCj4gLS0t LS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogTGlzYSBEdQ0KPiBTZW50OiAyMDEz5bm0 N+aciDI05pelIDk6MzENCj4gVG86ICdCb2IgTGl1Jw0KPiBDYzogbGludXgtbW1Aa3ZhY2sub3Jn OyBDaHJpc3RvcGggTGFtZXRlcjsgTWVsIEdvcm1hbg0KPiBTdWJqZWN0OiBSRTogUG9zc2libGUg ZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0/DQo+DQo+IERlYXIgQm9iDQo+ICAgICBUaGFuayB5 b3Ugc28gbXVjaCBmb3IgdGhlIGNhcmVmdWwgcmV2aWV3LCBZZXMsIGl0J3MgYSB0eXBvLCBJIG1l YW4gem9uZS0+YWxsX3VucmVjbGFpbWFibGUgPSAwLg0KPiAgICAgWW91IG1lbnRpb25lZCBhZGQg dGhlIGNoZWNrIGluIGtzd2FwZF9zaHJpbmtfem9uZSgpLCBzb3JyeSB0aGF0IEkgZGlkbid0IGZp bmQgdGhpcyBmdW5jdGlvbiBpbiBrZXJuZWwzLjQgb3Iga2VybmVsMy45Lg0KPiAgICAgSXMgdGhp cyBmdW5jdGlvbiBjYWxsZWQgaW4gZGlyZWN0X3JlY2xhaW0/DQo+ICAgICBBcyBJIG1lbnRpb25l ZCB0aGlzIGlzc3VlIGhhcHBlbmVkIGFmdGVyIGtzd2FwZCB0aHJlYWQgc2xlZXAsIGlmIGl0IG9u bHkgY2FsbGVkIGluIGtzd2FwZCwgdGhlbiBJIHRoaW5rIGl0IGNhbid0IGhlbHAuDQo+DQo+IFRo YW5rcyENCj4NCj4gQmVzdCBSZWdhcmRzDQo+IExpc2EgRHUNCj4NCj4NCj4gLS0tLS1PcmlnaW5h bCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogQm9iIExpdSBbbWFpbHRvOmxsaXViYm9AZ21haWwuY29t XQ0KPiBTZW50OiAyMDEz5bm0N+aciDI05pelIDk6MTgNCj4gVG86IExpc2EgRHUNCj4gQ2M6IGxp bnV4LW1tQGt2YWNrLm9yZzsgQ2hyaXN0b3BoIExhbWV0ZXI7IE1lbCBHb3JtYW4NCj4gU3ViamVj dDogUmU6IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNsYWltPw0KPg0KPiBPbiBUdWUs IEp1bCAyMywgMjAxMyBhdCAxMjo1OCBQTSwgTGlzYSBEdSA8Y2xkdUBtYXJ2ZWxsLmNvbT4gd3Jv dGU6DQo+PiBEZWFyIFNpcjoNCj4+DQo+PiBDdXJyZW50bHkgSSBtZXQgYSBwb3NzaWJsZSBkZWFk bG9vcCBpbiBkaXJlY3QgcmVjbGFpbS4gQWZ0ZXIgcnVuIHBsZW50eSBvZg0KPj4gdGhlIGFwcGxp Y2F0aW9uLCBzeXN0ZW0gcnVuIGludG8gYSBzdGF0dXMgdGhhdCBzeXN0ZW0gbWVtb3J5IGlzIHZl cnkNCj4+IGZyYWdtZW50aXplZC4gTGlrZSBvbmx5IG9yZGVyLTAgYW5kIG9yZGVyLTEgbWVtb3J5 IGxlZnQuDQo+Pg0KPj4gVGhlbiBvbmUgcHJvY2VzcyByZXF1aXJlZCBhIG9yZGVyLTIgYnVmZmVy IGJ1dCBpdCBlbnRlciBhbiBlbmRsZXNzIGRpcmVjdA0KPj4gcmVjbGFpbS4gRnJvbSBteSB0cmFj ZSBsb2csIEkgY2FuIHNlZSB0aGlzIGxvb3AgYWxyZWFkeSBvdmVyIDIwMCwwMDAgdGltZXMuDQo+ PiBLc3dhcGQgd2FzIGZpcnN0IHdha2UgdXAgYW5kIHRoZW4gZ28gYmFjayB0byBzbGVlcCBhcyBp dCBjYW5ub3QgcmViYWxhbmNlDQo+PiB0aGlzIG9yZGVy4oCZcyBtZW1vcnkuIEJ1dCB6b25lLT5h bGxfdW5yZWNsYWltYWJsZSByZW1haW5zIDEuDQo+Pg0KPj4gVGhvdWdoIGRpcmVjdF9yZWNsYWlt IGV2ZXJ5IHRpbWUgcmV0dXJucyBubyBwYWdlcywgYnV0IGFzDQo+PiB6b25lLT5hbGxfdW5yZWNs YWltYWJsZSA9IDEsIHNvIGl0IGxvb3AgYWdhaW4gYW5kIGFnYWluLiBFdmVuIHdoZW4NCj4+IHpv bmUtPnBhZ2VzX3NjYW5uZWQgYWxzbyBiZWNvbWVzIHZlcnkgbGFyZ2UuIEl0IHdpbGwgYmxvY2sg dGhlIHByb2Nlc3MgZm9yDQo+PiBsb25nIHRpbWUsIHVudGlsIHNvbWUgd2F0Y2hkb2cgdGhyZWFk IGRldGVjdCB0aGlzIGFuZCBraWxsIHRoaXMgcHJvY2Vzcy4NCj4+IFRob3VnaCBpdOKAmXMgaW4g X19hbGxvY19wYWdlc19zbG93cGF0aCwgYnV0IGl04oCZcyB0b28gc2xvdyByaWdodD8gTWF5YmUg Y29zdA0KPj4gb3ZlciA1MCBzZWNvbmRzIG9yIGV2ZW4gbW9yZS4NCj4NCj4gWW91IG11c3QgYmUg bWVhbiB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSA9IDA/DQo+DQo+Pg0KPj4gSSB0aGluayBpdOKA mXMgbm90IGFzIGV4cGVjdGVkIHJpZ2h0PyAgQ2FuIHdlIGFsc28gYWRkIGJlbG93IGNoZWNrIGlu IHRoZQ0KPj4gZnVuY3Rpb24gYWxsX3VucmVjbGFpbWFibGUoKSB0byB0ZXJtaW5hdGUgdGhpcyBs b29wPw0KPj4NCj4+DQo+Pg0KPj4gQEAgLTIzNTUsNiArMjM1NSw4IEBAIHN0YXRpYyBib29sIGFs bF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6b25lbGlzdA0KPj4gKnpvbmVsaXN0LA0KPj4NCj4+ICAg ICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPj4NCj4+ICAgICAgICAgICAgICAgICBp ZiAoIXpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KPj4NCj4+ICAgICAgICAgICAgICAgICAgICAg ICAgIHJldHVybiBmYWxzZTsNCj4+DQo+PiArICAgICAgICAgICAgICAgaWYgKHNjLT5ucl9yZWNs YWltZWQgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCj4+DQo+PiArICAgICAgICAg ICAgICAgICAgICAgICByZXR1cm4gdHJ1ZTsNCj4+DQo+DQo+IEhvdyBhYm91dCByZXBsYWNlIHRo ZSBjaGVja2luZyBpbiBrc3dhcGRfc2hyaW5rX3pvbmUoKT8NCj4NCj4gQEAgLTI4MjQsNyArMjgy NCw3IEBAIHN0YXRpYyBib29sIGtzd2FwZF9zaHJpbmtfem9uZShzdHJ1Y3Qgem9uZSAqem9uZSwN Cj4gICAgICAgICAvKiBBY2NvdW50IGZvciB0aGUgbnVtYmVyIG9mIHBhZ2VzIGF0dGVtcHRlZCB0 byByZWNsYWltICovDQo+ICAgICAgICAgKm5yX2F0dGVtcHRlZCArPSBzYy0+bnJfdG9fcmVjbGFp bTsNCj4NCj4gLSAgICAgICBpZiAobnJfc2xhYiA9PSAwICYmICF6b25lX3JlY2xhaW1hYmxlKHpv bmUpKQ0KPiArICAgICAgIGlmIChzYy0+bnJfcmVjbGFpbWVkID09IDAgJiYgIXpvbmVfcmVjbGFp bWFibGUoem9uZSkpDQo+ICAgICAgICAgICAgICAgICB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSA9 IDE7DQo+DQo+ICAgICAgICAgem9uZV9jbGVhcl9mbGFnKHpvbmUsIFpPTkVfV1JJVEVCQUNLKTsN Cj4NCj4NCj4gSSB0aGluayB0aGUgY3VycmVudCBjaGVjayBpcyB3cm9uZywgcmVjbGFpbWVkIGEg c2xhYiBkb2Vzbid0IG1lYW4NCj4gcmVjbGFpbWVkIGEgcGFnZS4NCj4NCj4gLS0NCj4gUmVnYXJk cywNCj4gLS1Cb2INCg0KDQoNCi0tIA0KUmVnYXJkcywNCi0tQm9iDQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx183.postini.com [74.125.245.183]) by kanga.kvack.org (Postfix) with SMTP id F21EA6B0031 for ; Thu, 25 Jul 2013 14:14:22 -0400 (EDT) Received: by mail-oa0-f52.google.com with SMTP id g12so5030519oah.25 for ; Thu, 25 Jul 2013 11:14:22 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> From: KOSAKI Motohiro Date: Thu, 25 Jul 2013 14:14:01 -0400 Message-ID: Subject: Re: Possible deadloop in direct reclaim? Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: Lisa Du , "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman > How about replace the checking in kswapd_shrink_zone()? > > @@ -2824,7 +2824,7 @@ static bool kswapd_shrink_zone(struct zone *zone, > /* Account for the number of pages attempted to reclaim */ > *nr_attempted += sc->nr_to_reclaim; > > - if (nr_slab == 0 && !zone_reclaimable(zone)) > + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) > zone->all_unreclaimable = 1; > > zone_clear_flag(zone, ZONE_WRITEBACK); > > > I think the current check is wrong, reclaimed a slab doesn't mean > reclaimed a page. The code is correct, at least, it works as intentional. page reclaim status is checked by zone_reclaimable() and slab shrinking status is checked by nr_slab. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx180.postini.com [74.125.245.180]) by kanga.kvack.org (Postfix) with SMTP id BBB176B0031 for ; Thu, 25 Jul 2013 14:19:41 -0400 (EDT) Received: by mail-oa0-f51.google.com with SMTP id i4so5036621oah.24 for ; Thu, 25 Jul 2013 11:19:40 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> From: KOSAKI Motohiro Date: Thu, 25 Jul 2013 14:19:20 -0400 Message-ID: Subject: Re: Possible deadloop in direct reclaim? Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu On Tue, Jul 23, 2013 at 9:21 PM, Lisa Du wrote: > Dear Christoph > Thanks a lot for your comment. When this issue happen I just trigger a kernel panic and got the kdump. > From the kdump, I got the global variable pg_data_t congit_page_data. From this structure, I can see in normal zone, only order-0's nr_free = 18442, order-1's nr_free = 367, all the other order's nr_free is 0. Don't you use compaction? Of if use, please get a log by tracepoints. We need to know why it doesn't work. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx113.postini.com [74.125.245.113]) by kanga.kvack.org (Postfix) with SMTP id ED9CA6B0031 for ; Thu, 25 Jul 2013 21:11:34 -0400 (EDT) From: Lisa Du Date: Thu, 25 Jul 2013 18:11:08 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu RGVhciBLT1NBS0kNCiAgIEluIG15IHRlc3QsIEkgZGlkbid0IHNldCBjb21wYWN0aW9uLiBNYXli ZSBjb21wYWN0aW9uIGlzIGhlbHBmdWwgdG8gYXZvaWQgdGhpcyBpc3N1ZS4gSSBjYW4gaGF2ZSB0 cnkgbGF0ZXIuDQogICBJbiBteSBtaW5kIENPTkZJR19DT01QQUNUSU9OIGlzIGFuIG9wdGlvbmFs IGNvbmZpZ3VyYXRpb24gcmlnaHQ/IA0KICAgSWYgd2UgZG9uJ3QgdXNlLCBhbmQgbWV0IHN1Y2gg YW4gaXNzdWUsIGhvdyBzaG91bGQgd2UgZGVhbCB3aXRoIHN1Y2ggaW5maW5pdGUgbG9vcD8NCg0K ICAgSSBtYWRlIGEgY2hhbmdlIGluIGFsbF9yZWNsYWltYWJsZSgpIGZ1bmN0aW9uLCBwYXNzZWQg b3Zlcm5pZ2h0IHRlc3RzLCBwbGVhc2UgaGVscCByZXZpZXcsIHRoYW5rcyBpbiBhZHZhbmNlIQ0K QEAgLTIzNTMsNyArMjM1Myw5IEBAIHN0YXRpYyBib29sIGFsbF91bnJlY2xhaW1hYmxlKHN0cnVj dCB6b25lbGlzdCAqem9uZWxpc3QsDQogICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsN CiAgICAgICAgICAgICAgICBpZiAoIWNwdXNldF96b25lX2FsbG93ZWRfaGFyZHdhbGwoem9uZSwg R0ZQX0tFUk5FTCkpDQogICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsNCi0gICAgICAg ICAgICAgICBpZiAoIXpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KKyAgICAgICAgICAgICAgIGlm ICh6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkNCisgICAgICAgICAgICAgICAgICAgICAgIGNvbnRp bnVlOw0KKyAgICAgICAgICAgICAgIGlmICh6b25lX3JlY2xhaW1hYmxlKHpvbmUpKQ0KICAgICAg ICAgICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOw0KICAgICAgICB9DQoNClRoYW5rcyENCg0K QmVzdCBSZWdhcmRzDQpMaXNhIER1DQoNCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZy b206IEtPU0FLSSBNb3RvaGlybyBbbWFpbHRvOmtvc2FraS5tb3RvaGlyb0BnbWFpbC5jb21dIA0K U2VudDogMjAxM8TqN9TCMjbI1SAyOjE5DQpUbzogTGlzYSBEdQ0KQ2M6IENocmlzdG9waCBMYW1l dGVyOyBsaW51eC1tbUBrdmFjay5vcmc7IE1lbCBHb3JtYW47IEJvYiBMaXUNClN1YmplY3Q6IFJl OiBQb3NzaWJsZSBkZWFkbG9vcCBpbiBkaXJlY3QgcmVjbGFpbT8NCg0KT24gVHVlLCBKdWwgMjMs IDIwMTMgYXQgOToyMSBQTSwgTGlzYSBEdSA8Y2xkdUBtYXJ2ZWxsLmNvbT4gd3JvdGU6DQo+IERl YXIgQ2hyaXN0b3BoDQo+ICAgIFRoYW5rcyBhIGxvdCBmb3IgeW91ciBjb21tZW50LiBXaGVuIHRo aXMgaXNzdWUgaGFwcGVuIEkganVzdCB0cmlnZ2VyIGEga2VybmVsIHBhbmljIGFuZCBnb3QgdGhl IGtkdW1wLg0KPiBGcm9tIHRoZSBrZHVtcCwgSSBnb3QgdGhlIGdsb2JhbCB2YXJpYWJsZSBwZ19k YXRhX3QgY29uZ2l0X3BhZ2VfZGF0YS4gRnJvbSB0aGlzIHN0cnVjdHVyZSwgSSBjYW4gc2VlIGlu IG5vcm1hbCB6b25lLCBvbmx5IG9yZGVyLTAncyBucl9mcmVlID0gMTg0NDIsIG9yZGVyLTEncyBu cl9mcmVlID0gMzY3LCBhbGwgdGhlIG90aGVyIG9yZGVyJ3MgbnJfZnJlZSBpcyAwLg0KDQpEb24n dCB5b3UgdXNlIGNvbXBhY3Rpb24/IE9mIGlmIHVzZSwgcGxlYXNlIGdldCBhIGxvZyBieSB0cmFj ZXBvaW50cy4NCldlIG5lZWQgdG8ga25vdyB3aHkgaXQgZG9lc24ndCB3b3JrLg0K -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx202.postini.com [74.125.245.202]) by kanga.kvack.org (Postfix) with SMTP id B86246B0031 for ; Thu, 25 Jul 2013 21:22:15 -0400 (EDT) Received: by mail-ve0-f177.google.com with SMTP id cz10so906658veb.22 for ; Thu, 25 Jul 2013 18:22:14 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Date: Fri, 26 Jul 2013 09:22:14 +0800 Message-ID: Subject: Re: Possible deadloop in direct reclaim? From: Bob Liu Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Lisa Du , "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman Hi Kosaki, On Fri, Jul 26, 2013 at 2:14 AM, KOSAKI Motohiro wrote: >> How about replace the checking in kswapd_shrink_zone()? >> >> @@ -2824,7 +2824,7 @@ static bool kswapd_shrink_zone(struct zone *zone, >> /* Account for the number of pages attempted to reclaim */ >> *nr_attempted += sc->nr_to_reclaim; >> >> - if (nr_slab == 0 && !zone_reclaimable(zone)) >> + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) >> zone->all_unreclaimable = 1; >> >> zone_clear_flag(zone, ZONE_WRITEBACK); >> >> >> I think the current check is wrong, reclaimed a slab doesn't mean >> reclaimed a page. > > The code is correct, at least, it works as intentional. page reclaim > status is checked by zone_reclaimable() and slab shrinking status is > checked by nr_slab. I'm afraid in some special cases, nr_slab = 1 or any small number which means we reclaimed some slab objects. Then we don't set zone->all_unreclaimeabled =1. But even though we reclaimed some slab objects, there may be no pages freed. Because one page may contain several objects. If we reclaimed some slab objects but without actual pages, we need to set zone->all_unreclaimeabled=1! So I think we should check sc->nr_reclaimed == 0 instead of nr_slab == 0. -- Regards, --Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx164.postini.com [74.125.245.164]) by kanga.kvack.org (Postfix) with SMTP id 384C56B0031 for ; Sun, 28 Jul 2013 21:32:51 -0400 (EDT) From: Lisa Du Date: Sun, 28 Jul 2013 18:32:46 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B6301CCE2@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du , KOSAKI Motohiro Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu RGVhciBLb3Nha2kNCiAgIERvIHlvdSBoYXZlIHRoZSBjaGFuY2UgdG8gcmV2aWV3IG15IGNoYW5n ZSBpbiB0aGUgZnVuY3Rpb24gYWxsX3VucmVjbGFpbWFibGUoKT8NCkBAIC0yMzUzLDcgKzIzNTMs OSBAQCBzdGF0aWMgYm9vbCBhbGxfdW5yZWNsYWltYWJsZShzdHJ1Y3Qgem9uZWxpc3QgKnpvbmVs aXN0LA0KICAgICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQogICAgICAgICAgICAgICAg aWYgKCFjcHVzZXRfem9uZV9hbGxvd2VkX2hhcmR3YWxsKHpvbmUsIEdGUF9LRVJORUwpKQ0KICAg ICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQotICAgICAgICAgICAgICAgaWYgKCF6b25l LT5hbGxfdW5yZWNsYWltYWJsZSkNCisgICAgICAgICAgICAgICBpZiAoem9uZS0+YWxsX3VucmVj bGFpbWFibGUpDQorICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsNCisgICAgICAgICAg ICAgICBpZiAoem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCiAgICAgICAgICAgICAgICAgICAgICAg IHJldHVybiBmYWxzZTsNCiAgICAgICAgfQ0KICAgSW4gbXkgdGVzdCwgaXQgaGVscGVkIHRvIGF2 b2lkIHRoZSBpbmZpbml0ZSBsb29wIGluIGRpcmVjdF9yZWNsYWltIHBhdGgsIGFuZCBJIHRoaW5r IGl0IHNob3VsZCBhbHNvIGF2b2lkIHRoZSBrZXJuZWwgaGFuZ2luZyB1cCBpc3N1ZSB5b3UgbWV0 IGluIHRoZSBjb21taXQ6IDkyOWJlYTdjNzE0MjIwLg0KICAgSW4gYSB3b3JkLCBJIHRoaW5rIG5l aXRoZXIgY2hlY2sgdGhlIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlIG5vciB6b25lX3JlY2xhaW1h YmxlKCkgaXMgZW5vdWdoIGluIHRoZSBmdW5jdGlvbiBhbGxfdW5yZWNsYWltYWJsZSgpLCBzbyBz aGFsbCB3ZSBjaGVjayBib3RoIHRvIGNvbmZpcm0gaWYgYSB6b25lIGlzIGFsbF91bnJlY2xhaW1h YmxlPw0KDQpUaGFua3MhDQoNCkJlc3QgUmVnYXJkcw0KTGlzYSBEdQ0KDQotLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KRnJvbTogTGlzYSBEdSANClNlbnQ6IDIwMTPE6jfUwjI2yNUgOToxMQ0K VG86ICdLT1NBS0kgTW90b2hpcm8nDQpDYzogQ2hyaXN0b3BoIExhbWV0ZXI7IGxpbnV4LW1tQGt2 YWNrLm9yZzsgTWVsIEdvcm1hbjsgQm9iIExpdQ0KU3ViamVjdDogUkU6IFBvc3NpYmxlIGRlYWRs b29wIGluIGRpcmVjdCByZWNsYWltPw0KDQpEZWFyIEtPU0FLSQ0KICAgSW4gbXkgdGVzdCwgSSBk aWRuJ3Qgc2V0IGNvbXBhY3Rpb24uIE1heWJlIGNvbXBhY3Rpb24gaXMgaGVscGZ1bCB0byBhdm9p ZCB0aGlzIGlzc3VlLiBJIGNhbiBoYXZlIHRyeSBsYXRlci4NCiAgIEluIG15IG1pbmQgQ09ORklH X0NPTVBBQ1RJT04gaXMgYW4gb3B0aW9uYWwgY29uZmlndXJhdGlvbiByaWdodD8gDQogICBJZiB3 ZSBkb24ndCB1c2UsIGFuZCBtZXQgc3VjaCBhbiBpc3N1ZSwgaG93IHNob3VsZCB3ZSBkZWFsIHdp dGggc3VjaCBpbmZpbml0ZSBsb29wPw0KDQogICBJIG1hZGUgYSBjaGFuZ2UgaW4gYWxsX3JlY2xh aW1hYmxlKCkgZnVuY3Rpb24sIHBhc3NlZCBvdmVybmlnaHQgdGVzdHMsIHBsZWFzZSBoZWxwIHJl dmlldywgdGhhbmtzIGluIGFkdmFuY2UhDQpAQCAtMjM1Myw3ICsyMzUzLDkgQEAgc3RhdGljIGJv b2wgYWxsX3VucmVjbGFpbWFibGUoc3RydWN0IHpvbmVsaXN0ICp6b25lbGlzdCwNCiAgICAgICAg ICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KICAgICAgICAgICAgICAgIGlmICghY3B1c2V0X3pv bmVfYWxsb3dlZF9oYXJkd2FsbCh6b25lLCBHRlBfS0VSTkVMKSkNCiAgICAgICAgICAgICAgICAg ICAgICAgIGNvbnRpbnVlOw0KLSAgICAgICAgICAgICAgIGlmICghem9uZS0+YWxsX3VucmVjbGFp bWFibGUpDQorICAgICAgICAgICAgICAgaWYgKHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KKyAg ICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQorICAgICAgICAgICAgICAgaWYgKHpvbmVf cmVjbGFpbWFibGUoem9uZSkpDQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gZmFsc2U7 DQogICAgICAgIH0NCg0KVGhhbmtzIQ0KDQpCZXN0IFJlZ2FyZHMNCkxpc2EgRHUNCg0KDQotLS0t LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogS09TQUtJIE1vdG9oaXJvIFttYWlsdG86a29z YWtpLm1vdG9oaXJvQGdtYWlsLmNvbV0gDQpTZW50OiAyMDEzxOo31MIyNsjVIDI6MTkNClRvOiBM aXNhIER1DQpDYzogQ2hyaXN0b3BoIExhbWV0ZXI7IGxpbnV4LW1tQGt2YWNrLm9yZzsgTWVsIEdv cm1hbjsgQm9iIExpdQ0KU3ViamVjdDogUmU6IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCBy ZWNsYWltPw0KDQpPbiBUdWUsIEp1bCAyMywgMjAxMyBhdCA5OjIxIFBNLCBMaXNhIER1IDxjbGR1 QG1hcnZlbGwuY29tPiB3cm90ZToNCj4gRGVhciBDaHJpc3RvcGgNCj4gICAgVGhhbmtzIGEgbG90 IGZvciB5b3VyIGNvbW1lbnQuIFdoZW4gdGhpcyBpc3N1ZSBoYXBwZW4gSSBqdXN0IHRyaWdnZXIg YSBrZXJuZWwgcGFuaWMgYW5kIGdvdCB0aGUga2R1bXAuDQo+IEZyb20gdGhlIGtkdW1wLCBJIGdv dCB0aGUgZ2xvYmFsIHZhcmlhYmxlIHBnX2RhdGFfdCBjb25naXRfcGFnZV9kYXRhLiBGcm9tIHRo aXMgc3RydWN0dXJlLCBJIGNhbiBzZWUgaW4gbm9ybWFsIHpvbmUsIG9ubHkgb3JkZXItMCdzIG5y X2ZyZWUgPSAxODQ0Miwgb3JkZXItMSdzIG5yX2ZyZWUgPSAzNjcsIGFsbCB0aGUgb3RoZXIgb3Jk ZXIncyBucl9mcmVlIGlzIDAuDQoNCkRvbid0IHlvdSB1c2UgY29tcGFjdGlvbj8gT2YgaWYgdXNl LCBwbGVhc2UgZ2V0IGEgbG9nIGJ5IHRyYWNlcG9pbnRzLg0KV2UgbmVlZCB0byBrbm93IHdoeSBp dCBkb2Vzbid0IHdvcmsuDQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx150.postini.com [74.125.245.150]) by kanga.kvack.org (Postfix) with SMTP id 4DDE46B0031 for ; Mon, 29 Jul 2013 12:43:36 -0400 (EDT) Received: by mail-vc0-f171.google.com with SMTP id ij15so2252996vcb.2 for ; Mon, 29 Jul 2013 09:43:35 -0700 (PDT) Message-ID: <51F69BD7.2060407@gmail.com> Date: Mon, 29 Jul 2013 12:44:07 -0400 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: Possible deadloop in direct reclaim? References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: KOSAKI Motohiro , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu (7/25/13 9:11 PM), Lisa Du wrote: > Dear KOSAKI > In my test, I didn't set compaction. Maybe compaction is helpful to avoid this issue. I can have try later. > In my mind CONFIG_COMPACTION is an optional configuration right? Right. But if you don't set it, application must NOT use >1 order allocations. It doesn't work and it is expected result. That's your application mistake. > If we don't use, and met such an issue, how should we deal with such infinite loop? > > I made a change in all_reclaimable() function, passed overnight tests, please help review, thanks in advance! > @@ -2353,7 +2353,9 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > continue; > - if (!zone->all_unreclaimable) > + if (zone->all_unreclaimable) > + continue; > + if (zone_reclaimable(zone)) > return false; Please tell me why you chaned here. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx161.postini.com [74.125.245.161]) by kanga.kvack.org (Postfix) with SMTP id E6EDE6B0031 for ; Mon, 29 Jul 2013 12:45:44 -0400 (EDT) Received: by mail-vb0-f54.google.com with SMTP id q14so894377vbe.27 for ; Mon, 29 Jul 2013 09:45:43 -0700 (PDT) Message-ID: <51F69C59.10307@gmail.com> Date: Mon, 29 Jul 2013 12:46:17 -0400 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: Possible deadloop in direct reclaim? References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com>

In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: KOSAKI Motohiro , Lisa Du , "linux-mm@kvack.org" , Christoph Lameter , Mel Gorman (7/25/13 9:22 PM), Bob Liu wrote: > Hi Kosaki, > > On Fri, Jul 26, 2013 at 2:14 AM, KOSAKI Motohiro > wrote: >>> How about replace the checking in kswapd_shrink_zone()? >>> >>> @@ -2824,7 +2824,7 @@ static bool kswapd_shrink_zone(struct zone *zone, >>> /* Account for the number of pages attempted to reclaim */ >>> *nr_attempted += sc->nr_to_reclaim; >>> >>> - if (nr_slab == 0 && !zone_reclaimable(zone)) >>> + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) >>> zone->all_unreclaimable = 1; >>> >>> zone_clear_flag(zone, ZONE_WRITEBACK); >>> >>> >>> I think the current check is wrong, reclaimed a slab doesn't mean >>> reclaimed a page. >> >> The code is correct, at least, it works as intentional. page reclaim >> status is checked by zone_reclaimable() and slab shrinking status is >> checked by nr_slab. > > I'm afraid in some special cases, nr_slab = 1 or any small number > which means we reclaimed some slab objects. > Then we don't set zone->all_unreclaimeabled =1. > > But even though we reclaimed some slab objects, there may be no pages freed. > Because one page may contain several objects. Right. This is a limitation of current slab shrinker's implementation. We are welcome you contribution this area. > If we reclaimed some slab objects but without actual pages, we need to > set zone->all_unreclaimeabled=1! > So I think we should check sc->nr_reclaimed == 0 instead of nr_slab == 0. sc->nr_reclaimed doesn't check how much pages freed from slab. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx112.postini.com [74.125.245.112]) by kanga.kvack.org (Postfix) with SMTP id A2FB06B0031 for ; Mon, 29 Jul 2013 21:27:31 -0400 (EDT) From: Lisa Du Date: Mon, 29 Jul 2013 18:27:26 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B6301D0B2@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> In-Reply-To: <51F69BD7.2060407@gmail.com> Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu , Neil Zhang LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEtPU0FLSSBNb3RvaGlybyBbbWFpbHRv Omtvc2FraS5tb3RvaGlyb0BnbWFpbC5jb21dIA0KU2VudDogMjAxM8TqN9TCMzDI1SAwOjQ0DQpU bzogTGlzYSBEdQ0KQ2M6IEtPU0FLSSBNb3RvaGlybzsgQ2hyaXN0b3BoIExhbWV0ZXI7IGxpbnV4 LW1tQGt2YWNrLm9yZzsgTWVsIEdvcm1hbjsgQm9iIExpdQ0KU3ViamVjdDogUmU6IFBvc3NpYmxl IGRlYWRsb29wIGluIGRpcmVjdCByZWNsYWltPw0KDQooNy8yNS8xMyA5OjExIFBNKSwgTGlzYSBE dSB3cm90ZToNCj4gRGVhciBLT1NBS0kNCj4gICAgIEluIG15IHRlc3QsIEkgZGlkbid0IHNldCBj b21wYWN0aW9uLiBNYXliZSBjb21wYWN0aW9uIGlzIGhlbHBmdWwgdG8gYXZvaWQgdGhpcyBpc3N1 ZS4gSSBjYW4gaGF2ZSB0cnkgbGF0ZXIuDQo+ICAgICBJbiBteSBtaW5kIENPTkZJR19DT01QQUNU SU9OIGlzIGFuIG9wdGlvbmFsIGNvbmZpZ3VyYXRpb24gcmlnaHQ/DQoNClJpZ2h0LiBCdXQgaWYg eW91IGRvbid0IHNldCBpdCwgYXBwbGljYXRpb24gbXVzdCBOT1QgdXNlID4xIG9yZGVyIGFsbG9j YXRpb25zLiBJdCBkb2Vzbid0IHdvcmsgYW5kIGl0IGlzIGV4cGVjdGVkDQpyZXN1bHQuDQpUaGF0 J3MgeW91ciBhcHBsaWNhdGlvbiBtaXN0YWtlLg0KRGVhciBLb3Nha2ksIEkgaGF2ZSB0d28gcXVl c3Rpb25zIG9uIHlvdXIgZXhwbGFuYXRpb246IGEpIHlvdSBzYWlkIGlmIGRvbid0IHNldCBDT05G SUdfQ09NUEFUSU9OLCBhcHBsaWNhdGlvbiBtdXN0IE5PVCB1c2UgPjEgb3JkZXIgYWxsb2NhdGlv bnMsIGlzIHRoZXJlIGFueSBkb2N1bWVudGF0aW9uIGZvciB0aGlzIHRoZW9yeT8gIGIpIE15IG9y ZGVyLTIgYWxsb2NhdGlvbiBub3QgY29tZXMgZnJvbSBhcHBsaWNhdGlvbiwgYnV0IGZyb20gZG9f Zm9yayB3aGljaCBpcyBpbiBrZXJuZWwgc3BhY2UsIGluIG15IG1pbmQgd2hlbiBhIHBhcmVudCBw cm9jZXNzIGZvcmtzIGEgY2hpbGQgcHJvY2VzcywgaXQgbmVlZCB0byBhbGxvY2F0ZSBhIG9yZGVy LTIgbWVtb3J5LCBpZiBhKSBpcyByaWdodCwgdGhlbiBDT05GSUdfQ09NUEFUSU9OIHNob3VsZCBi ZSBhIE1VU1QgY29uZmlndXJhdGlvbiBmb3IgbGludXgga2VybmVsIGJ1dCBub3Qgb3B0aW9uYWw/ IA0KPiAgICAgSWYgd2UgZG9uJ3QgdXNlLCBhbmQgbWV0IHN1Y2ggYW4gaXNzdWUsIGhvdyBzaG91 bGQgd2UgZGVhbCB3aXRoIHN1Y2ggaW5maW5pdGUgbG9vcD8NCj4gDQo+ICAgICBJIG1hZGUgYSBj aGFuZ2UgaW4gYWxsX3JlY2xhaW1hYmxlKCkgZnVuY3Rpb24sIHBhc3NlZCBvdmVybmlnaHQgdGVz dHMsIHBsZWFzZSBoZWxwIHJldmlldywgdGhhbmtzIGluIGFkdmFuY2UhDQo+IEBAIC0yMzUzLDcg KzIzNTMsOSBAQCBzdGF0aWMgYm9vbCBhbGxfdW5yZWNsYWltYWJsZShzdHJ1Y3Qgem9uZWxpc3Qg KnpvbmVsaXN0LA0KPiAgICAgICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQo+ICAgICAg ICAgICAgICAgICAgaWYgKCFjcHVzZXRfem9uZV9hbGxvd2VkX2hhcmR3YWxsKHpvbmUsIEdGUF9L RVJORUwpKQ0KPiAgICAgICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQo+IC0gICAgICAg ICAgICAgICBpZiAoIXpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KPiArICAgICAgICAgICAgICAg aWYgKHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KPiArICAgICAgICAgICAgICAgICAgICAgICBj b250aW51ZTsNCj4gKyAgICAgICAgICAgICAgIGlmICh6b25lX3JlY2xhaW1hYmxlKHpvbmUpKQ0K PiAgICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOw0KDQpQbGVhc2UgdGVsbCBt ZSB3aHkgeW91IGNoYW5lZCBoZXJlLg0KVGhlIG9yaWdpbmFsIGNoZWNrIGlzIG9uY2UgZm91bmQg em9uZS0+YWxsX3VucmVjbGFpbWFibGUgaXMgZmFsc2UsIGl0IHdpbGwgcmV0dXJuIGZhbHNlLCB0 aGVuIGl0IHdpbGwgc2V0IGRpZF9zb21lX3Byb2dyZXNzIG5vbi16ZXJvLg0KVGhlbiBhbm90aGVy IGxvb3Agb2YgZGlyZWN0X3JlY2xhaW1lZCBwZXJmb3JtZWQuIEJ1dCBJIHRoaW5rIHpvbmUtPmFs bF91bnJlY2xhaW1hYmxlIGlzIG5vdCBhbHdheXMgcmVsaWFibGUgc3VjaCBhcyBpbiBteSBjYXNl LCBrc3dhcGQgZ28gdG8gc2xlZXAgYW5kIG5vIG9uZSB3aWxsIGNoYW5nZSB0aGlzIGZsYWcuIFdl IHNob3VsZCBhbHNvIGNoZWNrIHpvbmVfcmVjbGFpbWFsYmUoem9uZSkgaWYgem9uZS0+YWxsX3Vu cmVjbGFpbWFsYmUgPSAwIHRvIGRvdWJsZSBjb25maXJtIGlmIGEgem9uZSBpcyByZWNsYWltYWJs ZTsNClRoaXMgY2hhbmdlIGFsc28gYXZvaWQgdGhlIGlzc3VlIHlvdSBkZXNjcmliZWQgaW4gYmVs b3cgY29tbWl0Og0KY29tbWl0IDkyOWJlYTdjNzE0MjIwZmM3NmNlM2Y3NWJlZjkwNTY0NzdjMjhl NzQNCkF1dGhvcjogS09TQUtJIE1vdG9oaXJvIDxrb3Nha2kubW90b2hpcm9AanAuZnVqaXRzdS5j b20+DQpEYXRlOiAgIFRodSBBcHIgMTQgMTU6MjI6MTIgMjAxMSAtMDcwMA0KDQogICAgdm1zY2Fu OiBhbGxfdW5yZWNsYWltYWJsZSgpIHVzZSB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBhcyBhIG5h bWUNCg== -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx194.postini.com [74.125.245.194]) by kanga.kvack.org (Postfix) with SMTP id 5A7C66B0031 for ; Wed, 31 Jul 2013 22:24:33 -0400 (EDT) From: Lisa Du Date: Wed, 31 Jul 2013 19:24:19 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> In-Reply-To: <51F69BD7.2060407@gmail.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu Dear Kosaki Would you please help to check my comment as below: >(7/25/13 9:11 PM), Lisa Du wrote: >> Dear KOSAKI >> In my test, I didn't set compaction. Maybe compaction is helpful to >avoid this issue. I can have try later. >> In my mind CONFIG_COMPACTION is an optional configuration >right? > >Right. But if you don't set it, application must NOT use >1 order allocati= ons. >It doesn't work and it is expected >result. >That's your application mistake. Dear Kosaki, I have two questions on your explanation: a) you said if don't set CONFIG_COMPATION, application must NOT use >1 orde= r allocations, is there any documentation for this theory? =20 b) My order-2 allocation not comes from application, but from do_fork which= is in kernel space, in my mind when a parent process forks a child process= , it need to allocate a order-2 memory, if a) is right, then CONFIG_COMPATI= ON should be a MUST configuration for linux kernel but not optional? > >> If we don't use, and met such an issue, how should we deal with >such infinite loop? >> >> I made a change in all_reclaimable() function, passed overnight test= s, >please help review, thanks in advance! >> @@ -2353,7 +2353,9 @@ static bool all_unreclaimable(struct zonelist >*zonelist, >> continue; >> if (!cpuset_zone_allowed_hardwall(zone, >GFP_KERNEL)) >> continue; >> - if (!zone->all_unreclaimable) >> + if (zone->all_unreclaimable) >> + continue; >> + if (zone_reclaimable(zone)) >> return false; > >Please tell me why you chaned here. The original check is once found zone->all_unreclaimable is false, it will = return false, then it will set did_some_progress non-zero. Then another loo= p of direct_reclaimed performed. But I think zone->all_unreclaimable is not= always reliable such as in my case, kswapd go to sleep and no one will cha= nge this flag. We should also check zone_reclaimalbe(zone) if zone->all_unr= eclaimalbe =3D 0 to double confirm if a zone is reclaimable; This change al= so avoid the issue you described in below commit: commit 929bea7c714220fc76ce3f75bef9056477c28e74 Author: KOSAKI Motohiro Date: Thu Apr 14 15:22:12 2011 -0700 vmscan: all_unreclaimable() use zone->all_unreclaimable as a name > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx164.postini.com [74.125.245.164]) by kanga.kvack.org (Postfix) with SMTP id CDEF26B0031 for ; Wed, 31 Jul 2013 22:45:23 -0400 (EDT) Received: by mail-ye0-f174.google.com with SMTP id q9so576291yen.33 for ; Wed, 31 Jul 2013 19:45:22 -0700 (PDT) Message-ID: <51F9CBC0.2020006@gmail.com> Date: Wed, 31 Jul 2013 22:45:20 -0400 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: Possible deadloop in direct reclaim? References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: KOSAKI Motohiro , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu (7/31/13 10:24 PM), Lisa Du wrote: > Dear Kosaki > Would you please help to check my comment as below: >> (7/25/13 9:11 PM), Lisa Du wrote: >>> Dear KOSAKI >>> In my test, I didn't set compaction. Maybe compaction is helpful to >> avoid this issue. I can have try later. >>> In my mind CONFIG_COMPACTION is an optional configuration >> right? >> >> Right. But if you don't set it, application must NOT use >1 order allocations. >> It doesn't work and it is expected >> result. >> That's your application mistake. > Dear Kosaki, I have two questions on your explanation: > a) you said if don't set CONFIG_COMPATION, application must NOT use >1 order allocations, is there any documentation for this theory? Sorry I don't understand what "this" mean. I mean, Even though you use desktop or server machine, no compaction kernel easily makes no order-2 situations. Then, our in-kernel subsystems don't use order-2 allocations as far as possible. > b) My order-2 allocation not comes from application, but from do_fork which is in kernel space, in my mind when a parent process forks a child process, it need to allocate a order-2 memory, if a) is right, then CONFIG_COMPATION should be a MUST configuration for linux kernel but not optional? ??? fork alloc order-1 memory for stack. Where and why alloc order-2? If it is arch specific code, please contact arch maintainer. >> >>> If we don't use, and met such an issue, how should we deal with >> such infinite loop? >>> >>> I made a change in all_reclaimable() function, passed overnight tests, >> please help review, thanks in advance! >>> @@ -2353,7 +2353,9 @@ static bool all_unreclaimable(struct zonelist >> *zonelist, >>> continue; >>> if (!cpuset_zone_allowed_hardwall(zone, >> GFP_KERNEL)) >>> continue; >>> - if (!zone->all_unreclaimable) >>> + if (zone->all_unreclaimable) >>> + continue; >>> + if (zone_reclaimable(zone)) >>> return false; >> >> Please tell me why you chaned here. > The original check is once found zone->all_unreclaimable is false, it will return false, then >it will set did_some_progress non-zero. Then another loop of direct_reclaimed performed. > But I think zone->all_unreclaimable is not always reliable such as in my case, kswapd go to > sleep and no one will change this flag. We should also check zone_reclaimalbe(zone) if > zone->all_unreclaimalbe = 0 to double confirm if a zone is reclaimable; This change also > avoid the issue you described in below commit: Please read more older code. Your pointed code is temporary change and I changed back for fixing bugs. If you look at the status in middle direct reclaim, we can't avoid race condition from multi direct reclaim issues. Moreover, if kswapd doesn't awaken, it is a problem. This is a reason why current code behave as you described. I agree we should fix your issue as far as possible. But I can't agree your analysis. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx179.postini.com [74.125.245.179]) by kanga.kvack.org (Postfix) with SMTP id C98DF6B0031 for ; Thu, 1 Aug 2013 00:22:09 -0400 (EDT) Message-ID: <51F9E265.7030503@oracle.com> Date: Thu, 01 Aug 2013 12:21:57 +0800 From: Bob Liu MIME-Version: 1.0 Subject: Re: Possible deadloop in direct reclaim? References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> In-Reply-To: <51F9CBC0.2020006@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Lisa Du , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu Hi KOSAKI, On 08/01/2013 10:45 AM, KOSAKI Motohiro wrote: > > Please read more older code. Your pointed code is temporary change and I > changed back for fixing > bugs. > If you look at the status in middle direct reclaim, we can't avoid race > condition from multi direct > reclaim issues. Moreover, if kswapd doesn't awaken, it is a problem. > This is a reason why current code > behave as you described. > I agree we should fix your issue as far as possible. But I can't agree > your analysis. > I found this thread: mm, vmscan: fix do_try_to_free_pages() livelock https://lkml.org/lkml/2012/6/14/74 I think that's the same issue Lisa met. But I didn't find out why your patch didn't get merged? There were already many acks. -- Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx192.postini.com [74.125.245.192]) by kanga.kvack.org (Postfix) with SMTP id 46CE66B0031 for ; Thu, 1 Aug 2013 01:24:26 -0400 (EDT) From: Lisa Du Date: Wed, 31 Jul 2013 22:19:53 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE028@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> In-Reply-To: <51F9CBC0.2020006@gmail.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro , "linux@arm.linux.org.uk" Cc: Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu , Neil Zhang Loop in Russel King. Would you please help to comment below questions Mr Motohiro asked about fo= rk allocating order-2 memory? Thanks in advance! >(7/31/13 10:24 PM), Lisa Du wrote: >> Dear Kosaki >> Would you please help to check my comment as below: >>> (7/25/13 9:11 PM), Lisa Du wrote: >>>> Dear KOSAKI >>>> In my test, I didn't set compaction. Maybe compaction is helpful >to >>> avoid this issue. I can have try later. >>>> In my mind CONFIG_COMPACTION is an optional configuration >>> right? >>> >>> Right. But if you don't set it, application must NOT use >1 order >allocations. >>> It doesn't work and it is expected >>> result. >>> That's your application mistake. >> Dear Kosaki, I have two questions on your explanation: >> a) you said if don't set CONFIG_COMPATION, application must NOT use >1 >order allocations, is there any documentation > for this theory? > >Sorry I don't understand what "this" mean. I mean, Even though you use >desktop or server machine, no compaction kernel >easily makes no order-2 situations. >Then, our in-kernel subsystems don't use order-2 allocations as far as >possible. Thanks, now I got your point.=20 > > >> b) My order-2 allocation not comes from application, but from do_fork >which is in kernel space, > in my mind when a parent process forks a child process, it need to >allocate a order-2 memory, > if a) is right, then CONFIG_COMPATION should be a MUST configuration >for linux kernel but not optional? > >??? >fork alloc order-1 memory for stack. Where and why alloc order-2? If it is >arch specific code, please >contact arch maintainer. Yes arch do_fork allocate order-2 memory when copy_process.=20 Hi, Russel What's your opinion about this question? =20 If we really need order-2 memory for fork, then we'd better set CONFIG_COMP= ATION right? > > > >>> >>>> If we don't use, and met such an issue, how should we deal with >>> such infinite loop? >>>> >>>> I made a change in all_reclaimable() function, passed overnight >tests, >>> please help review, thanks in advance! >>>> @@ -2353,7 +2353,9 @@ static bool all_unreclaimable(struct zonelist >>> *zonelist, >>>> continue; >>>> if (!cpuset_zone_allowed_hardwall(zone, >>> GFP_KERNEL)) >>>> continue; >>>> - if (!zone->all_unreclaimable) >>>> + if (zone->all_unreclaimable) >>>> + continue; >>>> + if (zone_reclaimable(zone)) >>>> return false; >>> >>> Please tell me why you chaned here. >> The original check is once found zone->all_unreclaimable is false, it wi= ll >return false, then >>it will set did_some_progress non-zero. Then another loop of >direct_reclaimed performed. >> But I think zone->all_unreclaimable is not always reliable such as in m= y >case, kswapd go to >> sleep and no one will change this flag. We should also check >zone_reclaimalbe(zone) if >> zone->all_unreclaimalbe =3D 0 to double confirm if a zone is reclaimabl= e; >This change also >> avoid the issue you described in below commit: > >Please read more older code. Your pointed code is temporary change and I >changed back for fixing >bugs. >If you look at the status in middle direct reclaim, we can't avoid race >condition from multi direct >reclaim issues. Moreover, if kswapd doesn't awaken, it is a problem. This = is >a reason why current code >behave as you described. >I agree we should fix your issue as far as possible. But I can't agree you= r >analysis. I read the code you modified which check the zone->all_unreclaimable instea= d of zone_reclaimable(zone); (In the commit 929bea7c714 vmscan: all_unreclaimable() use zone->all_unrecl= aimable as a name) Your patch was trying to fix the issue of zone->all_unreclaimable =3D 1, bu= t zone->pages_scanned =3D 0 which result all_unreclaimable() return false. Is there anything else I missed or misunderstanding? In my change, I'll first check zone->all_unreclaimable, if it was set 1, th= en I wouldn't check zone->pages_scanned value. My point is zone->all_unreclaimable =3D 0 doesn't mean this zone is always = reclaimable. As zone->all_unreclaimable can only be set in kswapd. And kswapd already fully scan all zones and still can't rebalance the syste= m for high-order allocations. Instead it recheck all watermarks at order-0= , and watermarks ok will let kswapd back to sleep. Unfortunately, Kswapd do= esn't awaken because long time no higher order allocation wake it up. But t= his process continue direct reclaim again and again as zone->all_unreclaima= ble remains 0. So I also checked the zone->pages_scanned when zone->all_unreclaimable =3D = 0, if zone_reclaimable() return true, then it's really reclaimable for dire= ct reclaimer. This change would break your bug fix right? Thanks Bob's finding, I read through below thread, and the patch your are t= rying to fix is the same issue as mine: mm, vmscan: fix do_try_to_free_pages() livelock https://lkml.org/lkml/2012/6/14/74 I have the same question as Bob, you already find this issue, why this patc= h wasn't got merged?=20 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx197.postini.com [74.125.245.197]) by kanga.kvack.org (Postfix) with SMTP id C27D96B0032 for ; Thu, 1 Aug 2013 01:43:11 -0400 (EDT) Date: Thu, 1 Aug 2013 14:43:38 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130801054338.GD19540@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" Hello, On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: > Dear Sir: > Currently I met a possible deadloop in direct reclaim. After run plenty of the application, system run into a status that system memory is very fragmentized. Like only order-0 and order-1 memory left. > Then one process required a order-2 buffer but it enter an endless direct reclaim. From my trace log, I can see this loop already over 200,000 times. Kswapd was first wake up and then go back to sleep as it cannot rebalance this order's memory. But zone->all_unreclaimable remains 1. > Though direct_reclaim every time returns no pages, but as zone->all_unreclaimable = 1, so it loop again and again. Even when zone->pages_scanned also becomes very large. It will block the process for long time, until some watchdog thread detect this and kill this process. Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe cost over 50 seconds or even more. > I think it's not as expected right? Can we also add below check in the function all_unreclaimable() to terminate this loop? > > @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!zone->all_unreclaimable) > return false; > + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) > + return true; > } > BTW: I'm using kernel3.4, I also try to search in the kernel3.9, didn't see a possible fix for such issue. Or is anyone also met such issue before? Any comment will be welcomed, looking forward to your reply! > > Thanks! I'd like to ask somethigs. 1. Do you have enabled swap? 2. Do you enable CONFIG_COMPACTION? 3. Could we get your zoneinfo via cat /proc/zoneinfo? 4. If you disabled watchdog thread, you could see OOM sometime although it takes very long time? > > Best Regards > Lisa Du > -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx143.postini.com [74.125.245.143]) by kanga.kvack.org (Postfix) with SMTP id 0616F6B0034 for ; Thu, 1 Aug 2013 02:14:58 -0400 (EDT) From: Lisa Du Date: Wed, 31 Jul 2013 23:13:07 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> In-Reply-To: <20130801054338.GD19540@bbox> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: "linux-mm@kvack.org" >On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: >> Dear Sir: >> Currently I met a possible deadloop in direct reclaim. After run plenty = of >the application, system run into a status that system memory is very >fragmentized. Like only order-0 and order-1 memory left. >> Then one process required a order-2 buffer but it enter an endless direc= t >reclaim. From my trace log, I can see this loop already over 200,000 times= . >Kswapd was first wake up and then go back to sleep as it cannot rebalance >this order's memory. But zone->all_unreclaimable remains 1. >> Though direct_reclaim every time returns no pages, but as >zone->all_unreclaimable =3D 1, so it loop again and again. Even when >zone->pages_scanned also becomes very large. It will block the process for >long time, until some watchdog thread detect this and kill this process. >Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe cost >over 50 seconds or even more. >> I think it's not as expected right? Can we also add below check in the >function all_unreclaimable() to terminate this loop? >> >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist >*zonelist, >> continue; >> if (!zone->all_unreclaimable) >> return false; >> + if (sc->nr_reclaimed =3D=3D 0 && !zone_reclaimable(zone)= ) >> + return true; >> } >> BTW: I'm using kernel3.4, I also try to search in the kernel3.9= , >didn't see a possible fix for such issue. Or is anyone also met such issue >before? Any comment will be welcomed, looking forward to your reply! >> >> Thanks! > >I'd like to ask somethigs. > >1. Do you have enabled swap? I set CONFIG_SWAP=3Dy, but I didn't really have a swap partition, that mean= s my swap buffer size is 0; >2. Do you enable CONFIG_COMPACTION? No, I didn't enable; >3. Could we get your zoneinfo via cat /proc/zoneinfo? I dump some info from ramdump, please review: crash> kmem -z NODE: 0 ZONE: 0 ADDR: c08460c0 NAME: "Normal" SIZE: 192512 PRESENT: 182304 MIN/LOW/HIGH: 853/1066/1279 VM_STAT: NR_FREE_PAGES: 16092 NR_INACTIVE_ANON: 17 NR_ACTIVE_ANON: 55091 NR_INACTIVE_FILE: 17 NR_ACTIVE_FILE: 17 NR_UNEVICTABLE: 0 NR_MLOCK: 0 NR_ANON_PAGES: 55077 NR_FILE_MAPPED: 42 NR_FILE_PAGES: 69 NR_FILE_DIRTY: 0 NR_WRITEBACK: 0 NR_SLAB_RECLAIMABLE: 1226 NR_SLAB_UNRECLAIMABLE: 9373 NR_PAGETABLE: 2776 NR_KERNEL_STACK: 798 NR_UNSTABLE_NFS: 0 NR_BOUNCE: 0 NR_VMSCAN_WRITE: 91 NR_VMSCAN_IMMEDIATE: 115381 NR_WRITEBACK_TEMP: 0 NR_ISOLATED_ANON: 0 NR_ISOLATED_FILE: 0 NR_SHMEM: 31 NR_DIRTIED: 15256 NR_WRITTEN: 11981 NR_ANON_TRANSPARENT_HUGEPAGES: 0 NODE: 0 ZONE: 1 ADDR: c08464c0 NAME: "HighMem" SIZE: 69632 PRESENT: 69088 MIN/LOW/HIGH: 67/147/228 VM_STAT: NR_FREE_PAGES: 161 NR_INACTIVE_ANON: 104 NR_ACTIVE_ANON: 46114 NR_INACTIVE_FILE: 9722 NR_ACTIVE_FILE: 12263 NR_UNEVICTABLE: 168 NR_MLOCK: 0 NR_ANON_PAGES: 46102 NR_FILE_MAPPED: 12227 NR_FILE_PAGES: 22270 NR_FILE_DIRTY: 1 NR_WRITEBACK: 0 NR_SLAB_RECLAIMABLE: 0 NR_SLAB_UNRECLAIMABLE: 0 NR_PAGETABLE: 0 NR_KERNEL_STACK: 0 NR_UNSTABLE_NFS: 0 NR_BOUNCE: 0 NR_VMSCAN_WRITE: 0 NR_VMSCAN_IMMEDIATE: 0 NR_WRITEBACK_TEMP: 0 NR_ISOLATED_ANON: 0 NR_ISOLATED_FILE: 0 NR_SHMEM: 117 NR_DIRTIED: 7364 NR_WRITTEN: 6989 NR_ANON_TRANSPARENT_HUGEPAGES: 0 ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 0 Normal 192512 16092 c1200000 0 0 =20 AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES 0 4k c08460f0 3 3 0 4k c08460f8 436 436 0 4k c0846100 15237 15237 0 4k c0846108 0 0 0 4k c0846110 0 0 1 8k c084611c 39 78 1 8k c0846124 0 0 1 8k c084612c 169 338 1 8k c0846134 0 0 1 8k c084613c 0 0 2 16k c0846148 0 0 2 16k c0846150 0 0 2 16k c0846158 0 0 ---------Normal zone all order > 1 has no free pages ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 1 HighMem 69632 161 c17e0000 2f000000 192512 =20 AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES 0 4k c08464f0 12 12 0 4k c08464f8 0 0 0 4k c0846500 14 14 0 4k c0846508 3 3 0 4k c0846510 0 0 1 8k c084651c 0 0 1 8k c0846524 0 0 1 8k c084652c 0 0 2 16k c0846548 0 0 2 16k c0846550 0 0 2 16k c0846558 0 0 2 16k c0846560 1 4 2 16k c0846568 0 0 5 128k c08465cc 0 0 5 128k c08465d4 0 0 5 128k c08465dc 0 0 5 128k c08465e4 4 128 5 128k c08465ec 0 0 ------Other's all zero Some other zone information I dump from pglist_data { watermark =3D {853, 1066, 1279},=20 percpu_drift_mark =3D 0,=20 lowmem_reserve =3D {0, 2159, 2159},=20 dirty_balance_reserve =3D 3438,=20 pageset =3D 0xc07f6144,=20 lock =3D { { rlock =3D { raw_lock =3D { lock =3D 0 },=20 break_lock =3D 0 } } }, =20 all_unreclaimable =3D 0, reclaim_stat =3D { recent_rotated =3D {903355, 960912},=20 recent_scanned =3D {932404, 2462017} },=20 pages_scanned =3D 84231, inactive_ratio =3D 1,=20 _pad2_ =3D { x =3D 0xc0846480 "@" },=20 wait_table =3D 0xc1a00040,=20 wait_table_hash_nr_entries =3D 1024,=20 wait_table_bits =3D 10,=20 zone_pgdat =3D 0xc08460c0,=20 zone_start_pfn =3D 0,=20 spanned_pages =3D 192512,=20 present_pages =3D 182304,=20 name =3D 0xc06f1d46 "Normal" } { watermark =3D {67, 147, 228},=20 percpu_drift_mark =3D 0,=20 lowmem_reserve =3D {0, 0, 0},=20 dirty_balance_reserve =3D 228,=20 pageset =3D 0xc07f6184,=20 lock =3D { { rlock =3D { raw_lock =3D { lock =3D 0 },=20 break_lock =3D 0 } } },=20 all_unreclaimable =3D 0, reclaim_stat =3D { recent_rotated =3D {272514, 28087},=20 recent_scanned =3D {287521, 110478} },=20 pages_scanned =3D 0,=20 flags =3D 0, } kswapd =3D 0xe02d4f00,=20 kswapd_max_order =3D 0,=20 classzone_idx =3D ZONE_HIGHMEM >4. If you disabled watchdog thread, you could see OOM sometime > although it takes very long time? I haven't try to disable watchdog, in my case, when watchdog triggered, it = means process already blocked over 60s. And during these 60s, I didn't see OOM happen. > > >> >> Best Regards >> Lisa Du >> > >-- >Kind regards, >Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx106.postini.com [74.125.245.106]) by kanga.kvack.org (Postfix) with SMTP id 520FB6B005C for ; Thu, 1 Aug 2013 03:33:04 -0400 (EDT) Date: Thu, 1 Aug 2013 16:33:30 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130801073330.GG19540@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro On Wed, Jul 31, 2013 at 11:13:07PM -0700, Lisa Du wrote: > >On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: > >> Dear Sir: > >> Currently I met a possible deadloop in direct reclaim. After run plenty of > >the application, system run into a status that system memory is very > >fragmentized. Like only order-0 and order-1 memory left. > >> Then one process required a order-2 buffer but it enter an endless direct > >reclaim. From my trace log, I can see this loop already over 200,000 times. > >Kswapd was first wake up and then go back to sleep as it cannot rebalance > >this order's memory. But zone->all_unreclaimable remains 1. > >> Though direct_reclaim every time returns no pages, but as > >zone->all_unreclaimable = 1, so it loop again and again. Even when > >zone->pages_scanned also becomes very large. It will block the process for > >long time, until some watchdog thread detect this and kill this process. > >Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe cost > >over 50 seconds or even more. > >> I think it's not as expected right? Can we also add below check in the > >function all_unreclaimable() to terminate this loop? > >> > >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist > >*zonelist, > >> continue; > >> if (!zone->all_unreclaimable) > >> return false; > >> + if (sc->nr_reclaimed == 0 && !zone_reclaimable(zone)) > >> + return true; > >> } > >> BTW: I'm using kernel3.4, I also try to search in the kernel3.9, > >didn't see a possible fix for such issue. Or is anyone also met such issue > >before? Any comment will be welcomed, looking forward to your reply! > >> > >> Thanks! > > > >I'd like to ask somethigs. > > > >1. Do you have enabled swap? > I set CONFIG_SWAP=y, but I didn't really have a swap partition, that means my swap buffer size is 0; > >2. Do you enable CONFIG_COMPACTION? > No, I didn't enable; > >3. Could we get your zoneinfo via cat /proc/zoneinfo? > I dump some info from ramdump, please review: Thanks for the information. You said order-2 allocation was failed so I will assume preferred zone is normal zone, not high zone because high order allocation in kernel side isn't from high zone. > crash> kmem -z > NODE: 0 ZONE: 0 ADDR: c08460c0 NAME: "Normal" > SIZE: 192512 PRESENT: 182304 MIN/LOW/HIGH: 853/1066/1279 712M normal memory. > VM_STAT: > NR_FREE_PAGES: 16092 There are plenty of free pages over high watermark but there are heavy fragmentation as I see below information. So, kswapd doesn't scan this zone loop iteration is done with order-2. I mean kswapd will scan this zone with order-0 if first iteration is done by this order = sc.order = 0; goto loop_again; But this time, zone_watermark_ok_safe with testorder = 0 on normal zone is always true so that scanning of zone will be skipped. It means kswapd never set zone->unreclaimable to 1. > NR_INACTIVE_ANON: 17 > NR_ACTIVE_ANON: 55091 > NR_INACTIVE_FILE: 17 > NR_ACTIVE_FILE: 17 > NR_UNEVICTABLE: 0 > NR_MLOCK: 0 > NR_ANON_PAGES: 55077 There are about 200M anon pages and few file pages. You don't have swap so that reclaimer couldn't go far. > NR_FILE_MAPPED: 42 > NR_FILE_PAGES: 69 > NR_FILE_DIRTY: 0 > NR_WRITEBACK: 0 > NR_SLAB_RECLAIMABLE: 1226 > NR_SLAB_UNRECLAIMABLE: 9373 > NR_PAGETABLE: 2776 > NR_KERNEL_STACK: 798 > NR_UNSTABLE_NFS: 0 > NR_BOUNCE: 0 > NR_VMSCAN_WRITE: 91 > NR_VMSCAN_IMMEDIATE: 115381 > NR_WRITEBACK_TEMP: 0 > NR_ISOLATED_ANON: 0 > NR_ISOLATED_FILE: 0 > NR_SHMEM: 31 > NR_DIRTIED: 15256 > NR_WRITTEN: 11981 > NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > NODE: 0 ZONE: 1 ADDR: c08464c0 NAME: "HighMem" > SIZE: 69632 PRESENT: 69088 MIN/LOW/HIGH: 67/147/228 > VM_STAT: > NR_FREE_PAGES: 161 Reclaimer should reclaim this zone. > NR_INACTIVE_ANON: 104 > NR_ACTIVE_ANON: 46114 > NR_INACTIVE_FILE: 9722 > NR_ACTIVE_FILE: 12263 It seems there are lots of room to evict file pages. > NR_UNEVICTABLE: 168 > NR_MLOCK: 0 > NR_ANON_PAGES: 46102 > NR_FILE_MAPPED: 12227 > NR_FILE_PAGES: 22270 > NR_FILE_DIRTY: 1 > NR_WRITEBACK: 0 > NR_SLAB_RECLAIMABLE: 0 > NR_SLAB_UNRECLAIMABLE: 0 > NR_PAGETABLE: 0 > NR_KERNEL_STACK: 0 > NR_UNSTABLE_NFS: 0 > NR_BOUNCE: 0 > NR_VMSCAN_WRITE: 0 > NR_VMSCAN_IMMEDIATE: 0 > NR_WRITEBACK_TEMP: 0 > NR_ISOLATED_ANON: 0 > NR_ISOLATED_FILE: 0 > NR_SHMEM: 117 > NR_DIRTIED: 7364 > NR_WRITTEN: 6989 > NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR > 0 Normal 192512 16092 c1200000 0 0 > AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > 0 4k c08460f0 3 3 > 0 4k c08460f8 436 436 > 0 4k c0846100 15237 15237 > 0 4k c0846108 0 0 > 0 4k c0846110 0 0 > 1 8k c084611c 39 78 > 1 8k c0846124 0 0 > 1 8k c084612c 169 338 > 1 8k c0846134 0 0 > 1 8k c084613c 0 0 > 2 16k c0846148 0 0 > 2 16k c0846150 0 0 > 2 16k c0846158 0 0 > ---------Normal zone all order > 1 has no free pages > ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR > 1 HighMem 69632 161 c17e0000 2f000000 192512 > AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > 0 4k c08464f0 12 12 > 0 4k c08464f8 0 0 > 0 4k c0846500 14 14 > 0 4k c0846508 3 3 > 0 4k c0846510 0 0 > 1 8k c084651c 0 0 > 1 8k c0846524 0 0 > 1 8k c084652c 0 0 > 2 16k c0846548 0 0 > 2 16k c0846550 0 0 > 2 16k c0846558 0 0 > 2 16k c0846560 1 4 > 2 16k c0846568 0 0 > 5 128k c08465cc 0 0 > 5 128k c08465d4 0 0 > 5 128k c08465dc 0 0 > 5 128k c08465e4 4 128 > 5 128k c08465ec 0 0 > ------Other's all zero > > Some other zone information I dump from pglist_data > { > watermark = {853, 1066, 1279}, > percpu_drift_mark = 0, > lowmem_reserve = {0, 2159, 2159}, > dirty_balance_reserve = 3438, > pageset = 0xc07f6144, > lock = { > { > rlock = { > raw_lock = { > lock = 0 > }, > break_lock = 0 > } > } > }, > all_unreclaimable = 0, > reclaim_stat = { > recent_rotated = {903355, 960912}, > recent_scanned = {932404, 2462017} > }, > pages_scanned = 84231, Most of scan happens in direct reclaim path, I guess but direct reclaim couldn't reclaim any pages due to lack of swap device. It means we have to set zone->all_unreclaimable in direct reclaim path, too. Below patch fix your problem? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx133.postini.com [74.125.245.133]) by kanga.kvack.org (Postfix) with SMTP id 2CD716B0032 for ; Thu, 1 Aug 2013 04:24:17 -0400 (EDT) From: Lisa Du Date: Thu, 1 Aug 2013 01:20:34 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> In-Reply-To: <20130801073330.GG19540@bbox> Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: "linux-mm@kvack.org" , KOSAKI Motohiro Pi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogTWluY2hhbiBLaW0gW21haWx0bzpt aW5jaGFuQGtlcm5lbC5vcmddDQo+U2VudDogMjAxM8TqONTCMcjVIDE1OjM0DQo+VG86IExpc2Eg RHUNCj5DYzogbGludXgtbW1Aa3ZhY2sub3JnOyBLT1NBS0kgTW90b2hpcm8NCj5TdWJqZWN0OiBS ZTogUG9zc2libGUgZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0/DQo+DQo+T24gV2VkLCBKdWwg MzEsIDIwMTMgYXQgMTE6MTM6MDdQTSAtMDcwMCwgTGlzYSBEdSB3cm90ZToNCj4+ID5PbiBNb24s IEp1bCAyMiwgMjAxMyBhdCAwOTo1ODoxN1BNIC0wNzAwLCBMaXNhIER1IHdyb3RlOg0KPj4gPj4g RGVhciBTaXI6DQo+PiA+PiBDdXJyZW50bHkgSSBtZXQgYSBwb3NzaWJsZSBkZWFkbG9vcCBpbiBk aXJlY3QgcmVjbGFpbS4gQWZ0ZXIgcnVuIHBsZW50eQ0KPm9mDQo+PiA+dGhlIGFwcGxpY2F0aW9u LCBzeXN0ZW0gcnVuIGludG8gYSBzdGF0dXMgdGhhdCBzeXN0ZW0gbWVtb3J5IGlzIHZlcnkNCj4+ ID5mcmFnbWVudGl6ZWQuIExpa2Ugb25seSBvcmRlci0wIGFuZCBvcmRlci0xIG1lbW9yeSBsZWZ0 Lg0KPj4gPj4gVGhlbiBvbmUgcHJvY2VzcyByZXF1aXJlZCBhIG9yZGVyLTIgYnVmZmVyIGJ1dCBp dCBlbnRlciBhbiBlbmRsZXNzDQo+ZGlyZWN0DQo+PiA+cmVjbGFpbS4gRnJvbSBteSB0cmFjZSBs b2csIEkgY2FuIHNlZSB0aGlzIGxvb3AgYWxyZWFkeSBvdmVyIDIwMCwwMDANCj50aW1lcy4NCj4+ ID5Lc3dhcGQgd2FzIGZpcnN0IHdha2UgdXAgYW5kIHRoZW4gZ28gYmFjayB0byBzbGVlcCBhcyBp dCBjYW5ub3QNCj5yZWJhbGFuY2UNCj4+ID50aGlzIG9yZGVyJ3MgbWVtb3J5LiBCdXQgem9uZS0+ YWxsX3VucmVjbGFpbWFibGUgcmVtYWlucyAxLg0KPj4gPj4gVGhvdWdoIGRpcmVjdF9yZWNsYWlt IGV2ZXJ5IHRpbWUgcmV0dXJucyBubyBwYWdlcywgYnV0IGFzDQo+PiA+em9uZS0+YWxsX3VucmVj bGFpbWFibGUgPSAxLCBzbyBpdCBsb29wIGFnYWluIGFuZCBhZ2Fpbi4gRXZlbiB3aGVuDQo+PiA+ em9uZS0+cGFnZXNfc2Nhbm5lZCBhbHNvIGJlY29tZXMgdmVyeSBsYXJnZS4gSXQgd2lsbCBibG9j ayB0aGUgcHJvY2Vzcw0KPmZvcg0KPj4gPmxvbmcgdGltZSwgdW50aWwgc29tZSB3YXRjaGRvZyB0 aHJlYWQgZGV0ZWN0IHRoaXMgYW5kIGtpbGwgdGhpcyBwcm9jZXNzLg0KPj4gPlRob3VnaCBpdCdz IGluIF9fYWxsb2NfcGFnZXNfc2xvd3BhdGgsIGJ1dCBpdCdzIHRvbyBzbG93IHJpZ2h0PyBNYXli ZQ0KPmNvc3QNCj4+ID5vdmVyIDUwIHNlY29uZHMgb3IgZXZlbiBtb3JlLg0KPj4gPj4gSSB0aGlu ayBpdCdzIG5vdCBhcyBleHBlY3RlZCByaWdodD8gIENhbiB3ZSBhbHNvIGFkZCBiZWxvdyBjaGVj ayBpbiB0aGUNCj4+ID5mdW5jdGlvbiBhbGxfdW5yZWNsYWltYWJsZSgpIHRvIHRlcm1pbmF0ZSB0 aGlzIGxvb3A/DQo+PiA+Pg0KPj4gPj4gQEAgLTIzNTUsNiArMjM1NSw4IEBAIHN0YXRpYyBib29s IGFsbF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6b25lbGlzdA0KPj4gPip6b25lbGlzdCwNCj4+ID4+ ICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPj4gPj4gICAgICAgICAgICAgICAg IGlmICghem9uZS0+YWxsX3VucmVjbGFpbWFibGUpDQo+PiA+PiAgICAgICAgICAgICAgICAgICAg ICAgICByZXR1cm4gZmFsc2U7DQo+PiA+PiArICAgICAgICAgICAgICAgaWYgKHNjLT5ucl9yZWNs YWltZWQgPT0gMA0KPiYmICF6b25lX3JlY2xhaW1hYmxlKHpvbmUpKQ0KPj4gPj4gKyAgICAgICAg ICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7DQo+PiA+PiAgICAgICAgIH0NCj4+ID4+ICAgICAg ICAgIEJUVzogSSdtIHVzaW5nIGtlcm5lbDMuNCwgSSBhbHNvIHRyeSB0byBzZWFyY2ggaW4gdGhl DQo+a2VybmVsMy45LA0KPj4gPmRpZG4ndCBzZWUgYSBwb3NzaWJsZSBmaXggZm9yIHN1Y2ggaXNz dWUuIE9yIGlzIGFueW9uZSBhbHNvIG1ldCBzdWNoIGlzc3VlDQo+PiA+YmVmb3JlPyBBbnkgY29t bWVudCB3aWxsIGJlIHdlbGNvbWVkLCBsb29raW5nIGZvcndhcmQgdG8geW91ciByZXBseSENCj4+ ID4+DQo+PiA+PiBUaGFua3MhDQo+PiA+DQo+PiA+SSdkIGxpa2UgdG8gYXNrIHNvbWV0aGlncy4N Cj4+ID4NCj4+ID4xLiBEbyB5b3UgaGF2ZSBlbmFibGVkIHN3YXA/DQo+PiBJIHNldCBDT05GSUdf U1dBUD15LCBidXQgSSBkaWRuJ3QgcmVhbGx5IGhhdmUgYSBzd2FwIHBhcnRpdGlvbiwgdGhhdA0K Pm1lYW5zIG15IHN3YXAgYnVmZmVyIHNpemUgaXMgMDsNCj4+ID4yLiBEbyB5b3UgZW5hYmxlIENP TkZJR19DT01QQUNUSU9OPw0KPj4gTm8sIEkgZGlkbid0IGVuYWJsZTsNCj4+ID4zLiBDb3VsZCB3 ZSBnZXQgeW91ciB6b25laW5mbyB2aWEgY2F0IC9wcm9jL3pvbmVpbmZvPw0KPj4gSSBkdW1wIHNv bWUgaW5mbyBmcm9tIHJhbWR1bXAsIHBsZWFzZSByZXZpZXc6DQo+DQo+VGhhbmtzIGZvciB0aGUg aW5mb3JtYXRpb24uDQo+WW91IHNhaWQgb3JkZXItMiBhbGxvY2F0aW9uIHdhcyBmYWlsZWQgc28g SSB3aWxsIGFzc3VtZSBwcmVmZXJyZWQgem9uZQ0KPmlzIG5vcm1hbCB6b25lLCBub3QgaGlnaCB6 b25lIGJlY2F1c2UgaGlnaCBvcmRlciBhbGxvY2F0aW9uIGluIGtlcm5lbCBzaWRlDQo+aXNuJ3Qg ZnJvbSBoaWdoIHpvbmUuDQpZZXMsIHRoYXQncyByaWdodCENCj4NCj4+IGNyYXNoPiBrbWVtIC16 DQo+PiBOT0RFOiAwICBaT05FOiAwICBBRERSOiBjMDg0NjBjMCAgTkFNRTogIk5vcm1hbCINCj4+ ICAgU0laRTogMTkyNTEyICBQUkVTRU5UOiAxODIzMDQgIE1JTi9MT1cvSElHSDogODUzLzEwNjYv MTI3OQ0KPg0KPjcxMk0gbm9ybWFsIG1lbW9yeS4NCj4NCj4+ICAgVk1fU1RBVDoNCj4+ICAgICAg ICAgICBOUl9GUkVFX1BBR0VTOiAxNjA5Mg0KPg0KPlRoZXJlIGFyZSBwbGVudHkgb2YgZnJlZSBw YWdlcyBvdmVyIGhpZ2ggd2F0ZXJtYXJrIGJ1dCB0aGVyZSBhcmUgaGVhdnkNCj5mcmFnbWVudGF0 aW9uIGFzIEkgc2VlIGJlbG93IGluZm9ybWF0aW9uLg0KPg0KPlNvLCBrc3dhcGQgZG9lc24ndCBz Y2FuIHRoaXMgem9uZSBsb29wIGl0ZXJhdGlvbiBpcyBkb25lIHdpdGggb3JkZXItMi4NCj5JIG1l YW4ga3N3YXBkIHdpbGwgc2NhbiB0aGlzIHpvbmUgd2l0aCBvcmRlci0wIGlmIGZpcnN0IGl0ZXJh dGlvbiBpcw0KPmRvbmUgYnkgdGhpcw0KPg0KPiAgICAgICAgb3JkZXIgPSBzYy5vcmRlciA9IDA7 DQo+DQo+ICAgICAgICBnb3RvIGxvb3BfYWdhaW47DQo+DQo+QnV0IHRoaXMgdGltZSwgem9uZV93 YXRlcm1hcmtfb2tfc2FmZSB3aXRoIHRlc3RvcmRlciA9IDAgb24gbm9ybWFsIHpvbmUNCj5pcyBh bHdheXMgdHJ1ZSBzbyB0aGF0IHNjYW5uaW5nIG9mIHpvbmUgd2lsbCBiZSBza2lwcGVkLiBJdCBt ZWFucyBrc3dhcGQNCj5uZXZlciBzZXQgem9uZS0+dW5yZWNsYWltYWJsZSB0byAxLg0KWWVzLCBk ZWZpbml0ZWx5IQ0KPg0KPj4gICAgICAgIE5SX0lOQUNUSVZFX0FOT046IDE3DQo+PiAgICAgICAg ICBOUl9BQ1RJVkVfQU5PTjogNTUwOTENCj4+ICAgICAgICBOUl9JTkFDVElWRV9GSUxFOiAxNw0K Pj4gICAgICAgICAgTlJfQUNUSVZFX0ZJTEU6IDE3DQo+PiAgICAgICAgICBOUl9VTkVWSUNUQUJM RTogMA0KPj4gICAgICAgICAgICAgICAgTlJfTUxPQ0s6IDANCj4+ICAgICAgICAgICBOUl9BTk9O X1BBR0VTOiA1NTA3Nw0KPg0KPlRoZXJlIGFyZSBhYm91dCAyMDBNIGFub24gcGFnZXMgYW5kIGZl dyBmaWxlIHBhZ2VzLg0KPllvdSBkb24ndCBoYXZlIHN3YXAgc28gdGhhdCByZWNsYWltZXIgY291 bGRuJ3QgZ28gZmFyLg0KPg0KPj4gICAgICAgICAgTlJfRklMRV9NQVBQRUQ6IDQyDQo+PiAgICAg ICAgICAgTlJfRklMRV9QQUdFUzogNjkNCj4+ICAgICAgICAgICBOUl9GSUxFX0RJUlRZOiAwDQo+ PiAgICAgICAgICAgIE5SX1dSSVRFQkFDSzogMA0KPj4gICAgIE5SX1NMQUJfUkVDTEFJTUFCTEU6 IDEyMjYNCj4+ICAgTlJfU0xBQl9VTlJFQ0xBSU1BQkxFOiA5MzczDQo+PiAgICAgICAgICAgIE5S X1BBR0VUQUJMRTogMjc3Ng0KPj4gICAgICAgICBOUl9LRVJORUxfU1RBQ0s6IDc5OA0KPj4gICAg ICAgICBOUl9VTlNUQUJMRV9ORlM6IDANCj4+ICAgICAgICAgICAgICAgTlJfQk9VTkNFOiAwDQo+ PiAgICAgICAgIE5SX1ZNU0NBTl9XUklURTogOTENCj4+ICAgICBOUl9WTVNDQU5fSU1NRURJQVRF OiAxMTUzODENCj4+ICAgICAgIE5SX1dSSVRFQkFDS19URU1QOiAwDQo+PiAgICAgICAgTlJfSVNP TEFURURfQU5PTjogMA0KPj4gICAgICAgIE5SX0lTT0xBVEVEX0ZJTEU6IDANCj4+ICAgICAgICAg ICAgICAgIE5SX1NITUVNOiAzMQ0KPj4gICAgICAgICAgICAgIE5SX0RJUlRJRUQ6IDE1MjU2DQo+ PiAgICAgICAgICAgICAgTlJfV1JJVFRFTjogMTE5ODENCj4+IE5SX0FOT05fVFJBTlNQQVJFTlRf SFVHRVBBR0VTOiAwDQo+Pg0KPj4gTk9ERTogMCAgWk9ORTogMSAgQUREUjogYzA4NDY0YzAgIE5B TUU6ICJIaWdoTWVtIg0KPj4gICBTSVpFOiA2OTYzMiAgUFJFU0VOVDogNjkwODggIE1JTi9MT1cv SElHSDogNjcvMTQ3LzIyOA0KPj4gICBWTV9TVEFUOg0KPj4gICAgICAgICAgIE5SX0ZSRUVfUEFH RVM6IDE2MQ0KPg0KPlJlY2xhaW1lciBzaG91bGQgcmVjbGFpbSB0aGlzIHpvbmUuDQo+DQo+PiAg ICAgICAgTlJfSU5BQ1RJVkVfQU5PTjogMTA0DQo+PiAgICAgICAgICBOUl9BQ1RJVkVfQU5PTjog NDYxMTQNCj4+ICAgICAgICBOUl9JTkFDVElWRV9GSUxFOiA5NzIyDQo+PiAgICAgICAgICBOUl9B Q1RJVkVfRklMRTogMTIyNjMNCj4NCj5JdCBzZWVtcyB0aGVyZSBhcmUgbG90cyBvZiByb29tIHRv IGV2aWN0IGZpbGUgcGFnZXMuDQo+DQo+PiAgICAgICAgICBOUl9VTkVWSUNUQUJMRTogMTY4DQo+ PiAgICAgICAgICAgICAgICBOUl9NTE9DSzogMA0KPj4gICAgICAgICAgIE5SX0FOT05fUEFHRVM6 IDQ2MTAyDQo+PiAgICAgICAgICBOUl9GSUxFX01BUFBFRDogMTIyMjcNCj4+ICAgICAgICAgICBO Ul9GSUxFX1BBR0VTOiAyMjI3MA0KPj4gICAgICAgICAgIE5SX0ZJTEVfRElSVFk6IDENCj4+ICAg ICAgICAgICAgTlJfV1JJVEVCQUNLOiAwDQo+PiAgICAgTlJfU0xBQl9SRUNMQUlNQUJMRTogMA0K Pj4gICBOUl9TTEFCX1VOUkVDTEFJTUFCTEU6IDANCj4+ICAgICAgICAgICAgTlJfUEFHRVRBQkxF OiAwDQo+PiAgICAgICAgIE5SX0tFUk5FTF9TVEFDSzogMA0KPj4gICAgICAgICBOUl9VTlNUQUJM RV9ORlM6IDANCj4+ICAgICAgICAgICAgICAgTlJfQk9VTkNFOiAwDQo+PiAgICAgICAgIE5SX1ZN U0NBTl9XUklURTogMA0KPj4gICAgIE5SX1ZNU0NBTl9JTU1FRElBVEU6IDANCj4+ICAgICAgIE5S X1dSSVRFQkFDS19URU1QOiAwDQo+PiAgICAgICAgTlJfSVNPTEFURURfQU5PTjogMA0KPj4gICAg ICAgIE5SX0lTT0xBVEVEX0ZJTEU6IDANCj4+ICAgICAgICAgICAgICAgIE5SX1NITUVNOiAxMTcN Cj4+ICAgICAgICAgICAgICBOUl9ESVJUSUVEOiA3MzY0DQo+PiAgICAgICAgICAgICAgTlJfV1JJ VFRFTjogNjk4OQ0KPj4gTlJfQU5PTl9UUkFOU1BBUkVOVF9IVUdFUEFHRVM6IDANCj4+DQo+PiBa T05FICBOQU1FICAgICAgICBTSVpFICAgIEZSRUUgIE1FTV9NQVAgICBTVEFSVF9QQUREUg0KPlNU QVJUX01BUE5SDQo+PiAgIDAgICBOb3JtYWwgICAgMTkyNTEyICAgMTYwOTIgIGMxMjAwMDAwICAg ICAgIDAgICAgICAgICAgICAwDQo+PiBBUkVBICAgIFNJWkUgIEZSRUVfQVJFQV9TVFJVQ1QgIEJM T0NLUyAgUEFHRVMNCj4+ICAgMCAgICAgICA0ayAgICAgIGMwODQ2MGYwICAgICAgICAgICAzICAg ICAgMw0KPj4gICAwICAgICAgIDRrICAgICAgYzA4NDYwZjggICAgICAgICA0MzYgICAgNDM2DQo+ PiAgIDAgICAgICAgNGsgICAgICBjMDg0NjEwMCAgICAgICAxNTIzNyAgMTUyMzcNCj4+ICAgMCAg ICAgICA0ayAgICAgIGMwODQ2MTA4ICAgICAgICAgICAwICAgICAgMA0KPj4gICAwICAgICAgIDRr ICAgICAgYzA4NDYxMTAgICAgICAgICAgIDAgICAgICAwDQo+PiAgIDEgICAgICAgOGsgICAgICBj MDg0NjExYyAgICAgICAgICAzOSAgICAgNzgNCj4+ICAgMSAgICAgICA4ayAgICAgIGMwODQ2MTI0 ICAgICAgICAgICAwICAgICAgMA0KPj4gICAxICAgICAgIDhrICAgICAgYzA4NDYxMmMgICAgICAg ICAxNjkgICAgMzM4DQo+PiAgIDEgICAgICAgOGsgICAgICBjMDg0NjEzNCAgICAgICAgICAgMCAg ICAgIDANCj4+ICAgMSAgICAgICA4ayAgICAgIGMwODQ2MTNjICAgICAgICAgICAwICAgICAgMA0K Pj4gICAyICAgICAgMTZrICAgICAgYzA4NDYxNDggICAgICAgICAgIDAgICAgICAwDQo+PiAgIDIg ICAgICAxNmsgICAgICBjMDg0NjE1MCAgICAgICAgICAgMCAgICAgIDANCj4+ICAgMiAgICAgIDE2 ayAgICAgIGMwODQ2MTU4ICAgICAgICAgICAwICAgICAgMA0KPj4gLS0tLS0tLS0tTm9ybWFsIHpv bmUgYWxsIG9yZGVyID4gMSBoYXMgbm8gZnJlZSBwYWdlcw0KPj4gWk9ORSAgTkFNRSAgICAgICAg U0laRSAgICBGUkVFICBNRU1fTUFQICAgU1RBUlRfUEFERFINCj5TVEFSVF9NQVBOUg0KPj4gICAx ICAgSGlnaE1lbSAgICA2OTYzMiAgICAgMTYxICBjMTdlMDAwMCAgICAyZjAwMDAwMA0KPjE5MjUx Mg0KPj4gQVJFQSAgICBTSVpFICBGUkVFX0FSRUFfU1RSVUNUICBCTE9DS1MgIFBBR0VTDQo+PiAg IDAgICAgICAgNGsgICAgICBjMDg0NjRmMCAgICAgICAgICAxMiAgICAgMTINCj4+ICAgMCAgICAg ICA0ayAgICAgIGMwODQ2NGY4ICAgICAgICAgICAwICAgICAgMA0KPj4gICAwICAgICAgIDRrICAg ICAgYzA4NDY1MDAgICAgICAgICAgMTQgICAgIDE0DQo+PiAgIDAgICAgICAgNGsgICAgICBjMDg0 NjUwOCAgICAgICAgICAgMyAgICAgIDMNCj4+ICAgMCAgICAgICA0ayAgICAgIGMwODQ2NTEwICAg ICAgICAgICAwICAgICAgMA0KPj4gICAxICAgICAgIDhrICAgICAgYzA4NDY1MWMgICAgICAgICAg IDAgICAgICAwDQo+PiAgIDEgICAgICAgOGsgICAgICBjMDg0NjUyNCAgICAgICAgICAgMCAgICAg IDANCj4+ICAgMSAgICAgICA4ayAgICAgIGMwODQ2NTJjICAgICAgICAgICAwICAgICAgMA0KPj4g ICAyICAgICAgMTZrICAgICAgYzA4NDY1NDggICAgICAgICAgIDAgICAgICAwDQo+PiAgIDIgICAg ICAxNmsgICAgICBjMDg0NjU1MCAgICAgICAgICAgMCAgICAgIDANCj4+ICAgMiAgICAgIDE2ayAg ICAgIGMwODQ2NTU4ICAgICAgICAgICAwICAgICAgMA0KPj4gICAyICAgICAgMTZrICAgICAgYzA4 NDY1NjAgICAgICAgICAgIDEgICAgICA0DQo+PiAgIDIgICAgICAxNmsgICAgICBjMDg0NjU2OCAg ICAgICAgICAgMCAgICAgIDANCj4+ICAgNSAgICAgMTI4ayAgICAgIGMwODQ2NWNjICAgICAgICAg ICAwICAgICAgMA0KPj4gICA1ICAgICAxMjhrICAgICAgYzA4NDY1ZDQgICAgICAgICAgIDAgICAg ICAwDQo+PiAgIDUgICAgIDEyOGsgICAgICBjMDg0NjVkYyAgICAgICAgICAgMCAgICAgIDANCj4+ ICAgNSAgICAgMTI4ayAgICAgIGMwODQ2NWU0ICAgICAgICAgICA0ICAgIDEyOA0KPj4gICA1ICAg ICAxMjhrICAgICAgYzA4NDY1ZWMgICAgICAgICAgIDAgICAgICAwDQo+PiAtLS0tLS1PdGhlcidz IGFsbCB6ZXJvDQo+Pg0KPj4gU29tZSBvdGhlciB6b25lIGluZm9ybWF0aW9uIEkgZHVtcCBmcm9t IHBnbGlzdF9kYXRhDQo+PiB7DQo+PiAJd2F0ZXJtYXJrID0gezg1MywgMTA2NiwgMTI3OX0sDQo+ PiAgICAgICBwZXJjcHVfZHJpZnRfbWFyayA9IDAsDQo+PiAgICAgICBsb3dtZW1fcmVzZXJ2ZSA9 IHswLCAyMTU5LCAyMTU5fSwNCj4+ICAgICAgIGRpcnR5X2JhbGFuY2VfcmVzZXJ2ZSA9IDM0Mzgs DQo+PiAgICAgICBwYWdlc2V0ID0gMHhjMDdmNjE0NCwNCj4+ICAgICAgIGxvY2sgPSB7DQo+PiAg ICAgICAgIHsNCj4+ICAgICAgICAgICBybG9jayA9IHsNCj4+ICAgICAgICAgICAgIHJhd19sb2Nr ID0gew0KPj4gICAgICAgICAgICAgICBsb2NrID0gMA0KPj4gICAgICAgICAgICAgfSwNCj4+ICAg ICAgICAgICAgIGJyZWFrX2xvY2sgPSAwDQo+PiAgICAgICAgICAgfQ0KPj4gICAgICAgICB9DQo+ PiAgICAgICB9LA0KPj4gCWFsbF91bnJlY2xhaW1hYmxlID0gMCwNCj4+ICAgICAgIHJlY2xhaW1f c3RhdCA9IHsNCj4+ICAgICAgICAgcmVjZW50X3JvdGF0ZWQgPSB7OTAzMzU1LCA5NjA5MTJ9LA0K Pj4gICAgICAgICByZWNlbnRfc2Nhbm5lZCA9IHs5MzI0MDQsIDI0NjIwMTd9DQo+PiAgICAgICB9 LA0KPj4gICAgICAgcGFnZXNfc2Nhbm5lZCA9IDg0MjMxLA0KPg0KPk1vc3Qgb2Ygc2NhbiBoYXBw ZW5zIGluIGRpcmVjdCByZWNsYWltIHBhdGgsIEkgZ3Vlc3MNCj5idXQgZGlyZWN0IHJlY2xhaW0g Y291bGRuJ3QgcmVjbGFpbSBhbnkgcGFnZXMgZHVlIHRvIGxhY2sgb2Ygc3dhcCBkZXZpY2UuDQo+ DQo+SXQgbWVhbnMgd2UgaGF2ZSB0byBzZXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUgaW4gZGly ZWN0IHJlY2xhaW0gcGF0aCwNCj50b28uDQo+QmVsb3cgcGF0Y2ggZml4IHlvdXIgcHJvYmxlbT8N ClllcywgeW91ciBwYXRjaCBzaG91bGQgZml4IG15IHByb2JsZW0hIA0KQWN0dWFsbHkgSSBhbHNv IGRpZCBhbm90aGVyIHBhdGNoLCBhZnRlciB0ZXN0LCBzaG91bGQgYWxzbyBmaXggbXkgaXNzdWUs IA0KYnV0IEkgZGlkbid0IHNldCB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBpbiBkaXJlY3QgcmVj bGFpbSBwYXRoIGFzIHlvdSwgDQpqdXN0IGRvdWJsZSBjaGVjayB6b25lX3JlY2xhaW1hYmxlKCkg c3RhdHVzIGluIGFsbF91bnJlY2xhaW1hYmxlKCkgZnVuY3Rpb24uDQpNYXliZSB5b3VyIHBhdGNo IGlzIGJldHRlciENCg0KY29tbWl0IDI2ZDJiNjBkMDYyMzQ2ODNhODE2NjZkYTU1MTI5ZjljOTgy MjcxYTUNCkF1dGhvcjogTGlzYSBEdSA8Y2xkdUBtYXJ2ZWxsLmNvbT4NCkRhdGU6ICAgVGh1IEF1 ZyAxIDEwOjE2OjMyIDIwMTMgKzA4MDANCg0KICAgIG1tOiBmaXggaW5maW5pdGUgZGlyZWN0X3Jl Y2xhaW0gd2hlbiBtZW1vcnkgaXMgdmVyeSBmcmFnbWVudGl6ZWQNCiAgICANCiAgICBsYXRlc3Qg YWxsX3VucmVjbGFpbWFibGUgY2hlY2sgaW4gZGlyZWN0IHJlY2xhaW0gaXMgdGhlIGZvbGxvd2lu ZyBjb21taXQuDQogICAgMjAxMSBBcHIgMTQ7IGNvbW1pdCA5MjliZWE3Yzsgdm1zY2FuOiAgYWxs X3VucmVjbGFpbWFibGUoKSB1c2UNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgem9u ZS0+YWxsX3VucmVjbGFpbWFibGUgYXMgYSBuYW1lDQogICAgYW5kIGluIGFkZGl0aW9uLCBhZGQg b29tX2tpbGxlcl9kaXNhYmxlZCBjaGVjayB0byBhdm9pZCByZWludHJvZHVjZSB0aGUNCiAgICBp c3N1ZSBvZiBjb21taXQgZDE5MDgzNjIgKCJ2bXNjYW46IGNoZWNrIGFsbF91bnJlY2xhaW1hYmxl IGluIGRpcmVjdCByZWNsYWltIHBhdGgiKS4NCiAgICANCiAgICBCdXQgZXhjZXB0IHRoZSBoaWJl cm5hdGlvbiBjYXNlIGluIHdoaWNoIGtzd2FwZCBpcyBmcmVlemVkLCB0aGVyZSdzIGFsc28gb3Ro ZXIgY2FzZQ0KICAgIHdoaWNoIG1heSBsZWFkIGluZmluaXRlIGxvb3AgaW4gZGlyZWN0IHJlbGFp bS4gSW4gYSByZWFsIHRlc3QsIGRpcmVjdF9yZWxhaW1lciBkaWQNCiAgICBvdmVyIDIwMDAwMCB0 aW1lcyByZWJhbGFuY2UgaW4gX19hbGxvY19wYWdlc19zbG93cGF0aCgpLCBzbyB0aGlzIHByb2Nl c3Mgd2lsbCBiZQ0KICAgIGJsb2NrZWQgdW50aWwgd2F0Y2hkb2cgZGV0ZWN0IGFuZCBraWxsIGl0 LiBUaGUgcm9vdCBjYXVzZSBpcyBhcyBiZWxvdzoNCiAgICANCiAgICBJZiBzeXN0ZW0gbWVtb3J5 IGlzIHZlcnkgZnJhZ21lbnRpemVkIGxpa2Ugb25seSBvcmRlci0wIGFuZCBvcmRlci0xIGxlZnQs DQogICAga3N3YXBkIHdpbGwgZ28gdG8gc2xlZXAgYXMgc3lzdGVtIGNhbm4ndCByZWJhbGFuY2Vk IGZvciBoaWdoLW9yZGVyIGFsbG9jYXRpb25zLg0KICAgIEJ1dCBkaXJlY3RfcmVjbGFpbSBzdGls bCB3b3JrcyBmb3IgaGlnaGVyIG9yZGVyIHJlcXVlc3QuIFNvIHpvbmVzIGNhbiBiZWNvbWUgYSBz dGF0ZQ0KICAgIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlID0gMCBidXQgem9uZS0+cGFnZXNfc2Nh bm5lZCA+IHpvbmVfcmVjbGFpbWFibGVfcGFnZXMoem9uZSkgKiA2Lg0KICAgIEluIHRoaXMgY2Fz ZSBpZiBhIHByb2Nlc3MgbGlrZSBkb19mb3JrIHRyeSB0byBhbGxvY2F0ZSBhbiBvcmRlci0yIG1l bW9yeSB3aGljaCBpcyBub3QNCiAgICBhIENPU1RMWV9PUkRFUiwgYXMgZGlyZWN0X3JlY2xhaW0g YWx3YXlzIHNhaWQgaXQgZGlkX3NvbWVfcHJvZ3Jlc3MsIHNvIHJlYmFsYW5jZSBhZ2Fpbg0KICAg IGFuZCBhZ2FpbiBpbiBfX2FsbG9jX3BhZ2VzX3Nsb3dwYXRoKCkuIFRoaXMgaXNzdWUgaXMgZWFz aWx5IGhhcHBlbiBpbiBubyBzd2FwIGFuZCBubw0KICAgIGNvbXBhY3Rpb24gZW52aXJvbWVudC4N CiAgICANCiAgICBTbyBhZGQgZnVydGh1ciBjaGVjayBpbiBhbGxfdW5yZWNsYWltYWJsZSgpIHRv IGF2b2lkIHN1Y2ggY2FzZS4NCiAgICANCiAgICBDaGFuZ2UtSWQ6IElkMzI2NmI0N2M2M2Y1Yjk2 YWFiNDY2ZmQ5ZjFmNDRkMzdlMTZjZGNiDQogICAgU2lnbmVkLW9mZi1ieTogTGlzYSBEdSA8Y2xk dUBtYXJ2ZWxsLmNvbT4NCg0KZGlmZiAtLWdpdCBhL21tL3Ztc2Nhbi5jIGIvbW0vdm1zY2FuLmMN CmluZGV4IDJjZmYwZDQuLjM0NTgyZDkgMTAwNjQ0DQotLS0gYS9tbS92bXNjYW4uYw0KKysrIGIv bW0vdm1zY2FuLmMNCkBAIC0yMzAxLDcgKzIzMDEsOSBAQCBzdGF0aWMgYm9vbCBhbGxfdW5yZWNs YWltYWJsZShzdHJ1Y3Qgem9uZWxpc3QgKnpvbmVsaXN0LA0KICAgICAgICAgICAgICAgICAgICAg ICAgY29udGludWU7DQogICAgICAgICAgICAgICAgaWYgKCFjcHVzZXRfem9uZV9hbGxvd2VkX2hh cmR3YWxsKHpvbmUsIEdGUF9LRVJORUwpKQ0KICAgICAgICAgICAgICAgICAgICAgICAgY29udGlu dWU7DQotICAgICAgICAgICAgICAgaWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkNCisgICAg ICAgICAgICAgICBpZiAoem9uZS0+YWxsX3VucmVjbGFpbWFibGUpDQorICAgICAgICAgICAgICAg ICAgICAgICBjb250aW51ZTsNCisgICAgICAgICAgICAgICBpZiAoem9uZV9yZWNsYWltYWJsZSh6 b25lKSkNCiAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBmYWxzZTsNCiAgICAgICAgfQ0K Pg0KPkZyb20gYTVkODIxNTliOThmM2Q5MGMyZjlmZjllNDg2Njk5ZmI0YzY3Y2NlZCBNb24gU2Vw IDE3IDAwOjAwOjAwDQo+MjAwMQ0KPkZyb206IE1pbmNoYW4gS2ltIDxtaW5jaGFuQGtlcm5lbC5v cmc+DQo+RGF0ZTogVGh1LCAxIEF1ZyAyMDEzIDE2OjE4OjAwICswOTAwDQo+U3ViamVjdDpbUEFU Q0hdIG1tOiBzZXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUgaW4gZGlyZWN0IHJlY2xhaW0NCj4g cGF0aA0KPg0KPkxpc2EgcmVwb3J0ZWQgdGhlcmUgYXJlIGxvdHMgb2YgZnJlZSBwYWdlcyBpbiBh IHpvbmUgYnV0IG1vc3Qgb2YgdGhlbQ0KPmlzIG9yZGVyLTAgcGFnZXMgc28gaXQgbWVhbnMgdGhl IHpvbmUgaXMgaGVhdmlseSBmcmFnZW1lbnRlZC4NCj5UaGVuLCBoaWdoIG9yZGVyIGFsbG9jYXRp b24gY291bGQgbWFrZSBkaXJlY3QgcmVjbGFpbSBwYXRoJ3Nsb25nIHN0YWxsKA0KPmV4LCA1MCBz ZWNvbmQpIGluIG5vIHN3YXAgYW5kIG5vIGNvbXBhY3Rpb24gZW52aXJvbm1lbnQuDQo+DQo+VGhl IHJlYXNvbiBpcyBrc3dhcGQgY2FuIHNraXAgdGhlIHpvbmUncyBzY2FubmluZyBiZWNhdXNlIHRo ZSB6b25lDQo+aXMgbG90cyBvZiBmcmVlIHBhZ2VzIGFuZCBrc3dhcGQgY2hhbmdlcyBzY2Fubmlu ZyBvcmRlciBmcm9tIGhpZ2gtb3JkZXINCj50byAwLW9yZGVyIGFmdGVyIGhpcyBmaXJzdCBpdGVy YXRpb24gaXMgZG9uZSBiZWNhdXNlIGtzd2FwZCB0aGluaw0KPm9yZGVyLTAgYWxsb2NhdGlvbiBp cyB0aGUgbW9zdCBpbXBvcnRhbnQuDQo+TG9vayBhdCA3M2NlMDJlOSBpbiBkZXRhaWwuDQo+DQo+ VGhlIHByb2JsZW0gZnJvbSB0aGF0IGlzIHRoYXQgb25seSBrc3dhcGQgY2FuIHNldCB6b25lLT5h bGxfdW5yZWNsYWltYWJsZQ0KPnRvIDEgYXQgdGhlIG1vbWVudCBzbyBkaXJlY3QgcmVjbGFpbSBw YXRoIHNob3VsZCBsb29wIGZvcmV2ZXIgdW50aWwgYSBnaG9zdA0KPmNhbiBzZXQgdGhlIHpvbmUt PmFsbF91bnJlY2xhaW1hYmxlIHRvIDEuDQo+DQo+VGhpcyBwYXRjaCBtYWtlcyBkaXJlY3QgcmVj bGFpbSBwYXRoIHRvIHNldCB6b25lLT5hbGxfdW5yZWNsYWltYWJsZQ0KPnRvIGF2b2lkIGluZmlu aXRlIGxvb3AuIFNvIG5vdyB3ZSBkb24ndCBuZWVkIGEgZ2hvc3QuDQo+DQo+UmVwb3J0ZWQtYnk6 IExpc2EgRHUgPGNsZHVAbWFydmVsbC5jb20+DQo+U2lnbmVkLW9mZi1ieTogTWluY2hhbiBLaW0g PG1pbmNoYW5Aa2VybmVsLm9yZz4NCj4tLS0NCj4gbW0vdm1zY2FuLmMgfCAgIDI5ICsrKysrKysr KysrKysrKysrKysrKysrKysrKystDQo+IDEgZmlsZSBjaGFuZ2VkLCAyOCBpbnNlcnRpb25zKCsp LCAxIGRlbGV0aW9uKC0pDQo+DQo+ZGlmZiAtLWdpdCBhL21tL3Ztc2Nhbi5jIGIvbW0vdm1zY2Fu LmMNCj5pbmRleCAzM2RjMjU2Li5mOTU3ZTg3IDEwMDY0NA0KPi0tLSBhL21tL3Ztc2Nhbi5jDQo+ KysrIGIvbW0vdm1zY2FuLmMNCj5AQCAtMjMxNyw2ICsyMzE3LDIzIEBAIHN0YXRpYyBib29sIGFs bF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6b25lbGlzdA0KPip6b25lbGlzdCwNCj4gCXJldHVybiB0 cnVlOw0KPiB9DQo+DQo+K3N0YXRpYyB2b2lkIGNoZWNrX3pvbmVzX3VucmVjbGFpbWFibGUoc3Ry dWN0IHpvbmVsaXN0ICp6b25lbGlzdCwNCj4rCQkJCQlzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykN Cj4rew0KPisJc3RydWN0IHpvbmVyZWYgKno7DQo+KwlzdHJ1Y3Qgem9uZSAqem9uZTsNCj4rDQo+ Kwlmb3JfZWFjaF96b25lX3pvbmVsaXN0X25vZGVtYXNrKHpvbmUsIHosIHpvbmVsaXN0LA0KPisJ CQlnZnBfem9uZShzYy0+Z2ZwX21hc2spLCBzYy0+bm9kZW1hc2spIHsNCj4rCQlpZiAoIXBvcHVs YXRlZF96b25lKHpvbmUpKQ0KPisJCQljb250aW51ZTsNCj4rCQlpZiAoIWNwdXNldF96b25lX2Fs bG93ZWRfaGFyZHdhbGwoem9uZSwgR0ZQX0tFUk5FTCkpDQo+KwkJCWNvbnRpbnVlOw0KPisJCWlm ICghem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCj4rCQkJem9uZS0+YWxsX3VucmVjbGFpbWFibGUg PSAxOw0KPisJfQ0KPit9DQo+Kw0KPiAvKg0KPiAgKiBUaGlzIGlzIHRoZSBtYWluIGVudHJ5IHBv aW50IHRvIGRpcmVjdCBwYWdlIHJlY2xhaW0uDQo+ICAqDQo+QEAgLTIzNzAsNyArMjM4NywxNyBA QCBzdGF0aWMgdW5zaWduZWQgbG9uZw0KPmRvX3RyeV90b19mcmVlX3BhZ2VzKHN0cnVjdCB6b25l bGlzdCAqem9uZWxpc3QsDQo+IAkJCQlscnVfcGFnZXMgKz0gem9uZV9yZWNsYWltYWJsZV9wYWdl cyh6b25lKTsNCj4gCQkJfQ0KPg0KPi0JCQlzaHJpbmtfc2xhYihzaHJpbmssIHNjLT5ucl9zY2Fu bmVkLCBscnVfcGFnZXMpOw0KPisJCQkvKg0KPisJCQkgKiBXaGVuIGEgem9uZSBoYXMgZW5vdWdo IG9yZGVyLTAgZnJlZSBtZW1vcnkgYnV0DQo+KwkJCSAqIHpvbmUgaXMgaGVhdmlseSBmcmFnbWVu dGVkIGFuZCB3ZSBuZWVkIGhpZ2ggb3JkZXINCj4rCQkJICogcGFnZSBmcm9tIHRoZSB6b25lLCBr c3dhcGQgY291bGQgc2tpcCB0aGUgem9uZQ0KPisJCQkgKiBhZnRlciBmaXJzdCBpdGVyYXRpb24g d2l0aCBoaWdoIG9yZGVyLiBTbywga3N3YXBkDQo+KwkJCSAqIG5ldmVyIHNldCB0aGUgem9uZS0+ YWxsX3VucmVjbGFpbWFibGUgdG8gMSBzbw0KPisJCQkgKiBkaXJlY3QgcmVjbGFpbSBwYXRoIG5l ZWRzIHRoZSBjaGVjay4NCj4rCQkJICovDQo+KwkJCWlmICghc2hyaW5rX3NsYWIoc2hyaW5rLCBz Yy0+bnJfc2Nhbm5lZCwgbHJ1X3BhZ2VzKSkNCj4rCQkJCWNoZWNrX3pvbmVzX3VucmVjbGFpbWFi bGUoem9uZWxpc3QsIHNjKTsNCj4rDQo+IAkJCWlmIChyZWNsYWltX3N0YXRlKSB7DQo+IAkJCQlz Yy0+bnJfcmVjbGFpbWVkICs9IHJlY2xhaW1fc3RhdGUtPnJlY2xhaW1lZF9zbGFiOw0KPiAJCQkJ cmVjbGFpbV9zdGF0ZS0+cmVjbGFpbWVkX3NsYWIgPSAwOw0KPi0tDQo+MS43LjkuNQ0KPg0KPi0t DQo+S2luZCByZWdhcmRzLA0KPk1pbmNoYW4gS2ltDQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx196.postini.com [74.125.245.196]) by kanga.kvack.org (Postfix) with SMTP id D64AB6B0031 for ; Thu, 1 Aug 2013 04:42:32 -0400 (EDT) Date: Thu, 1 Aug 2013 17:42:59 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130801084259.GA32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro On Thu, Aug 01, 2013 at 01:20:34AM -0700, Lisa Du wrote: > >-----Original Message----- > >From: Minchan Kim [mailto:minchan@kernel.org] > >Sent: 2013a1'8ae??1ae?JPY 15:34 > >To: Lisa Du > >Cc: linux-mm@kvack.org; KOSAKI Motohiro > >Subject: Re: Possible deadloop in direct reclaim? > > > >On Wed, Jul 31, 2013 at 11:13:07PM -0700, Lisa Du wrote: > >> >On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: > >> >> Dear Sir: > >> >> Currently I met a possible deadloop in direct reclaim. After run plenty > >of > >> >the application, system run into a status that system memory is very > >> >fragmentized. Like only order-0 and order-1 memory left. > >> >> Then one process required a order-2 buffer but it enter an endless > >direct > >> >reclaim. From my trace log, I can see this loop already over 200,000 > >times. > >> >Kswapd was first wake up and then go back to sleep as it cannot > >rebalance > >> >this order's memory. But zone->all_unreclaimable remains 1. > >> >> Though direct_reclaim every time returns no pages, but as > >> >zone->all_unreclaimable = 1, so it loop again and again. Even when > >> >zone->pages_scanned also becomes very large. It will block the process > >for > >> >long time, until some watchdog thread detect this and kill this process. > >> >Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe > >cost > >> >over 50 seconds or even more. > >> >> I think it's not as expected right? Can we also add below check in the > >> >function all_unreclaimable() to terminate this loop? > >> >> > >> >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist > >> >*zonelist, > >> >> continue; > >> >> if (!zone->all_unreclaimable) > >> >> return false; > >> >> + if (sc->nr_reclaimed == 0 > >&& !zone_reclaimable(zone)) > >> >> + return true; > >> >> } > >> >> BTW: I'm using kernel3.4, I also try to search in the > >kernel3.9, > >> >didn't see a possible fix for such issue. Or is anyone also met such issue > >> >before? Any comment will be welcomed, looking forward to your reply! > >> >> > >> >> Thanks! > >> > > >> >I'd like to ask somethigs. > >> > > >> >1. Do you have enabled swap? > >> I set CONFIG_SWAP=y, but I didn't really have a swap partition, that > >means my swap buffer size is 0; > >> >2. Do you enable CONFIG_COMPACTION? > >> No, I didn't enable; > >> >3. Could we get your zoneinfo via cat /proc/zoneinfo? > >> I dump some info from ramdump, please review: > > > >Thanks for the information. > >You said order-2 allocation was failed so I will assume preferred zone > >is normal zone, not high zone because high order allocation in kernel side > >isn't from high zone. > Yes, that's right! > > > >> crash> kmem -z > >> NODE: 0 ZONE: 0 ADDR: c08460c0 NAME: "Normal" > >> SIZE: 192512 PRESENT: 182304 MIN/LOW/HIGH: 853/1066/1279 > > > >712M normal memory. > > > >> VM_STAT: > >> NR_FREE_PAGES: 16092 > > > >There are plenty of free pages over high watermark but there are heavy > >fragmentation as I see below information. > > > >So, kswapd doesn't scan this zone loop iteration is done with order-2. > >I mean kswapd will scan this zone with order-0 if first iteration is > >done by this > > > > order = sc.order = 0; > > > > goto loop_again; > > > >But this time, zone_watermark_ok_safe with testorder = 0 on normal zone > >is always true so that scanning of zone will be skipped. It means kswapd > >never set zone->unreclaimable to 1. > Yes, definitely! > > > >> NR_INACTIVE_ANON: 17 > >> NR_ACTIVE_ANON: 55091 > >> NR_INACTIVE_FILE: 17 > >> NR_ACTIVE_FILE: 17 > >> NR_UNEVICTABLE: 0 > >> NR_MLOCK: 0 > >> NR_ANON_PAGES: 55077 > > > >There are about 200M anon pages and few file pages. > >You don't have swap so that reclaimer couldn't go far. > > > >> NR_FILE_MAPPED: 42 > >> NR_FILE_PAGES: 69 > >> NR_FILE_DIRTY: 0 > >> NR_WRITEBACK: 0 > >> NR_SLAB_RECLAIMABLE: 1226 > >> NR_SLAB_UNRECLAIMABLE: 9373 > >> NR_PAGETABLE: 2776 > >> NR_KERNEL_STACK: 798 > >> NR_UNSTABLE_NFS: 0 > >> NR_BOUNCE: 0 > >> NR_VMSCAN_WRITE: 91 > >> NR_VMSCAN_IMMEDIATE: 115381 > >> NR_WRITEBACK_TEMP: 0 > >> NR_ISOLATED_ANON: 0 > >> NR_ISOLATED_FILE: 0 > >> NR_SHMEM: 31 > >> NR_DIRTIED: 15256 > >> NR_WRITTEN: 11981 > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > >> > >> NODE: 0 ZONE: 1 ADDR: c08464c0 NAME: "HighMem" > >> SIZE: 69632 PRESENT: 69088 MIN/LOW/HIGH: 67/147/228 > >> VM_STAT: > >> NR_FREE_PAGES: 161 > > > >Reclaimer should reclaim this zone. > > > >> NR_INACTIVE_ANON: 104 > >> NR_ACTIVE_ANON: 46114 > >> NR_INACTIVE_FILE: 9722 > >> NR_ACTIVE_FILE: 12263 > > > >It seems there are lots of room to evict file pages. > > > >> NR_UNEVICTABLE: 168 > >> NR_MLOCK: 0 > >> NR_ANON_PAGES: 46102 > >> NR_FILE_MAPPED: 12227 > >> NR_FILE_PAGES: 22270 > >> NR_FILE_DIRTY: 1 > >> NR_WRITEBACK: 0 > >> NR_SLAB_RECLAIMABLE: 0 > >> NR_SLAB_UNRECLAIMABLE: 0 > >> NR_PAGETABLE: 0 > >> NR_KERNEL_STACK: 0 > >> NR_UNSTABLE_NFS: 0 > >> NR_BOUNCE: 0 > >> NR_VMSCAN_WRITE: 0 > >> NR_VMSCAN_IMMEDIATE: 0 > >> NR_WRITEBACK_TEMP: 0 > >> NR_ISOLATED_ANON: 0 > >> NR_ISOLATED_FILE: 0 > >> NR_SHMEM: 117 > >> NR_DIRTIED: 7364 > >> NR_WRITTEN: 6989 > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > >> > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > >START_MAPNR > >> 0 Normal 192512 16092 c1200000 0 0 > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > >> 0 4k c08460f0 3 3 > >> 0 4k c08460f8 436 436 > >> 0 4k c0846100 15237 15237 > >> 0 4k c0846108 0 0 > >> 0 4k c0846110 0 0 > >> 1 8k c084611c 39 78 > >> 1 8k c0846124 0 0 > >> 1 8k c084612c 169 338 > >> 1 8k c0846134 0 0 > >> 1 8k c084613c 0 0 > >> 2 16k c0846148 0 0 > >> 2 16k c0846150 0 0 > >> 2 16k c0846158 0 0 > >> ---------Normal zone all order > 1 has no free pages > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > >START_MAPNR > >> 1 HighMem 69632 161 c17e0000 2f000000 > >192512 > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > >> 0 4k c08464f0 12 12 > >> 0 4k c08464f8 0 0 > >> 0 4k c0846500 14 14 > >> 0 4k c0846508 3 3 > >> 0 4k c0846510 0 0 > >> 1 8k c084651c 0 0 > >> 1 8k c0846524 0 0 > >> 1 8k c084652c 0 0 > >> 2 16k c0846548 0 0 > >> 2 16k c0846550 0 0 > >> 2 16k c0846558 0 0 > >> 2 16k c0846560 1 4 > >> 2 16k c0846568 0 0 > >> 5 128k c08465cc 0 0 > >> 5 128k c08465d4 0 0 > >> 5 128k c08465dc 0 0 > >> 5 128k c08465e4 4 128 > >> 5 128k c08465ec 0 0 > >> ------Other's all zero > >> > >> Some other zone information I dump from pglist_data > >> { > >> watermark = {853, 1066, 1279}, > >> percpu_drift_mark = 0, > >> lowmem_reserve = {0, 2159, 2159}, > >> dirty_balance_reserve = 3438, > >> pageset = 0xc07f6144, > >> lock = { > >> { > >> rlock = { > >> raw_lock = { > >> lock = 0 > >> }, > >> break_lock = 0 > >> } > >> } > >> }, > >> all_unreclaimable = 0, > >> reclaim_stat = { > >> recent_rotated = {903355, 960912}, > >> recent_scanned = {932404, 2462017} > >> }, > >> pages_scanned = 84231, > > > >Most of scan happens in direct reclaim path, I guess > >but direct reclaim couldn't reclaim any pages due to lack of swap device. > > > >It means we have to set zone->all_unreclaimable in direct reclaim path, > >too. > >Below patch fix your problem? > Yes, your patch should fix my problem! > Actually I also did another patch, after test, should also fix my issue, > but I didn't set zone->all_unreclaimable in direct reclaim path as you, > just double check zone_reclaimable() status in all_unreclaimable() function. > Maybe your patch is better! Nope. I think your patch is better. :) Just thing is anlaysis of the problem and description and I think we could do better but unfortunately, I don't have enough time today so I will see tomorrow. Just nitpick below. Thanks. > > commit 26d2b60d06234683a81666da55129f9c982271a5 > Author: Lisa Du > Date: Thu Aug 1 10:16:32 2013 +0800 > > mm: fix infinite direct_reclaim when memory is very fragmentized > > latest all_unreclaimable check in direct reclaim is the following commit. > 2011 Apr 14; commit 929bea7c; vmscan: all_unreclaimable() use > zone->all_unreclaimable as a name > and in addition, add oom_killer_disabled check to avoid reintroduce the > issue of commit d1908362 ("vmscan: check all_unreclaimable in direct reclaim path"). > > But except the hibernation case in which kswapd is freezed, there's also other case > which may lead infinite loop in direct relaim. In a real test, direct_relaimer did > over 200000 times rebalance in __alloc_pages_slowpath(), so this process will be > blocked until watchdog detect and kill it. The root cause is as below: > > If system memory is very fragmentized like only order-0 and order-1 left, > kswapd will go to sleep as system cann't rebalanced for high-order allocations. > But direct_reclaim still works for higher order request. So zones can become a state > zone->all_unreclaimable = 0 but zone->pages_scanned > zone_reclaimable_pages(zone) * 6. > In this case if a process like do_fork try to allocate an order-2 memory which is not > a COSTLY_ORDER, as direct_reclaim always said it did_some_progress, so rebalance again > and again in __alloc_pages_slowpath(). This issue is easily happen in no swap and no > compaction enviroment. > > So add furthur check in all_unreclaimable() to avoid such case. > > Change-Id: Id3266b47c63f5b96aab466fd9f1f44d37e16cdcb > Signed-off-by: Lisa Du > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 2cff0d4..34582d9 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2301,7 +2301,9 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > continue; > - if (!zone->all_unreclaimable) > + if (zone->all_unreclaimable) > + continue; Nitpick: If we use zone_reclaimable(), above check is redundant and gain is very tiny because this path is already slow. > + if (zone_reclaimable(zone)) > return false; > } > > > >From a5d82159b98f3d90c2f9ff9e486699fb4c67cced Mon Sep 17 00:00:00 > >2001 > >From: Minchan Kim > >Date: Thu, 1 Aug 2013 16:18:00 +0900 > >Subject:[PATCH] mm: set zone->all_unreclaimable in direct reclaim > > path > > > >Lisa reported there are lots of free pages in a zone but most of them > >is order-0 pages so it means the zone is heavily fragemented. > >Then, high order allocation could make direct reclaim path'slong stall( > >ex, 50 second) in no swap and no compaction environment. > > > >The reason is kswapd can skip the zone's scanning because the zone > >is lots of free pages and kswapd changes scanning order from high-order > >to 0-order after his first iteration is done because kswapd think > >order-0 allocation is the most important. > >Look at 73ce02e9 in detail. > > > >The problem from that is that only kswapd can set zone->all_unreclaimable > >to 1 at the moment so direct reclaim path should loop forever until a ghost > >can set the zone->all_unreclaimable to 1. > > > >This patch makes direct reclaim path to set zone->all_unreclaimable > >to avoid infinite loop. So now we don't need a ghost. > > > >Reported-by: Lisa Du > >Signed-off-by: Minchan Kim > >--- > > mm/vmscan.c | 29 ++++++++++++++++++++++++++++- > > 1 file changed, 28 insertions(+), 1 deletion(-) > > > >diff --git a/mm/vmscan.c b/mm/vmscan.c > >index 33dc256..f957e87 100644 > >--- a/mm/vmscan.c > >+++ b/mm/vmscan.c > >@@ -2317,6 +2317,23 @@ static bool all_unreclaimable(struct zonelist > >*zonelist, > > return true; > > } > > > >+static void check_zones_unreclaimable(struct zonelist *zonelist, > >+ struct scan_control *sc) > >+{ > >+ struct zoneref *z; > >+ struct zone *zone; > >+ > >+ for_each_zone_zonelist_nodemask(zone, z, zonelist, > >+ gfp_zone(sc->gfp_mask), sc->nodemask) { > >+ if (!populated_zone(zone)) > >+ continue; > >+ if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > >+ continue; > >+ if (!zone_reclaimable(zone)) > >+ zone->all_unreclaimable = 1; > >+ } > >+} > >+ > > /* > > * This is the main entry point to direct page reclaim. > > * > >@@ -2370,7 +2387,17 @@ static unsigned long > >do_try_to_free_pages(struct zonelist *zonelist, > > lru_pages += zone_reclaimable_pages(zone); > > } > > > >- shrink_slab(shrink, sc->nr_scanned, lru_pages); > >+ /* > >+ * When a zone has enough order-0 free memory but > >+ * zone is heavily fragmented and we need high order > >+ * page from the zone, kswapd could skip the zone > >+ * after first iteration with high order. So, kswapd > >+ * never set the zone->all_unreclaimable to 1 so > >+ * direct reclaim path needs the check. > >+ */ > >+ if (!shrink_slab(shrink, sc->nr_scanned, lru_pages)) > >+ check_zones_unreclaimable(zonelist, sc); > >+ > > if (reclaim_state) { > > sc->nr_reclaimed += reclaim_state->reclaimed_slab; > > reclaim_state->reclaimed_slab = 0; > >-- > >1.7.9.5 > > > >-- > >Kind regards, > >Minchan Kim -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx194.postini.com [74.125.245.194]) by kanga.kvack.org (Postfix) with SMTP id 676946B0031 for ; Thu, 1 Aug 2013 04:57:13 -0400 (EDT) Date: Thu, 1 Aug 2013 09:56:53 +0100 From: Russell King - ARM Linux Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130801085653.GD24642@n2100.arm.linux.org.uk> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BE028@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BE028@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: KOSAKI Motohiro , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu , Neil Zhang On Wed, Jul 31, 2013 at 10:19:53PM -0700, Lisa Du wrote: > >fork alloc order-1 memory for stack. Where and why alloc order-2? If it is > >arch specific code, please > >contact arch maintainer. > Yes arch do_fork allocate order-2 memory when copy_process. > Hi, Russel > What's your opinion about this question? > If we really need order-2 memory for fork, then we'd better set > CONFIG_COMPATION right? Well, I gave up trying to read the original messages because the quoting style is a total mess, so I don't have a full understanding of what the issue is. However, we have always required order-2 memory for fork, going back to the 1.x kernel days - it's fundamental to ARM to have that. The order-2 allocation os for the 1st level page table. No order-2 allocation, no page tables for the new thread. Looking at this commit: commit 05106e6a54aed321191b4bb5c9ee09538cbad3b1 Author: Rik van Riel Date: Mon Oct 8 16:33:03 2012 -0700 mm: enable CONFIG_COMPACTION by default Now that lumpy reclaim has been removed, compaction is the only way left to free up contiguous memory areas. It is time to just enable CONFIG_COMPACTION by default. it seems to indicate that everyone should have this enabled - however, the way the change has been done, anyone building from defconfigs before that change will not have that option enabled. So yes, this option should be turned on. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx192.postini.com [74.125.245.192]) by kanga.kvack.org (Postfix) with SMTP id 620E86B0031 for ; Thu, 1 Aug 2013 21:05:31 -0400 (EDT) From: Lisa Du Date: Thu, 1 Aug 2013 18:03:32 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE39C@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> In-Reply-To: <20130801084259.GA32486@bbox> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: "linux-mm@kvack.org" , KOSAKI Motohiro Pi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogTWluY2hhbiBLaW0gW21haWx0bzpt aW5jaGFuQGtlcm5lbC5vcmddDQo+U2VudDogMjAxM+W5tDjmnIgx5pelIDE2OjQzDQo+VG86IExp c2EgRHUNCj5DYzogbGludXgtbW1Aa3ZhY2sub3JnOyBLT1NBS0kgTW90b2hpcm8NCj5TdWJqZWN0 OiBSZTogUG9zc2libGUgZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0/DQo+DQo+T24gVGh1LCBB dWcgMDEsIDIwMTMgYXQgMDE6MjA6MzRBTSAtMDcwMCwgTGlzYSBEdSB3cm90ZToNCj4+ID4tLS0t LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4gPkZyb206IE1pbmNoYW4gS2ltIFttYWlsdG86bWlu Y2hhbkBrZXJuZWwub3JnXQ0KPj4gPlNlbnQ6IDIwMTPlubQ45pyIMeaXpSAxNTozNA0KPj4gPlRv OiBMaXNhIER1DQo+PiA+Q2M6IGxpbnV4LW1tQGt2YWNrLm9yZzsgS09TQUtJIE1vdG9oaXJvDQo+ PiA+U3ViamVjdDogUmU6IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNsYWltPw0KPj4g Pg0KPj4gPk9uIFdlZCwgSnVsIDMxLCAyMDEzIGF0IDExOjEzOjA3UE0gLTA3MDAsIExpc2EgRHUg d3JvdGU6DQo+PiA+PiA+T24gTW9uLCBKdWwgMjIsIDIwMTMgYXQgMDk6NTg6MTdQTSAtMDcwMCwg TGlzYSBEdSB3cm90ZToNCj4+ID4+ID4+IERlYXIgU2lyOg0KPj4gPj4gPj4gQ3VycmVudGx5IEkg bWV0IGEgcG9zc2libGUgZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0uIEFmdGVyIHJ1bg0KPnBs ZW50eQ0KPj4gPm9mDQo+PiA+PiA+dGhlIGFwcGxpY2F0aW9uLCBzeXN0ZW0gcnVuIGludG8gYSBz dGF0dXMgdGhhdCBzeXN0ZW0gbWVtb3J5IGlzIHZlcnkNCj4+ID4+ID5mcmFnbWVudGl6ZWQuIExp a2Ugb25seSBvcmRlci0wIGFuZCBvcmRlci0xIG1lbW9yeSBsZWZ0Lg0KPj4gPj4gPj4gVGhlbiBv bmUgcHJvY2VzcyByZXF1aXJlZCBhIG9yZGVyLTIgYnVmZmVyIGJ1dCBpdCBlbnRlciBhbiBlbmRs ZXNzDQo+PiA+ZGlyZWN0DQo+PiA+PiA+cmVjbGFpbS4gRnJvbSBteSB0cmFjZSBsb2csIEkgY2Fu IHNlZSB0aGlzIGxvb3AgYWxyZWFkeSBvdmVyIDIwMCwwMDANCj4+ID50aW1lcy4NCj4+ID4+ID5L c3dhcGQgd2FzIGZpcnN0IHdha2UgdXAgYW5kIHRoZW4gZ28gYmFjayB0byBzbGVlcCBhcyBpdCBj YW5ub3QNCj4+ID5yZWJhbGFuY2UNCj4+ID4+ID50aGlzIG9yZGVyJ3MgbWVtb3J5LiBCdXQgem9u ZS0+YWxsX3VucmVjbGFpbWFibGUgcmVtYWlucyAxLg0KPj4gPj4gPj4gVGhvdWdoIGRpcmVjdF9y ZWNsYWltIGV2ZXJ5IHRpbWUgcmV0dXJucyBubyBwYWdlcywgYnV0IGFzDQo+PiA+PiA+em9uZS0+ YWxsX3VucmVjbGFpbWFibGUgPSAxLCBzbyBpdCBsb29wIGFnYWluIGFuZCBhZ2Fpbi4gRXZlbiB3 aGVuDQo+PiA+PiA+em9uZS0+cGFnZXNfc2Nhbm5lZCBhbHNvIGJlY29tZXMgdmVyeSBsYXJnZS4g SXQgd2lsbCBibG9jayB0aGUNCj5wcm9jZXNzDQo+PiA+Zm9yDQo+PiA+PiA+bG9uZyB0aW1lLCB1 bnRpbCBzb21lIHdhdGNoZG9nIHRocmVhZCBkZXRlY3QgdGhpcyBhbmQga2lsbCB0aGlzDQo+cHJv Y2Vzcy4NCj4+ID4+ID5UaG91Z2ggaXQncyBpbiBfX2FsbG9jX3BhZ2VzX3Nsb3dwYXRoLCBidXQg aXQncyB0b28gc2xvdyByaWdodD8gTWF5YmUNCj4+ID5jb3N0DQo+PiA+PiA+b3ZlciA1MCBzZWNv bmRzIG9yIGV2ZW4gbW9yZS4NCj4+ID4+ID4+IEkgdGhpbmsgaXQncyBub3QgYXMgZXhwZWN0ZWQg cmlnaHQ/ICBDYW4gd2UgYWxzbyBhZGQgYmVsb3cgY2hlY2sgaW4NCj50aGUNCj4+ID4+ID5mdW5j dGlvbiBhbGxfdW5yZWNsYWltYWJsZSgpIHRvIHRlcm1pbmF0ZSB0aGlzIGxvb3A/DQo+PiA+PiA+ Pg0KPj4gPj4gPj4gQEAgLTIzNTUsNiArMjM1NSw4IEBAIHN0YXRpYyBib29sIGFsbF91bnJlY2xh aW1hYmxlKHN0cnVjdA0KPnpvbmVsaXN0DQo+PiA+PiA+KnpvbmVsaXN0LA0KPj4gPj4gPj4gICAg ICAgICAgICAgICAgICAgICAgICAgY29udGludWU7DQo+PiA+PiA+PiAgICAgICAgICAgICAgICAg aWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkNCj4+ID4+ID4+ICAgICAgICAgICAgICAgICAg ICAgICAgIHJldHVybiBmYWxzZTsNCj4+ID4+ID4+ICsgICAgICAgICAgICAgICBpZiAoc2MtPm5y X3JlY2xhaW1lZCA9PSAwDQo+PiA+JiYgIXpvbmVfcmVjbGFpbWFibGUoem9uZSkpDQo+PiA+PiA+ PiArICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gdHJ1ZTsNCj4+ID4+ID4+ICAgICAgICAg fQ0KPj4gPj4gPj4gICAgICAgICAgQlRXOiBJJ20gdXNpbmcga2VybmVsMy40LCBJIGFsc28gdHJ5 IHRvIHNlYXJjaCBpbiB0aGUNCj4+ID5rZXJuZWwzLjksDQo+PiA+PiA+ZGlkbid0IHNlZSBhIHBv c3NpYmxlIGZpeCBmb3Igc3VjaCBpc3N1ZS4gT3IgaXMgYW55b25lIGFsc28gbWV0IHN1Y2gNCj5p c3N1ZQ0KPj4gPj4gPmJlZm9yZT8gQW55IGNvbW1lbnQgd2lsbCBiZSB3ZWxjb21lZCwgbG9va2lu ZyBmb3J3YXJkIHRvIHlvdXINCj5yZXBseSENCj4+ID4+ID4+DQo+PiA+PiA+PiBUaGFua3MhDQo+ PiA+PiA+DQo+PiA+PiA+SSdkIGxpa2UgdG8gYXNrIHNvbWV0aGlncy4NCj4+ID4+ID4NCj4+ID4+ ID4xLiBEbyB5b3UgaGF2ZSBlbmFibGVkIHN3YXA/DQo+PiA+PiBJIHNldCBDT05GSUdfU1dBUD15 LCBidXQgSSBkaWRuJ3QgcmVhbGx5IGhhdmUgYSBzd2FwIHBhcnRpdGlvbiwgdGhhdA0KPj4gPm1l YW5zIG15IHN3YXAgYnVmZmVyIHNpemUgaXMgMDsNCj4+ID4+ID4yLiBEbyB5b3UgZW5hYmxlIENP TkZJR19DT01QQUNUSU9OPw0KPj4gPj4gTm8sIEkgZGlkbid0IGVuYWJsZTsNCj4+ID4+ID4zLiBD b3VsZCB3ZSBnZXQgeW91ciB6b25laW5mbyB2aWEgY2F0IC9wcm9jL3pvbmVpbmZvPw0KPj4gPj4g SSBkdW1wIHNvbWUgaW5mbyBmcm9tIHJhbWR1bXAsIHBsZWFzZSByZXZpZXc6DQo+PiA+DQo+PiA+ VGhhbmtzIGZvciB0aGUgaW5mb3JtYXRpb24uDQo+PiA+WW91IHNhaWQgb3JkZXItMiBhbGxvY2F0 aW9uIHdhcyBmYWlsZWQgc28gSSB3aWxsIGFzc3VtZSBwcmVmZXJyZWQgem9uZQ0KPj4gPmlzIG5v cm1hbCB6b25lLCBub3QgaGlnaCB6b25lIGJlY2F1c2UgaGlnaCBvcmRlciBhbGxvY2F0aW9uIGlu IGtlcm5lbA0KPnNpZGUNCj4+ID5pc24ndCBmcm9tIGhpZ2ggem9uZS4NCj4+IFllcywgdGhhdCdz IHJpZ2h0IQ0KPj4gPg0KPj4gPj4gY3Jhc2g+IGttZW0gLXoNCj4+ID4+IE5PREU6IDAgIFpPTkU6 IDAgIEFERFI6IGMwODQ2MGMwICBOQU1FOiAiTm9ybWFsIg0KPj4gPj4gICBTSVpFOiAxOTI1MTIg IFBSRVNFTlQ6IDE4MjMwNCAgTUlOL0xPVy9ISUdIOiA4NTMvMTA2Ni8xMjc5DQo+PiA+DQo+PiA+ NzEyTSBub3JtYWwgbWVtb3J5Lg0KPj4gPg0KPj4gPj4gICBWTV9TVEFUOg0KPj4gPj4gICAgICAg ICAgIE5SX0ZSRUVfUEFHRVM6IDE2MDkyDQo+PiA+DQo+PiA+VGhlcmUgYXJlIHBsZW50eSBvZiBm cmVlIHBhZ2VzIG92ZXIgaGlnaCB3YXRlcm1hcmsgYnV0IHRoZXJlIGFyZSBoZWF2eQ0KPj4gPmZy YWdtZW50YXRpb24gYXMgSSBzZWUgYmVsb3cgaW5mb3JtYXRpb24uDQo+PiA+DQo+PiA+U28sIGtz d2FwZCBkb2Vzbid0IHNjYW4gdGhpcyB6b25lIGxvb3AgaXRlcmF0aW9uIGlzIGRvbmUgd2l0aCBv cmRlci0yLg0KPj4gPkkgbWVhbiBrc3dhcGQgd2lsbCBzY2FuIHRoaXMgem9uZSB3aXRoIG9yZGVy LTAgaWYgZmlyc3QgaXRlcmF0aW9uIGlzDQo+PiA+ZG9uZSBieSB0aGlzDQo+PiA+DQo+PiA+ICAg ICAgICBvcmRlciA9IHNjLm9yZGVyID0gMDsNCj4+ID4NCj4+ID4gICAgICAgIGdvdG8gbG9vcF9h Z2FpbjsNCj4+ID4NCj4+ID5CdXQgdGhpcyB0aW1lLCB6b25lX3dhdGVybWFya19va19zYWZlIHdp dGggdGVzdG9yZGVyID0gMCBvbiBub3JtYWwNCj56b25lDQo+PiA+aXMgYWx3YXlzIHRydWUgc28g dGhhdCBzY2FubmluZyBvZiB6b25lIHdpbGwgYmUgc2tpcHBlZC4gSXQgbWVhbnMga3N3YXBkDQo+ PiA+bmV2ZXIgc2V0IHpvbmUtPnVucmVjbGFpbWFibGUgdG8gMS4NCj4+IFllcywgZGVmaW5pdGVs eSENCj4+ID4NCj4+ID4+ICAgICAgICBOUl9JTkFDVElWRV9BTk9OOiAxNw0KPj4gPj4gICAgICAg ICAgTlJfQUNUSVZFX0FOT046IDU1MDkxDQo+PiA+PiAgICAgICAgTlJfSU5BQ1RJVkVfRklMRTog MTcNCj4+ID4+ICAgICAgICAgIE5SX0FDVElWRV9GSUxFOiAxNw0KPj4gPj4gICAgICAgICAgTlJf VU5FVklDVEFCTEU6IDANCj4+ID4+ICAgICAgICAgICAgICAgIE5SX01MT0NLOiAwDQo+PiA+PiAg ICAgICAgICAgTlJfQU5PTl9QQUdFUzogNTUwNzcNCj4+ID4NCj4+ID5UaGVyZSBhcmUgYWJvdXQg MjAwTSBhbm9uIHBhZ2VzIGFuZCBmZXcgZmlsZSBwYWdlcy4NCj4+ID5Zb3UgZG9uJ3QgaGF2ZSBz d2FwIHNvIHRoYXQgcmVjbGFpbWVyIGNvdWxkbid0IGdvIGZhci4NCj4+ID4NCj4+ID4+ICAgICAg ICAgIE5SX0ZJTEVfTUFQUEVEOiA0Mg0KPj4gPj4gICAgICAgICAgIE5SX0ZJTEVfUEFHRVM6IDY5 DQo+PiA+PiAgICAgICAgICAgTlJfRklMRV9ESVJUWTogMA0KPj4gPj4gICAgICAgICAgICBOUl9X UklURUJBQ0s6IDANCj4+ID4+ICAgICBOUl9TTEFCX1JFQ0xBSU1BQkxFOiAxMjI2DQo+PiA+PiAg IE5SX1NMQUJfVU5SRUNMQUlNQUJMRTogOTM3Mw0KPj4gPj4gICAgICAgICAgICBOUl9QQUdFVEFC TEU6IDI3NzYNCj4+ID4+ICAgICAgICAgTlJfS0VSTkVMX1NUQUNLOiA3OTgNCj4+ID4+ICAgICAg ICAgTlJfVU5TVEFCTEVfTkZTOiAwDQo+PiA+PiAgICAgICAgICAgICAgIE5SX0JPVU5DRTogMA0K Pj4gPj4gICAgICAgICBOUl9WTVNDQU5fV1JJVEU6IDkxDQo+PiA+PiAgICAgTlJfVk1TQ0FOX0lN TUVESUFURTogMTE1MzgxDQo+PiA+PiAgICAgICBOUl9XUklURUJBQ0tfVEVNUDogMA0KPj4gPj4g ICAgICAgIE5SX0lTT0xBVEVEX0FOT046IDANCj4+ID4+ICAgICAgICBOUl9JU09MQVRFRF9GSUxF OiAwDQo+PiA+PiAgICAgICAgICAgICAgICBOUl9TSE1FTTogMzENCj4+ID4+ICAgICAgICAgICAg ICBOUl9ESVJUSUVEOiAxNTI1Ng0KPj4gPj4gICAgICAgICAgICAgIE5SX1dSSVRURU46IDExOTgx DQo+PiA+PiBOUl9BTk9OX1RSQU5TUEFSRU5UX0hVR0VQQUdFUzogMA0KPj4gPj4NCj4+ID4+IE5P REU6IDAgIFpPTkU6IDEgIEFERFI6IGMwODQ2NGMwICBOQU1FOiAiSGlnaE1lbSINCj4+ID4+ICAg U0laRTogNjk2MzIgIFBSRVNFTlQ6IDY5MDg4ICBNSU4vTE9XL0hJR0g6IDY3LzE0Ny8yMjgNCj4+ ID4+ICAgVk1fU1RBVDoNCj4+ID4+ICAgICAgICAgICBOUl9GUkVFX1BBR0VTOiAxNjENCj4+ID4N Cj4+ID5SZWNsYWltZXIgc2hvdWxkIHJlY2xhaW0gdGhpcyB6b25lLg0KPj4gPg0KPj4gPj4gICAg ICAgIE5SX0lOQUNUSVZFX0FOT046IDEwNA0KPj4gPj4gICAgICAgICAgTlJfQUNUSVZFX0FOT046 IDQ2MTE0DQo+PiA+PiAgICAgICAgTlJfSU5BQ1RJVkVfRklMRTogOTcyMg0KPj4gPj4gICAgICAg ICAgTlJfQUNUSVZFX0ZJTEU6IDEyMjYzDQo+PiA+DQo+PiA+SXQgc2VlbXMgdGhlcmUgYXJlIGxv dHMgb2Ygcm9vbSB0byBldmljdCBmaWxlIHBhZ2VzLg0KPj4gPg0KPj4gPj4gICAgICAgICAgTlJf VU5FVklDVEFCTEU6IDE2OA0KPj4gPj4gICAgICAgICAgICAgICAgTlJfTUxPQ0s6IDANCj4+ID4+ ICAgICAgICAgICBOUl9BTk9OX1BBR0VTOiA0NjEwMg0KPj4gPj4gICAgICAgICAgTlJfRklMRV9N QVBQRUQ6IDEyMjI3DQo+PiA+PiAgICAgICAgICAgTlJfRklMRV9QQUdFUzogMjIyNzANCj4+ID4+ ICAgICAgICAgICBOUl9GSUxFX0RJUlRZOiAxDQo+PiA+PiAgICAgICAgICAgIE5SX1dSSVRFQkFD SzogMA0KPj4gPj4gICAgIE5SX1NMQUJfUkVDTEFJTUFCTEU6IDANCj4+ID4+ICAgTlJfU0xBQl9V TlJFQ0xBSU1BQkxFOiAwDQo+PiA+PiAgICAgICAgICAgIE5SX1BBR0VUQUJMRTogMA0KPj4gPj4g ICAgICAgICBOUl9LRVJORUxfU1RBQ0s6IDANCj4+ID4+ICAgICAgICAgTlJfVU5TVEFCTEVfTkZT OiAwDQo+PiA+PiAgICAgICAgICAgICAgIE5SX0JPVU5DRTogMA0KPj4gPj4gICAgICAgICBOUl9W TVNDQU5fV1JJVEU6IDANCj4+ID4+ICAgICBOUl9WTVNDQU5fSU1NRURJQVRFOiAwDQo+PiA+PiAg ICAgICBOUl9XUklURUJBQ0tfVEVNUDogMA0KPj4gPj4gICAgICAgIE5SX0lTT0xBVEVEX0FOT046 IDANCj4+ID4+ICAgICAgICBOUl9JU09MQVRFRF9GSUxFOiAwDQo+PiA+PiAgICAgICAgICAgICAg ICBOUl9TSE1FTTogMTE3DQo+PiA+PiAgICAgICAgICAgICAgTlJfRElSVElFRDogNzM2NA0KPj4g Pj4gICAgICAgICAgICAgIE5SX1dSSVRURU46IDY5ODkNCj4+ID4+IE5SX0FOT05fVFJBTlNQQVJF TlRfSFVHRVBBR0VTOiAwDQo+PiA+Pg0KPj4gPj4gWk9ORSAgTkFNRSAgICAgICAgU0laRSAgICBG UkVFICBNRU1fTUFQICAgU1RBUlRfUEFERFINCj4+ID5TVEFSVF9NQVBOUg0KPj4gPj4gICAwICAg Tm9ybWFsICAgIDE5MjUxMiAgIDE2MDkyICBjMTIwMDAwMCAgICAgICAwDQo+MA0KPj4gPj4gQVJF QSAgICBTSVpFICBGUkVFX0FSRUFfU1RSVUNUICBCTE9DS1MgIFBBR0VTDQo+PiA+PiAgIDAgICAg ICAgNGsgICAgICBjMDg0NjBmMCAgICAgICAgICAgMyAgICAgIDMNCj4+ID4+ICAgMCAgICAgICA0 ayAgICAgIGMwODQ2MGY4ICAgICAgICAgNDM2ICAgIDQzNg0KPj4gPj4gICAwICAgICAgIDRrICAg ICAgYzA4NDYxMDAgICAgICAgMTUyMzcgIDE1MjM3DQo+PiA+PiAgIDAgICAgICAgNGsgICAgICBj MDg0NjEwOCAgICAgICAgICAgMCAgICAgIDANCj4+ID4+ICAgMCAgICAgICA0ayAgICAgIGMwODQ2 MTEwICAgICAgICAgICAwICAgICAgMA0KPj4gPj4gICAxICAgICAgIDhrICAgICAgYzA4NDYxMWMg ICAgICAgICAgMzkgICAgIDc4DQo+PiA+PiAgIDEgICAgICAgOGsgICAgICBjMDg0NjEyNCAgICAg ICAgICAgMCAgICAgIDANCj4+ID4+ICAgMSAgICAgICA4ayAgICAgIGMwODQ2MTJjICAgICAgICAg MTY5ICAgIDMzOA0KPj4gPj4gICAxICAgICAgIDhrICAgICAgYzA4NDYxMzQgICAgICAgICAgIDAg ICAgICAwDQo+PiA+PiAgIDEgICAgICAgOGsgICAgICBjMDg0NjEzYyAgICAgICAgICAgMCAgICAg IDANCj4+ID4+ICAgMiAgICAgIDE2ayAgICAgIGMwODQ2MTQ4ICAgICAgICAgICAwICAgICAgMA0K Pj4gPj4gICAyICAgICAgMTZrICAgICAgYzA4NDYxNTAgICAgICAgICAgIDAgICAgICAwDQo+PiA+ PiAgIDIgICAgICAxNmsgICAgICBjMDg0NjE1OCAgICAgICAgICAgMCAgICAgIDANCj4+ID4+IC0t LS0tLS0tLU5vcm1hbCB6b25lIGFsbCBvcmRlciA+IDEgaGFzIG5vIGZyZWUgcGFnZXMNCj4+ID4+ IFpPTkUgIE5BTUUgICAgICAgIFNJWkUgICAgRlJFRSAgTUVNX01BUCAgIFNUQVJUX1BBRERSDQo+ PiA+U1RBUlRfTUFQTlINCj4+ID4+ICAgMSAgIEhpZ2hNZW0gICAgNjk2MzIgICAgIDE2MSAgYzE3 ZTAwMDAgICAgMmYwMDAwMDANCj4+ID4xOTI1MTINCj4+ID4+IEFSRUEgICAgU0laRSAgRlJFRV9B UkVBX1NUUlVDVCAgQkxPQ0tTICBQQUdFUw0KPj4gPj4gICAwICAgICAgIDRrICAgICAgYzA4NDY0 ZjAgICAgICAgICAgMTIgICAgIDEyDQo+PiA+PiAgIDAgICAgICAgNGsgICAgICBjMDg0NjRmOCAg ICAgICAgICAgMCAgICAgIDANCj4+ID4+ICAgMCAgICAgICA0ayAgICAgIGMwODQ2NTAwICAgICAg ICAgIDE0ICAgICAxNA0KPj4gPj4gICAwICAgICAgIDRrICAgICAgYzA4NDY1MDggICAgICAgICAg IDMgICAgICAzDQo+PiA+PiAgIDAgICAgICAgNGsgICAgICBjMDg0NjUxMCAgICAgICAgICAgMCAg ICAgIDANCj4+ID4+ICAgMSAgICAgICA4ayAgICAgIGMwODQ2NTFjICAgICAgICAgICAwICAgICAg MA0KPj4gPj4gICAxICAgICAgIDhrICAgICAgYzA4NDY1MjQgICAgICAgICAgIDAgICAgICAwDQo+ PiA+PiAgIDEgICAgICAgOGsgICAgICBjMDg0NjUyYyAgICAgICAgICAgMCAgICAgIDANCj4+ID4+ ICAgMiAgICAgIDE2ayAgICAgIGMwODQ2NTQ4ICAgICAgICAgICAwICAgICAgMA0KPj4gPj4gICAy ICAgICAgMTZrICAgICAgYzA4NDY1NTAgICAgICAgICAgIDAgICAgICAwDQo+PiA+PiAgIDIgICAg ICAxNmsgICAgICBjMDg0NjU1OCAgICAgICAgICAgMCAgICAgIDANCj4+ID4+ICAgMiAgICAgIDE2 ayAgICAgIGMwODQ2NTYwICAgICAgICAgICAxICAgICAgNA0KPj4gPj4gICAyICAgICAgMTZrICAg ICAgYzA4NDY1NjggICAgICAgICAgIDAgICAgICAwDQo+PiA+PiAgIDUgICAgIDEyOGsgICAgICBj MDg0NjVjYyAgICAgICAgICAgMCAgICAgIDANCj4+ID4+ICAgNSAgICAgMTI4ayAgICAgIGMwODQ2 NWQ0ICAgICAgICAgICAwICAgICAgMA0KPj4gPj4gICA1ICAgICAxMjhrICAgICAgYzA4NDY1ZGMg ICAgICAgICAgIDAgICAgICAwDQo+PiA+PiAgIDUgICAgIDEyOGsgICAgICBjMDg0NjVlNCAgICAg ICAgICAgNCAgICAxMjgNCj4+ID4+ICAgNSAgICAgMTI4ayAgICAgIGMwODQ2NWVjICAgICAgICAg ICAwICAgICAgMA0KPj4gPj4gLS0tLS0tT3RoZXIncyBhbGwgemVybw0KPj4gPj4NCj4+ID4+IFNv bWUgb3RoZXIgem9uZSBpbmZvcm1hdGlvbiBJIGR1bXAgZnJvbSBwZ2xpc3RfZGF0YQ0KPj4gPj4g ew0KPj4gPj4gICB3YXRlcm1hcmsgPSB7ODUzLCAxMDY2LCAxMjc5fSwNCj4+ID4+ICAgICAgIHBl cmNwdV9kcmlmdF9tYXJrID0gMCwNCj4+ID4+ICAgICAgIGxvd21lbV9yZXNlcnZlID0gezAsIDIx NTksIDIxNTl9LA0KPj4gPj4gICAgICAgZGlydHlfYmFsYW5jZV9yZXNlcnZlID0gMzQzOCwNCj4+ ID4+ICAgICAgIHBhZ2VzZXQgPSAweGMwN2Y2MTQ0LA0KPj4gPj4gICAgICAgbG9jayA9IHsNCj4+ ID4+ICAgICAgICAgew0KPj4gPj4gICAgICAgICAgIHJsb2NrID0gew0KPj4gPj4gICAgICAgICAg ICAgcmF3X2xvY2sgPSB7DQo+PiA+PiAgICAgICAgICAgICAgIGxvY2sgPSAwDQo+PiA+PiAgICAg ICAgICAgICB9LA0KPj4gPj4gICAgICAgICAgICAgYnJlYWtfbG9jayA9IDANCj4+ID4+ICAgICAg ICAgICB9DQo+PiA+PiAgICAgICAgIH0NCj4+ID4+ICAgICAgIH0sDQo+PiA+PiAgIGFsbF91bnJl Y2xhaW1hYmxlID0gMCwNCj4+ID4+ICAgICAgIHJlY2xhaW1fc3RhdCA9IHsNCj4+ID4+ICAgICAg ICAgcmVjZW50X3JvdGF0ZWQgPSB7OTAzMzU1LCA5NjA5MTJ9LA0KPj4gPj4gICAgICAgICByZWNl bnRfc2Nhbm5lZCA9IHs5MzI0MDQsIDI0NjIwMTd9DQo+PiA+PiAgICAgICB9LA0KPj4gPj4gICAg ICAgcGFnZXNfc2Nhbm5lZCA9IDg0MjMxLA0KPj4gPg0KPj4gPk1vc3Qgb2Ygc2NhbiBoYXBwZW5z IGluIGRpcmVjdCByZWNsYWltIHBhdGgsIEkgZ3Vlc3MNCj4+ID5idXQgZGlyZWN0IHJlY2xhaW0g Y291bGRuJ3QgcmVjbGFpbSBhbnkgcGFnZXMgZHVlIHRvIGxhY2sgb2Ygc3dhcCBkZXZpY2UuDQo+ PiA+DQo+PiA+SXQgbWVhbnMgd2UgaGF2ZSB0byBzZXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUg aW4gZGlyZWN0IHJlY2xhaW0gcGF0aCwNCj4+ID50b28uDQo+PiA+QmVsb3cgcGF0Y2ggZml4IHlv dXIgcHJvYmxlbT8NCj4+IFllcywgeW91ciBwYXRjaCBzaG91bGQgZml4IG15IHByb2JsZW0hDQo+ PiBBY3R1YWxseSBJIGFsc28gZGlkIGFub3RoZXIgcGF0Y2gsIGFmdGVyIHRlc3QsIHNob3VsZCBh bHNvIGZpeCBteSBpc3N1ZSwNCj4+IGJ1dCBJIGRpZG4ndCBzZXQgem9uZS0+YWxsX3VucmVjbGFp bWFibGUgaW4gZGlyZWN0IHJlY2xhaW0gcGF0aCBhcyB5b3UsDQo+PiBqdXN0IGRvdWJsZSBjaGVj ayB6b25lX3JlY2xhaW1hYmxlKCkgc3RhdHVzIGluIGFsbF91bnJlY2xhaW1hYmxlKCkNCj5mdW5j dGlvbi4NCj4+IE1heWJlIHlvdXIgcGF0Y2ggaXMgYmV0dGVyIQ0KPg0KPk5vcGUuIEkgdGhpbmsg eW91ciBwYXRjaCBpcyBiZXR0ZXIuIDopDQo+SnVzdCB0aGluZyBpcyBhbmxheXNpcyBvZiB0aGUg cHJvYmxlbSBhbmQgZGVzY3JpcHRpb24gYW5kIEkgdGhpbmsgd2UgY291bGQNCj5kbw0KPmJldHRl ciBidXQgdW5mb3J0dW5hdGVseSwgSSBkb24ndCBoYXZlIGVub3VnaCB0aW1lIHRvZGF5IHNvIEkg d2lsbCBzZWUNCj50b21vcnJvdy4NCj5KdXN0IG5pdHBpY2sgYmVsb3cuDQo+DQo+VGhhbmtzLg0K Pg0KPj4NCj4+IGNvbW1pdCAyNmQyYjYwZDA2MjM0NjgzYTgxNjY2ZGE1NTEyOWY5Yzk4MjI3MWE1 DQo+PiBBdXRob3I6IExpc2EgRHUgPGNsZHVAbWFydmVsbC5jb20+DQo+PiBEYXRlOiAgIFRodSBB dWcgMSAxMDoxNjozMiAyMDEzICswODAwDQo+Pg0KPj4gICAgIG1tOiBmaXggaW5maW5pdGUgZGly ZWN0X3JlY2xhaW0gd2hlbiBtZW1vcnkgaXMgdmVyeSBmcmFnbWVudGl6ZWQNCj4+DQo+PiAgICAg bGF0ZXN0IGFsbF91bnJlY2xhaW1hYmxlIGNoZWNrIGluIGRpcmVjdCByZWNsYWltIGlzIHRoZSBm b2xsb3dpbmcNCj5jb21taXQuDQo+PiAgICAgMjAxMSBBcHIgMTQ7IGNvbW1pdCA5MjliZWE3Yzsg dm1zY2FuOiAgYWxsX3VucmVjbGFpbWFibGUoKSB1c2UNCj4+ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgem9uZS0+YWxsX3VucmVjbGFpbWFibGUgYXMgYSBuYW1lDQo+PiAgICAgYW5k IGluIGFkZGl0aW9uLCBhZGQgb29tX2tpbGxlcl9kaXNhYmxlZCBjaGVjayB0byBhdm9pZCByZWlu dHJvZHVjZQ0KPnRoZQ0KPj4gICAgIGlzc3VlIG9mIGNvbW1pdCBkMTkwODM2MiAoInZtc2Nhbjog Y2hlY2sgYWxsX3VucmVjbGFpbWFibGUgaW4NCj5kaXJlY3QgcmVjbGFpbSBwYXRoIikuDQo+Pg0K Pj4gICAgIEJ1dCBleGNlcHQgdGhlIGhpYmVybmF0aW9uIGNhc2UgaW4gd2hpY2gga3N3YXBkIGlz IGZyZWV6ZWQsIHRoZXJlJ3MNCj5hbHNvIG90aGVyIGNhc2UNCj4+ICAgICB3aGljaCBtYXkgbGVh ZCBpbmZpbml0ZSBsb29wIGluIGRpcmVjdCByZWxhaW0uIEluIGEgcmVhbCB0ZXN0LA0KPmRpcmVj dF9yZWxhaW1lciBkaWQNCj4+ICAgICBvdmVyIDIwMDAwMCB0aW1lcyByZWJhbGFuY2UgaW4gX19h bGxvY19wYWdlc19zbG93cGF0aCgpLCBzbyB0aGlzDQo+cHJvY2VzcyB3aWxsIGJlDQo+PiAgICAg YmxvY2tlZCB1bnRpbCB3YXRjaGRvZyBkZXRlY3QgYW5kIGtpbGwgaXQuIFRoZSByb290IGNhdXNl IGlzIGFzIGJlbG93Og0KPj4NCj4+ICAgICBJZiBzeXN0ZW0gbWVtb3J5IGlzIHZlcnkgZnJhZ21l bnRpemVkIGxpa2Ugb25seSBvcmRlci0wIGFuZCBvcmRlci0xDQo+bGVmdCwNCj4+ICAgICBrc3dh cGQgd2lsbCBnbyB0byBzbGVlcCBhcyBzeXN0ZW0gY2Fubid0IHJlYmFsYW5jZWQgZm9yIGhpZ2gt b3JkZXINCj5hbGxvY2F0aW9ucy4NCj4+ICAgICBCdXQgZGlyZWN0X3JlY2xhaW0gc3RpbGwgd29y a3MgZm9yIGhpZ2hlciBvcmRlciByZXF1ZXN0LiBTbyB6b25lcyBjYW4NCj5iZWNvbWUgYSBzdGF0 ZQ0KPj4gICAgIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlID0gMCBidXQgem9uZS0+cGFnZXNfc2Nh bm5lZCA+DQo+em9uZV9yZWNsYWltYWJsZV9wYWdlcyh6b25lKSAqIDYuDQo+PiAgICAgSW4gdGhp cyBjYXNlIGlmIGEgcHJvY2VzcyBsaWtlIGRvX2ZvcmsgdHJ5IHRvIGFsbG9jYXRlIGFuIG9yZGVy LTINCj5tZW1vcnkgd2hpY2ggaXMgbm90DQo+PiAgICAgYSBDT1NUTFlfT1JERVIsIGFzIGRpcmVj dF9yZWNsYWltIGFsd2F5cyBzYWlkIGl0DQo+ZGlkX3NvbWVfcHJvZ3Jlc3MsIHNvIHJlYmFsYW5j ZSBhZ2Fpbg0KPj4gICAgIGFuZCBhZ2FpbiBpbiBfX2FsbG9jX3BhZ2VzX3Nsb3dwYXRoKCkuIFRo aXMgaXNzdWUgaXMgZWFzaWx5IGhhcHBlbiBpbg0KPm5vIHN3YXAgYW5kIG5vDQo+PiAgICAgY29t cGFjdGlvbiBlbnZpcm9tZW50Lg0KPj4NCj4+ICAgICBTbyBhZGQgZnVydGh1ciBjaGVjayBpbiBh bGxfdW5yZWNsYWltYWJsZSgpIHRvIGF2b2lkIHN1Y2ggY2FzZS4NCj4+DQo+PiAgICAgQ2hhbmdl LUlkOiBJZDMyNjZiNDdjNjNmNWI5NmFhYjQ2NmZkOWYxZjQ0ZDM3ZTE2Y2RjYg0KPj4gICAgIFNp Z25lZC1vZmYtYnk6IExpc2EgRHUgPGNsZHVAbWFydmVsbC5jb20+DQo+Pg0KPj4gZGlmZiAtLWdp dCBhL21tL3Ztc2Nhbi5jIGIvbW0vdm1zY2FuLmMNCj4+IGluZGV4IDJjZmYwZDQuLjM0NTgyZDkg MTAwNjQ0DQo+PiAtLS0gYS9tbS92bXNjYW4uYw0KPj4gKysrIGIvbW0vdm1zY2FuLmMNCj4+IEBA IC0yMzAxLDcgKzIzMDEsOSBAQCBzdGF0aWMgYm9vbCBhbGxfdW5yZWNsYWltYWJsZShzdHJ1Y3Qg em9uZWxpc3QNCj4qem9uZWxpc3QsDQo+PiAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51 ZTsNCj4+ICAgICAgICAgICAgICAgICBpZiAoIWNwdXNldF96b25lX2FsbG93ZWRfaGFyZHdhbGwo em9uZSwNCj5HRlBfS0VSTkVMKSkNCj4+ICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVl Ow0KPj4gLSAgICAgICAgICAgICAgIGlmICghem9uZS0+YWxsX3VucmVjbGFpbWFibGUpDQo+PiAr ICAgICAgICAgICAgICAgaWYgKHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KPj4gKyAgICAgICAg ICAgICAgICAgICAgICAgY29udGludWU7DQo+DQo+Tml0cGljazogSWYgd2UgdXNlIHpvbmVfcmVj bGFpbWFibGUoKSwgYWJvdmUgY2hlY2sgaXMgcmVkdW5kYW50IGFuZA0KPmdhaW4gaXMgdmVyeSB0 aW55IGJlY2F1c2UgdGhpcyBwYXRoIGlzIGFscmVhZHkgc2xvdy4NClllcywgSSBhZ3JlZSwgSSBh ZGQgYWJvdmUgY2hlY2sganVzdCB3YW50IHRvIGF2b2lkIHRoZSBpc3N1ZSBLb3Nha2kgbWV0IHdo aWNoIGZpeCBieSB0aGUgY29tbWl0IDkyOWJlYTdjLg0KSW4gc2hvcnQsIHRvIGF2b2lkIHRoZSBj YXNlIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlID0gMSwgYnV0IHpvbmUtPnBhZ2VzX3NjYW5uZWQg PSAwLCBzbyBvbmx5IGNoZWNrIHpvbmVfcmVjbGFpbWFibGUoKSBzaG91bGQgbm90IGVub3VnaC4N Cj4NCj4+ICsgICAgICAgICAgICAgICBpZiAoem9uZV9yZWNsYWltYWJsZSh6b25lKSkNCj4+ICAg ICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBmYWxzZTsNCj4+ICAgICAgICAgfQ0KPj4gPg0K Pj4gPkZyb20gYTVkODIxNTliOThmM2Q5MGMyZjlmZjllNDg2Njk5ZmI0YzY3Y2NlZCBNb24gU2Vw IDE3IDAwOjAwOjAwDQo+PiA+MjAwMQ0KPj4gPkZyb206IE1pbmNoYW4gS2ltIDxtaW5jaGFuQGtl cm5lbC5vcmc+DQo+PiA+RGF0ZTogVGh1LCAxIEF1ZyAyMDEzIDE2OjE4OjAwICswOTAwDQo+PiA+ U3ViamVjdDpbUEFUQ0hdIG1tOiBzZXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUgaW4gZGlyZWN0 IHJlY2xhaW0NCj4+ID4gcGF0aA0KPj4gPg0KPj4gPkxpc2EgcmVwb3J0ZWQgdGhlcmUgYXJlIGxv dHMgb2YgZnJlZSBwYWdlcyBpbiBhIHpvbmUgYnV0IG1vc3Qgb2YgdGhlbQ0KPj4gPmlzIG9yZGVy LTAgcGFnZXMgc28gaXQgbWVhbnMgdGhlIHpvbmUgaXMgaGVhdmlseSBmcmFnZW1lbnRlZC4NCj4+ ID5UaGVuLCBoaWdoIG9yZGVyIGFsbG9jYXRpb24gY291bGQgbWFrZSBkaXJlY3QgcmVjbGFpbSBw YXRoJ3Nsb25nIHN0YWxsKA0KPj4gPmV4LCA1MCBzZWNvbmQpIGluIG5vIHN3YXAgYW5kIG5vIGNv bXBhY3Rpb24gZW52aXJvbm1lbnQuDQo+PiA+DQo+PiA+VGhlIHJlYXNvbiBpcyBrc3dhcGQgY2Fu IHNraXAgdGhlIHpvbmUncyBzY2FubmluZyBiZWNhdXNlIHRoZSB6b25lDQo+PiA+aXMgbG90cyBv ZiBmcmVlIHBhZ2VzIGFuZCBrc3dhcGQgY2hhbmdlcyBzY2FubmluZyBvcmRlciBmcm9tIGhpZ2gt b3JkZXINCj4+ID50byAwLW9yZGVyIGFmdGVyIGhpcyBmaXJzdCBpdGVyYXRpb24gaXMgZG9uZSBi ZWNhdXNlIGtzd2FwZCB0aGluaw0KPj4gPm9yZGVyLTAgYWxsb2NhdGlvbiBpcyB0aGUgbW9zdCBp bXBvcnRhbnQuDQo+PiA+TG9vayBhdCA3M2NlMDJlOSBpbiBkZXRhaWwuDQo+PiA+DQo+PiA+VGhl IHByb2JsZW0gZnJvbSB0aGF0IGlzIHRoYXQgb25seSBrc3dhcGQgY2FuIHNldA0KPnpvbmUtPmFs bF91bnJlY2xhaW1hYmxlDQo+PiA+dG8gMSBhdCB0aGUgbW9tZW50IHNvIGRpcmVjdCByZWNsYWlt IHBhdGggc2hvdWxkIGxvb3AgZm9yZXZlciB1bnRpbCBhDQo+Z2hvc3QNCj4+ID5jYW4gc2V0IHRo ZSB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSB0byAxLg0KPj4gPg0KPj4gPlRoaXMgcGF0Y2ggbWFr ZXMgZGlyZWN0IHJlY2xhaW0gcGF0aCB0byBzZXQgem9uZS0+YWxsX3VucmVjbGFpbWFibGUNCj4+ ID50byBhdm9pZCBpbmZpbml0ZSBsb29wLiBTbyBub3cgd2UgZG9uJ3QgbmVlZCBhIGdob3N0Lg0K Pj4gPg0KPj4gPlJlcG9ydGVkLWJ5OiBMaXNhIER1IDxjbGR1QG1hcnZlbGwuY29tPg0KPj4gPlNp Z25lZC1vZmYtYnk6IE1pbmNoYW4gS2ltIDxtaW5jaGFuQGtlcm5lbC5vcmc+DQo+PiA+LS0tDQo+ PiA+IG1tL3Ztc2Nhbi5jIHwgICAyOSArKysrKysrKysrKysrKysrKysrKysrKysrKysrLQ0KPj4g PiAxIGZpbGUgY2hhbmdlZCwgMjggaW5zZXJ0aW9ucygrKSwgMSBkZWxldGlvbigtKQ0KPj4gPg0K Pj4gPmRpZmYgLS1naXQgYS9tbS92bXNjYW4uYyBiL21tL3Ztc2Nhbi5jDQo+PiA+aW5kZXggMzNk YzI1Ni4uZjk1N2U4NyAxMDA2NDQNCj4+ID4tLS0gYS9tbS92bXNjYW4uYw0KPj4gPisrKyBiL21t L3Ztc2Nhbi5jDQo+PiA+QEAgLTIzMTcsNiArMjMxNywyMyBAQCBzdGF0aWMgYm9vbCBhbGxfdW5y ZWNsYWltYWJsZShzdHJ1Y3Qgem9uZWxpc3QNCj4+ID4qem9uZWxpc3QsDQo+PiA+ICAgIHJldHVy biB0cnVlOw0KPj4gPiB9DQo+PiA+DQo+PiA+K3N0YXRpYyB2b2lkIGNoZWNrX3pvbmVzX3VucmVj bGFpbWFibGUoc3RydWN0IHpvbmVsaXN0ICp6b25lbGlzdCwNCj4+ID4rICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykNCj4+ID4rew0KPj4g PisgICBzdHJ1Y3Qgem9uZXJlZiAqejsNCj4+ID4rICAgc3RydWN0IHpvbmUgKnpvbmU7DQo+PiA+ Kw0KPj4gPisgICBmb3JfZWFjaF96b25lX3pvbmVsaXN0X25vZGVtYXNrKHpvbmUsIHosIHpvbmVs aXN0LA0KPj4gPisgICAgICAgICAgICAgICAgICAgZ2ZwX3pvbmUoc2MtPmdmcF9tYXNrKSwgc2Mt Pm5vZGVtYXNrKSB7DQo+PiA+KyAgICAgICAgICAgaWYgKCFwb3B1bGF0ZWRfem9uZSh6b25lKSkN Cj4+ID4rICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPj4gPisgICAgICAgICAgIGlmICgh Y3B1c2V0X3pvbmVfYWxsb3dlZF9oYXJkd2FsbCh6b25lLCBHRlBfS0VSTkVMKSkNCj4+ID4rICAg ICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPj4gPisgICAgICAgICAgIGlmICghem9uZV9yZWNs YWltYWJsZSh6b25lKSkNCj4+ID4rICAgICAgICAgICAgICAgICAgIHpvbmUtPmFsbF91bnJlY2xh aW1hYmxlID0gMTsNCj4+ID4rICAgfQ0KPj4gPit9DQo+PiA+Kw0KPj4gPiAvKg0KPj4gPiAgKiBU aGlzIGlzIHRoZSBtYWluIGVudHJ5IHBvaW50IHRvIGRpcmVjdCBwYWdlIHJlY2xhaW0uDQo+PiA+ ICAqDQo+PiA+QEAgLTIzNzAsNyArMjM4NywxNyBAQCBzdGF0aWMgdW5zaWduZWQgbG9uZw0KPj4g PmRvX3RyeV90b19mcmVlX3BhZ2VzKHN0cnVjdCB6b25lbGlzdCAqem9uZWxpc3QsDQo+PiA+ICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGxydV9wYWdlcyArPSB6b25lX3JlY2xhaW1hYmxlX3Bh Z2VzKHpvbmUpOw0KPj4gPiAgICAgICAgICAgICAgICAgICAgfQ0KPj4gPg0KPj4gPi0gICAgICAg ICAgICAgICAgICAgc2hyaW5rX3NsYWIoc2hyaW5rLCBzYy0+bnJfc2Nhbm5lZCwgbHJ1X3BhZ2Vz KTsNCj4+ID4rICAgICAgICAgICAgICAgICAgIC8qDQo+PiA+KyAgICAgICAgICAgICAgICAgICAg KiBXaGVuIGEgem9uZSBoYXMgZW5vdWdoIG9yZGVyLTAgZnJlZSBtZW1vcnkgYnV0DQo+PiA+KyAg ICAgICAgICAgICAgICAgICAgKiB6b25lIGlzIGhlYXZpbHkgZnJhZ21lbnRlZCBhbmQgd2UgbmVl ZCBoaWdoIG9yZGVyDQo+PiA+KyAgICAgICAgICAgICAgICAgICAgKiBwYWdlIGZyb20gdGhlIHpv bmUsIGtzd2FwZCBjb3VsZCBza2lwIHRoZSB6b25lDQo+PiA+KyAgICAgICAgICAgICAgICAgICAg KiBhZnRlciBmaXJzdCBpdGVyYXRpb24gd2l0aCBoaWdoIG9yZGVyLiBTbywga3N3YXBkDQo+PiA+ KyAgICAgICAgICAgICAgICAgICAgKiBuZXZlciBzZXQgdGhlIHpvbmUtPmFsbF91bnJlY2xhaW1h YmxlIHRvIDEgc28NCj4+ID4rICAgICAgICAgICAgICAgICAgICAqIGRpcmVjdCByZWNsYWltIHBh dGggbmVlZHMgdGhlIGNoZWNrLg0KPj4gPisgICAgICAgICAgICAgICAgICAgICovDQo+PiA+KyAg ICAgICAgICAgICAgICAgICBpZiAoIXNocmlua19zbGFiKHNocmluaywgc2MtPm5yX3NjYW5uZWQs IGxydV9wYWdlcykpDQo+PiA+KyAgICAgICAgICAgICAgICAgICAgICAgICAgIGNoZWNrX3pvbmVz X3VucmVjbGFpbWFibGUoem9uZWxpc3QsIHNjKTsNCj4+ID4rDQo+PiA+ICAgICAgICAgICAgICAg ICAgICBpZiAocmVjbGFpbV9zdGF0ZSkgew0KPj4gPiAgICAgICAgICAgICAgICAgICAgICAgICAg ICBzYy0+bnJfcmVjbGFpbWVkICs9IHJlY2xhaW1fc3RhdGUtPnJlY2xhaW1lZF9zbGFiOw0KPj4g PiAgICAgICAgICAgICAgICAgICAgICAgICAgICByZWNsYWltX3N0YXRlLT5yZWNsYWltZWRfc2xh YiA9IDA7DQo+PiA+LS0NCj4+ID4xLjcuOS41DQo+PiA+DQo+PiA+LS0NCj4+ID5LaW5kIHJlZ2Fy ZHMsDQo+PiA+TWluY2hhbiBLaW0NCj4NCj4tLQ0KPktpbmQgcmVnYXJkcywNCj5NaW5jaGFuIEtp bQ0K -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx135.postini.com [74.125.245.135]) by kanga.kvack.org (Postfix) with SMTP id CD1296B0031 for ; Thu, 1 Aug 2013 21:22:44 -0400 (EDT) From: Lisa Du Date: Thu, 1 Aug 2013 18:18:43 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE3B0@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BE028@SC-VEXCH4.marvell.com> <20130801085653.GD24642@n2100.arm.linux.org.uk> In-Reply-To: <20130801085653.GD24642@n2100.arm.linux.org.uk> Content-Language: en-US Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Russell King - ARM Linux Cc: KOSAKI Motohiro , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu , Neil Zhang Pi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogUnVzc2VsbCBLaW5nIC0gQVJNIExp bnV4IFttYWlsdG86bGludXhAYXJtLmxpbnV4Lm9yZy51a10NCj5TZW50OiAyMDEzxOo41MIxyNUg MTY6NTcNCj5UbzogTGlzYSBEdQ0KPkNjOiBLT1NBS0kgTW90b2hpcm87IENocmlzdG9waCBMYW1l dGVyOyBsaW51eC1tbUBrdmFjay5vcmc7IE1lbA0KPkdvcm1hbjsgQm9iIExpdTsgTmVpbCBaaGFu Zw0KPlN1YmplY3Q6IFJlOiBQb3NzaWJsZSBkZWFkbG9vcCBpbiBkaXJlY3QgcmVjbGFpbT8NCj4N Cj5PbiBXZWQsIEp1bCAzMSwgMjAxMyBhdCAxMDoxOTo1M1BNIC0wNzAwLCBMaXNhIER1IHdyb3Rl Og0KPj4gPmZvcmsgYWxsb2Mgb3JkZXItMSBtZW1vcnkgZm9yIHN0YWNrLiBXaGVyZSBhbmQgd2h5 IGFsbG9jIG9yZGVyLTI/IElmIGl0IGlzDQo+PiA+YXJjaCBzcGVjaWZpYyBjb2RlLCBwbGVhc2UN Cj4+ID5jb250YWN0IGFyY2ggbWFpbnRhaW5lci4NCj4+IFllcyBhcmNoIGRvX2ZvcmsgYWxsb2Nh dGUgb3JkZXItMiBtZW1vcnkgd2hlbiBjb3B5X3Byb2Nlc3MuDQo+PiBIaSwgUnVzc2VsDQo+PiBX aGF0J3MgeW91ciBvcGluaW9uIGFib3V0IHRoaXMgcXVlc3Rpb24/DQo+PiBJZiB3ZSByZWFsbHkg bmVlZCBvcmRlci0yIG1lbW9yeSBmb3IgZm9yaywgdGhlbiB3ZSdkIGJldHRlciBzZXQNCj4+IENP TkZJR19DT01QQVRJT04gcmlnaHQ/DQo+DQo+V2VsbCwgSSBnYXZlIHVwIHRyeWluZyB0byByZWFk IHRoZSBvcmlnaW5hbCBtZXNzYWdlcyBiZWNhdXNlIHRoZSBxdW90aW5nDQo+c3R5bGUgaXMgYSB0 b3RhbCBtZXNzLCBzbyBJIGRvbid0IGhhdmUgYSBmdWxsIHVuZGVyc3RhbmRpbmcgb2Ygd2hhdCB0 aGUNCj5pc3N1ZSBpcy4NCkknbSByZWFsbHkgc29ycnkgZm9yIG15IHF1b3Rpbmcgc3R5bGUsIEkn bGwgYXZvaWQgc3VjaCBpc3N1ZSBpbiBmdXR1cmUhDQo+DQo+SG93ZXZlciwgd2UgaGF2ZSBhbHdh eXMgcmVxdWlyZWQgb3JkZXItMiBtZW1vcnkgZm9yIGZvcmssIGdvaW5nIGJhY2sgdG8NCj50aGUg MS54IGtlcm5lbCBkYXlzIC0gaXQncyBmdW5kYW1lbnRhbCB0byBBUk0gdG8gaGF2ZSB0aGF0LiAg VGhlIG9yZGVyLTINCj5hbGxvY2F0aW9uIG9zIGZvciB0aGUgMXN0IGxldmVsIHBhZ2UgdGFibGUu ICBObyBvcmRlci0yIGFsbG9jYXRpb24sIG5vDQo+cGFnZSB0YWJsZXMgZm9yIHRoZSBuZXcgdGhy ZWFkLg0KPg0KPkxvb2tpbmcgYXQgdGhpcyBjb21taXQ6DQo+DQo+Y29tbWl0IDA1MTA2ZTZhNTRh ZWQzMjExOTFiNGJiNWM5ZWUwOTUzOGNiYWQzYjENCj5BdXRob3I6IFJpayB2YW4gUmllbCA8cmll bEByZWRoYXQuY29tPg0KPkRhdGU6ICAgTW9uIE9jdCA4IDE2OjMzOjAzIDIwMTIgLTA3MDANCj4N Cj4gICAgbW06IGVuYWJsZSBDT05GSUdfQ09NUEFDVElPTiBieSBkZWZhdWx0DQo+DQo+ICAgIE5v dyB0aGF0IGx1bXB5IHJlY2xhaW0gaGFzIGJlZW4gcmVtb3ZlZCwgY29tcGFjdGlvbiBpcyB0aGUg b25seQ0KPndheSBsZWZ0DQo+ICAgIHRvIGZyZWUgdXAgY29udGlndW91cyBtZW1vcnkgYXJlYXMu ICBJdCBpcyB0aW1lIHRvIGp1c3QgZW5hYmxlDQo+ICAgIENPTkZJR19DT01QQUNUSU9OIGJ5IGRl ZmF1bHQuDQo+DQo+aXQgc2VlbXMgdG8gaW5kaWNhdGUgdGhhdCBldmVyeW9uZSBzaG91bGQgaGF2 ZSB0aGlzIGVuYWJsZWQgLSBob3dldmVyLA0KPnRoZSB3YXkgdGhlIGNoYW5nZSBoYXMgYmVlbiBk b25lLCBhbnlvbmUgYnVpbGRpbmcgZnJvbSBkZWZjb25maWdzIGJlZm9yZQ0KPnRoYXQgY2hhbmdl IHdpbGwgbm90IGhhdmUgdGhhdCBvcHRpb24gZW5hYmxlZC4NCj4NCj5TbyB5ZXMsIHRoaXMgb3B0 aW9uIHNob3VsZCBiZSB0dXJuZWQgb24uDQpUaGFua3MgUnVzc2VsISANCkkgdGhpbmsgSSBoYXZl IGdvdCB0aGUgaW5mb3JtYXRpb24gSSB3YW50LiBSZWFsbHkgYXBwcmVjaWF0ZSBmb3IgeW91ciBl eHBsYW5hdGlvbiENCg== -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx111.postini.com [74.125.245.111]) by kanga.kvack.org (Postfix) with SMTP id 91D576B0031 for ; Thu, 1 Aug 2013 22:25:58 -0400 (EDT) Date: Fri, 2 Aug 2013 11:26:28 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130802015241.GB32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20130801084259.GA32486@bbox> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro Hello Lisa and KOSAKI, Lisa's quote style is very hard to follow so I'd like to write at bottom as ignoring line by line rule. Lisa, please correct your MUA. On Thu, Aug 01, 2013 at 05:42:59PM +0900, Minchan Kim wrote: > On Thu, Aug 01, 2013 at 01:20:34AM -0700, Lisa Du wrote: > > >-----Original Message----- > > >From: Minchan Kim [mailto:minchan@kernel.org] > > >Sent: 2013a1'8ae??1ae?JPY 15:34 > > >To: Lisa Du > > >Cc: linux-mm@kvack.org; KOSAKI Motohiro > > >Subject: Re: Possible deadloop in direct reclaim? > > > > > >On Wed, Jul 31, 2013 at 11:13:07PM -0700, Lisa Du wrote: > > >> >On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: > > >> >> Dear Sir: > > >> >> Currently I met a possible deadloop in direct reclaim. After run plenty > > >of > > >> >the application, system run into a status that system memory is very > > >> >fragmentized. Like only order-0 and order-1 memory left. > > >> >> Then one process required a order-2 buffer but it enter an endless > > >direct > > >> >reclaim. From my trace log, I can see this loop already over 200,000 > > >times. > > >> >Kswapd was first wake up and then go back to sleep as it cannot > > >rebalance > > >> >this order's memory. But zone->all_unreclaimable remains 1. > > >> >> Though direct_reclaim every time returns no pages, but as > > >> >zone->all_unreclaimable = 1, so it loop again and again. Even when > > >> >zone->pages_scanned also becomes very large. It will block the process > > >for > > >> >long time, until some watchdog thread detect this and kill this process. > > >> >Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe > > >cost > > >> >over 50 seconds or even more. > > >> >> I think it's not as expected right? Can we also add below check in the > > >> >function all_unreclaimable() to terminate this loop? > > >> >> > > >> >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist > > >> >*zonelist, > > >> >> continue; > > >> >> if (!zone->all_unreclaimable) > > >> >> return false; > > >> >> + if (sc->nr_reclaimed == 0 > > >&& !zone_reclaimable(zone)) > > >> >> + return true; > > >> >> } > > >> >> BTW: I'm using kernel3.4, I also try to search in the > > >kernel3.9, > > >> >didn't see a possible fix for such issue. Or is anyone also met such issue > > >> >before? Any comment will be welcomed, looking forward to your reply! > > >> >> > > >> >> Thanks! > > >> > > > >> >I'd like to ask somethigs. > > >> > > > >> >1. Do you have enabled swap? > > >> I set CONFIG_SWAP=y, but I didn't really have a swap partition, that > > >means my swap buffer size is 0; > > >> >2. Do you enable CONFIG_COMPACTION? > > >> No, I didn't enable; > > >> >3. Could we get your zoneinfo via cat /proc/zoneinfo? > > >> I dump some info from ramdump, please review: > > > > > >Thanks for the information. > > >You said order-2 allocation was failed so I will assume preferred zone > > >is normal zone, not high zone because high order allocation in kernel side > > >isn't from high zone. > > Yes, that's right! > > > > > >> crash> kmem -z > > >> NODE: 0 ZONE: 0 ADDR: c08460c0 NAME: "Normal" > > >> SIZE: 192512 PRESENT: 182304 MIN/LOW/HIGH: 853/1066/1279 > > > > > >712M normal memory. > > > > > >> VM_STAT: > > >> NR_FREE_PAGES: 16092 > > > > > >There are plenty of free pages over high watermark but there are heavy > > >fragmentation as I see below information. > > > > > >So, kswapd doesn't scan this zone loop iteration is done with order-2. > > >I mean kswapd will scan this zone with order-0 if first iteration is > > >done by this > > > > > > order = sc.order = 0; > > > > > > goto loop_again; > > > > > >But this time, zone_watermark_ok_safe with testorder = 0 on normal zone > > >is always true so that scanning of zone will be skipped. It means kswapd > > >never set zone->unreclaimable to 1. > > Yes, definitely! > > > > > >> NR_INACTIVE_ANON: 17 > > >> NR_ACTIVE_ANON: 55091 > > >> NR_INACTIVE_FILE: 17 > > >> NR_ACTIVE_FILE: 17 > > >> NR_UNEVICTABLE: 0 > > >> NR_MLOCK: 0 > > >> NR_ANON_PAGES: 55077 > > > > > >There are about 200M anon pages and few file pages. > > >You don't have swap so that reclaimer couldn't go far. > > > > > >> NR_FILE_MAPPED: 42 > > >> NR_FILE_PAGES: 69 > > >> NR_FILE_DIRTY: 0 > > >> NR_WRITEBACK: 0 > > >> NR_SLAB_RECLAIMABLE: 1226 > > >> NR_SLAB_UNRECLAIMABLE: 9373 > > >> NR_PAGETABLE: 2776 > > >> NR_KERNEL_STACK: 798 > > >> NR_UNSTABLE_NFS: 0 > > >> NR_BOUNCE: 0 > > >> NR_VMSCAN_WRITE: 91 > > >> NR_VMSCAN_IMMEDIATE: 115381 > > >> NR_WRITEBACK_TEMP: 0 > > >> NR_ISOLATED_ANON: 0 > > >> NR_ISOLATED_FILE: 0 > > >> NR_SHMEM: 31 > > >> NR_DIRTIED: 15256 > > >> NR_WRITTEN: 11981 > > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > >> > > >> NODE: 0 ZONE: 1 ADDR: c08464c0 NAME: "HighMem" > > >> SIZE: 69632 PRESENT: 69088 MIN/LOW/HIGH: 67/147/228 > > >> VM_STAT: > > >> NR_FREE_PAGES: 161 > > > > > >Reclaimer should reclaim this zone. > > > > > >> NR_INACTIVE_ANON: 104 > > >> NR_ACTIVE_ANON: 46114 > > >> NR_INACTIVE_FILE: 9722 > > >> NR_ACTIVE_FILE: 12263 > > > > > >It seems there are lots of room to evict file pages. > > > > > >> NR_UNEVICTABLE: 168 > > >> NR_MLOCK: 0 > > >> NR_ANON_PAGES: 46102 > > >> NR_FILE_MAPPED: 12227 > > >> NR_FILE_PAGES: 22270 > > >> NR_FILE_DIRTY: 1 > > >> NR_WRITEBACK: 0 > > >> NR_SLAB_RECLAIMABLE: 0 > > >> NR_SLAB_UNRECLAIMABLE: 0 > > >> NR_PAGETABLE: 0 > > >> NR_KERNEL_STACK: 0 > > >> NR_UNSTABLE_NFS: 0 > > >> NR_BOUNCE: 0 > > >> NR_VMSCAN_WRITE: 0 > > >> NR_VMSCAN_IMMEDIATE: 0 > > >> NR_WRITEBACK_TEMP: 0 > > >> NR_ISOLATED_ANON: 0 > > >> NR_ISOLATED_FILE: 0 > > >> NR_SHMEM: 117 > > >> NR_DIRTIED: 7364 > > >> NR_WRITTEN: 6989 > > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > >> > > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > > >START_MAPNR > > >> 0 Normal 192512 16092 c1200000 0 0 > > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > > >> 0 4k c08460f0 3 3 > > >> 0 4k c08460f8 436 436 > > >> 0 4k c0846100 15237 15237 > > >> 0 4k c0846108 0 0 > > >> 0 4k c0846110 0 0 > > >> 1 8k c084611c 39 78 > > >> 1 8k c0846124 0 0 > > >> 1 8k c084612c 169 338 > > >> 1 8k c0846134 0 0 > > >> 1 8k c084613c 0 0 > > >> 2 16k c0846148 0 0 > > >> 2 16k c0846150 0 0 > > >> 2 16k c0846158 0 0 > > >> ---------Normal zone all order > 1 has no free pages > > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > > >START_MAPNR > > >> 1 HighMem 69632 161 c17e0000 2f000000 > > >192512 > > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > > >> 0 4k c08464f0 12 12 > > >> 0 4k c08464f8 0 0 > > >> 0 4k c0846500 14 14 > > >> 0 4k c0846508 3 3 > > >> 0 4k c0846510 0 0 > > >> 1 8k c084651c 0 0 > > >> 1 8k c0846524 0 0 > > >> 1 8k c084652c 0 0 > > >> 2 16k c0846548 0 0 > > >> 2 16k c0846550 0 0 > > >> 2 16k c0846558 0 0 > > >> 2 16k c0846560 1 4 > > >> 2 16k c0846568 0 0 > > >> 5 128k c08465cc 0 0 > > >> 5 128k c08465d4 0 0 > > >> 5 128k c08465dc 0 0 > > >> 5 128k c08465e4 4 128 > > >> 5 128k c08465ec 0 0 > > >> ------Other's all zero > > >> > > >> Some other zone information I dump from pglist_data > > >> { > > >> watermark = {853, 1066, 1279}, > > >> percpu_drift_mark = 0, > > >> lowmem_reserve = {0, 2159, 2159}, > > >> dirty_balance_reserve = 3438, > > >> pageset = 0xc07f6144, > > >> lock = { > > >> { > > >> rlock = { > > >> raw_lock = { > > >> lock = 0 > > >> }, > > >> break_lock = 0 > > >> } > > >> } > > >> }, > > >> all_unreclaimable = 0, > > >> reclaim_stat = { > > >> recent_rotated = {903355, 960912}, > > >> recent_scanned = {932404, 2462017} > > >> }, > > >> pages_scanned = 84231, > > > > > >Most of scan happens in direct reclaim path, I guess > > >but direct reclaim couldn't reclaim any pages due to lack of swap device. > > > > > >It means we have to set zone->all_unreclaimable in direct reclaim path, > > >too. > > >Below patch fix your problem? > > Yes, your patch should fix my problem! > > Actually I also did another patch, after test, should also fix my issue, > > but I didn't set zone->all_unreclaimable in direct reclaim path as you, > > just double check zone_reclaimable() status in all_unreclaimable() function. > > Maybe your patch is better! > > Nope. I think your patch is better. :) > Just thing is anlaysis of the problem and description and I think we could do > better but unfortunately, I don't have enough time today so I will see tomorrow. > Just nitpick below. > > Thanks. > > > > > commit 26d2b60d06234683a81666da55129f9c982271a5 > > Author: Lisa Du > > Date: Thu Aug 1 10:16:32 2013 +0800 > > > > mm: fix infinite direct_reclaim when memory is very fragmentized > > > > latest all_unreclaimable check in direct reclaim is the following commit. > > 2011 Apr 14; commit 929bea7c; vmscan: all_unreclaimable() use > > zone->all_unreclaimable as a name > > and in addition, add oom_killer_disabled check to avoid reintroduce the > > issue of commit d1908362 ("vmscan: check all_unreclaimable in direct reclaim path"). > > > > But except the hibernation case in which kswapd is freezed, there's also other case > > which may lead infinite loop in direct relaim. In a real test, direct_relaimer did > > over 200000 times rebalance in __alloc_pages_slowpath(), so this process will be > > blocked until watchdog detect and kill it. The root cause is as below: > > > > If system memory is very fragmentized like only order-0 and order-1 left, > > kswapd will go to sleep as system cann't rebalanced for high-order allocations. > > But direct_reclaim still works for higher order request. So zones can become a state > > zone->all_unreclaimable = 0 but zone->pages_scanned > zone_reclaimable_pages(zone) * 6. > > In this case if a process like do_fork try to allocate an order-2 memory which is not > > a COSTLY_ORDER, as direct_reclaim always said it did_some_progress, so rebalance again > > and again in __alloc_pages_slowpath(). This issue is easily happen in no swap and no > > compaction enviroment. > > > > So add furthur check in all_unreclaimable() to avoid such case. > > > > Change-Id: Id3266b47c63f5b96aab466fd9f1f44d37e16cdcb > > Signed-off-by: Lisa Du > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 2cff0d4..34582d9 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -2301,7 +2301,9 @@ static bool all_unreclaimable(struct zonelist *zonelist, > > continue; > > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > > continue; > > - if (!zone->all_unreclaimable) > > + if (zone->all_unreclaimable) > > + continue; > > Nitpick: If we use zone_reclaimable(), above check is redundant and > gain is very tiny because this path is already slow. > > > + if (zone_reclaimable(zone)) > > return false; > > } > > > > > >From a5d82159b98f3d90c2f9ff9e486699fb4c67cced Mon Sep 17 00:00:00 > > >2001 > > >From: Minchan Kim > > >Date: Thu, 1 Aug 2013 16:18:00 +0900 > > >Subject:[PATCH] mm: set zone->all_unreclaimable in direct reclaim > > > path > > > > > >Lisa reported there are lots of free pages in a zone but most of them > > >is order-0 pages so it means the zone is heavily fragemented. > > >Then, high order allocation could make direct reclaim path'slong stall( > > >ex, 50 second) in no swap and no compaction environment. > > > > > >The reason is kswapd can skip the zone's scanning because the zone > > >is lots of free pages and kswapd changes scanning order from high-order > > >to 0-order after his first iteration is done because kswapd think > > >order-0 allocation is the most important. > > >Look at 73ce02e9 in detail. > > > > > >The problem from that is that only kswapd can set zone->all_unreclaimable > > >to 1 at the moment so direct reclaim path should loop forever until a ghost > > >can set the zone->all_unreclaimable to 1. > > > > > >This patch makes direct reclaim path to set zone->all_unreclaimable > > >to avoid infinite loop. So now we don't need a ghost. > > > > > >Reported-by: Lisa Du > > >Signed-off-by: Minchan Kim > > >--- > > > mm/vmscan.c | 29 ++++++++++++++++++++++++++++- > > > 1 file changed, 28 insertions(+), 1 deletion(-) > > > > > >diff --git a/mm/vmscan.c b/mm/vmscan.c > > >index 33dc256..f957e87 100644 > > >--- a/mm/vmscan.c > > >+++ b/mm/vmscan.c > > >@@ -2317,6 +2317,23 @@ static bool all_unreclaimable(struct zonelist > > >*zonelist, > > > return true; > > > } > > > > > >+static void check_zones_unreclaimable(struct zonelist *zonelist, > > >+ struct scan_control *sc) > > >+{ > > >+ struct zoneref *z; > > >+ struct zone *zone; > > >+ > > >+ for_each_zone_zonelist_nodemask(zone, z, zonelist, > > >+ gfp_zone(sc->gfp_mask), sc->nodemask) { > > >+ if (!populated_zone(zone)) > > >+ continue; > > >+ if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > > >+ continue; > > >+ if (!zone_reclaimable(zone)) > > >+ zone->all_unreclaimable = 1; > > >+ } > > >+} > > >+ > > > /* > > > * This is the main entry point to direct page reclaim. > > > * > > >@@ -2370,7 +2387,17 @@ static unsigned long > > >do_try_to_free_pages(struct zonelist *zonelist, > > > lru_pages += zone_reclaimable_pages(zone); > > > } > > > > > >- shrink_slab(shrink, sc->nr_scanned, lru_pages); > > >+ /* > > >+ * When a zone has enough order-0 free memory but > > >+ * zone is heavily fragmented and we need high order > > >+ * page from the zone, kswapd could skip the zone > > >+ * after first iteration with high order. So, kswapd > > >+ * never set the zone->all_unreclaimable to 1 so > > >+ * direct reclaim path needs the check. > > >+ */ > > >+ if (!shrink_slab(shrink, sc->nr_scanned, lru_pages)) > > >+ check_zones_unreclaimable(zonelist, sc); > > >+ > > > if (reclaim_state) { > > > sc->nr_reclaimed += reclaim_state->reclaimed_slab; > > > reclaim_state->reclaimed_slab = 0; > > >-- > > >1.7.9.5 > > > > > >-- > > >Kind regards, > > >Minchan Kim > I reviewed current mmotm because recently Mel changed kswapd a lot and all_unreclaimable patch history today. What I see is recent mmotm has a same problem, too if system have no swap and no compaction. Of course, compaction is default yes option so we could recommend to enable if system works well but it's up to user and we should avoid direct reclaim hang although user disable compaction. When I see the patch history, real culprit is 929bea7c. " zone->all_unreclaimable and zone->pages_scanned are neigher atomic variables nor protected by lock. Therefore zones can become a state of zone->page_scanned=0 and zone->all_unreclaimable=1. In this case, current all_unreclaimable() return false even though zone->all_unreclaimabe=1." I understand the problem but apparently, it makes Lisa's problem because kswapd can give up balancing when high order allocation happens to prevent excessive reclaim with assuming the process requested high order allocation can do direct reclaim/compaction. But what if the process can't reclaim by no swap but lots of anon pages and can't compact by !CONFIG_COMPACTION? In such system, OOM kill is natural but not hang. So, a solution we can fix simply introduces zone_reclaimable check again in all_unreclaimabe() like this. What do you think about it? It's a same patch Lisa posted so we should give a credit to her/him(Sorry I'm not sure) if we agree thie approach. Lisa, If KOSAKI agree with this, could you resend this patch with your SOB? Thanks. diff --git a/mm/vmscan.c b/mm/vmscan.c index a3bf7fd..78f46d8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2367,7 +2367,15 @@ static bool all_unreclaimable(struct zonelist *zonelist, continue; if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) continue; - if (!zone->all_unreclaimable) + /* + * zone->page_scanned and could be raced so we need + * dobule check by zone->all_unreclaimable. Morever, kswapd + * could skip (zone->all_unreclaimable = 1) if the zone + * is heavily fragmented but enough free pages to meet + * high watermark. In such case, kswapd never set + * all_unreclaimable to 1 so we need zone_reclaimable, too. + */ + if (!zone->all_unreclaimable || zone_reclaimable(zone)) return false; } -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx152.postini.com [74.125.245.152]) by kanga.kvack.org (Postfix) with SMTP id EA0646B0034 for ; Thu, 1 Aug 2013 22:32:56 -0400 (EDT) Date: Fri, 2 Aug 2013 11:33:26 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130802023326.GC32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> <20130802015241.GB32486@bbox> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20130802015241.GB32486@bbox> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro On Fri, Aug 02, 2013 at 11:26:28AM +0900, Minchan Kim wrote: > Hello Lisa and KOSAKI, > > Lisa's quote style is very hard to follow so I'd like to write at bottom > as ignoring line by line rule. > > Lisa, please correct your MUA. > > On Thu, Aug 01, 2013 at 05:42:59PM +0900, Minchan Kim wrote: > > On Thu, Aug 01, 2013 at 01:20:34AM -0700, Lisa Du wrote: > > > >-----Original Message----- > > > >From: Minchan Kim [mailto:minchan@kernel.org] > > > >Sent: 2013a1'8ae??1ae?JPY 15:34 > > > >To: Lisa Du > > > >Cc: linux-mm@kvack.org; KOSAKI Motohiro > > > >Subject: Re: Possible deadloop in direct reclaim? > > > > > > > >On Wed, Jul 31, 2013 at 11:13:07PM -0700, Lisa Du wrote: > > > >> >On Mon, Jul 22, 2013 at 09:58:17PM -0700, Lisa Du wrote: > > > >> >> Dear Sir: > > > >> >> Currently I met a possible deadloop in direct reclaim. After run plenty > > > >of > > > >> >the application, system run into a status that system memory is very > > > >> >fragmentized. Like only order-0 and order-1 memory left. > > > >> >> Then one process required a order-2 buffer but it enter an endless > > > >direct > > > >> >reclaim. From my trace log, I can see this loop already over 200,000 > > > >times. > > > >> >Kswapd was first wake up and then go back to sleep as it cannot > > > >rebalance > > > >> >this order's memory. But zone->all_unreclaimable remains 1. > > > >> >> Though direct_reclaim every time returns no pages, but as > > > >> >zone->all_unreclaimable = 1, so it loop again and again. Even when > > > >> >zone->pages_scanned also becomes very large. It will block the process > > > >for > > > >> >long time, until some watchdog thread detect this and kill this process. > > > >> >Though it's in __alloc_pages_slowpath, but it's too slow right? Maybe > > > >cost > > > >> >over 50 seconds or even more. > > > >> >> I think it's not as expected right? Can we also add below check in the > > > >> >function all_unreclaimable() to terminate this loop? > > > >> >> > > > >> >> @@ -2355,6 +2355,8 @@ static bool all_unreclaimable(struct zonelist > > > >> >*zonelist, > > > >> >> continue; > > > >> >> if (!zone->all_unreclaimable) > > > >> >> return false; > > > >> >> + if (sc->nr_reclaimed == 0 > > > >&& !zone_reclaimable(zone)) > > > >> >> + return true; > > > >> >> } > > > >> >> BTW: I'm using kernel3.4, I also try to search in the > > > >kernel3.9, > > > >> >didn't see a possible fix for such issue. Or is anyone also met such issue > > > >> >before? Any comment will be welcomed, looking forward to your reply! > > > >> >> > > > >> >> Thanks! > > > >> > > > > >> >I'd like to ask somethigs. > > > >> > > > > >> >1. Do you have enabled swap? > > > >> I set CONFIG_SWAP=y, but I didn't really have a swap partition, that > > > >means my swap buffer size is 0; > > > >> >2. Do you enable CONFIG_COMPACTION? > > > >> No, I didn't enable; > > > >> >3. Could we get your zoneinfo via cat /proc/zoneinfo? > > > >> I dump some info from ramdump, please review: > > > > > > > >Thanks for the information. > > > >You said order-2 allocation was failed so I will assume preferred zone > > > >is normal zone, not high zone because high order allocation in kernel side > > > >isn't from high zone. > > > Yes, that's right! > > > > > > > >> crash> kmem -z > > > >> NODE: 0 ZONE: 0 ADDR: c08460c0 NAME: "Normal" > > > >> SIZE: 192512 PRESENT: 182304 MIN/LOW/HIGH: 853/1066/1279 > > > > > > > >712M normal memory. > > > > > > > >> VM_STAT: > > > >> NR_FREE_PAGES: 16092 > > > > > > > >There are plenty of free pages over high watermark but there are heavy > > > >fragmentation as I see below information. > > > > > > > >So, kswapd doesn't scan this zone loop iteration is done with order-2. > > > >I mean kswapd will scan this zone with order-0 if first iteration is > > > >done by this > > > > > > > > order = sc.order = 0; > > > > > > > > goto loop_again; > > > > > > > >But this time, zone_watermark_ok_safe with testorder = 0 on normal zone > > > >is always true so that scanning of zone will be skipped. It means kswapd > > > >never set zone->unreclaimable to 1. > > > Yes, definitely! > > > > > > > >> NR_INACTIVE_ANON: 17 > > > >> NR_ACTIVE_ANON: 55091 > > > >> NR_INACTIVE_FILE: 17 > > > >> NR_ACTIVE_FILE: 17 > > > >> NR_UNEVICTABLE: 0 > > > >> NR_MLOCK: 0 > > > >> NR_ANON_PAGES: 55077 > > > > > > > >There are about 200M anon pages and few file pages. > > > >You don't have swap so that reclaimer couldn't go far. > > > > > > > >> NR_FILE_MAPPED: 42 > > > >> NR_FILE_PAGES: 69 > > > >> NR_FILE_DIRTY: 0 > > > >> NR_WRITEBACK: 0 > > > >> NR_SLAB_RECLAIMABLE: 1226 > > > >> NR_SLAB_UNRECLAIMABLE: 9373 > > > >> NR_PAGETABLE: 2776 > > > >> NR_KERNEL_STACK: 798 > > > >> NR_UNSTABLE_NFS: 0 > > > >> NR_BOUNCE: 0 > > > >> NR_VMSCAN_WRITE: 91 > > > >> NR_VMSCAN_IMMEDIATE: 115381 > > > >> NR_WRITEBACK_TEMP: 0 > > > >> NR_ISOLATED_ANON: 0 > > > >> NR_ISOLATED_FILE: 0 > > > >> NR_SHMEM: 31 > > > >> NR_DIRTIED: 15256 > > > >> NR_WRITTEN: 11981 > > > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > > >> > > > >> NODE: 0 ZONE: 1 ADDR: c08464c0 NAME: "HighMem" > > > >> SIZE: 69632 PRESENT: 69088 MIN/LOW/HIGH: 67/147/228 > > > >> VM_STAT: > > > >> NR_FREE_PAGES: 161 > > > > > > > >Reclaimer should reclaim this zone. > > > > > > > >> NR_INACTIVE_ANON: 104 > > > >> NR_ACTIVE_ANON: 46114 > > > >> NR_INACTIVE_FILE: 9722 > > > >> NR_ACTIVE_FILE: 12263 > > > > > > > >It seems there are lots of room to evict file pages. > > > > > > > >> NR_UNEVICTABLE: 168 > > > >> NR_MLOCK: 0 > > > >> NR_ANON_PAGES: 46102 > > > >> NR_FILE_MAPPED: 12227 > > > >> NR_FILE_PAGES: 22270 > > > >> NR_FILE_DIRTY: 1 > > > >> NR_WRITEBACK: 0 > > > >> NR_SLAB_RECLAIMABLE: 0 > > > >> NR_SLAB_UNRECLAIMABLE: 0 > > > >> NR_PAGETABLE: 0 > > > >> NR_KERNEL_STACK: 0 > > > >> NR_UNSTABLE_NFS: 0 > > > >> NR_BOUNCE: 0 > > > >> NR_VMSCAN_WRITE: 0 > > > >> NR_VMSCAN_IMMEDIATE: 0 > > > >> NR_WRITEBACK_TEMP: 0 > > > >> NR_ISOLATED_ANON: 0 > > > >> NR_ISOLATED_FILE: 0 > > > >> NR_SHMEM: 117 > > > >> NR_DIRTIED: 7364 > > > >> NR_WRITTEN: 6989 > > > >> NR_ANON_TRANSPARENT_HUGEPAGES: 0 > > > >> > > > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > > > >START_MAPNR > > > >> 0 Normal 192512 16092 c1200000 0 0 > > > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > > > >> 0 4k c08460f0 3 3 > > > >> 0 4k c08460f8 436 436 > > > >> 0 4k c0846100 15237 15237 > > > >> 0 4k c0846108 0 0 > > > >> 0 4k c0846110 0 0 > > > >> 1 8k c084611c 39 78 > > > >> 1 8k c0846124 0 0 > > > >> 1 8k c084612c 169 338 > > > >> 1 8k c0846134 0 0 > > > >> 1 8k c084613c 0 0 > > > >> 2 16k c0846148 0 0 > > > >> 2 16k c0846150 0 0 > > > >> 2 16k c0846158 0 0 > > > >> ---------Normal zone all order > 1 has no free pages > > > >> ZONE NAME SIZE FREE MEM_MAP START_PADDR > > > >START_MAPNR > > > >> 1 HighMem 69632 161 c17e0000 2f000000 > > > >192512 > > > >> AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES > > > >> 0 4k c08464f0 12 12 > > > >> 0 4k c08464f8 0 0 > > > >> 0 4k c0846500 14 14 > > > >> 0 4k c0846508 3 3 > > > >> 0 4k c0846510 0 0 > > > >> 1 8k c084651c 0 0 > > > >> 1 8k c0846524 0 0 > > > >> 1 8k c084652c 0 0 > > > >> 2 16k c0846548 0 0 > > > >> 2 16k c0846550 0 0 > > > >> 2 16k c0846558 0 0 > > > >> 2 16k c0846560 1 4 > > > >> 2 16k c0846568 0 0 > > > >> 5 128k c08465cc 0 0 > > > >> 5 128k c08465d4 0 0 > > > >> 5 128k c08465dc 0 0 > > > >> 5 128k c08465e4 4 128 > > > >> 5 128k c08465ec 0 0 > > > >> ------Other's all zero > > > >> > > > >> Some other zone information I dump from pglist_data > > > >> { > > > >> watermark = {853, 1066, 1279}, > > > >> percpu_drift_mark = 0, > > > >> lowmem_reserve = {0, 2159, 2159}, > > > >> dirty_balance_reserve = 3438, > > > >> pageset = 0xc07f6144, > > > >> lock = { > > > >> { > > > >> rlock = { > > > >> raw_lock = { > > > >> lock = 0 > > > >> }, > > > >> break_lock = 0 > > > >> } > > > >> } > > > >> }, > > > >> all_unreclaimable = 0, > > > >> reclaim_stat = { > > > >> recent_rotated = {903355, 960912}, > > > >> recent_scanned = {932404, 2462017} > > > >> }, > > > >> pages_scanned = 84231, > > > > > > > >Most of scan happens in direct reclaim path, I guess > > > >but direct reclaim couldn't reclaim any pages due to lack of swap device. > > > > > > > >It means we have to set zone->all_unreclaimable in direct reclaim path, > > > >too. > > > >Below patch fix your problem? > > > Yes, your patch should fix my problem! > > > Actually I also did another patch, after test, should also fix my issue, > > > but I didn't set zone->all_unreclaimable in direct reclaim path as you, > > > just double check zone_reclaimable() status in all_unreclaimable() function. > > > Maybe your patch is better! > > > > Nope. I think your patch is better. :) > > Just thing is anlaysis of the problem and description and I think we could do > > better but unfortunately, I don't have enough time today so I will see tomorrow. > > Just nitpick below. > > > > Thanks. > > > > > > > > commit 26d2b60d06234683a81666da55129f9c982271a5 > > > Author: Lisa Du > > > Date: Thu Aug 1 10:16:32 2013 +0800 > > > > > > mm: fix infinite direct_reclaim when memory is very fragmentized > > > > > > latest all_unreclaimable check in direct reclaim is the following commit. > > > 2011 Apr 14; commit 929bea7c; vmscan: all_unreclaimable() use > > > zone->all_unreclaimable as a name > > > and in addition, add oom_killer_disabled check to avoid reintroduce the > > > issue of commit d1908362 ("vmscan: check all_unreclaimable in direct reclaim path"). > > > > > > But except the hibernation case in which kswapd is freezed, there's also other case > > > which may lead infinite loop in direct relaim. In a real test, direct_relaimer did > > > over 200000 times rebalance in __alloc_pages_slowpath(), so this process will be > > > blocked until watchdog detect and kill it. The root cause is as below: > > > > > > If system memory is very fragmentized like only order-0 and order-1 left, > > > kswapd will go to sleep as system cann't rebalanced for high-order allocations. > > > But direct_reclaim still works for higher order request. So zones can become a state > > > zone->all_unreclaimable = 0 but zone->pages_scanned > zone_reclaimable_pages(zone) * 6. > > > In this case if a process like do_fork try to allocate an order-2 memory which is not > > > a COSTLY_ORDER, as direct_reclaim always said it did_some_progress, so rebalance again > > > and again in __alloc_pages_slowpath(). This issue is easily happen in no swap and no > > > compaction enviroment. > > > > > > So add furthur check in all_unreclaimable() to avoid such case. > > > > > > Change-Id: Id3266b47c63f5b96aab466fd9f1f44d37e16cdcb > > > Signed-off-by: Lisa Du > > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > > index 2cff0d4..34582d9 100644 > > > --- a/mm/vmscan.c > > > +++ b/mm/vmscan.c > > > @@ -2301,7 +2301,9 @@ static bool all_unreclaimable(struct zonelist *zonelist, > > > continue; > > > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > > > continue; > > > - if (!zone->all_unreclaimable) > > > + if (zone->all_unreclaimable) > > > + continue; > > > > Nitpick: If we use zone_reclaimable(), above check is redundant and > > gain is very tiny because this path is already slow. > > > > > + if (zone_reclaimable(zone)) > > > return false; > > > } > > > > > > > >From a5d82159b98f3d90c2f9ff9e486699fb4c67cced Mon Sep 17 00:00:00 > > > >2001 > > > >From: Minchan Kim > > > >Date: Thu, 1 Aug 2013 16:18:00 +0900 > > > >Subject:[PATCH] mm: set zone->all_unreclaimable in direct reclaim > > > > path > > > > > > > >Lisa reported there are lots of free pages in a zone but most of them > > > >is order-0 pages so it means the zone is heavily fragemented. > > > >Then, high order allocation could make direct reclaim path'slong stall( > > > >ex, 50 second) in no swap and no compaction environment. > > > > > > > >The reason is kswapd can skip the zone's scanning because the zone > > > >is lots of free pages and kswapd changes scanning order from high-order > > > >to 0-order after his first iteration is done because kswapd think > > > >order-0 allocation is the most important. > > > >Look at 73ce02e9 in detail. > > > > > > > >The problem from that is that only kswapd can set zone->all_unreclaimable > > > >to 1 at the moment so direct reclaim path should loop forever until a ghost > > > >can set the zone->all_unreclaimable to 1. > > > > > > > >This patch makes direct reclaim path to set zone->all_unreclaimable > > > >to avoid infinite loop. So now we don't need a ghost. > > > > > > > >Reported-by: Lisa Du > > > >Signed-off-by: Minchan Kim > > > >--- > > > > mm/vmscan.c | 29 ++++++++++++++++++++++++++++- > > > > 1 file changed, 28 insertions(+), 1 deletion(-) > > > > > > > >diff --git a/mm/vmscan.c b/mm/vmscan.c > > > >index 33dc256..f957e87 100644 > > > >--- a/mm/vmscan.c > > > >+++ b/mm/vmscan.c > > > >@@ -2317,6 +2317,23 @@ static bool all_unreclaimable(struct zonelist > > > >*zonelist, > > > > return true; > > > > } > > > > > > > >+static void check_zones_unreclaimable(struct zonelist *zonelist, > > > >+ struct scan_control *sc) > > > >+{ > > > >+ struct zoneref *z; > > > >+ struct zone *zone; > > > >+ > > > >+ for_each_zone_zonelist_nodemask(zone, z, zonelist, > > > >+ gfp_zone(sc->gfp_mask), sc->nodemask) { > > > >+ if (!populated_zone(zone)) > > > >+ continue; > > > >+ if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > > > >+ continue; > > > >+ if (!zone_reclaimable(zone)) > > > >+ zone->all_unreclaimable = 1; > > > >+ } > > > >+} > > > >+ > > > > /* > > > > * This is the main entry point to direct page reclaim. > > > > * > > > >@@ -2370,7 +2387,17 @@ static unsigned long > > > >do_try_to_free_pages(struct zonelist *zonelist, > > > > lru_pages += zone_reclaimable_pages(zone); > > > > } > > > > > > > >- shrink_slab(shrink, sc->nr_scanned, lru_pages); > > > >+ /* > > > >+ * When a zone has enough order-0 free memory but > > > >+ * zone is heavily fragmented and we need high order > > > >+ * page from the zone, kswapd could skip the zone > > > >+ * after first iteration with high order. So, kswapd > > > >+ * never set the zone->all_unreclaimable to 1 so > > > >+ * direct reclaim path needs the check. > > > >+ */ > > > >+ if (!shrink_slab(shrink, sc->nr_scanned, lru_pages)) > > > >+ check_zones_unreclaimable(zonelist, sc); > > > >+ > > > > if (reclaim_state) { > > > > sc->nr_reclaimed += reclaim_state->reclaimed_slab; > > > > reclaim_state->reclaimed_slab = 0; > > > >-- > > > >1.7.9.5 > > > > > > > >-- > > > >Kind regards, > > > >Minchan Kim > > > > I reviewed current mmotm because recently Mel changed kswapd a lot and > all_unreclaimable patch history today. > What I see is recent mmotm has a same problem, too if system have no swap > and no compaction. Of course, compaction is default yes option so we could > recommend to enable if system works well but it's up to user and we should ^^^ typo if system want work well > avoid direct reclaim hang although user disable compaction. > > When I see the patch history, real culprit is 929bea7c. > > " zone->all_unreclaimable and zone->pages_scanned are neigher atomic > variables nor protected by lock. Therefore zones can become a state of > zone->page_scanned=0 and zone->all_unreclaimable=1. In this case, current > all_unreclaimable() return false even though zone->all_unreclaimabe=1." > > I understand the problem but apparently, it makes Lisa's problem because > kswapd can give up balancing when high order allocation happens to prevent > excessive reclaim with assuming the process requested high order allocation > can do direct reclaim/compaction. But what if the process can't reclaim > by no swap but lots of anon pages and can't compact by !CONFIG_COMPACTION? > > In such system, OOM kill is natural but not hang. > So, a solution we can fix simply introduces zone_reclaimable check again in > all_unreclaimabe() like this. > > What do you think about it? > > It's a same patch Lisa posted so we should give a credit > to her/him(Sorry I'm not sure) if we agree thie approach. > > Lisa, If KOSAKI agree with this, could you resend this patch with your SOB? > > Thanks. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index a3bf7fd..78f46d8 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2367,7 +2367,15 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > continue; > - if (!zone->all_unreclaimable) > + /* > + * zone->page_scanned and could be raced so we need ^^^ typo: please remove it. > + * dobule check by zone->all_unreclaimable. Morever, kswapd > + * could skip (zone->all_unreclaimable = 1) if the zone > + * is heavily fragmented but enough free pages to meet > + * high watermark. In such case, kswapd never set > + * all_unreclaimable to 1 so we need zone_reclaimable, too. > + */ > + if (!zone->all_unreclaimable || zone_reclaimable(zone)) > return false; > } > > > > -- > Kind regards, > Minchan Kim > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx123.postini.com [74.125.245.123]) by kanga.kvack.org (Postfix) with SMTP id 393A16B0031 for ; Thu, 1 Aug 2013 23:20:51 -0400 (EDT) From: Lisa Du Date: Thu, 1 Aug 2013 20:17:56 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE43E@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> <20130802015241.GB32486@bbox> In-Reply-To: <20130802015241.GB32486@bbox> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: "linux-mm@kvack.org" , KOSAKI Motohiro , Bob Liu Pi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogTWluY2hhbiBLaW0gW21haWx0bzpt aW5jaGFuQGtlcm5lbC5vcmddDQo+U2VudDogMjAxM+W5tDjmnIgy5pelIDEwOjI2DQo+VG86IExp c2EgRHUNCj5DYzogbGludXgtbW1Aa3ZhY2sub3JnOyBLT1NBS0kgTW90b2hpcm8NCj5TdWJqZWN0 OiBSZTogUG9zc2libGUgZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0/DQo+DQo+SGVsbG8gTGlz YSBhbmQgS09TQUtJLA0KPg0KPkxpc2EncyBxdW90ZSBzdHlsZSBpcyB2ZXJ5IGhhcmQgdG8gZm9s bG93IHNvIEknZCBsaWtlIHRvIHdyaXRlIGF0IGJvdHRvbQ0KPmFzIGlnbm9yaW5nIGxpbmUgYnkg bGluZSBydWxlLg0KPg0KPkxpc2EsIHBsZWFzZSBjb3JyZWN0IHlvdXIgTVVBLg0KSSdtIHJlYWxs eSBzb3JyeSBmb3IgbXkgcXVvdGUgc3R5bGUsIHdpbGwgaW1wcm92ZSBpdCBpbiBteSBmb2xsb3dp bmcgbWFpbHMuDQo+DQo+DQo+SSByZXZpZXdlZCBjdXJyZW50IG1tb3RtIGJlY2F1c2UgcmVjZW50 bHkgTWVsIGNoYW5nZWQga3N3YXBkIGEgbG90IGFuZA0KPmFsbF91bnJlY2xhaW1hYmxlIHBhdGNo IGhpc3RvcnkgdG9kYXkuDQo+V2hhdCBJIHNlZSBpcyByZWNlbnQgbW1vdG0gaGFzIGEgc2FtZSBw cm9ibGVtLCB0b28gaWYgc3lzdGVtIGhhdmUgbm8gc3dhcA0KPmFuZCBubyBjb21wYWN0aW9uLiBP ZiBjb3Vyc2UsIGNvbXBhY3Rpb24gaXMgZGVmYXVsdCB5ZXMgb3B0aW9uIHNvIHdlIGNvdWxkDQo+ cmVjb21tZW5kIHRvIGVuYWJsZSBpZiBzeXN0ZW0gd29ya3Mgd2VsbCBidXQgaXQncyB1cCB0byB1 c2VyIGFuZCB3ZSBzaG91bGQNCj5hdm9pZCBkaXJlY3QgcmVjbGFpbSBoYW5nIGFsdGhvdWdoIHVz ZXIgZGlzYWJsZSBjb21wYWN0aW9uLg0KPg0KPldoZW4gSSBzZWUgdGhlIHBhdGNoIGhpc3Rvcnks IHJlYWwgY3VscHJpdCBpcyA5MjliZWE3Yy4NCj4NCj4iICB6b25lLT5hbGxfdW5yZWNsYWltYWJs ZSBhbmQgem9uZS0+cGFnZXNfc2Nhbm5lZCBhcmUgbmVpZ2hlciBhdG9taWMNCj4gICAgdmFyaWFi bGVzIG5vciBwcm90ZWN0ZWQgYnkgbG9jay4gIFRoZXJlZm9yZSB6b25lcyBjYW4gYmVjb21lIGEg c3RhdGUgb2YNCj4gICAgem9uZS0+cGFnZV9zY2FubmVkPTAgYW5kIHpvbmUtPmFsbF91bnJlY2xh aW1hYmxlPTEuICBJbiB0aGlzIGNhc2UsIGN1cnJlbnQNCj4gICAgYWxsX3VucmVjbGFpbWFibGUo KSByZXR1cm4gZmFsc2UgZXZlbiB0aG91Z2ggem9uZS0+YWxsX3VucmVjbGFpbWFiZT0xLiINCj4N Cj5JIHVuZGVyc3RhbmQgdGhlIHByb2JsZW0gYnV0IGFwcGFyZW50bHksIGl0IG1ha2VzIExpc2En cyBwcm9ibGVtIGJlY2F1c2UNCj5rc3dhcGQgY2FuIGdpdmUgdXAgYmFsYW5jaW5nIHdoZW4gaGln aCBvcmRlciBhbGxvY2F0aW9uIGhhcHBlbnMgdG8gcHJldmVudA0KPmV4Y2Vzc2l2ZSByZWNsYWlt IHdpdGggYXNzdW1pbmcgdGhlIHByb2Nlc3MgcmVxdWVzdGVkIGhpZ2ggb3JkZXIgYWxsb2NhdGlv bg0KPmNhbiBkbyBkaXJlY3QgcmVjbGFpbS9jb21wYWN0aW9uLiBCdXQgd2hhdCBpZiB0aGUgcHJv Y2VzcyBjYW4ndCByZWNsYWltDQo+Ynkgbm8gc3dhcCBidXQgbG90cyBvZiBhbm9uIHBhZ2VzIGFu ZCBjYW4ndCBjb21wYWN0IGJ5ICFDT05GSUdfQ09NUEFDVElPTj8NCj4NCj5JbiBzdWNoIHN5c3Rl bSwgT09NIGtpbGwgaXMgbmF0dXJhbCBidXQgbm90IGhhbmcuDQo+U28sIGEgc29sdXRpb24gd2Ug Y2FuIGZpeCBzaW1wbHkgaW50cm9kdWNlcyB6b25lX3JlY2xhaW1hYmxlIGNoZWNrIGFnYWluIGlu DQo+YWxsX3VucmVjbGFpbWFiZSgpIGxpa2UgdGhpcy4NCj4NCj5XaGF0IGRvIHlvdSB0aGluayBh Ym91dCBpdD8NCj4NCj5JdCdzIGEgc2FtZSBwYXRjaCBMaXNhIHBvc3RlZCBzbyB3ZSBzaG91bGQg Z2l2ZSBhIGNyZWRpdA0KPnRvIGhlci9oaW0oU29ycnkgSSdtIG5vdCBzdXJlKSBpZiB3ZSBhZ3Jl ZSB0aGllIGFwcHJvYWNoLg0KPg0KPkxpc2EsIElmIEtPU0FLSSBhZ3JlZSB3aXRoIHRoaXMsIGNv dWxkIHlvdSByZXNlbmQgdGhpcyBwYXRjaCB3aXRoIHlvdXIgU09CPw0KPg0KPlRoYW5rcy4NCj4N Cj5kaWZmIC0tZ2l0IGEvbW0vdm1zY2FuLmMgYi9tbS92bXNjYW4uYw0KPmluZGV4IGEzYmY3ZmQu Ljc4ZjQ2ZDggMTAwNjQ0DQo+LS0tIGEvbW0vdm1zY2FuLmMNCj4rKysgYi9tbS92bXNjYW4uYw0K PkBAIC0yMzY3LDcgKzIzNjcsMTUgQEAgc3RhdGljIGJvb2wgYWxsX3VucmVjbGFpbWFibGUoc3Ry dWN0IHpvbmVsaXN0ICp6b25lbGlzdCwNCj4gCQkJY29udGludWU7DQo+IAkJaWYgKCFjcHVzZXRf em9uZV9hbGxvd2VkX2hhcmR3YWxsKHpvbmUsIEdGUF9LRVJORUwpKQ0KPiAJCQljb250aW51ZTsN Cj4tCQlpZiAoIXpvbmUtPmFsbF91bnJlY2xhaW1hYmxlKQ0KPisJCS8qDQo+KwkJICogem9uZS0+ cGFnZV9zY2FubmVkIGFuZCBjb3VsZCBiZSByYWNlZCBzbyB3ZSBuZWVkDQo+KwkJICogZG9idWxl IGNoZWNrIGJ5IHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlLiBNb3JldmVyLCBrc3dhcGQNCj4rCQkg KiBjb3VsZCBza2lwICh6b25lLT5hbGxfdW5yZWNsYWltYWJsZSA9IDEpIGlmIHRoZSB6b25lDQo+ KwkJICogaXMgaGVhdmlseSBmcmFnbWVudGVkIGJ1dCBlbm91Z2ggZnJlZSBwYWdlcyB0byBtZWV0 DQo+KwkJICogaGlnaCB3YXRlcm1hcmsuIEluIHN1Y2ggY2FzZSwga3N3YXBkIG5ldmVyIHNldA0K PisJCSAqIGFsbF91bnJlY2xhaW1hYmxlIHRvIDEgc28gd2UgbmVlZCB6b25lX3JlY2xhaW1hYmxl LCB0b28uDQo+KwkJICovDQo+KwkJaWYgKCF6b25lLT5hbGxfdW5yZWNsYWltYWJsZSB8fCB6b25l X3JlY2xhaW1hYmxlKHpvbmUpKQ0KPiAJCQlyZXR1cm4gZmFsc2U7DQo+IAl9DQogICBJJ20gYWZy YWlkIHRoaXMgcGF0Y2ggbWF5IGNhbid0IGhlbHAuDQogICB6b25lLT5hbGxfdW5yZWNsYWltYWJs ZSA9IDAgd2lsbCBhbHdheXMgcmVzdWx0IHRoZSBmYWxzZSByZXR1cm4sDQogICB6b25lX3JlY2xh aW1hYmxlKHpvbmUpIGNoZWNrIHdvdWxkbid0IHRha2UgZWZmZWN0IG5vIG1hdHRlcg0KICAgaXQn cyB0cnVlIG9mIGZhbHNlIHJpZ2h0Pw0KDQpBbHNvIEJvYiBmb3VuZCBiZWxvdyB0aHJlYWQsIHNl ZW1zIEtvc2FraSBhbHNvIGZvdW5kIHNhbWUgaXNzdWU6DQptbSwgdm1zY2FuOiBmaXggZG9fdHJ5 X3RvX2ZyZWVfcGFnZXMoKSBsaXZlbG9jaw0KaHR0cHM6Ly9sa21sLm9yZy9sa21sLzIwMTIvNi8x NC83NA0KDQo+DQo+DQo+DQo+LS0NCj5LaW5kIHJlZ2FyZHMsDQo+TWluY2hhbiBLaW0NCg== -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx188.postini.com [74.125.245.188]) by kanga.kvack.org (Postfix) with SMTP id ECA126B0031 for ; Thu, 1 Aug 2013 23:53:03 -0400 (EDT) Date: Fri, 2 Aug 2013 12:53:33 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130802035333.GD32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> <20130802015241.GB32486@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE43E@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BE43E@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro , Bob Liu On Thu, Aug 01, 2013 at 08:17:56PM -0700, Lisa Du wrote: > >-----Original Message----- > >From: Minchan Kim [mailto:minchan@kernel.org] > >Sent: 2013a1'8ae??2ae?JPY 10:26 > >To: Lisa Du > >Cc: linux-mm@kvack.org; KOSAKI Motohiro > >Subject: Re: Possible deadloop in direct reclaim? > > > >Hello Lisa and KOSAKI, > > > >Lisa's quote style is very hard to follow so I'd like to write at bottom > >as ignoring line by line rule. > > > >Lisa, please correct your MUA. > I'm really sorry for my quote style, will improve it in my following mails. > > > > > >I reviewed current mmotm because recently Mel changed kswapd a lot and > >all_unreclaimable patch history today. > >What I see is recent mmotm has a same problem, too if system have no swap > >and no compaction. Of course, compaction is default yes option so we could > >recommend to enable if system works well but it's up to user and we should > >avoid direct reclaim hang although user disable compaction. > > > >When I see the patch history, real culprit is 929bea7c. > > > >" zone->all_unreclaimable and zone->pages_scanned are neigher atomic > > variables nor protected by lock. Therefore zones can become a state of > > zone->page_scanned=0 and zone->all_unreclaimable=1. In this case, current > > all_unreclaimable() return false even though zone->all_unreclaimabe=1." > > > >I understand the problem but apparently, it makes Lisa's problem because > >kswapd can give up balancing when high order allocation happens to prevent > >excessive reclaim with assuming the process requested high order allocation > >can do direct reclaim/compaction. But what if the process can't reclaim > >by no swap but lots of anon pages and can't compact by !CONFIG_COMPACTION? > > > >In such system, OOM kill is natural but not hang. > >So, a solution we can fix simply introduces zone_reclaimable check again in > >all_unreclaimabe() like this. > > > >What do you think about it? > > > >It's a same patch Lisa posted so we should give a credit > >to her/him(Sorry I'm not sure) if we agree thie approach. > > > >Lisa, If KOSAKI agree with this, could you resend this patch with your SOB? > > > >Thanks. > > > >diff --git a/mm/vmscan.c b/mm/vmscan.c > >index a3bf7fd..78f46d8 100644 > >--- a/mm/vmscan.c > >+++ b/mm/vmscan.c > >@@ -2367,7 +2367,15 @@ static bool all_unreclaimable(struct zonelist *zonelist, > > continue; > > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > > continue; > >- if (!zone->all_unreclaimable) > >+ /* > >+ * zone->page_scanned and could be raced so we need > >+ * dobule check by zone->all_unreclaimable. Morever, kswapd > >+ * could skip (zone->all_unreclaimable = 1) if the zone > >+ * is heavily fragmented but enough free pages to meet > >+ * high watermark. In such case, kswapd never set > >+ * all_unreclaimable to 1 so we need zone_reclaimable, too. > >+ */ > >+ if (!zone->all_unreclaimable || zone_reclaimable(zone)) > > return false; > > } > I'm afraid this patch may can't help. > zone->all_unreclaimable = 0 will always result the false return, > zone_reclaimable(zone) check wouldn't take effect no matter > it's true of false right? You're right. It was not what I want but check both conditions. > > Also Bob found below thread, seems Kosaki also found same issue: > mm, vmscan: fix do_try_to_free_pages() livelock > https://lkml.org/lkml/2012/6/14/74 I remember it and AFAIRC, I had a concern because description was too vague without detailed example and I fixed Aaditya's problem with another approach. That's why it wasn't merged at that time. Now, we have a real problem and analysis so I think KOSAKI's patch makes perfect to me. Lisa, Could you resend KOSAKI's patch with more detailed description? > > > > > > > > >-- > >Kind regards, > >Minchan Kim -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx152.postini.com [74.125.245.152]) by kanga.kvack.org (Postfix) with SMTP id 2E71B6B0031 for ; Fri, 2 Aug 2013 04:08:50 -0400 (EDT) From: Lisa Du Date: Fri, 2 Aug 2013 01:08:50 -0700 Subject: RE: Possible deadloop in direct reclaim? Message-ID: <89813612683626448B837EE5A0B6A7CB3B630BE511@SC-VEXCH4.marvell.com> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> <20130802015241.GB32486@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE43E@SC-VEXCH4.marvell.com> <20130802035333.GD32486@bbox> In-Reply-To: <20130802035333.GD32486@bbox> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: "linux-mm@kvack.org" , KOSAKI Motohiro , Bob Liu Pi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogTWluY2hhbiBLaW0gW21haWx0bzpt aW5jaGFuQGtlcm5lbC5vcmddDQo+U2VudDogMjAxM+W5tDjmnIgy5pelIDExOjU0DQo+VG86IExp c2EgRHUNCj5DYzogbGludXgtbW1Aa3ZhY2sub3JnOyBLT1NBS0kgTW90b2hpcm87IEJvYiBMaXUN Cj5TdWJqZWN0OiBSZTogUG9zc2libGUgZGVhZGxvb3AgaW4gZGlyZWN0IHJlY2xhaW0/DQo+DQo+ T24gVGh1LCBBdWcgMDEsIDIwMTMgYXQgMDg6MTc6NTZQTSAtMDcwMCwgTGlzYSBEdSB3cm90ZToN Cj4+ID4tLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4gPkZyb206IE1pbmNoYW4gS2ltIFtt YWlsdG86bWluY2hhbkBrZXJuZWwub3JnXQ0KPj4gPlNlbnQ6IDIwMTPlubQ45pyIMuaXpSAxMDoy Ng0KPj4gPlRvOiBMaXNhIER1DQo+PiA+Q2M6IGxpbnV4LW1tQGt2YWNrLm9yZzsgS09TQUtJIE1v dG9oaXJvDQo+PiA+U3ViamVjdDogUmU6IFBvc3NpYmxlIGRlYWRsb29wIGluIGRpcmVjdCByZWNs YWltPw0KDQoNCj4+ID5JIHJldmlld2VkIGN1cnJlbnQgbW1vdG0gYmVjYXVzZSByZWNlbnRseSBN ZWwgY2hhbmdlZCBrc3dhcGQgYSBsb3QgYW5kDQo+PiA+YWxsX3VucmVjbGFpbWFibGUgcGF0Y2gg aGlzdG9yeSB0b2RheS4NCj4+ID5XaGF0IEkgc2VlIGlzIHJlY2VudCBtbW90bSBoYXMgYSBzYW1l IHByb2JsZW0sIHRvbyBpZiBzeXN0ZW0gaGF2ZSBubyBzd2FwDQo+PiA+YW5kIG5vIGNvbXBhY3Rp b24uIE9mIGNvdXJzZSwgY29tcGFjdGlvbiBpcyBkZWZhdWx0IHllcyBvcHRpb24gc28gd2UgY291 bGQNCj4+ID5yZWNvbW1lbmQgdG8gZW5hYmxlIGlmIHN5c3RlbSB3b3JrcyB3ZWxsIGJ1dCBpdCdz IHVwIHRvIHVzZXIgYW5kIHdlIHNob3VsZA0KPj4gPmF2b2lkIGRpcmVjdCByZWNsYWltIGhhbmcg YWx0aG91Z2ggdXNlciBkaXNhYmxlIGNvbXBhY3Rpb24uDQo+PiA+DQo+PiA+V2hlbiBJIHNlZSB0 aGUgcGF0Y2ggaGlzdG9yeSwgcmVhbCBjdWxwcml0IGlzIDkyOWJlYTdjLg0KPj4gPg0KPj4gPiIg IHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlIGFuZCB6b25lLT5wYWdlc19zY2FubmVkIGFyZSBuZWln aGVyIGF0b21pYw0KPj4gPiAgICB2YXJpYWJsZXMgbm9yIHByb3RlY3RlZCBieSBsb2NrLiAgVGhl cmVmb3JlIHpvbmVzIGNhbiBiZWNvbWUgYSBzdGF0ZSBvZg0KPj4gPiAgICB6b25lLT5wYWdlX3Nj YW5uZWQ9MCBhbmQgem9uZS0+YWxsX3VucmVjbGFpbWFibGU9MS4gIEluIHRoaXMgY2FzZSwgY3Vy cmVudA0KPj4gPiAgICBhbGxfdW5yZWNsYWltYWJsZSgpIHJldHVybiBmYWxzZSBldmVuIHRob3Vn aCB6b25lLT5hbGxfdW5yZWNsYWltYWJlPTEuIg0KPj4gPg0KPj4gPkkgdW5kZXJzdGFuZCB0aGUg cHJvYmxlbSBidXQgYXBwYXJlbnRseSwgaXQgbWFrZXMgTGlzYSdzIHByb2JsZW0gYmVjYXVzZQ0K Pj4gPmtzd2FwZCBjYW4gZ2l2ZSB1cCBiYWxhbmNpbmcgd2hlbiBoaWdoIG9yZGVyIGFsbG9jYXRp b24gaGFwcGVucyB0byBwcmV2ZW50DQo+PiA+ZXhjZXNzaXZlIHJlY2xhaW0gd2l0aCBhc3N1bWlu ZyB0aGUgcHJvY2VzcyByZXF1ZXN0ZWQgaGlnaCBvcmRlciBhbGxvY2F0aW9uDQo+PiA+Y2FuIGRv IGRpcmVjdCByZWNsYWltL2NvbXBhY3Rpb24uIEJ1dCB3aGF0IGlmIHRoZSBwcm9jZXNzIGNhbid0 IHJlY2xhaW0NCj4+ID5ieSBubyBzd2FwIGJ1dCBsb3RzIG9mIGFub24gcGFnZXMgYW5kIGNhbid0 IGNvbXBhY3QgYnkgIUNPTkZJR19DT01QQUNUSU9OPw0KPj4gPg0KPj4NCj4+IEFsc28gQm9iIGZv dW5kIGJlbG93IHRocmVhZCwgc2VlbXMgS29zYWtpIGFsc28gZm91bmQgc2FtZSBpc3N1ZToNCj4+ IG1tLCB2bXNjYW46IGZpeCBkb190cnlfdG9fZnJlZV9wYWdlcygpIGxpdmVsb2NrDQo+PiBodHRw czovL2xrbWwub3JnL2xrbWwvMjAxMi82LzE0Lzc0DQo+DQo+SSByZW1lbWJlciBpdCBhbmQgQUZB SVJDLCBJIGhhZCBhIGNvbmNlcm4gYmVjYXVzZSBkZXNjcmlwdGlvbiB3YXMNCj50b28gdmFndWUg d2l0aG91dCBkZXRhaWxlZCBleGFtcGxlIGFuZCBJIGZpeGVkIEFhZGl0eWEncyBwcm9ibGVtIHdp dGgNCj5hbm90aGVyIGFwcHJvYWNoLiBUaGF0J3Mgd2h5IGl0IHdhc24ndCBtZXJnZWQgYXQgdGhh dCB0aW1lLg0KPg0KPk5vdywgd2UgaGF2ZSBhIHJlYWwgcHJvYmxlbSBhbmQgYW5hbHlzaXMgc28g SSB0aGluayBLT1NBS0kncyBwYXRjaCBtYWtlcw0KPnBlcmZlY3QgdG8gbWUuDQo+DQo+TGlzYSwg Q291bGQgeW91IHJlc2VuZCBLT1NBS0kncyBwYXRjaCB3aXRoIG1vcmUgZGV0YWlsZWQgZGVzY3Jp cHRpb24/DQoNCkhpLCBNaW5jaGFuIGFuZCBLb3Nha2kNCldvdWxkIHlvdSBwbGVhc2UgaGVscCBj aGVjayBiZWxvdyBwYXRjaCBJIHJlc2VuZCBiYXNlZCBvbiBwcmV2aW91cyBLb3Nha2kncyBwYXRj aD8NCkknbSBub3Qgc3VyZSBpZiB0aGUgZGVzY3JpcHRpb24gaXMgY2xlYXIgZW5vdWdoLCBwbGVh c2UgbGV0IG1lIGtub3cgaWYgeW91IGhhdmUgYW55IGNvbW1lbnRzLg0KTWFueSB0aGFua3MhDQpG cm9tIDJkZmUxMzc2NjVhNjk0ZGNjNzRhZTljOGEyNzY0MWIwNjE5MGYzNDQgTW9uIFNlcCAxNyAw MDowMDowMCAyMDAxDQpGcm9tOiBMaXNhIER1IDxjbGR1QG1hcnZlbGwuY29tPg0KRGF0ZTogRnJp LCAyIEF1ZyAyMDEzIDE0OjM3OjMxICswODAwDQpTdWJqZWN0OiBbUEFUQ0hdIG1tOiB2bXNjYW46 IGZpeCBkb190cnlfdG9fZnJlZV9wYWdlcygpIGxpdmVsb2NrDQoNCkN1cnJlbnRseSwgSSBmb3Vu ZCBzeXN0ZW0gY2FuIGVudGVyIGEgc3RhdGUgdGhhdCB0aGVyZSBhcmUgbG90cw0Kb2YgZnJlZSBw YWdlcyBpbiBhIHpvbmUgYnV0IG9ubHkgb3JkZXItMCBhbmQgb3JkZXItMSBwYWdlcyB3aGljaA0K bWVhbnMgdGhlIHpvbmUgaXMgaGVhdmlseSBmcmFnbWVudGVkLCB0aGVuIGhpZ2ggb3JkZXIgYWxs b2NhdGlvbg0KY291bGQgbWFrZSBkaXJlY3QgcmVjbGFpbSBwYXRoJ3MgbG9uZyBzdGFsbChleCwg NjAgc2Vjb25kcykNCmVzcGVjaWFsbHkgaW4gbm8gc3dhcCBhbmQgbm8gY29tcGFjaXRvbiBlbnZp cm9tZW50Lg0KDQpUaGUgcmVhc29uIGlzIGRvX3RyeV90b19mcmVlX3BhZ2VzIGVudGVyIGxpdmUg bG9jazoNCg0Ka3N3YXBkIHdpbGwgZ28gdG8gc2xlZXAgaWYgdGhlIHpvbmVzIGhhdmUgYmVlbiBm dWxseSBzY2FubmVkDQphbmQgYXJlIHN0aWxsIG5vdCBiYWxhbmNlZC4gQXMga3N3YXBkIHRoaW5r cyB0aGVyZSdzIGxpdHRsZSBwb2ludA0KdHJ5aW5nIGFsbCBvdmVyIGFnYWluIHRvIGF2b2lkIGlu ZmluaXRlIGxvb3AuIEluc3RlYWQgaXQgY2hhbmdlcw0Kb3JkZXIgZnJvbSBoaWdoLW9yZGVyIHRv IDAtb3JkZXIgYmVjYXVzZSBrc3dhcGQgdGhpbmsgb3JkZXItMCBpcyB0aGUNCm1vc3QgaW1wb3J0 YW50LiBMb29rIGF0IDczY2UwMmU5IGluIGRldGFpbC4gSWYgd2F0ZXJtYXJrcyBhcmUgb2ssDQpr c3dhcGQgd2lsbCBnbyBiYWNrIHRvIHNsZWVwIGFuZCBtYXkgbGVhdmUgem9uZS0+YWxsX3VucmVj bGFpbWFibGUgPSAwLg0KSXQgYXNzdW1lIGhpZ2gtb3JkZXIgdXNlcnMgY2FuIHN0aWxsIHBlcmZv cm0gZGlyZWN0IHJlY2xhaW0gaWYgdGhleSB3aXNoLg0KDQpEaXJlY3QgcmVjbGFpbSBjb250aW51 ZSB0byByZWNsYWltIGZvciBhIGhpZ2ggb3JkZXIgd2hpY2ggaXMgbm90IGENCkNPU1RMWV9PUkRF UiB3aXRob3V0IG9vbS1raWxsZXIgdW50aWwga3N3YXBkIHR1cm4gb24gem9uZS0+YWxsX3VucmVj bGFpbWJsZS4NClRoaXMgaXMgYmVjYXVzZSB0byBhdm9pZCB0b28gZWFybHkgb29tLWtpbGwuIFNv IGl0IG1lYW5zIGRpcmVjdF9yZWNsYWltDQpkZXBlbmRzIG9uIGtzd2FwZCB0byBicmVhayB0aGlz IGxvb3AuDQoNCkluIHdvcnN0IGNhc2UsIGRpcmVjdC1yZWNsYWltIG1heSBjb250aW51ZSB0byBw YWdlIHJlY2xhaW0gZm9yZXZlcg0Kd2hlbiBrc3dhcGQgc2xlZXBzIGZvcmV2ZXIgdW50aWwgc29t ZW9uZSBsaWtlIHdhdGNoZG9nIGRldGVjdCBhbmQgZmluYWxseQ0Ka2lsbCB0aGUgcHJvY2Vzcy4N Cg0KV2UgY2FuJ3QgdHVybiBvbiB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSBmcm9tIGRpcmVjdCBy ZWNsYWltIHBhdGgNCmJlY2F1c2UgZGlyZWN0IHJlY2xhaW0gcGF0aCBkb24ndCB0YWtlIGFueSBs b2NrIGFuZCB0aGlzIHdheSBpcyByYWN5Lg0KVGh1cyB0aGlzIHBhdGNoIHJlbW92ZXMgem9uZS0+ YWxsX3VucmVjbGFpbWFibGUgZmllbGQgY29tcGxldGVseSBhbmQNCnJlY2FsY3VsYXRlcyB6b25l IHJlY2xhaW1hYmxlIHN0YXRlIGV2ZXJ5IHRpbWUuDQoNCk5vdGU6IHdlIGNhbid0IHRha2UgdGhl IGlkZWEgdGhhdCBkaXJlY3QtcmVjbGFpbSBzZWUgem9uZS0+cGFnZXNfc2Nhbm5lZA0KZGlyZWN0 bHkgYW5kIGtzd2FwZCBjb250aW51ZSB0byB1c2Ugem9uZS0+YWxsX3VucmVjbGFpbWFibGUuIEJl Y2F1c2UsIGl0DQppcyByYWN5LiBjb21taXQgOTI5YmVhN2M3MSAodm1zY2FuOiBhbGxfdW5yZWNs YWltYWJsZSgpIHVzZQ0Kem9uZS0+YWxsX3VucmVjbGFpbWFibGUgYXMgYSBuYW1lKSBkZXNjcmli ZXMgdGhlIGRldGFpbC4NCg0KQ2hhbmdlLUlkOiBJZjNiNDRlMzNlNDAwYzFkYjBlNDJhNWUyZmM5 ZWJjN2EyNjVmMmFhZQ0KQ2M6IEFhZGl0eWEgS3VtYXIgPGFhZGl0eWEua3VtYXIuMzBAZ21haWwu Y29tPg0KQ2M6IFlpbmcgSGFuIDx5aW5naGFuQGdvb2dsZS5jb20+DQpDYzogTmljayBQaWdnaW4g PG5waWdnaW5AZ21haWwuY29tPg0KQWNrZWQtYnk6IFJpayB2YW4gUmllbCA8cmllbEByZWRoYXQu Y29tPg0KQ2M6IE1pY2hhbCBIb2NrbyA8bWhvY2tvQHN1c2UuY3o+DQpDYzogSm9oYW5uZXMgV2Vp bmVyIDxoYW5uZXNAY21weGNoZy5vcmc+DQpDYzogTWVsIEdvcm1hbiA8bWVsQGNzbi51bC5pZT4N CkNjOiBLQU1FWkFXQSBIaXJveXVraSA8a2FtZXphd2EuaGlyb3l1QGpwLmZ1aml0c3UuY29tPg0K Q2M6IE1pbmNoYW4gS2ltIDxtaW5jaGFuLmtpbUBnbWFpbC5jb20+DQpDYzogQm9iIExpdSA8bGxp dWJib0BnbWFpbC5jb20+DQpTaWduZWQtb2ZmLWJ5OiBLT1NBS0kgTW90b2hpcm8gPGtvc2FraS5t b3RvaGlyb0BqcC5mdWppdHN1LmNvbT4NClNpZ25lZC1vZmYtYnk6IExpc2EgRHUgPGNsZHVAbWFy dmVsbC5jb20+DQotLS0NCiBpbmNsdWRlL2xpbnV4L21tX2lubGluZS5oIHwgICAyMCArKysrKysr KysrKysrKysrKysrKw0KIGluY2x1ZGUvbGludXgvbW16b25lLmggICAgfCAgICAxIC0NCiBpbmNs dWRlL2xpbnV4L3Ztc3RhdC5oICAgIHwgICAgMSAtDQogbW0vcGFnZS13cml0ZWJhY2suYyAgICAg ICB8ICAgIDEgKw0KIG1tL3BhZ2VfYWxsb2MuYyAgICAgICAgICAgfCAgICA1ICsrLS0tDQogbW0v dm1zY2FuLmMgICAgICAgICAgICAgICB8ICAgNDMgKysrKysrKysrLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLQ0KIG1tL3Ztc3RhdC5jICAgICAgICAgICAgICAgfCAgICAzICsrLQ0K IDcgZmlsZXMgY2hhbmdlZCwgMzQgaW5zZXJ0aW9ucygrKSwgNDAgZGVsZXRpb25zKC0pDQoNCmRp ZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21tX2lubGluZS5oIGIvaW5jbHVkZS9saW51eC9tbV9p bmxpbmUuaA0KaW5kZXggMTM5N2NjZi4uZTIxMmZhZSAxMDA2NDQNCi0tLSBhL2luY2x1ZGUvbGlu dXgvbW1faW5saW5lLmgNCisrKyBiL2luY2x1ZGUvbGludXgvbW1faW5saW5lLmgNCkBAIC0yLDYg KzIsNyBAQA0KICNkZWZpbmUgTElOVVhfTU1fSU5MSU5FX0gNCiANCiAjaW5jbHVkZSA8bGludXgv aHVnZV9tbS5oPg0KKyNpbmNsdWRlIDxsaW51eC9zd2FwLmg+DQogDQogLyoqDQogICogcGFnZV9p c19maWxlX2NhY2hlIC0gc2hvdWxkIHRoZSBwYWdlIGJlIG9uIGEgZmlsZSBMUlUgb3IgYW5vbiBM UlU/DQpAQCAtOTksNCArMTAwLDIzIEBAIHN0YXRpYyBfX2Fsd2F5c19pbmxpbmUgZW51bSBscnVf bGlzdCBwYWdlX2xydShzdHJ1Y3QgcGFnZSAqcGFnZSkNCiAJcmV0dXJuIGxydTsNCiB9DQogDQor c3RhdGljIGlubGluZSB1bnNpZ25lZCBsb25nIHpvbmVfcmVjbGFpbWFibGVfcGFnZXMoc3RydWN0 IHpvbmUgKnpvbmUpDQorew0KKwlpbnQgbnI7DQorDQorCW5yID0gem9uZV9wYWdlX3N0YXRlKHpv bmUsIE5SX0FDVElWRV9GSUxFKSArDQorCSAgICAgem9uZV9wYWdlX3N0YXRlKHpvbmUsIE5SX0lO QUNUSVZFX0ZJTEUpOw0KKw0KKwlpZiAoZ2V0X25yX3N3YXBfcGFnZXMoKSA+IDApDQorCQluciAr PSB6b25lX3BhZ2Vfc3RhdGUoem9uZSwgTlJfQUNUSVZFX0FOT04pICsNCisJCSAgICAgIHpvbmVf cGFnZV9zdGF0ZSh6b25lLCBOUl9JTkFDVElWRV9BTk9OKTsNCisNCisJcmV0dXJuIG5yOw0KK30N CisNCitzdGF0aWMgaW5saW5lIGJvb2wgem9uZV9yZWNsYWltYWJsZShzdHJ1Y3Qgem9uZSAqem9u ZSkNCit7DQorCXJldHVybiB6b25lLT5wYWdlc19zY2FubmVkIDwgem9uZV9yZWNsYWltYWJsZV9w YWdlcyh6b25lKSAqIDY7DQorfQ0KKw0KICNlbmRpZg0KZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGlu dXgvbW16b25lLmggYi9pbmNsdWRlL2xpbnV4L21tem9uZS5oDQppbmRleCBhZjRhM2I3Li5lODM1 OTc0IDEwMDY0NA0KLS0tIGEvaW5jbHVkZS9saW51eC9tbXpvbmUuaA0KKysrIGIvaW5jbHVkZS9s aW51eC9tbXpvbmUuaA0KQEAgLTM1Miw3ICszNTIsNiBAQCBzdHJ1Y3Qgem9uZSB7DQogCSAqIGZy ZWUgYXJlYXMgb2YgZGlmZmVyZW50IHNpemVzDQogCSAqLw0KIAlzcGlubG9ja190CQlsb2NrOw0K LQlpbnQgICAgICAgICAgICAgICAgICAgICBhbGxfdW5yZWNsYWltYWJsZTsgLyogQWxsIHBhZ2Vz IHBpbm5lZCAqLw0KICNpZiBkZWZpbmVkIENPTkZJR19DT01QQUNUSU9OIHx8IGRlZmluZWQgQ09O RklHX0NNQQ0KIAkvKiBTZXQgdG8gdHJ1ZSB3aGVuIHRoZSBQR19taWdyYXRlX3NraXAgYml0cyBz aG91bGQgYmUgY2xlYXJlZCAqLw0KIAlib29sCQkJY29tcGFjdF9ibG9ja3NraXBfZmx1c2g7DQpk aWZmIC0tZ2l0IGEvaW5jbHVkZS9saW51eC92bXN0YXQuaCBiL2luY2x1ZGUvbGludXgvdm1zdGF0 LmgNCmluZGV4IGM1ODY2NzkuLjZmZmYwMDQgMTAwNjQ0DQotLS0gYS9pbmNsdWRlL2xpbnV4L3Zt c3RhdC5oDQorKysgYi9pbmNsdWRlL2xpbnV4L3Ztc3RhdC5oDQpAQCAtMTQzLDcgKzE0Myw2IEBA IHN0YXRpYyBpbmxpbmUgdW5zaWduZWQgbG9uZyB6b25lX3BhZ2Vfc3RhdGVfc25hcHNob3Qoc3Ry dWN0IHpvbmUgKnpvbmUsDQogfQ0KIA0KIGV4dGVybiB1bnNpZ25lZCBsb25nIGdsb2JhbF9yZWNs YWltYWJsZV9wYWdlcyh2b2lkKTsNCi1leHRlcm4gdW5zaWduZWQgbG9uZyB6b25lX3JlY2xhaW1h YmxlX3BhZ2VzKHN0cnVjdCB6b25lICp6b25lKTsNCiANCiAjaWZkZWYgQ09ORklHX05VTUENCiAv Kg0KZGlmZiAtLWdpdCBhL21tL3BhZ2Utd3JpdGViYWNrLmMgYi9tbS9wYWdlLXdyaXRlYmFjay5j DQppbmRleCAzZjBjODk1Li42MmJmZDkyIDEwMDY0NA0KLS0tIGEvbW0vcGFnZS13cml0ZWJhY2su Yw0KKysrIGIvbW0vcGFnZS13cml0ZWJhY2suYw0KQEAgLTM2LDYgKzM2LDcgQEANCiAjaW5jbHVk ZSA8bGludXgvcGFnZXZlYy5oPg0KICNpbmNsdWRlIDxsaW51eC90aW1lci5oPg0KICNpbmNsdWRl IDxsaW51eC9zY2hlZC9ydC5oPg0KKyNpbmNsdWRlIDxsaW51eC9tbV9pbmxpbmUuaD4NCiAjaW5j bHVkZSA8dHJhY2UvZXZlbnRzL3dyaXRlYmFjay5oPg0KIA0KIC8qDQpkaWZmIC0tZ2l0IGEvbW0v cGFnZV9hbGxvYy5jIGIvbW0vcGFnZV9hbGxvYy5jDQppbmRleCBiMTAwMjU1Li4xOWExOGMwIDEw MDY0NA0KLS0tIGEvbW0vcGFnZV9hbGxvYy5jDQorKysgYi9tbS9wYWdlX2FsbG9jLmMNCkBAIC02 MCw2ICs2MCw3IEBADQogI2luY2x1ZGUgPGxpbnV4L3BhZ2UtZGVidWctZmxhZ3MuaD4NCiAjaW5j bHVkZSA8bGludXgvaHVnZXRsYi5oPg0KICNpbmNsdWRlIDxsaW51eC9zY2hlZC9ydC5oPg0KKyNp bmNsdWRlIDxsaW51eC9tbV9pbmxpbmUuaD4NCiANCiAjaW5jbHVkZSA8YXNtL3NlY3Rpb25zLmg+ DQogI2luY2x1ZGUgPGFzbS90bGJmbHVzaC5oPg0KQEAgLTY0Nyw3ICs2NDgsNiBAQCBzdGF0aWMg dm9pZCBmcmVlX3BjcHBhZ2VzX2J1bGsoc3RydWN0IHpvbmUgKnpvbmUsIGludCBjb3VudCwNCiAJ aW50IHRvX2ZyZWUgPSBjb3VudDsNCiANCiAJc3Bpbl9sb2NrKCZ6b25lLT5sb2NrKTsNCi0Jem9u ZS0+YWxsX3VucmVjbGFpbWFibGUgPSAwOw0KIAl6b25lLT5wYWdlc19zY2FubmVkID0gMDsNCiAN CiAJd2hpbGUgKHRvX2ZyZWUpIHsNCkBAIC02OTYsNyArNjk2LDYgQEAgc3RhdGljIHZvaWQgZnJl ZV9vbmVfcGFnZShzdHJ1Y3Qgem9uZSAqem9uZSwgc3RydWN0IHBhZ2UgKnBhZ2UsIGludCBvcmRl ciwNCiAJCQkJaW50IG1pZ3JhdGV0eXBlKQ0KIHsNCiAJc3Bpbl9sb2NrKCZ6b25lLT5sb2NrKTsN Ci0Jem9uZS0+YWxsX3VucmVjbGFpbWFibGUgPSAwOw0KIAl6b25lLT5wYWdlc19zY2FubmVkID0g MDsNCiANCiAJX19mcmVlX29uZV9wYWdlKHBhZ2UsIHpvbmUsIG9yZGVyLCBtaWdyYXRldHlwZSk7 DQpAQCAtMzA5NSw3ICszMDk0LDcgQEAgdm9pZCBzaG93X2ZyZWVfYXJlYXModW5zaWduZWQgaW50 IGZpbHRlcikNCiAJCQlLKHpvbmVfcGFnZV9zdGF0ZSh6b25lLCBOUl9GUkVFX0NNQV9QQUdFUykp LA0KIAkJCUsoem9uZV9wYWdlX3N0YXRlKHpvbmUsIE5SX1dSSVRFQkFDS19URU1QKSksDQogCQkJ em9uZS0+cGFnZXNfc2Nhbm5lZCwNCi0JCQkoem9uZS0+YWxsX3VucmVjbGFpbWFibGUgPyAieWVz IiA6ICJubyIpDQorCQkJKCF6b25lX3JlY2xhaW1hYmxlKHpvbmUpID8gInllcyIgOiAibm8iKQ0K IAkJCSk7DQogCQlwcmludGsoImxvd21lbV9yZXNlcnZlW106Iik7DQogCQlmb3IgKGkgPSAwOyBp IDwgTUFYX05SX1pPTkVTOyBpKyspDQpkaWZmIC0tZ2l0IGEvbW0vdm1zY2FuLmMgYi9tbS92bXNj YW4uYw0KaW5kZXggMzQ1ODJkOS4uNzUwMWQxZSAxMDA2NDQNCi0tLSBhL21tL3Ztc2Nhbi5jDQor KysgYi9tbS92bXNjYW4uYw0KQEAgLTE3ODksNyArMTc4OSw3IEBAIHN0YXRpYyB2b2lkIGdldF9z Y2FuX2NvdW50KHN0cnVjdCBscnV2ZWMgKmxydXZlYywgc3RydWN0IHNjYW5fY29udHJvbCAqc2Ms DQogCSAqIGxhdGVuY2llcywgc28gaXQncyBiZXR0ZXIgdG8gc2NhbiBhIG1pbmltdW0gYW1vdW50 IHRoZXJlIGFzDQogCSAqIHdlbGwuDQogCSAqLw0KLQlpZiAoY3VycmVudF9pc19rc3dhcGQoKSAm JiB6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkNCisJaWYgKGN1cnJlbnRfaXNfa3N3YXBkKCkgJiYg IXpvbmVfcmVjbGFpbWFibGUoem9uZSkpDQogCQlmb3JjZV9zY2FuID0gdHJ1ZTsNCiAJaWYgKCFn bG9iYWxfcmVjbGFpbShzYykpDQogCQlmb3JjZV9zY2FuID0gdHJ1ZTsNCkBAIC0yMjQ0LDggKzIy NDQsOCBAQCBzdGF0aWMgYm9vbCBzaHJpbmtfem9uZXMoc3RydWN0IHpvbmVsaXN0ICp6b25lbGlz dCwgc3RydWN0IHNjYW5fY29udHJvbCAqc2MpDQogCQlpZiAoZ2xvYmFsX3JlY2xhaW0oc2MpKSB7 DQogCQkJaWYgKCFjcHVzZXRfem9uZV9hbGxvd2VkX2hhcmR3YWxsKHpvbmUsIEdGUF9LRVJORUwp KQ0KIAkJCQljb250aW51ZTsNCi0JCQlpZiAoem9uZS0+YWxsX3VucmVjbGFpbWFibGUgJiYNCi0J CQkJCXNjLT5wcmlvcml0eSAhPSBERUZfUFJJT1JJVFkpDQorCQkJaWYgKCF6b25lX3JlY2xhaW1h YmxlKHpvbmUpICYmDQorCQkJICAgIHNjLT5wcmlvcml0eSAhPSBERUZfUFJJT1JJVFkpDQogCQkJ CWNvbnRpbnVlOwkvKiBMZXQga3N3YXBkIHBvbGwgaXQgKi8NCiAJCQlpZiAoSVNfRU5BQkxFRChD T05GSUdfQ09NUEFDVElPTikpIHsNCiAJCQkJLyoNCkBAIC0yMjgzLDExICsyMjgzLDYgQEAgc3Rh dGljIGJvb2wgc2hyaW5rX3pvbmVzKHN0cnVjdCB6b25lbGlzdCAqem9uZWxpc3QsIHN0cnVjdCBz Y2FuX2NvbnRyb2wgKnNjKQ0KIAlyZXR1cm4gYWJvcnRlZF9yZWNsYWltOw0KIH0NCiANCi1zdGF0 aWMgYm9vbCB6b25lX3JlY2xhaW1hYmxlKHN0cnVjdCB6b25lICp6b25lKQ0KLXsNCi0JcmV0dXJu IHpvbmUtPnBhZ2VzX3NjYW5uZWQgPCB6b25lX3JlY2xhaW1hYmxlX3BhZ2VzKHpvbmUpICogNjsN Ci19DQotDQogLyogQWxsIHpvbmVzIGluIHpvbmVsaXN0IGFyZSB1bnJlY2xhaW1hYmxlPyAqLw0K IHN0YXRpYyBib29sIGFsbF91bnJlY2xhaW1hYmxlKHN0cnVjdCB6b25lbGlzdCAqem9uZWxpc3Qs DQogCQlzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykNCkBAIC0yMzAxLDggKzIyOTYsNiBAQCBzdGF0 aWMgYm9vbCBhbGxfdW5yZWNsYWltYWJsZShzdHJ1Y3Qgem9uZWxpc3QgKnpvbmVsaXN0LA0KIAkJ CWNvbnRpbnVlOw0KIAkJaWYgKCFjcHVzZXRfem9uZV9hbGxvd2VkX2hhcmR3YWxsKHpvbmUsIEdG UF9LRVJORUwpKQ0KIAkJCWNvbnRpbnVlOw0KLQkJaWYgKHpvbmUtPmFsbF91bnJlY2xhaW1hYmxl KQ0KLQkJCWNvbnRpbnVlOw0KIAkJaWYgKHpvbmVfcmVjbGFpbWFibGUoem9uZSkpDQogCQkJcmV0 dXJuIGZhbHNlOw0KIAl9DQpAQCAtMjcxNCw3ICsyNzA3LDcgQEAgc3RhdGljIGJvb2wgcGdkYXRf YmFsYW5jZWQocGdfZGF0YV90ICpwZ2RhdCwgaW50IG9yZGVyLCBpbnQgY2xhc3N6b25lX2lkeCkN CiAJCSAqIERFRl9QUklPUklUWS4gRWZmZWN0aXZlbHksIGl0IGNvbnNpZGVycyB0aGVtIGJhbGFu Y2VkIHNvDQogCQkgKiB0aGV5IG11c3QgYmUgY29uc2lkZXJlZCBiYWxhbmNlZCBoZXJlIGFzIHdl bGwhDQogCQkgKi8NCi0JCWlmICh6b25lLT5hbGxfdW5yZWNsYWltYWJsZSkgew0KKwkJaWYgKCF6 b25lX3JlY2xhaW1hYmxlKHpvbmUpKSB7DQogCQkJYmFsYW5jZWRfcGFnZXMgKz0gem9uZS0+bWFu YWdlZF9wYWdlczsNCiAJCQljb250aW51ZTsNCiAJCX0NCkBAIC0yNzc1LDcgKzI3NjgsNiBAQCBz dGF0aWMgYm9vbCBrc3dhcGRfc2hyaW5rX3pvbmUoc3RydWN0IHpvbmUgKnpvbmUsDQogCQkJICAg ICAgIHVuc2lnbmVkIGxvbmcgbHJ1X3BhZ2VzLA0KIAkJCSAgICAgICB1bnNpZ25lZCBsb25nICpu cl9hdHRlbXB0ZWQpDQogew0KLQl1bnNpZ25lZCBsb25nIG5yX3NsYWI7DQogCWludCB0ZXN0b3Jk ZXIgPSBzYy0+b3JkZXI7DQogCXVuc2lnbmVkIGxvbmcgYmFsYW5jZV9nYXA7DQogCXN0cnVjdCBy ZWNsYWltX3N0YXRlICpyZWNsYWltX3N0YXRlID0gY3VycmVudC0+cmVjbGFpbV9zdGF0ZTsNCkBA IC0yODIwLDE1ICsyODEyLDEyIEBAIHN0YXRpYyBib29sIGtzd2FwZF9zaHJpbmtfem9uZShzdHJ1 Y3Qgem9uZSAqem9uZSwNCiAJc2hyaW5rX3pvbmUoem9uZSwgc2MpOw0KIA0KIAlyZWNsYWltX3N0 YXRlLT5yZWNsYWltZWRfc2xhYiA9IDA7DQotCW5yX3NsYWIgPSBzaHJpbmtfc2xhYigmc2hyaW5r LCBzYy0+bnJfc2Nhbm5lZCwgbHJ1X3BhZ2VzKTsNCisJc2hyaW5rX3NsYWIoJnNocmluaywgc2Mt Pm5yX3NjYW5uZWQsIGxydV9wYWdlcyk7DQogCXNjLT5ucl9yZWNsYWltZWQgKz0gcmVjbGFpbV9z dGF0ZS0+cmVjbGFpbWVkX3NsYWI7DQogDQogCS8qIEFjY291bnQgZm9yIHRoZSBudW1iZXIgb2Yg cGFnZXMgYXR0ZW1wdGVkIHRvIHJlY2xhaW0gKi8NCiAJKm5yX2F0dGVtcHRlZCArPSBzYy0+bnJf dG9fcmVjbGFpbTsNCiANCi0JaWYgKG5yX3NsYWIgPT0gMCAmJiAhem9uZV9yZWNsYWltYWJsZSh6 b25lKSkNCi0JCXpvbmUtPmFsbF91bnJlY2xhaW1hYmxlID0gMTsNCi0NCiAJem9uZV9jbGVhcl9m bGFnKHpvbmUsIFpPTkVfV1JJVEVCQUNLKTsNCiANCiAJLyoNCkBAIC0yODM3LDcgKzI4MjYsNyBA QCBzdGF0aWMgYm9vbCBrc3dhcGRfc2hyaW5rX3pvbmUoc3RydWN0IHpvbmUgKnpvbmUsDQogCSAq IEJESXMgYnV0IGFzIHByZXNzdXJlIGlzIHJlbGlldmVkLCBzcGVjdWxhdGl2ZWx5IGF2b2lkIGNv bmdlc3Rpb24NCiAJICogd2FpdHMuDQogCSAqLw0KLQlpZiAoIXpvbmUtPmFsbF91bnJlY2xhaW1h YmxlICYmDQorCWlmICh6b25lX3JlY2xhaW1hYmxlKHpvbmUpICYmDQogCSAgICB6b25lX2JhbGFu Y2VkKHpvbmUsIHRlc3RvcmRlciwgMCwgY2xhc3N6b25lX2lkeCkpIHsNCiAJCXpvbmVfY2xlYXJf ZmxhZyh6b25lLCBaT05FX0NPTkdFU1RFRCk7DQogCQl6b25lX2NsZWFyX2ZsYWcoem9uZSwgWk9O RV9UQUlMX0xSVV9ESVJUWSk7DQpAQCAtMjkwMyw3ICsyODkyLDcgQEAgc3RhdGljIHVuc2lnbmVk IGxvbmcgYmFsYW5jZV9wZ2RhdChwZ19kYXRhX3QgKnBnZGF0LCBpbnQgb3JkZXIsDQogCQkJaWYg KCFwb3B1bGF0ZWRfem9uZSh6b25lKSkNCiAJCQkJY29udGludWU7DQogDQotCQkJaWYgKHpvbmUt PmFsbF91bnJlY2xhaW1hYmxlICYmDQorCQkJaWYgKCF6b25lX3JlY2xhaW1hYmxlKHpvbmUpICYm DQogCQkJICAgIHNjLnByaW9yaXR5ICE9IERFRl9QUklPUklUWSkNCiAJCQkJY29udGludWU7DQog DQpAQCAtMjk4Miw3ICsyOTcxLDcgQEAgc3RhdGljIHVuc2lnbmVkIGxvbmcgYmFsYW5jZV9wZ2Rh dChwZ19kYXRhX3QgKnBnZGF0LCBpbnQgb3JkZXIsDQogCQkJaWYgKCFwb3B1bGF0ZWRfem9uZSh6 b25lKSkNCiAJCQkJY29udGludWU7DQogDQotCQkJaWYgKHpvbmUtPmFsbF91bnJlY2xhaW1hYmxl ICYmDQorCQkJaWYgKCF6b25lX3JlY2xhaW1hYmxlKHpvbmUpICYmDQogCQkJICAgIHNjLnByaW9y aXR5ICE9IERFRl9QUklPUklUWSkNCiAJCQkJY29udGludWU7DQogDQpAQCAtMzI2NywyMCArMzI1 Niw2IEBAIHVuc2lnbmVkIGxvbmcgZ2xvYmFsX3JlY2xhaW1hYmxlX3BhZ2VzKHZvaWQpDQogCXJl dHVybiBucjsNCiB9DQogDQotdW5zaWduZWQgbG9uZyB6b25lX3JlY2xhaW1hYmxlX3BhZ2VzKHN0 cnVjdCB6b25lICp6b25lKQ0KLXsNCi0JaW50IG5yOw0KLQ0KLQluciA9IHpvbmVfcGFnZV9zdGF0 ZSh6b25lLCBOUl9BQ1RJVkVfRklMRSkgKw0KLQkgICAgIHpvbmVfcGFnZV9zdGF0ZSh6b25lLCBO Ul9JTkFDVElWRV9GSUxFKTsNCi0NCi0JaWYgKGdldF9ucl9zd2FwX3BhZ2VzKCkgPiAwKQ0KLQkJ bnIgKz0gem9uZV9wYWdlX3N0YXRlKHpvbmUsIE5SX0FDVElWRV9BTk9OKSArDQotCQkgICAgICB6 b25lX3BhZ2Vfc3RhdGUoem9uZSwgTlJfSU5BQ1RJVkVfQU5PTik7DQotDQotCXJldHVybiBucjsN Ci19DQotDQogI2lmZGVmIENPTkZJR19ISUJFUk5BVElPTg0KIC8qDQogICogVHJ5IHRvIGZyZWUg YG5yX3RvX3JlY2xhaW0nIG9mIG1lbW9yeSwgc3lzdGVtLXdpZGUsIGFuZCByZXR1cm4gdGhlIG51 bWJlciBvZg0KQEAgLTM1NzgsNyArMzU1Myw3IEBAIGludCB6b25lX3JlY2xhaW0oc3RydWN0IHpv bmUgKnpvbmUsIGdmcF90IGdmcF9tYXNrLCB1bnNpZ25lZCBpbnQgb3JkZXIpDQogCSAgICB6b25l X3BhZ2Vfc3RhdGUoem9uZSwgTlJfU0xBQl9SRUNMQUlNQUJMRSkgPD0gem9uZS0+bWluX3NsYWJf cGFnZXMpDQogCQlyZXR1cm4gWk9ORV9SRUNMQUlNX0ZVTEw7DQogDQotCWlmICh6b25lLT5hbGxf dW5yZWNsYWltYWJsZSkNCisJaWYgKCF6b25lX3JlY2xhaW1hYmxlKHpvbmUpKQ0KIAkJcmV0dXJu IFpPTkVfUkVDTEFJTV9GVUxMOw0KIA0KIAkvKg0KZGlmZiAtLWdpdCBhL21tL3Ztc3RhdC5jIGIv bW0vdm1zdGF0LmMNCmluZGV4IDIwYzJlZjQuLmM0OGY3NWIgMTAwNjQ0DQotLS0gYS9tbS92bXN0 YXQuYw0KKysrIGIvbW0vdm1zdGF0LmMNCkBAIC0xOSw2ICsxOSw3IEBADQogI2luY2x1ZGUgPGxp bnV4L21hdGg2NC5oPg0KICNpbmNsdWRlIDxsaW51eC93cml0ZWJhY2suaD4NCiAjaW5jbHVkZSA8 bGludXgvY29tcGFjdGlvbi5oPg0KKyNpbmNsdWRlIDxsaW51eC9tbV9pbmxpbmUuaD4NCiANCiAj aWZkZWYgQ09ORklHX1ZNX0VWRU5UX0NPVU5URVJTDQogREVGSU5FX1BFUl9DUFUoc3RydWN0IHZt X2V2ZW50X3N0YXRlLCB2bV9ldmVudF9zdGF0ZXMpID0ge3swfX07DQpAQCAtMTA1Miw3ICsxMDUz LDcgQEAgc3RhdGljIHZvaWQgem9uZWluZm9fc2hvd19wcmludChzdHJ1Y3Qgc2VxX2ZpbGUgKm0s IHBnX2RhdGFfdCAqcGdkYXQsDQogCQkgICAiXG4gIGFsbF91bnJlY2xhaW1hYmxlOiAldSINCiAJ CSAgICJcbiAgc3RhcnRfcGZuOiAgICAgICAgICVsdSINCiAJCSAgICJcbiAgaW5hY3RpdmVfcmF0 aW86ICAgICV1IiwNCi0JCSAgIHpvbmUtPmFsbF91bnJlY2xhaW1hYmxlLA0KKwkJICAgIXpvbmVf cmVjbGFpbWFibGUoem9uZSksDQogCQkgICB6b25lLT56b25lX3N0YXJ0X3BmbiwNCiAJCSAgIHpv bmUtPmluYWN0aXZlX3JhdGlvKTsNCiAJc2VxX3B1dGMobSwgJ1xuJyk7DQotLSANCjEuNy4wLjQN Cg0KPg0KPj4NCj4+ID4NCj4+ID4NCj4+ID4NCj4+ID4tLQ0KPj4gPktpbmQgcmVnYXJkcywNCj4+ ID5NaW5jaGFuIEtpbQ0KPg0KPi0tDQo+S2luZCByZWdhcmRzLA0KPk1pbmNoYW4gS2ltDQo= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx151.postini.com [74.125.245.151]) by kanga.kvack.org (Postfix) with SMTP id B8CEC6B0031 for ; Sat, 3 Aug 2013 17:22:08 -0400 (EDT) Received: by mail-qa0-f52.google.com with SMTP id bq6so305563qab.4 for ; Sat, 03 Aug 2013 14:22:07 -0700 (PDT) Message-ID: <51FD7483.5060504@gmail.com> Date: Sat, 03 Aug 2013 17:22:11 -0400 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: Possible deadloop in direct reclaim? References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> <51F9E265.7030503@oracle.com> In-Reply-To: <51F9E265.7030503@oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: KOSAKI Motohiro , Lisa Du , Christoph Lameter , "linux-mm@kvack.org" , Mel Gorman , Bob Liu (8/1/13 12:21 AM), Bob Liu wrote: > Hi KOSAKI, > > On 08/01/2013 10:45 AM, KOSAKI Motohiro wrote: > >> >> Please read more older code. Your pointed code is temporary change and I >> changed back for fixing >> bugs. >> If you look at the status in middle direct reclaim, we can't avoid race >> condition from multi direct >> reclaim issues. Moreover, if kswapd doesn't awaken, it is a problem. >> This is a reason why current code >> behave as you described. >> I agree we should fix your issue as far as possible. But I can't agree >> your analysis. >> > > I found this thread: > mm, vmscan: fix do_try_to_free_pages() livelock > https://lkml.org/lkml/2012/6/14/74 > > I think that's the same issue Lisa met. > > But I didn't find out why your patch didn't get merged? > There were already many acks. Just because I misunderstood the patch has already been merged. OK, I'll resend this. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx155.postini.com [74.125.245.155]) by kanga.kvack.org (Postfix) with SMTP id 43B3A6B0031 for ; Sun, 4 Aug 2013 19:47:01 -0400 (EDT) Date: Mon, 5 Aug 2013 08:47:40 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130804234740.GG32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <20130801054338.GD19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE04E@SC-VEXCH4.marvell.com> <20130801073330.GG19540@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE0E3@SC-VEXCH4.marvell.com> <20130801084259.GA32486@bbox> <20130802015241.GB32486@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE43E@SC-VEXCH4.marvell.com> <20130802035333.GD32486@bbox> <89813612683626448B837EE5A0B6A7CB3B630BE511@SC-VEXCH4.marvell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <89813612683626448B837EE5A0B6A7CB3B630BE511@SC-VEXCH4.marvell.com> Sender: owner-linux-mm@kvack.org List-ID: To: Lisa Du Cc: "linux-mm@kvack.org" , KOSAKI Motohiro , Bob Liu Hello, On Fri, Aug 02, 2013 at 01:08:50AM -0700, Lisa Du wrote: > >-----Original Message----- > >From: Minchan Kim [mailto:minchan@kernel.org] > >Sent: 2013a1'8ae??2ae?JPY 11:54 > >To: Lisa Du > >Cc: linux-mm@kvack.org; KOSAKI Motohiro; Bob Liu > >Subject: Re: Possible deadloop in direct reclaim? > > > >On Thu, Aug 01, 2013 at 08:17:56PM -0700, Lisa Du wrote: > >> >-----Original Message----- > >> >From: Minchan Kim [mailto:minchan@kernel.org] > >> >Sent: 2013a1'8ae??2ae?JPY 10:26 > >> >To: Lisa Du > >> >Cc: linux-mm@kvack.org; KOSAKI Motohiro > >> >Subject: Re: Possible deadloop in direct reclaim? > > > >> >I reviewed current mmotm because recently Mel changed kswapd a lot and > >> >all_unreclaimable patch history today. > >> >What I see is recent mmotm has a same problem, too if system have no swap > >> >and no compaction. Of course, compaction is default yes option so we could > >> >recommend to enable if system works well but it's up to user and we should > >> >avoid direct reclaim hang although user disable compaction. > >> > > >> >When I see the patch history, real culprit is 929bea7c. > >> > > >> >" zone->all_unreclaimable and zone->pages_scanned are neigher atomic > >> > variables nor protected by lock. Therefore zones can become a state of > >> > zone->page_scanned=0 and zone->all_unreclaimable=1. In this case, current > >> > all_unreclaimable() return false even though zone->all_unreclaimabe=1." > >> > > >> >I understand the problem but apparently, it makes Lisa's problem because > >> >kswapd can give up balancing when high order allocation happens to prevent > >> >excessive reclaim with assuming the process requested high order allocation > >> >can do direct reclaim/compaction. But what if the process can't reclaim > >> >by no swap but lots of anon pages and can't compact by !CONFIG_COMPACTION? > >> > > >> > >> Also Bob found below thread, seems Kosaki also found same issue: > >> mm, vmscan: fix do_try_to_free_pages() livelock > >> https://lkml.org/lkml/2012/6/14/74 > > > >I remember it and AFAIRC, I had a concern because description was > >too vague without detailed example and I fixed Aaditya's problem with > >another approach. That's why it wasn't merged at that time. > > > >Now, we have a real problem and analysis so I think KOSAKI's patch makes > >perfect to me. > > > >Lisa, Could you resend KOSAKI's patch with more detailed description? > > Hi, Minchan and Kosaki > Would you please help check below patch I resend based on previous Kosaki's patch? > I'm not sure if the description is clear enough, please let me know if you have any comments. > Many thanks! > From 2dfe137665a694dcc74ae9c8a27641b06190f344 Mon Sep 17 00:00:00 2001 > From: Lisa Du > Date: Fri, 2 Aug 2013 14:37:31 +0800 > Subject: [PATCH] mm: vmscan: fix do_try_to_free_pages() livelock > > Currently, I found system can enter a state that there are lots > of free pages in a zone but only order-0 and order-1 pages which > means the zone is heavily fragmented, then high order allocation > could make direct reclaim path's long stall(ex, 60 seconds) > especially in no swap and no compaciton enviroment. > > The reason is do_try_to_free_pages enter live lock: > > kswapd will go to sleep if the zones have been fully scanned > and are still not balanced. As kswapd thinks there's little point > trying all over again to avoid infinite loop. Instead it changes > order from high-order to 0-order because kswapd think order-0 is the > most important. Look at 73ce02e9 in detail. If watermarks are ok, > kswapd will go back to sleep and may leave zone->all_unreclaimable = 0. > It assume high-order users can still perform direct reclaim if they wish. > > Direct reclaim continue to reclaim for a high order which is not a > COSTLY_ORDER without oom-killer until kswapd turn on zone->all_unreclaimble. > This is because to avoid too early oom-kill. So it means direct_reclaim > depends on kswapd to break this loop. > > In worst case, direct-reclaim may continue to page reclaim forever > when kswapd sleeps forever until someone like watchdog detect and finally > kill the process. > > We can't turn on zone->all_unreclaimable from direct reclaim path > because direct reclaim path don't take any lock and this way is racy. > Thus this patch removes zone->all_unreclaimable field completely and > recalculates zone reclaimable state every time. > > Note: we can't take the idea that direct-reclaim see zone->pages_scanned > directly and kswapd continue to use zone->all_unreclaimable. Because, it > is racy. commit 929bea7c71 (vmscan: all_unreclaimable() use > zone->all_unreclaimable as a name) describes the detail. > > Change-Id: If3b44e33e400c1db0e42a5e2fc9ebc7a265f2aae Remove Change-ID. And please write down explicitly "It's based KOSAKI's work and I rewrite the description" In addtion, write down "The problem happend v3.4 but it seems the problem still lives in current tree because ..." Otherwise, looks good to me. If you respin, please send a mail with new thread for akpm to be confused. Thanks! > Cc: Aaditya Kumar > Cc: Ying Han > Cc: Nick Piggin > Acked-by: Rik van Riel > Cc: Michal Hocko > Cc: Johannes Weiner > Cc: Mel Gorman > Cc: KAMEZAWA Hiroyuki > Cc: Minchan Kim > Cc: Bob Liu > Signed-off-by: KOSAKI Motohiro > Signed-off-by: Lisa Du > --- > include/linux/mm_inline.h | 20 ++++++++++++++++++++ > include/linux/mmzone.h | 1 - > include/linux/vmstat.h | 1 - > mm/page-writeback.c | 1 + > mm/page_alloc.c | 5 ++--- > mm/vmscan.c | 43 +++++++++---------------------------------- > mm/vmstat.c | 3 ++- > 7 files changed, 34 insertions(+), 40 deletions(-) > > diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h > index 1397ccf..e212fae 100644 > --- a/include/linux/mm_inline.h > +++ b/include/linux/mm_inline.h > @@ -2,6 +2,7 @@ > #define LINUX_MM_INLINE_H > > #include > +#include > > /** > * page_is_file_cache - should the page be on a file LRU or anon LRU? > @@ -99,4 +100,23 @@ static __always_inline enum lru_list page_lru(struct page *page) > return lru; > } > > +static inline unsigned long zone_reclaimable_pages(struct zone *zone) > +{ > + int nr; > + > + nr = zone_page_state(zone, NR_ACTIVE_FILE) + > + zone_page_state(zone, NR_INACTIVE_FILE); > + > + if (get_nr_swap_pages() > 0) > + nr += zone_page_state(zone, NR_ACTIVE_ANON) + > + zone_page_state(zone, NR_INACTIVE_ANON); > + > + return nr; > +} > + > +static inline bool zone_reclaimable(struct zone *zone) > +{ > + return zone->pages_scanned < zone_reclaimable_pages(zone) * 6; > +} > + > #endif > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h > index af4a3b7..e835974 100644 > --- a/include/linux/mmzone.h > +++ b/include/linux/mmzone.h > @@ -352,7 +352,6 @@ struct zone { > * free areas of different sizes > */ > spinlock_t lock; > - int all_unreclaimable; /* All pages pinned */ > #if defined CONFIG_COMPACTION || defined CONFIG_CMA > /* Set to true when the PG_migrate_skip bits should be cleared */ > bool compact_blockskip_flush; > diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h > index c586679..6fff004 100644 > --- a/include/linux/vmstat.h > +++ b/include/linux/vmstat.h > @@ -143,7 +143,6 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone, > } > > extern unsigned long global_reclaimable_pages(void); > -extern unsigned long zone_reclaimable_pages(struct zone *zone); > > #ifdef CONFIG_NUMA > /* > diff --git a/mm/page-writeback.c b/mm/page-writeback.c > index 3f0c895..62bfd92 100644 > --- a/mm/page-writeback.c > +++ b/mm/page-writeback.c > @@ -36,6 +36,7 @@ > #include > #include > #include > +#include > #include > > /* > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index b100255..19a18c0 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -60,6 +60,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -647,7 +648,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, > int to_free = count; > > spin_lock(&zone->lock); > - zone->all_unreclaimable = 0; > zone->pages_scanned = 0; > > while (to_free) { > @@ -696,7 +696,6 @@ static void free_one_page(struct zone *zone, struct page *page, int order, > int migratetype) > { > spin_lock(&zone->lock); > - zone->all_unreclaimable = 0; > zone->pages_scanned = 0; > > __free_one_page(page, zone, order, migratetype); > @@ -3095,7 +3094,7 @@ void show_free_areas(unsigned int filter) > K(zone_page_state(zone, NR_FREE_CMA_PAGES)), > K(zone_page_state(zone, NR_WRITEBACK_TEMP)), > zone->pages_scanned, > - (zone->all_unreclaimable ? "yes" : "no") > + (!zone_reclaimable(zone) ? "yes" : "no") > ); > printk("lowmem_reserve[]:"); > for (i = 0; i < MAX_NR_ZONES; i++) > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 34582d9..7501d1e 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1789,7 +1789,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, > * latencies, so it's better to scan a minimum amount there as > * well. > */ > - if (current_is_kswapd() && zone->all_unreclaimable) > + if (current_is_kswapd() && !zone_reclaimable(zone)) > force_scan = true; > if (!global_reclaim(sc)) > force_scan = true; > @@ -2244,8 +2244,8 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc) > if (global_reclaim(sc)) { > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > continue; > - if (zone->all_unreclaimable && > - sc->priority != DEF_PRIORITY) > + if (!zone_reclaimable(zone) && > + sc->priority != DEF_PRIORITY) > continue; /* Let kswapd poll it */ > if (IS_ENABLED(CONFIG_COMPACTION)) { > /* > @@ -2283,11 +2283,6 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc) > return aborted_reclaim; > } > > -static bool zone_reclaimable(struct zone *zone) > -{ > - return zone->pages_scanned < zone_reclaimable_pages(zone) * 6; > -} > - > /* All zones in zonelist are unreclaimable? */ > static bool all_unreclaimable(struct zonelist *zonelist, > struct scan_control *sc) > @@ -2301,8 +2296,6 @@ static bool all_unreclaimable(struct zonelist *zonelist, > continue; > if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL)) > continue; > - if (zone->all_unreclaimable) > - continue; > if (zone_reclaimable(zone)) > return false; > } > @@ -2714,7 +2707,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int classzone_idx) > * DEF_PRIORITY. Effectively, it considers them balanced so > * they must be considered balanced here as well! > */ > - if (zone->all_unreclaimable) { > + if (!zone_reclaimable(zone)) { > balanced_pages += zone->managed_pages; > continue; > } > @@ -2775,7 +2768,6 @@ static bool kswapd_shrink_zone(struct zone *zone, > unsigned long lru_pages, > unsigned long *nr_attempted) > { > - unsigned long nr_slab; > int testorder = sc->order; > unsigned long balance_gap; > struct reclaim_state *reclaim_state = current->reclaim_state; > @@ -2820,15 +2812,12 @@ static bool kswapd_shrink_zone(struct zone *zone, > shrink_zone(zone, sc); > > reclaim_state->reclaimed_slab = 0; > - nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages); > + shrink_slab(&shrink, sc->nr_scanned, lru_pages); > sc->nr_reclaimed += reclaim_state->reclaimed_slab; > > /* Account for the number of pages attempted to reclaim */ > *nr_attempted += sc->nr_to_reclaim; > > - if (nr_slab == 0 && !zone_reclaimable(zone)) > - zone->all_unreclaimable = 1; > - > zone_clear_flag(zone, ZONE_WRITEBACK); > > /* > @@ -2837,7 +2826,7 @@ static bool kswapd_shrink_zone(struct zone *zone, > * BDIs but as pressure is relieved, speculatively avoid congestion > * waits. > */ > - if (!zone->all_unreclaimable && > + if (zone_reclaimable(zone) && > zone_balanced(zone, testorder, 0, classzone_idx)) { > zone_clear_flag(zone, ZONE_CONGESTED); > zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY); > @@ -2903,7 +2892,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, > if (!populated_zone(zone)) > continue; > > - if (zone->all_unreclaimable && > + if (!zone_reclaimable(zone) && > sc.priority != DEF_PRIORITY) > continue; > > @@ -2982,7 +2971,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, > if (!populated_zone(zone)) > continue; > > - if (zone->all_unreclaimable && > + if (!zone_reclaimable(zone) && > sc.priority != DEF_PRIORITY) > continue; > > @@ -3267,20 +3256,6 @@ unsigned long global_reclaimable_pages(void) > return nr; > } > > -unsigned long zone_reclaimable_pages(struct zone *zone) > -{ > - int nr; > - > - nr = zone_page_state(zone, NR_ACTIVE_FILE) + > - zone_page_state(zone, NR_INACTIVE_FILE); > - > - if (get_nr_swap_pages() > 0) > - nr += zone_page_state(zone, NR_ACTIVE_ANON) + > - zone_page_state(zone, NR_INACTIVE_ANON); > - > - return nr; > -} > - > #ifdef CONFIG_HIBERNATION > /* > * Try to free `nr_to_reclaim' of memory, system-wide, and return the number of > @@ -3578,7 +3553,7 @@ int zone_reclaim(struct zone *zone, gfp_t gfp_mask, unsigned int order) > zone_page_state(zone, NR_SLAB_RECLAIMABLE) <= zone->min_slab_pages) > return ZONE_RECLAIM_FULL; > > - if (zone->all_unreclaimable) > + if (!zone_reclaimable(zone)) > return ZONE_RECLAIM_FULL; > > /* > diff --git a/mm/vmstat.c b/mm/vmstat.c > index 20c2ef4..c48f75b 100644 > --- a/mm/vmstat.c > +++ b/mm/vmstat.c > @@ -19,6 +19,7 @@ > #include > #include > #include > +#include > > #ifdef CONFIG_VM_EVENT_COUNTERS > DEFINE_PER_CPU(struct vm_event_state, vm_event_states) = {{0}}; > @@ -1052,7 +1053,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, > "\n all_unreclaimable: %u" > "\n start_pfn: %lu" > "\n inactive_ratio: %u", > - zone->all_unreclaimable, > + !zone_reclaimable(zone), > zone->zone_start_pfn, > zone->inactive_ratio); > seq_putc(m, '\n'); > -- > 1.7.0.4 > > > > >> > >> > > >> > > >> > > >> >-- > >> >Kind regards, > >> >Minchan Kim > > > >-- > >Kind regards, > >Minchan Kim -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx149.postini.com [74.125.245.149]) by kanga.kvack.org (Postfix) with SMTP id 4C3476B0031 for ; Sun, 4 Aug 2013 19:49:32 -0400 (EDT) Date: Mon, 5 Aug 2013 08:50:11 +0900 From: Minchan Kim Subject: Re: Possible deadloop in direct reclaim? Message-ID: <20130804235011.GH32486@bbox> References: <89813612683626448B837EE5A0B6A7CB3B62F8F272@SC-VEXCH4.marvell.com> <000001400d38469d-a121fb96-4483-483a-9d3e-fc552e413892-000000@email.amazonses.com> <89813612683626448B837EE5A0B6A7CB3B62F8F5C3@SC-VEXCH4.marvell.com> <89813612683626448B837EE5A0B6A7CB3B62F8FE33@SC-VEXCH4.marvell.com> <51F69BD7.2060407@gmail.com> <89813612683626448B837EE5A0B6A7CB3B630BDF99@SC-VEXCH4.marvell.com> <51F9CBC0.2020006@gmail.com> <51F9E265.7030503@oracle.com> <51FD7483.5060504@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51FD7483.5060504@gmail.com> Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: Bob Liu , Lisa Du , Christoph Lameter