From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: [PATCH 3/5] gpu/drm/ttm: Use mutex_trylock() to avoid deadlock inside shrinker functions. Date: Tue, 10 Jun 2014 15:17:41 -0400 Message-ID: <20140610191741.GA28523@phenom.dumpdata.com> References: <201405290647.DHI69200.HSFVFMFOJOLOQt@I-love.SAKURA.ne.jp> <201405292334.EAG00503.FLOOJFStHVQMFO@I-love.SAKURA.ne.jp> <20140530160824.GD3621@localhost.localdomain> <201405311158.DGE64002.QLOOHJSFFMVFOt@I-love.SAKURA.ne.jp> <201405311159.CHG64048.SOFLQHVtFOMFJO@I-love.SAKURA.ne.jp> <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by gabe.freedesktop.org (Postfix) with ESMTP id C16BB6E2C5 for ; Tue, 10 Jun 2014 12:17:54 -0700 (PDT) Content-Disposition: inline In-Reply-To: <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Tetsuo Handa Cc: linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, glommer@openvz.org, linux-mm@kvack.org, mgorman@suse.de, dchinner@redhat.com List-Id: dri-devel@lists.freedesktop.org T24gU2F0LCBNYXkgMzEsIDIwMTQgYXQgMTI6MDA6NDVQTSArMDkwMCwgVGV0c3VvIEhhbmRhIHdy b3RlOgo+ID5Gcm9tIDRlOGQxYTgzNjI5YzU5NjZiZmQ0MDFjNWYyMTg3MzU1NjI0MTk0ZjIgTW9u IFNlcCAxNyAwMDowMDowMCAyMDAxCj4gRnJvbTogVGV0c3VvIEhhbmRhIDxwZW5ndWluLWtlcm5l bEBJLWxvdmUuU0FLVVJBLm5lLmpwPgo+IERhdGU6IFNhdCwgMzEgTWF5IDIwMTQgMDk6NTk6NDQg KzA5MDAKPiBTdWJqZWN0OiBbUEFUQ0ggMy81XSBncHUvZHJtL3R0bTogVXNlIG11dGV4X3RyeWxv Y2soKSB0byBhdm9pZCBkZWFkbG9jayBpbnNpZGUgc2hyaW5rZXIgZnVuY3Rpb25zLgo+IAo+IEkg Y2FuIG9ic2VydmUgdGhhdCBSSEVMNyBlbnZpcm9ubWVudCBzdGFsbHMgd2l0aCAxMDAlIENQVSB1 c2FnZSB3aGVuIGEKPiBjZXJ0YWluIHR5cGUgb2YgbWVtb3J5IHByZXNzdXJlIGlzIGdpdmVuLiBX aGlsZSB0aGUgc2hyaW5rZXIgZnVuY3Rpb25zCj4gYXJlIGNhbGxlZCBieSBzaHJpbmtfc2xhYigp IGJlZm9yZSB0aGUgT09NIGtpbGxlciBpcyB0cmlnZ2VyZWQsIHRoZSBzdGFsbAo+IGxhc3RzIGZv ciBtYW55IG1pbnV0ZXMuCj4gCj4gT25lIG9mIHJlYXNvbnMgb2YgdGhpcyBzdGFsbCBpcyB0aGF0 Cj4gdHRtX2RtYV9wb29sX3Nocmlua19jb3VudCgpL3R0bV9kbWFfcG9vbF9zaHJpbmtfc2Nhbigp IGFyZSBjYWxsZWQgYW5kCj4gYXJlIGJsb2NrZWQgYXQgbXV0ZXhfbG9jaygmX21hbmFnZXItPmxv Y2spLiBHRlBfS0VSTkVMIGFsbG9jYXRpb24gd2l0aAo+IF9tYW5hZ2VyLT5sb2NrIGhlbGQgY2F1 c2VzIHNvbWVvbmUgKGluY2x1ZGluZyBrc3dhcGQpIHRvIGRlYWRsb2NrIHdoZW4KPiB0aGVzZSBm dW5jdGlvbnMgYXJlIGNhbGxlZCBkdWUgdG8gbWVtb3J5IHByZXNzdXJlLiBUaGlzIHBhdGNoIGNo YW5nZXMKPiAibXV0ZXhfbG9jaygpOyIgdG8gImlmICghbXV0ZXhfdHJ5bG9jaygpKSByZXR1cm4g Li4uOyIgaW4gb3JkZXIgdG8KPiBhdm9pZCBkZWFkbG9jay4KPiAKPiBTaWduZWQtb2ZmLWJ5OiBU ZXRzdW8gSGFuZGEgPHBlbmd1aW4ta2VybmVsQEktbG92ZS5TQUtVUkEubmUuanA+Cj4gQ2M6IHN0 YWJsZSA8c3RhYmxlQGtlcm5lbC5vcmc+IFszLjMrXQo+IC0tLQo+ICBkcml2ZXJzL2dwdS9kcm0v dHRtL3R0bV9wYWdlX2FsbG9jX2RtYS5jIHwgICAgNiArKysrLS0KPiAgMSBmaWxlcyBjaGFuZ2Vk LCA0IGluc2VydGlvbnMoKyksIDIgZGVsZXRpb25zKC0pCj4gCj4gZGlmZiAtLWdpdCBhL2RyaXZl cnMvZ3B1L2RybS90dG0vdHRtX3BhZ2VfYWxsb2NfZG1hLmMgYi9kcml2ZXJzL2dwdS9kcm0vdHRt L3R0bV9wYWdlX2FsbG9jX2RtYS5jCj4gaW5kZXggZDhlNTlmNy4uNjIwZGEzOSAxMDA2NDQKPiAt LS0gYS9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9wYWdlX2FsbG9jX2RtYS5jCj4gKysrIGIvZHJp dmVycy9ncHUvZHJtL3R0bS90dG1fcGFnZV9hbGxvY19kbWEuYwo+IEBAIC0xMDE0LDcgKzEwMTQs OCBAQCB0dG1fZG1hX3Bvb2xfc2hyaW5rX3NjYW4oc3RydWN0IHNocmlua2VyICpzaHJpbmssIHN0 cnVjdCBzaHJpbmtfY29udHJvbCAqc2MpCj4gIAlpZiAobGlzdF9lbXB0eSgmX21hbmFnZXItPnBv b2xzKSkKPiAgCQlyZXR1cm4gU0hSSU5LX1NUT1A7Cj4gIAo+IC0JbXV0ZXhfbG9jaygmX21hbmFn ZXItPmxvY2spOwo+ICsJaWYgKCFtdXRleF9sb2NrKCZfbWFuYWdlci0+bG9jaykpCj4gKwkJcmV0 dXJuIFNIUklOS19TVE9QOwoKSG1tLi4KCi9ob21lL2tvbnJhZC9saW51eC9kcml2ZXJzL2dwdS9k cm0vdHRtL3R0bV9wYWdlX2FsbG9jX2RtYS5jOiBJbiBmdW5jdGlvbiDigJh0dG1fZG1hX3Bvb2xf c2hyaW5rX3NjYW7igJk6Ci9ob21lL2tvbnJhZC9saW51eC9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0 bV9wYWdlX2FsbG9jX2RtYS5jOjEwMTU6MjogZXJyb3I6IGludmFsaWQgdXNlIG9mIHZvaWQgZXhw cmVzc2lvbgogIGlmICghbXV0ZXhfbG9jaygmX21hbmFnZXItPmxvY2spKQoKVGhpcyBpcyBiYXNl ZCBvbiB2My4xNSB3aXRoIHRoZXNlIHBhdGNoZXMuCgo+ICAJaWYgKCFfbWFuYWdlci0+bnBvb2xz KQo+ICAJCWdvdG8gb3V0Owo+ICAJcG9vbF9vZmZzZXQgPSArK3N0YXJ0X3Bvb2wgJSBfbWFuYWdl ci0+bnBvb2xzOwo+IEBAIC0xMDQ3LDcgKzEwNDgsOCBAQCB0dG1fZG1hX3Bvb2xfc2hyaW5rX2Nv dW50KHN0cnVjdCBzaHJpbmtlciAqc2hyaW5rLCBzdHJ1Y3Qgc2hyaW5rX2NvbnRyb2wgKnNjKQo+ ICAJc3RydWN0IGRldmljZV9wb29scyAqcDsKPiAgCXVuc2lnbmVkIGxvbmcgY291bnQgPSAwOwo+ ICAKPiAtCW11dGV4X2xvY2soJl9tYW5hZ2VyLT5sb2NrKTsKPiArCWlmICghbXV0ZXhfdHJ5bG9j aygmX21hbmFnZXItPmxvY2spKQo+ICsJCXJldHVybiAwOwo+ICAJbGlzdF9mb3JfZWFjaF9lbnRy eShwLCAmX21hbmFnZXItPnBvb2xzLCBwb29scykKPiAgCQljb3VudCArPSBwLT5wb29sLT5ucGFn ZXNfZnJlZTsKPiAgCW11dGV4X3VubG9jaygmX21hbmFnZXItPmxvY2spOwo+IC0tIAo+IDEuNy4x Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRyaS1kZXZl bCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwOi8vbGlz dHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f173.google.com (mail-ie0-f173.google.com [209.85.223.173]) by kanga.kvack.org (Postfix) with ESMTP id F3CCE6B0106 for ; Tue, 10 Jun 2014 15:17:53 -0400 (EDT) Received: by mail-ie0-f173.google.com with SMTP id y20so5157511ier.4 for ; Tue, 10 Jun 2014 12:17:53 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id d13si40319659icj.54.2014.06.10.12.17.52 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 10 Jun 2014 12:17:53 -0700 (PDT) Date: Tue, 10 Jun 2014 15:17:41 -0400 From: Konrad Rzeszutek Wilk Subject: Re: [PATCH 3/5] gpu/drm/ttm: Use mutex_trylock() to avoid deadlock inside shrinker functions. Message-ID: <20140610191741.GA28523@phenom.dumpdata.com> References: <201405290647.DHI69200.HSFVFMFOJOLOQt@I-love.SAKURA.ne.jp> <201405292334.EAG00503.FLOOJFStHVQMFO@I-love.SAKURA.ne.jp> <20140530160824.GD3621@localhost.localdomain> <201405311158.DGE64002.QLOOHJSFFMVFOt@I-love.SAKURA.ne.jp> <201405311159.CHG64048.SOFLQHVtFOMFJO@I-love.SAKURA.ne.jp> <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Tetsuo Handa Cc: dchinner@redhat.com, airlied@linux.ie, glommer@openvz.org, mgorman@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org On Sat, May 31, 2014 at 12:00:45PM +0900, Tetsuo Handa wrote: > >From 4e8d1a83629c5966bfd401c5f2187355624194f2 Mon Sep 17 00:00:00 2001 > From: Tetsuo Handa > Date: Sat, 31 May 2014 09:59:44 +0900 > Subject: [PATCH 3/5] gpu/drm/ttm: Use mutex_trylock() to avoid deadlock= inside shrinker functions. >=20 > I can observe that RHEL7 environment stalls with 100% CPU usage when a > certain type of memory pressure is given. While the shrinker functions > are called by shrink_slab() before the OOM killer is triggered, the sta= ll > lasts for many minutes. >=20 > One of reasons of this stall is that > ttm_dma_pool_shrink_count()/ttm_dma_pool_shrink_scan() are called and > are blocked at mutex_lock(&_manager->lock). GFP_KERNEL allocation with > _manager->lock held causes someone (including kswapd) to deadlock when > these functions are called due to memory pressure. This patch changes > "mutex_lock();" to "if (!mutex_trylock()) return ...;" in order to > avoid deadlock. >=20 > Signed-off-by: Tetsuo Handa > Cc: stable [3.3+] > --- > drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) >=20 > diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c b/drivers/gpu/drm= /ttm/ttm_page_alloc_dma.c > index d8e59f7..620da39 100644 > --- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > +++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > @@ -1014,7 +1014,8 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink,= struct shrink_control *sc) > if (list_empty(&_manager->pools)) > return SHRINK_STOP; > =20 > - mutex_lock(&_manager->lock); > + if (!mutex_lock(&_manager->lock)) > + return SHRINK_STOP; Hmm.. /home/konrad/linux/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c: In function = =E2=80=98ttm_dma_pool_shrink_scan=E2=80=99: /home/konrad/linux/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c:1015:2: error= : invalid use of void expression if (!mutex_lock(&_manager->lock)) This is based on v3.15 with these patches. > if (!_manager->npools) > goto out; > pool_offset =3D ++start_pool % _manager->npools; > @@ -1047,7 +1048,8 @@ ttm_dma_pool_shrink_count(struct shrinker *shrink= , struct shrink_control *sc) > struct device_pools *p; > unsigned long count =3D 0; > =20 > - mutex_lock(&_manager->lock); > + if (!mutex_trylock(&_manager->lock)) > + return 0; > list_for_each_entry(p, &_manager->pools, pools) > count +=3D p->pool->npages_free; > mutex_unlock(&_manager->lock); > --=20 > 1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753086AbaFJTSH (ORCPT ); Tue, 10 Jun 2014 15:18:07 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:44787 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751851AbaFJTSE convert rfc822-to-8bit (ORCPT ); Tue, 10 Jun 2014 15:18:04 -0400 Date: Tue, 10 Jun 2014 15:17:41 -0400 From: Konrad Rzeszutek Wilk To: Tetsuo Handa Cc: dchinner@redhat.com, airlied@linux.ie, glommer@openvz.org, mgorman@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: Re: [PATCH 3/5] gpu/drm/ttm: Use mutex_trylock() to avoid deadlock inside shrinker functions. Message-ID: <20140610191741.GA28523@phenom.dumpdata.com> References: <201405290647.DHI69200.HSFVFMFOJOLOQt@I-love.SAKURA.ne.jp> <201405292334.EAG00503.FLOOJFStHVQMFO@I-love.SAKURA.ne.jp> <20140530160824.GD3621@localhost.localdomain> <201405311158.DGE64002.QLOOHJSFFMVFOt@I-love.SAKURA.ne.jp> <201405311159.CHG64048.SOFLQHVtFOMFJO@I-love.SAKURA.ne.jp> <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201405311200.III57894.MLFOOFStQVHJFO@I-love.SAKURA.ne.jp> User-Agent: Mutt/1.5.23 (2014-03-12) Content-Transfer-Encoding: 8BIT X-Source-IP: acsinet21.oracle.com [141.146.126.237] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, May 31, 2014 at 12:00:45PM +0900, Tetsuo Handa wrote: > >From 4e8d1a83629c5966bfd401c5f2187355624194f2 Mon Sep 17 00:00:00 2001 > From: Tetsuo Handa > Date: Sat, 31 May 2014 09:59:44 +0900 > Subject: [PATCH 3/5] gpu/drm/ttm: Use mutex_trylock() to avoid deadlock inside shrinker functions. > > I can observe that RHEL7 environment stalls with 100% CPU usage when a > certain type of memory pressure is given. While the shrinker functions > are called by shrink_slab() before the OOM killer is triggered, the stall > lasts for many minutes. > > One of reasons of this stall is that > ttm_dma_pool_shrink_count()/ttm_dma_pool_shrink_scan() are called and > are blocked at mutex_lock(&_manager->lock). GFP_KERNEL allocation with > _manager->lock held causes someone (including kswapd) to deadlock when > these functions are called due to memory pressure. This patch changes > "mutex_lock();" to "if (!mutex_trylock()) return ...;" in order to > avoid deadlock. > > Signed-off-by: Tetsuo Handa > Cc: stable [3.3+] > --- > drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) > > diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > index d8e59f7..620da39 100644 > --- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > +++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > @@ -1014,7 +1014,8 @@ ttm_dma_pool_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > if (list_empty(&_manager->pools)) > return SHRINK_STOP; > > - mutex_lock(&_manager->lock); > + if (!mutex_lock(&_manager->lock)) > + return SHRINK_STOP; Hmm.. /home/konrad/linux/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c: In function ‘ttm_dma_pool_shrink_scan’: /home/konrad/linux/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c:1015:2: error: invalid use of void expression if (!mutex_lock(&_manager->lock)) This is based on v3.15 with these patches. > if (!_manager->npools) > goto out; > pool_offset = ++start_pool % _manager->npools; > @@ -1047,7 +1048,8 @@ ttm_dma_pool_shrink_count(struct shrinker *shrink, struct shrink_control *sc) > struct device_pools *p; > unsigned long count = 0; > > - mutex_lock(&_manager->lock); > + if (!mutex_trylock(&_manager->lock)) > + return 0; > list_for_each_entry(p, &_manager->pools, pools) > count += p->pool->npages_free; > mutex_unlock(&_manager->lock); > -- > 1.7.1