From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lucas Stach Subject: Re: [BUG 4.17] etnaviv-gpu f1840000.gpu: recover hung GPU! Date: Tue, 19 Jun 2018 14:28:46 +0200 Message-ID: <1529411326.7211.19.camel@pengutronix.de> References: <20180619094303.GE17671@n2100.armlinux.org.uk> <1529402956.7211.14.camel@pengutronix.de> <20180619110021.GF17671@n2100.armlinux.org.uk> <1529406689.7211.16.camel@pengutronix.de> <20180619114200.GG17671@n2100.armlinux.org.uk> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-ft3ZvzochtO3Uq4YCjHe" Return-path: Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 99AF589939 for ; Tue, 19 Jun 2018 12:28:50 +0000 (UTC) In-Reply-To: <20180619114200.GG17671@n2100.armlinux.org.uk> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Russell King - ARM Linux Cc: kernel@pengutronix.de, etnaviv@lists.freedesktop.org, dri-devel@lists.freedesktop.org, patchwork-lst@pengutronix.de List-Id: dri-devel@lists.freedesktop.org --=-ft3ZvzochtO3Uq4YCjHe Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Am Dienstag, den 19.06.2018, 12:42 +0100 schrieb Russell King - ARM Linux: > On Tue, Jun 19, 2018 at 01:11:29PM +0200, Lucas Stach wrote: > > Am Dienstag, den 19.06.2018, 12:00 +0100 schrieb Russell King - ARM Linux: > > > No, it's not "a really big job" - it's just that the Dove GC600 is not > > > fast enough to complete _two_ 1080p sized GPU operations within 500ms. > > > The preceeding job contained two blits - one of them a non-alphablend > > > copy of: > > > > > >                 00180000 04200780  0,24,1920,1056 -> 0,24,1920,1056 > > > > > > and one an alpha blended copy of: > > > > > >                 00000000 04380780  0,0,1920,1080 -> 0,0,1920,1080 > > > > > > This is (iirc) something I already fixed with the addition of the > > > progress detection back before etnaviv was merged into the mainline > > > kernel. > > > > I hadn't expected it to be this slow. I see that we might need to bring > > back the progress detection to fix the userspace regression, but I'm > > not fond of this, as it might lead to really bad QoS. > > Well, the choices are that or worse overall performance through having > to ignore the GPU entirely. > > > I would prefer userspace tracking the size of the blits and flushing > > the cmdstream at an appropriate time, so we don't end up with really > > long running jobs, but I'm not sure if this would be acceptable to > > you... > > The question becomes how to split up two operations.  Yes, we could > submit them individually, but if they're together taking in excess of > 500ms, then it's likely that individually, each operation will take in > excess of 250ms which is still a long time. > > In any case, I think we need to fix this for 4.17-stable and then try > to work (a) which operations are taking a long time, and (b) how to > solve this issue. Agreed. I'll look into bringing back the process detection for 4.17 stable. I'm still curious why the GC600 on the Dove is that slow. With performance like this moving a big(ish) window on the screen must be a horrible user experience. > Do we have any way to track how long each submitted job has actually > taken on the GPU?  (Eg, by recording the times that we receive the > events?)  It wouldn't be very accurate for small jobs, but given this > operation is taking so long, it would give an indication of how long > this operation is actually taking.  etnaviv doesn't appear to have > any tracepoints, which would've been ideal for that.  Maybe this is > a reason to add some? ;) See attached patch (which I apparently forgot to send out). The DRM GPU scheduler has some tracepoints, which might be helpful. The attached patch adds a drm_sched_job_run tracepoint when a job is queued in the hardware ring. Together with the existing drm_sched_process_job, this should get you an idea how long a job takes to process. Note that at any time up to 4 jobs are allowed in the hardware queue, so you need to match up the end times. Regards, Lucas --=-ft3ZvzochtO3Uq4YCjHe Content-Disposition: attachment; filename="0001-drm-sched-add-tracepoint-for-job-run.patch" Content-Type: text/x-patch; name="0001-drm-sched-add-tracepoint-for-job-run.patch"; charset="UTF-8" Content-Transfer-Encoding: base64 RnJvbSBhOWVjNDhkMWVlY2RkY2M5NTAxOGFkMzdlYmRmMTU0ZmZhN2NlOWE0IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBMdWNhcyBTdGFjaCA8bC5zdGFjaEBwZW5ndXRyb25peC5kZT4K RGF0ZTogRnJpLCA4IERlYyAyMDE3IDE4OjM1OjQzICswMTAwClN1YmplY3Q6IFtQQVRDSF0gZHJt L3NjaGVkOiBhZGQgdHJhY2Vwb2ludCBmb3Igam9iIHJ1bgoKV2hlbiB0cmFjaW5nIEdQVSBleGVj dXRpb24gaXQgaXMgdmVyeSBpbnRlcmVzdGluZyB0byBrbm93IHdoZW4gdGhlIGpvYiBnZXRzCmRl cXVldWVkIGZyb20gdGhlIHNvZnR3YXJlIHF1ZXVlIGFuZCBhZGRlZCB0byB0aGUgaGFyZHdhcmUg cmluZy4gQWRkIGEKdHJhY2Vwb2ludCB0byBhbGxvdyBlYXN5IGFjY2VzcyB0byB0aGlzIGluZm9y bWF0aW9uLgoKU2lnbmVkLW9mZi1ieTogTHVjYXMgU3RhY2ggPGwuc3RhY2hAcGVuZ3V0cm9uaXgu ZGU+Ci0tLQogZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9ncHVfc2NoZWR1bGVyLmMgfCAgMSAr CiBpbmNsdWRlL2RybS9ncHVfc2NoZWR1bGVyX3RyYWNlLmggICAgICAgICB8IDI3ICsrKysrKysr KysrKysrKysrKysrKysrCiAyIGZpbGVzIGNoYW5nZWQsIDI4IGluc2VydGlvbnMoKykKCmRpZmYg LS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL2dwdV9zY2hlZHVsZXIuYyBiL2RyaXZl cnMvZ3B1L2RybS9zY2hlZHVsZXIvZ3B1X3NjaGVkdWxlci5jCmluZGV4IDBkOTU4ODhjY2MzZS4u Y2VlY2FlZjY3ODAxIDEwMDY0NAotLS0gYS9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL2dwdV9z Y2hlZHVsZXIuYworKysgYi9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL2dwdV9zY2hlZHVsZXIu YwpAQCAtNjY2LDYgKzY2Niw3IEBAIHN0YXRpYyBpbnQgZHJtX3NjaGVkX21haW4odm9pZCAqcGFy YW0pCiAJCWRybV9zY2hlZF9qb2JfYmVnaW4oc2NoZWRfam9iKTsKIAogCQlmZW5jZSA9IHNjaGVk LT5vcHMtPnJ1bl9qb2Ioc2NoZWRfam9iKTsKKwkJdHJhY2VfZHJtX3NjaGVkX2pvYl9ydW4oc2No ZWRfam9iLCBlbnRpdHkpOwogCQlkcm1fc2NoZWRfZmVuY2Vfc2NoZWR1bGVkKHNfZmVuY2UpOwog CiAJCWlmIChmZW5jZSkgewpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9kcm0vZ3B1X3NjaGVkdWxlcl90 cmFjZS5oIGIvaW5jbHVkZS9kcm0vZ3B1X3NjaGVkdWxlcl90cmFjZS5oCmluZGV4IDA3ODllOGQw YTBlMS4uYzRkODM4NTdhZTAwIDEwMDY0NAotLS0gYS9pbmNsdWRlL2RybS9ncHVfc2NoZWR1bGVy X3RyYWNlLmgKKysrIGIvaW5jbHVkZS9kcm0vZ3B1X3NjaGVkdWxlcl90cmFjZS5oCkBAIC02MSw2 ICs2MSwzMyBAQCBUUkFDRV9FVkVOVChkcm1fc2NoZWRfam9iLAogCQkgICAgICBfX2VudHJ5LT5q b2JfY291bnQsIF9fZW50cnktPmh3X2pvYl9jb3VudCkKICk7CiAKK1RSQUNFX0VWRU5UKGRybV9z Y2hlZF9qb2JfcnVuLAorCSAgICBUUF9QUk9UTyhzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqc2NoZWRf am9iLCBzdHJ1Y3QgZHJtX3NjaGVkX2VudGl0eSAqZW50aXR5KSwKKwkgICAgVFBfQVJHUyhzY2hl ZF9qb2IsIGVudGl0eSksCisJICAgIFRQX1NUUlVDVF9fZW50cnkoCisJCQkgICAgIF9fZmllbGQo c3RydWN0IGRybV9zY2hlZF9lbnRpdHkgKiwgZW50aXR5KQorCQkJICAgICBfX2ZpZWxkKHN0cnVj dCBkbWFfZmVuY2UgKiwgZmVuY2UpCisJCQkgICAgIF9fZmllbGQoY29uc3QgY2hhciAqLCBuYW1l KQorCQkJICAgICBfX2ZpZWxkKHVpbnQ2NF90LCBpZCkKKwkJCSAgICAgX19maWVsZCh1MzIsIGpv Yl9jb3VudCkKKwkJCSAgICAgX19maWVsZChpbnQsIGh3X2pvYl9jb3VudCkKKwkJCSAgICAgKSwK KworCSAgICBUUF9mYXN0X2Fzc2lnbigKKwkJCSAgIF9fZW50cnktPmVudGl0eSA9IGVudGl0eTsK KwkJCSAgIF9fZW50cnktPmlkID0gc2NoZWRfam9iLT5pZDsKKwkJCSAgIF9fZW50cnktPmZlbmNl ID0gJnNjaGVkX2pvYi0+c19mZW5jZS0+ZmluaXNoZWQ7CisJCQkgICBfX2VudHJ5LT5uYW1lID0g c2NoZWRfam9iLT5zY2hlZC0+bmFtZTsKKwkJCSAgIF9fZW50cnktPmpvYl9jb3VudCA9IHNwc2Nf cXVldWVfY291bnQoJmVudGl0eS0+am9iX3F1ZXVlKTsKKwkJCSAgIF9fZW50cnktPmh3X2pvYl9j b3VudCA9IGF0b21pY19yZWFkKAorCQkJCSAgICZzY2hlZF9qb2ItPnNjaGVkLT5od19ycV9jb3Vu dCk7CisJCQkgICApLAorCSAgICBUUF9wcmludGsoImVudGl0eT0lcCwgaWQ9JWxsdSwgZmVuY2U9 JXAsIHJpbmc9JXMsIGpvYiBjb3VudDoldSwgaHcgam9iIGNvdW50OiVkIiwKKwkJICAgICAgX19l bnRyeS0+ZW50aXR5LCBfX2VudHJ5LT5pZCwKKwkJICAgICAgX19lbnRyeS0+ZmVuY2UsIF9fZW50 cnktPm5hbWUsCisJCSAgICAgIF9fZW50cnktPmpvYl9jb3VudCwgX19lbnRyeS0+aHdfam9iX2Nv dW50KQorKTsKKwogVFJBQ0VfRVZFTlQoZHJtX3NjaGVkX3Byb2Nlc3Nfam9iLAogCSAgICBUUF9Q Uk9UTyhzdHJ1Y3QgZHJtX3NjaGVkX2ZlbmNlICpmZW5jZSksCiAJICAgIFRQX0FSR1MoZmVuY2Up LAotLSAKMi4xNy4xCgo= --=-ft3ZvzochtO3Uq4YCjHe Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVs IG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlz dHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCg== --=-ft3ZvzochtO3Uq4YCjHe--