| Priority | medium |
|---|---|
| Bug ID | 73528 |
| Assignee | dri-devel@lists.freedesktop.org |
| Summary | Deferred lighting in Second Life causes system hiccups and screen flickering |
| Severity | major |
| Classification | Unclassified |
| OS | Linux (All) |
| Reporter | sonichedgehog_hyperblast00@yahoo.com |
| Hardware | x86-64 (AMD64) |
| Status | NEW |
| Version | git |
| Component | Drivers/DRI/R600 |
| Product | Mesa |
I compiled Mesa with libdrm from their GIT repositories (10.1, commit 2dc35a619c50139d07ad96fc4dfe456e5811c84e). I started up the Second Life viewer through it (Kokua x64, 3.6.12.30743, Dec 12 2013 05:51:00) and logged in. Upon enabling Advanced Lighting Model from Preferences - Graphics, the system froze and the screen started flickering, turning black then recovering every few seconds. The scene also got covered in pink pixels, meaning it also failed to render correctly. I was barely able to move the mouse pointer and shut the viewer down. Once I did, everything returned to normal. This does not happen with Mesa 9.2.3, and other Graphics options in the viewer work fine (like "Basic shaders" or "Atmospheric shaders"). Shadows were disabled and likely not a cause. If someone has Second Life, please try the deferred lighting system under the 10.1 development version of Mesa. Currently, I don't have any more information on the problem apart from having experienced the described symptoms.
(In reply to comment #1) > Can you bisect? Probably not. I just set up the GIT today for the first time, and compiling takes many minutes as it is. I can't estimate where it might have broken either, nor risk experiencing the problem too many times (given it almost brings the system down). I just know it works well in 9.2.3 but not 10.1.
(In reply to comment #2= ) > [...] compiling takes many minutes as it is. git bisect minimizes the number of times you need to compile Mesa to isolate the change introducing the problem. > I can't estimate where it might have broken eith= er, Your initial report says it's working with Mesa 9.2.3 but broken with curre= nt Git, that's good enough for bisecting. > nor risk experiencing the problem too many times= (given it almost > brings the system down). I don't think it's that dramatic. It sounds like the usual symptoms of the = GPU repeatedly hanging and being reset by the driver.
| What | Removed | Added |
|---|---|---|
| Component | Drivers/DRI/R600 | Drivers/Gallium/r600 |
I updated Mesa to latest GIT and tried this again last night (Kokua viewer 3.6.12 x64). The result was even more catastrophic: As soon the world was to begin rendering, the screen turned black and the monitor went in standby mode. The numlock / capslock leds on the keyboard also stopped toggling. I had to power off the computer to restart. This likely means a Kernel panic happened, so the problem is probably really bad.
Any news on this? With a new distribution upgrade coming slowly, I'm worried that Second Life might no longer work properly and even crash my system. Can anyone with a 6xxx series card try the latest Mesa GIT and attempt to reproduce the problem?
I compiled latest MESA GIT today and tried the SL viewer out once more. The issue still exists! I have however at least found a way to test it, that doesn't appear to crash my computer (have no prims in view and therefore rendered, have a command ready to instantly kill the viewer process). I shall see if I can test with different settings, in case it's a combination of factors. One thing I noticed additionally: I have Radeon DPM enabled, despite my Kernel requiring a boot parameter for that as that's not yet default. When the issue takes place, the fan on my video card also reach maximum RPM. Something clearly floods the video card and GPU beyond what they can handle.
I brought this up on IRC last night, and was advised to do a GIT bisect or an apitrace. Since I have no idea when the issue started happening and I can't risk crashing my machine dozens of times, I can't try GIT bisect myself. I was however able to do an apitrace, which also produces the crash when played back. As requested, I packed it as an xz archive and uploaded it. This should allow anyone who can't try the Second Life viewer themselves to reproduce the crash. http://www.sendspace.com/file/oixqur Unpack that somewhere, then run the command "glretrace ./do-not-directly-run-kokua-bin.trace" from the same directory. Be prepared for a system freeze, and only do this on a machine where it's safe! Also, post a comment here if that link stops working, so I can re-post the trace file if needed. I crashed my machine 5 times in order to record this. So I'd appreciate it if someone could make the effort worth it and please test this more in depth, to see where the issue really takes place in the MESA code.
First of all, please attach the Xorg.0.log file and the output= of glxinfo and dmesg. (In reply to comment #6) > One thing I noticed additionally: I have Radeon = DPM enabled, despite my > Kernel requiring a boot parameter for that as that's not yet default.<= /span > Does the problem also occur without DPM? BTW, in case your Mesa is built with --enable-r600-llvm-compiler, does the problem also occur if you disable the LLVM based shader compiler with the environment variable R600_LLVM=3D0 for the SL process? (In reply to comment #7) > Since I have no idea when the issue started happ= ening and I can't risk > crashing my machine dozens of times, I can't try GIT bisect myself. That's just not true, see comment #3= . Also (to preempt another concern you mentioned on IRC), bisecting normally doesn't require any knowledge of the code. > I was however able to do an apitrace, which also= produces the crash > when played back. Unfortunately, I'm not able to reproduce anything like what you describe wi= th that apitrace on my machines. > I crashed my machine 5 times in order to record = this. Five iterations of git bisect would go a long way towards isolating the problem. If you had started the bisection when you reported this bug, you w= ould have isolated the change introducing the problem for you a long time ago, e= ven if you only tested it once a day.
(In reply to comment #8) I haven't tried without DPM. I might sometime later. I shall also test R600_LLVM=0. Do I just set that as a variable in the same console before running the SL process, or is it a compile parameter? The problem likely existed months before I first tried running the SL viewer on a GIT version of MESA. The oldest version of Mesa where I don't get this occurring is the one my Linux distribution comes with, 9.2.3. I wouldn't even know where to begin bisecting in such an ocean of commits, especially considering the crashes. One more thing to mention if it's relevant: I also compile latest GIT Mesa with latest libdrm from GIT. Since by the time I first tried a development version of Mesa, it wouldn't compile against the libdrm-devel package in my system. Still, I use the free Radeon driver and X11 version provided by openSUSE 13.1. Could there be an incompatibility there? No, I can't risk testing a newer driver or X11 than my distribution offers, as this is my desktop and I could break it. I'll attach the logs you mentioned shortly after I post this comment.
Created attachment 99862 [details]
Xorg.0.log dating 26-05-2014
Created attachment 99863 [details]
glxinfo output, dating 26-05-2014
Created attachment 99864 [details]
dmesg output, dating 26-05-2014
(In reply to comment #9= ) > I shall also test R600_LLVM=3D0. Do I just set t= hat as a variable in the same > console before running the SL process, or is it a compile parameter? The former, but make sure it's actually visible in the SL process, either by exporting it or just prepending it on the SL command line. > I wouldn't even know where to begin bisecting in= such an ocean of commits, > [...] As I said before, the information in your original report is enough to get = the bisection started: git bisect bad 2dc35a619c50139d07ad96fc4dfe456e5811c84e git bisect good mesa-9.2.3 After that, git bisect will isolate the change introducing the problem after the minimum number of tests required, which is about a dozen (roughly the logarithm to the base of 2 of the number of commits between the commits abo= ve). At this point, there's no way to make progress on this bug without someone = who can reproduce the problem and investigate or at least bisect it. It's up to= you if you want to look for someone else who fits the bill, or just bisect yourself. > I also compile latest GIT Mesa with latest libdr= m from GIT. [...] Still, I > use the free Radeon driver and X11 version provided by openSUSE 13.1. = Could > there be an incompatibility there? That's unlikely, especially since Mesa 9.2.3 still works fine? > No, I can't risk testing a newer driver or X11 t= han my distribution offers, > as this is my desktop and I could break it. FYI, it's possible to test newer versions of all components without affecti= ng the system installed ones. But so far everything points to this being a Mesa bug.
BTW, if you capture the output of dmesg after the problem occu= rs, it should contain some information about the GPU lockups. But it'll be much harder to deduce the cause from that than from a bisection.
Today with its release, I upgraded to openSUSE 13.2, which has MESA 10.3.0. Despite being an official package, unlike my previous tests on repository versions of MESA, the issue is still there. Enabling Advanced Lighting in the SL viewer will freeze and crash the entire system immediately. I still have a trace from a while ago, which can be used to re-produce the issue without running SL yourself. As an xz archive, it seems to be reduced to 54 MB... but if I cannot post it here ask me for another place to upload it. Anyway, I really hope someone looks into this :(
| What | Removed | Added |
|---|---|---|
| Priority | medium | high |
| Version | git | 10.3 |
Okay, I just found an important clue: The issue is not limited to Second Life alone! I just tried to enable shaders in the open-source racing game Stuntrally (Vdrift Ogre) and got the exact same problem. The machine would freeze, the screen would flicker (monitor going in standby), but in this case it would unfreeze after a few seconds so I managed to turn shaders off. Additionally, I did more tests in Second Life several days ago, and found something interesting: If I look at the ground so barely anything is rendered, the shaders work perfectly well. But if the sky or too much detail come into view, it only takes a few seconds until the GPU suddenly hiccups and I can barely re-stabilize the system. I'm unclear whether this is a coincidence. But when the problem takes place, the fan on my video card is running at maximum speed, indicating the Radeon DPM module is stressing the card out. Note that the freezes existed before DPM was supported by the free driver, and came after a MESA update alone. My humble impression is that some shaders either create an infinite loop on the GPU due to a compilation bug, either they're so unoptimized that they flood the GPU with data it can't handle. It does seem to be triggered by specific objects coming into view... but I can't tell if that's because of the model complexity (high polygon count) or because they contain a "sick" shader which the video card attempts to render. Since this has been around for an year, and I gathered a few clues already, please allow me to set it to "high" priority to bring more awareness. I'm around and watching this, so I can do more tests... as long as it's nothing that would risk breaking my system (like changing system drivers).
The situation hasn't changed since comment #13: At this point, there's no way to make progress on this bug without someone = who can reproduce the problem and investigate or at least bisect it. It's up to= you if you want to look for someone else who fits the bill, or just bisect yourself.
(In reply to Michel D=C3=A4nzer from comment #17) This happens since an year ago. Commit 2dc35a619c50139d07ad96fc4dfe456e5811= c84e is when I first noticed the problem. I'm uncertain whether I can even still compile such an old version of MESA at this day. I shall attempt to however. In case I can't compile: Is there a place where I can download pre-compiled nightly / weekly builds of MESA at least, dating since last year?
Alright. I dusted up my local GIT clone of MESA, and started checking out and compiling various versions. So far I only looked at the tags, and found the releases between which the issue was added: mesa-9.2.5 (works fine) to mesa-10.0-rc1 (contains the problem). Now comes the painful process of narrowing down which commit between these two tags implements the problem. It might take some time, but I'll post here the moment I have a result.
It is done. After nearly two hours of bisecting, I managed to = track the exact commit which adds the freezes... at least in the case of Second Life. edbbfac6cfc634e697d7f981155a5072c52d77ac is the first bad commit commit edbbfac6cfc634e697d7f981155a5072c52d77ac Author: Grigori Goronzy <greg&#= 64;chown.ath.cx> Date: Wed Sep 11 01:41:40 2013 +0200 r600g: fast color clears for single-sample buffers Allocate a CMASK on demand and use it to fast clear single-sample colorbuffers. Both FBOs and window system colorbuffers are fast cleared. Expand as needed when colorbuffers are mapped or displayed on screen. v2: cosmetics, move transfer expansion into dma_blit Signed-off-by: Marek Ol=C5=A1=C3=A1k <marek.olsak@amd.com> :040000 040000 dbb6df4e7ed243043e46afc9b91c389b61549ef9 62ae46bd2049b7cb6e35e2dce2ba4725b413bdbe M src 56d9a397aa2dbee6b12e1bbe56be39f426e1e34d is the last good commit These are the changes in the bad commit: http://pastebin.com/raw.php?i=3DZ1RAGRcT
| What | Removed | Added |
|---|---|---|
| CC | greg@chown.ath.cx |