| Bug ID | 93629 |
|---|---|
| Summary | [NVE6] complete system freeze, PGRAPH engine fault on channel 2, SCHED_ERROR [ CTXSW_TIMEOUT ] |
| Product | xorg |
| Version | unspecified |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | normal |
| Priority | medium |
| Component | Driver/nouveau |
| Assignee | nouveau@lists.freedesktop.org |
| Reporter | matthias.h.nagel@gmail.com |
| QA Contact | xorg-team@lists.x.org |
Created attachment 120860 [details]
dmesg output after crash
The system randomly freezes. I can still move the mouse but can't do anything
else. I cannot switch VT, the three-finger salute does not work and system shut
down via ACPI does not work either. The only way of recovery is a hard switch
off.
The crash seems to occur more frequently, if I play videos (mplayer or Flash),
but this is not a requirement.
The crash does not occur, if I use the blob firmware
(nouveau.config="NvGrUseFw=1").
Graphic card: "ASUS GeForce GTX 660 PCI 3.0 2GB DDR5"
uname: "Linux 4.1.12-gentoo #2 SMP Wed Dec 23 19:28:09 CET 2015 x86_64 Intel(R)
Xeon(R) CPU E3-1230 v3 @ 3.30GHz GenuineIntel GNU/Linux"
Beginning of dmesg:
Jan 06 16:04:24 matthias-pc kernel: nouveau E[ PFIFO][0000:01:00.0]
SCHED_ERROR [ CTXSW_TIMEOUT ]
Jan 06 16:04:24 matthias-pc kernel: nouveau E[ PFIFO][0000:01:00.0] PGRAPH
engine fault on channel 2, recovering...
Jan 06 16:04:31 matthias-pc kernel: nouveau E[ PGRAPH][0000:01:00.0] TRAP ch 5
[0x023f926000 kwin[606]]
Jan 06 16:04:31 matthias-pc kernel: nouveau E[ PGRAPH][0000:01:00.0] ROP0
0x80000000 0x80000001
Jan 06 16:04:31 matthias-pc kernel: nouveau E[ PGRAPH][0000:01:00.0] ROP1
0x80000000 0x80000001
Jan 06 16:04:31 matthias-pc kernel: nouveau E[ PGRAPH][0000:01:00.0] ROP2
0x80000000 0x80000001
For a complete log and more system information see the attached files.
Created attachment 120861 [details]
dmesg boot messages
I doubt this is it, but this patch only made it into 4.1.13: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=930a50a45963c8da0a27fbee73ae462d8ea54aca The manifestation of that bug was different though, so ... probably unrelated. Wouldn't hurt to update though. There have also been slight updates to the ctxsw fw in kernel 4.4, so you could try 4.4-rc8 and see if the issues remain. (Also the driver got a substantial refactor/rewrite in kernel 4.3.) Either way, please check on latest to see if issues persist.
I also reported bug #93630 at the same time. Both problems occur in turns. I do not know if they are related, because the symptoms are completely different, but they both disappear using the blob firmware.
OK, I will test kernel 4.1.13 and 4.3.3. Both are available in the unstable branch of my distro. I cannot test 4.4.x yet. Let's see what I will find out.
The error also occurs with kernel 4.1.13 and 4.3.3. First time it happened while playing a H.264 video with mplayer, the second time while scrolling a LibreOffice document. Same dmesg messages.
Today, I finally managed to give kernel 4.4.0 a try. When I try to use the proprietary firmware I get to following boot error Jan 20 20:01:44 kernel: nouveau 0000:01:00.0: Direct firmware load for nvidia/gk106/fecs_inst.bin failed with error -2 Jan 20 20:01:44 kernel: nouveau 0000:01:00.0: gr: failed to load fecs_inst Where do I get this firmware from? Until now (<4.4.0) I used the following kernel parameters CONFIG_CMDLINE="nouveau.config=\"NvGrUseFw=1\"" CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="nouveau/nve6_fuc409c nouveau/nve6_fuc409d nouveau/nve6_fuc41ac nouveau/nve6_fuc41ad" CONFIG_EXTRA_FIRMWARE_DIR="firmware" and used the firmware found in bug 72180 in attachment https://bugs.freedesktop.org/attachment.cgi?id=90180
| What | Removed | Added |
|---|---|---|
| CC | gnurou@gmail.com, skeggsb@gmail.com |
The driver's expectations of firmware were changed in commit 8539b37acef73949861a16808b60cb8b5b9b3bab Author: Alexandre Courbot <acourbot@nvidia.com> Date: Tue Jun 23 15:16:01 2015 +0900 drm/nouveau/gr: use NVIDIA-provided external firmwares NVIDIA will officially start providing GR firmwares through linux-firmware for GPUs that require it. Change the GR firmware lookup function to use these files. Signed-off-by: Alexandre Courbot <acourbot@nvidia.com> Signed-off-by: Ben Skeggs <bskeggs@redhat.com> fuc409c -> fecs_inst fuc409d -> fecs_data fuc41ac -> gpccs_inst fuc41ad -> gpccs_data And it wants the files to live in nvidia/gk106/ (or whatever the relevant chip name). Ben's belief is that everyone should be fine with nouveau firmware, and the fact that the nvidia firmware works at all with nouveau is a fluke.
(In reply to Ilia Mirkin from comment #9) > fuc409c -> fecs_inst > fuc409d -> fecs_data > fuc41ac -> gpccs_inst > fuc41ad -> gpccs_data Sorry, all of those should have .bin at the end (i.e. fecs_inst.bin, etc)
Ok. Let's restate it in my own words to make sure I understood you correctly. But first, let me say that I already have tried 4.4.0 without proprietary firmware. To this end I omitted nouveau.config="NvGrUseFw=1". I experienced the same crash with freeze while scrolling a LibreOffice Writer document. So bad news the bug is still there. Now, I wanted to check if 4.4.0 with proprietary firmware works without crashing. For this purpose you want me to rename nouveau/fuc409c -> nvidia/fecs_inst nouveau/fuc409d -> nvidia/fecs_data nouveau/fuc41ac -> nvidia/gpccs_inst nouveau/fuc41ad -> nvidia/gpccs_data These are essentially the same files but with a different path and name. Then I change CONFIG_CMDLINE="nouveau.config=\"NvGrUseFw=1\"" CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="nvidia/fecs_inst nvidia/fecs_data nvidia/gpccs_inst nvidia/gpccs_data" CONFIG_EXTRA_FIRMWARE_DIR="firmware" After re-compilation I should have a new kernel that uses the proprietary fw and I should be able to check if the crash with freeze is still there or not.
(In reply to Matthias Nagel from comment #12) > ... and of course with a ".bin" at the end and a gk106 in between nvidia and the name, i.e. something that matches the error message it's giving you: Jan 20 20:01:44 kernel: nouveau 0000:01:00.0: Direct firmware load for nvidia/gk106/fecs_inst.bin failed with error -2
Hi i can confirm the problems with the GTX660 firmware. I'm using Debian 9.0 Stretch with Kernel 4.4. When using the nouveau firmware i got random freezes. After installing the nvidia driver, the freezes disappeared. I then found this bug report, an the solution with using the nvidia blob firmware is working like a charm Cédric
I think I am facing this issue. Kernel 4.4.5, Arch Linux, GeForce GTX 650 Ti Boost (first time using it). X will freeze almost immediately (typically right after my KDE session has loaded) with the following message: nouveau 0000:01:00.0: fifo: write fault at 00002a2000 engine 00 [GR] client 0f [GPC1/PROP_0] reason 02 [PTE] on channel 2 [023faf0000 Xorg[666]] nouveau 0000:01:00.0: fifo: gr engine fault on channel 2, recovering... If have just booted using the NVIDIA firmware (NvGrUseFW=1), and X is working fine so far. Will perform a few more boots and spend some time on it to confirm the issue is solved. This seems to be a GR firmware issue, any guidance on how I can help fixing it?
| What | Removed | Added |
|---|---|---|
| CC | dmoulding@me.com |
I'm experiencing this bug with kernel 4.4.6 and GK106 (GeForce GTX 645 OEM). For what it's worth, I've been experiencing this same bug for about two years, ever since kernel 3.12 (I was on kernel 3.10 prior to that and never experienced the bug). After each kernel upgrade I've done, I usually test the built-in firmware to see if the problem is still there. For a while, I thought the bug was gone as of the 4.4 kernel because I hadn't experienced the problem for a long time on 4.4 with the built-in firmware. But recently I upgraded my desktop environment to KDE Plasma 5 and now I've experienced the lockup twice in less than a 24-hour period. I just switched back to the proprietary firmware (after moving/renaming the binary blobs to match the names that the new version of the driver looks for). I'm willing to test the built-in firmware more, in order to help debug this longstanding problem. Is there something I can do to help gather more information (traces, enable debug logging, etc)?
(In reply to Dan Moulding from comment #16) > I'm experiencing this bug with kernel 4.4.6 and GK106 (GeForce GTX 645 OEM). > > For what it's worth, I've been experiencing this same bug for about two > years, ever since kernel 3.12 (I was on kernel 3.10 prior to that and never > experienced the bug). After each kernel upgrade I've done, I usually test > the built-in firmware to see if the problem is still there. For a while, I > thought the bug was gone as of the 4.4 kernel because I hadn't experienced > the problem for a long time on 4.4 with the built-in firmware. But recently > I upgraded my desktop environment to KDE Plasma 5 and now I've experienced > the lockup twice in less than a 24-hour period. > > I just switched back to the proprietary firmware (after moving/renaming the > binary blobs to match the names that the new version of the driver looks > for). > > I'm willing to test the built-in firmware more, in order to help debug this > longstanding problem. Is there something I can do to help gather more > information (traces, enable debug logging, etc)? if you are _sure_ that it never happened with 3.10 it would be really good if you git bisect the kernel and see which commit broke it. If you wait like one or two days for a freeze you should be through in a month :/ But maybe there is somebody who might know what it could be if it is really new with 3.12 (or 3.11)
| What | Removed | Added |
|---|---|---|
| See Also | https://bugs.freedesktop.org/show_bug.cgi?id=72180 |
Compiled and booted a 3.10 kernel in order to attempt a bisect, but the issue showed up on that version as well: [ 95.424646] nouveau E[ PGRAPH][0000:01:00.0] GPC ch 7 [0x000027f563 Xorg[1698]]: [ 95.424659] nouveau E[ PGRAPH][0000:01:00.0] GPC0/TP0/MP trap: INVALID_OPCODE [ 95.424671] nouveau E[ PGRAPH][0000:01:00.0] GPC2/TP0/MP trap: INVALID_OPCODE [ 95.424805] nouveau E[ PGRAPH][0000:01:00.0] GPC ch 7 [0x000027f563 Xorg[1698]]: [ 95.424813] nouveau E[ PGRAPH][0000:01:00.0] GPC1/TP0/MP trap: INVALID_OPCODE [ 95.424826] nouveau E[ PGRAPH][0000:01:00.0] GPC2/TP1/MP trap: INVALID_OPCODE [ 99.750909] nouveau W[ PFIFO][0000:01:00.0] unknown status 0x00000100 [ 102.082998] nouveau E[ PGRAPH][0000:01:00.0] ROP0 ch 5 [0x000027f71f Xorg[1698]] 0x80000000 0x80000001 [ 102.083000] nouveau E[ PGRAPH][0000:01:00.0] ROP1 ch 5 [0x000027f71f Xorg[1698]] 0x80000000 0x80000001 [ 102.083000] nouveau E[ PGRAPH][0000:01:00.0] ROP2 ch 5 [0x000027f71f Xorg[1698]] 0x80000000 0x80000001 [ 104.045757] nouveau W[ PFIFO][0000:01:00.0] unknown status 0x00000100 [ 108.340607] nouveau W[ PFIFO][0000:01:00.0] unknown status 0x00000100 Using the external firmware, I cannot reproduce the problem on this kernel either. So it seems like the NVE6 firmware has a long-standing bug.
Roy, NVE6 is rock-stable on my end using your patch. Feel free= to add my Tested-by: Alexandre Courbot <acourbot@nvidia.com>
Created = attachment 123609 [details] = [review] Race condition fix for nouveau firmware For anyone who experiences random hangs using the nouveau firmware and have spotted "SCHED_ERROR [ CTXSW_TIMEOUT ]" in the logs, please build= a kernel with the attached patch and run using the nouveau firmware. Minimum kernel requi= red is 4.6. I've tested this on one machine of my own and have a strong indication that this improves the situation, but unfortunately it is difficult to distingui= sh random infrequent hangs from no hangs.
Roy, I can also give the patch a try on my setup tomorrow and = will let you know how it goes.
Created attachment 123658
References:
In-Reply-To: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
Errors-To: nouveau-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
Sender: "Nouveau"
To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
List-Id: nouveau.vger.kernel.org
--===============0537284630==
Content-Type: multipart/alternative; boundary="14631365564.F01b95bfb.22544";
charset="UTF-8"
--14631365564.F01b95bfb.22544
Date: Fri, 13 May 2016 10:49:16 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://bugs.freedesktop.org/
Auto-Submitted: auto-generated
https://bugs.freedesktop.org/show_bug.cgi?id=3D93629
--- Comment #23 from Alexandre Courbot ---
Ben, your patch is also working ok on my end (tested on top of Karol's bran=
ch).
Tested-by: Alexandre Courbot
--=20
You are receiving this mail because:
You are the assignee for the bug.=
--14631365564.F01b95bfb.22544
Date: Fri, 13 May 2016 10:49:16 +0000
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://bugs.freedesktop.org/
Auto-Submitted: auto-generated