Kernel Oops on 3.14.66

Kernel Oops on 3.14.66

[Andreas Lampersperger]

Hello,

has anyone here a hint for me, what can cause this error.
The error occures highly sporadic on different machines with intel hd
graphics (ivb_gt1).
I did also some kernel coredumps and found out, that the failed
paging request came from drm_i915_gem_request->list.prev or ->list.next.

Thank you
Andreas

dmesg:

[13328.412403] BUG: unable to handle kernel paging request at 00c6d1fc
[13328.412439] IP: [<f8224eed>] i915_gem_free_request+0x10/0x81 [i915]
[13328.412440] *pde = 00000000
[13328.412443] Oops: 0002 [#1] PREEMPT SMP
[13328.412485] Modules linked in: snd_dummy snd_pcm snd_timer snd
soundcore ftdi_sio usbserial 8250 serial_core jhhw(O) fex(O) fe1(O)
jhint(O) heros(O) arc4 ecb md4 md5 nls_utf8 cifs hid_multitouch
kvm_intel kvm evdev cdc_acm nls_cp437 des_generic af_packet hik(O)
hsci(O) jhnc(O) sr_mod cdrom hldx(O) msr input_polldev i2c_dev w83781d
hwmon_vid i2c_i801 i2c_ali1535 rtc_cmos fuse nbd psmouse igb pch_gbe
ptp_pch e1000 e1000e ptp pps_core e100 mii tun pata_sch usb_storage
sd_mod sg pata_ali ide_gd_mod ide_core ahci libahci ata_piix libata
scsi_mod hid_generic usbmon usbkbd usbhid hid uhci_hcd ohci_pci
ohci_hcd xhci_hcd ehci_pci ehci_hcd usbcore i915 drm_kms_helper drm
firmware_class i2c_algo_bit button video thermal_sys hwmon
[13328.412554] CPU: 1 PID: 5341 Comm: Xorg Tainted: G           O
3.14.66-rt68-heros5 #2
[13328.412555] Hardware name: MC64XX/MC64XX, BIOS MC64XX_2.2.0.377 X64
04/22/2014
[13328.412557] task: efa84970 ti: e640a000 task.ti: e640a000
[13328.412559] EIP: 0060:[<f8224eed>] EFLAGS: 00213246 CPU: 1
[13328.412586] EIP is at i915_gem_free_request+0x10/0x81 [i915]
[13328.412588] EAX: 00c6d1fc EBX: cc303140 ECX: 00000000 EDX: cc3030dc
[13328.412589] ESI: 00000000 EDI: cc30315c EBP: e640bdd8 ESP: e640bdd0
[13328.412591]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[13328.412592] CR0: 80050033 CR2: 00c6d1fc CR3: 2644b000 CR4: 001407d0
[13328.412593] Stack:
[13328.412597]  f4dd4b24 00000000 e640be00 f8226740 00423d74 cc303140
f4dd4bcc e640a000
[13328.412601]  00423d76 f4d36d00 00000000 f4dd4b24 e640be14 f8226bc4
00000300 082f177c
[13328.412604]  f4d36d00 e640be88 f822a9d6 e640be6c 00000000 00005ea9
00000000 00000000
[13328.412605] Call Trace:
[13328.412638]  [<f8226740>] i915_gem_retire_requests_ring+0xac/0x106 [i915]
[13328.412667]  [<f8226bc4>] i915_gem_object_wait_rendering+0x3c/0x51 [i915]
[13328.412698]  [<f822a9d6>] i915_gem_pwrite_ioctl+0x36d/0x710 [i915]
[13328.412705]  [<c0545ea9>] ? preempt_count_add+0x69/0x9b
[13328.412709]  [<c0151317>] ? migrate_enable+0x11a/0x16b
[13328.412714]  [<c03b4724>] ? _copy_from_user+0x27/0x3a
[13328.412743]  [<f822a669>] ?
i915_gem_object_set_to_gtt_domain+0x15f/0x15f [i915]
[13328.412766]  [<f80f4962>] drm_ioctl+0x2aa/0x3a4 [drm]
[13328.412793]  [<f822a669>] ?
i915_gem_object_set_to_gtt_domain+0x15f/0x15f [i915]
[13328.412802]  [<c0375ef5>] ? avc_has_perm+0x83/0xd6
[13328.412820]  [<f80f46b8>] ? drm_core_reclaim_buffers+0x5b/0x5b [drm]
[13328.412825]  [<c020cc62>] do_vfs_ioctl+0x387/0x436
[13328.412828]  [<c037a1db>] ? file_has_perm+0x4d/0x6e
[13328.412832]  [<c037a5d0>] ? selinux_file_ioctl+0x9b/0x9e
[13328.412836]  [<c020cd4f>] SyS_ioctl+0x3e/0x63
[13328.412841]  [<c0548a00>] sysenter_do_call+0x12/0x12
[13328.412865] Code: fb c7 8a 93 a9 00 00 00 83 e2 fc 83 ca 02 88 93
a9 00 00 00 58 5a 8b 5d fc c9 c3 55 89 e5 56 53 89 c3 8b 50 1c 8b 40
20 89 42 04 <89> 10 8b 73 24 c7 43 1c 00 01 10 00 c7 43 20 00 02 20 00
85 f6
[13328.412891] EIP: [<f8224eed>] i915_gem_free_request+0x10/0x81
[i915] SS:ESP 0068:e640bdd0
[13328.412892] CR2: 0000000000c6d1fc


gdb log of kernel coredump:

i915_gem_free_request (request=0xcc303140) at
drivers/gpu/drm/i915/i915_gem.c:2367
(gdb) p request
$2 = (struct drm_i915_gem_request *) 0xcc303140
(gdb) p *request
$3 = {ring = 0xf4dd4b24, seqno = 4341108, head = 26224, tail = 26240,
ctx = 0xf4d58600, batch_obj = 0xf4d36d00, emitted_jiffies = 13029884,
list = {next = 0xcc3030dc, prev = 0x
c6d1fc}, file_priv = 0xefae4780, client_list = {next = 0xcc3030e8,
prev = 0xefae479c}}
(gdb) bt
#0  __list_del (next=<optimized out>, prev=<optimized out>) at
include/linux/list.h:89
#1  list_del (entry=<optimized out>) at include/linux/list.h:106
#2  i915_gem_free_request (request=0xcc303140) at
drivers/gpu/drm/i915/i915_gem.c:2367
#3  0xf8226740 in i915_gem_retire_requests_ring (ring=0xf4dd4b24) at
drivers/gpu/drm/i915/i915_gem.c:2498
#4  0xf8226bc4 in i915_gem_object_wait_rendering__tail
(ring=<optimized out>, obj=<optimized out>) at
drivers/gpu/drm/i915/i915_gem.c:1144
#5  i915_gem_object_wait_rendering (obj=0xf4d36d00,
readonly=<optimized out>) at drivers/gpu/drm/i915/i915_gem.c:1179
#6  0xf822a9d6 in i915_gem_shmem_pwrite (file=<optimized out>,
args=<optimized out>, obj=<optimized out>, dev=<optimized out>) at
drivers/gpu/drm/i915/i915_gem.c:752
#7  i915_gem_pwrite_ioctl (dev=<optimized out>, data=<optimized out>,
file=<optimized out>) at drivers/gpu/drm/i915/i915_gem.c:927
#8  0xf80f4962 in drm_ioctl (filp=<optimized out>, cmd=<optimized
out>, arg=<optimized out>) at drivers/gpu/drm/drm_drv.c:387
#9  0xc020cc62 in vfs_ioctl (arg=<optimized out>, cmd=<optimized out>,
filp=<optimized out>) at fs/ioctl.c:43
#10 do_vfs_ioctl (filp=0xf01af100, fd=12, cmd=<optimized out>,
arg=3215145920) at fs/ioctl.c:598
#11 0xc020cd4f in SYSC_ioctl (arg=<optimized out>, cmd=<optimized
out>, fd=<optimized out>) at fs/ioctl.c:613
#12 SyS_ioctl (fd=12, cmd=1075864669, arg=-1079821376) at fs/ioctl.c:604
#13 <signal handler called>
#14 0xffffe440 in ?? ()
Backtrace stopped: Cannot access memory at address 0xbfa33b58
(gdb)
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Kernel Oops on 3.14.66

[Dave Gordon]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

On 27/04/16 10:19, Andreas Lampersperger wrote:
> Hello,
>
> has anyone here a hint for me, what can cause this error.
> The error occures highly sporadic on different machines with intel hd
> graphics (ivb_gt1).
> I did also some kernel coredumps and found out, that the failed
> paging request came from drm_i915_gem_request->list.prev or ->list.next.
>
> Thank you
> Andreas

Try this patch.

.Dave.


[-- Attachment #2: 0001-initialise-request-list-head.patch --]
[-- Type: text/x-patch, Size: 1598 bytes --]

>From 6dde6b8b07239c68f32315faffa9d80b6e2d0aec Mon Sep 17 00:00:00 2001
From: Dave Gordon <david.s.gordon@intel.com>
Date: Fri, 22 Apr 2016 19:33:53 +0100
Subject: [PATCH] Request constructor should initialise list head
Organization: Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ

Users of the Linux kernel linked-list macros should remember that a
zeroed "struct list_head" is not a valid list and in particular is
not the same as an empty list. ALL list heads MUST be initialised!

Without this, any attempt to follow this link (e.g. during teardown)
before the request has been put on the list of pending requests (or
if is not put on the list, e.g. is cancelled instead) will result in
a NULL pointer dereference.

Note: the diff below claims this code goes in i915_gem_request_free(),
but that's just git's failure to parse the function headers correctly.
It obviously belongs in __i915_gem_request_alloc(); it seems to apply
correctly despite the incorrect function tag.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=93907 ?
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d493e79..8ce7d4d 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2751,6 +2751,7 @@ void i915_gem_request_free(struct kref *req_ref)
 	if (ret)
 		goto err;
 
+	INIT_LIST_HEAD(&req->list);
 	kref_init(&req->ref);
 	req->i915 = dev_priv;
 	req->engine = engine;
-- 
1.9.1


[-- Attachment #3: Type: text/plain, Size: 160 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.BAT: success for Kernel Oops on 3.14.66

[Patchwork]

== Series Details ==

Series: Kernel Oops on 3.14.66
URL   : https://patchwork.freedesktop.org/series/6394/
State : success

== Summary ==

Series 6394v1 Kernel Oops on 3.14.66
http://patchwork.freedesktop.org/api/1.0/series/6394/revisions/1/mbox/


bdw-nuci7        total:200  pass:188  dwarn:0   dfail:0   fail:0   skip:12 
bdw-ultra        total:200  pass:175  dwarn:0   dfail:0   fail:0   skip:25 
bsw-nuc-2        total:199  pass:158  dwarn:0   dfail:0   fail:0   skip:41 
byt-nuc          total:199  pass:158  dwarn:0   dfail:0   fail:0   skip:41 
hsw-brixbox      total:200  pass:174  dwarn:0   dfail:0   fail:0   skip:26 
hsw-gt2          total:200  pass:178  dwarn:0   dfail:0   fail:1   skip:21 
ilk-hp8440p      total:200  pass:139  dwarn:0   dfail:0   fail:0   skip:61 
ivb-t430s        total:200  pass:169  dwarn:0   dfail:0   fail:0   skip:31 
skl-i7k-2        total:200  pass:173  dwarn:0   dfail:0   fail:0   skip:27 
skl-nuci5        total:200  pass:189  dwarn:0   dfail:0   fail:0   skip:11 
snb-dellxps      total:200  pass:158  dwarn:0   dfail:0   fail:0   skip:42 
snb-x220t        total:200  pass:158  dwarn:0   dfail:0   fail:1   skip:41 

Results at /archive/results/CI_IGT_test/Patchwork_2086/

6da4af12ef4d4d82f4e0f2a4a960e15a74cced22 drm-intel-nightly: 2016y-04m-27d-08h-36m-12s UTC integration manifest
6c663ff Kernel Oops on 3.14.66

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Kernel Oops on 3.14.66

[Andreas Lampersperger]

[-- Attachment #1.1: Type: text/plain, Size: 829 bytes --]

Hi Dave,

thank you for the patch, but I'm using longtime kernel 3.14.66 and so the
patch does not fit.
Do you have any suggestions for that kernel?

Andreas

2016-04-27 12:03 GMT+02:00 Dave Gordon <david.s.gordon@intel.com>:

> On 27/04/16 10:19, Andreas Lampersperger wrote:
>
>> Hello,
>>
>> has anyone here a hint for me, what can cause this error.
>> The error occures highly sporadic on different machines with intel hd
>> graphics (ivb_gt1).
>> I did also some kernel coredumps and found out, that the failed
>> paging request came from drm_i915_gem_request->list.prev or ->list.next.
>>
>> Thank you
>> Andreas
>>
>
> Try this patch.
>
> .Dave.
>
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
>
>

[-- Attachment #1.2: Type: text/html, Size: 1641 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Kernel Oops on 3.14.66

[Dave Gordon]

> 2016-04-27 12:03 GMT+02:00 Dave Gordon <david.s.gordon@intel.com>:
>
>> On 27/04/16 10:19, Andreas Lampersperger wrote:
>>
>>> Hello,
>>>
>>> has anyone here a hint for me, what can cause this error.
>>> The error occures highly sporadic on different machines with intel hd
>>> graphics (ivb_gt1).
>>> I did also some kernel coredumps and found out, that the failed
>>> paging request came from drm_i915_gem_request->list.prev or ->list.next.
>>>
>>> Thank you
>>> Andreas
>>
>> Try this patch.
>>
>> .Dave.
>
On 27/04/16 13:20, Andreas Lampersperger wrote:
> Hi Dave,
>
> thank you for the patch, but I'm using longtime kernel 3.14.66 and so the
> patch does not fit.
> Do you have any suggestions for that kernel?
>
> Andreas

I have no idea what the code would look like in that version; do you 
have any way to get the git commit hash or tag of the i915 driver?

But essentially, if you have the kernel sources for that version, you 
could look for a line that allocates a "struct drm_i915_gem_request". It 
might be

     req = kmem_cache_zalloc(dev_priv->requests, GFP_KERNEL);

or in older versions it might be

     request = kzalloc(sizeof(*request), GFP_KERNEL);

or some other variation. It should of course be followed by a test for 
successful allocation, e.g.

     if (request == NULL)
         return -ENOMEM;

If you can find that line (or lines, at one time there were TWO such 
places), then you just have to add

     INIT_LIST_HEAD(&request->list);

after the test-for-NULL, or anywhere inline thereafter. For example:

     request = kzalloc(sizeof(*request), GFP_KERNEL);
     if (request == NULL)
         return -ENOMEM;

     ret = i915_gem_get_seqno(ring->dev, &request->seqno);
     if (ret) {
         kfree(request);
         return ret;
     }

+   INIT_LIST_HEAD(&request->list);
     kref_init(&request->ref);
     request->ring = ring;
     request->uniq = dev_private->request_uniq++;

HTH,
.Dave.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Kernel Oops on 3.14.66

[Andreas Lampersperger]

[-- Attachment #1.1: Type: text/plain, Size: 3142 bytes --]

Hi Dave,

thank you again.

I searched, where the memory for the request came from, it is set in line
2158 of i915_gem.c
<https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/gpu/drm/i915/i915_gem.c?id=refs/tags/v3.14.67#n2158>
and the kmalloc is in intel_ring_alloc_seqno() in intel_ringbuffer.c
<https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/gpu/drm/i915/intel_ringbuffer.c?id=refs/tags/v3.14.67#n1600>,
right?

So I will test the following patch for intel_ringbuffer.c:


intel_ring_alloc_seqno(struct intel_ring_buffer *ring)
{
    if (ring->outstanding_lazy_seqno)
        return 0;

    if (ring->preallocated_lazy_request == NULL) {
        struct drm_i915_gem_request *request;

        request = kmalloc(sizeof(*request), GFP_KERNEL);
        if (request == NULL)
            return -ENOMEM;
+
+       INIT_LIST_HEAD(&request->list);
        ring->preallocated_lazy_request = request;
    }

    return i915_gem_get_seqno(ring->dev, &ring->outstanding_lazy_seqno);
}
:

2016-04-27 20:28 GMT+02:00 Dave Gordon <david.s.gordon@intel.com>:

> 2016-04-27 12:03 GMT+02:00 Dave Gordon <david.s.gordon@intel.com>:
>>
>> On 27/04/16 10:19, Andreas Lampersperger wrote:
>>>
>>> Hello,
>>>>
>>>> has anyone here a hint for me, what can cause this error.
>>>> The error occures highly sporadic on different machines with intel hd
>>>> graphics (ivb_gt1).
>>>> I did also some kernel coredumps and found out, that the failed
>>>> paging request came from drm_i915_gem_request->list.prev or ->list.next.
>>>>
>>>> Thank you
>>>> Andreas
>>>>
>>>
>>> Try this patch.
>>>
>>> .Dave.
>>>
>>
>> On 27/04/16 13:20, Andreas Lampersperger wrote:
>
>> Hi Dave,
>>
>> thank you for the patch, but I'm using longtime kernel 3.14.66 and so the
>> patch does not fit.
>> Do you have any suggestions for that kernel?
>>
>> Andreas
>>
>
> I have no idea what the code would look like in that version; do you have
> any way to get the git commit hash or tag of the i915 driver?
>
> But essentially, if you have the kernel sources for that version, you
> could look for a line that allocates a "struct drm_i915_gem_request". It
> might be
>
>     req = kmem_cache_zalloc(dev_priv->requests, GFP_KERNEL);
>
> or in older versions it might be
>
>     request = kzalloc(sizeof(*request), GFP_KERNEL);
>
> or some other variation. It should of course be followed by a test for
> successful allocation, e.g.
>
>     if (request == NULL)
>         return -ENOMEM;
>
> If you can find that line (or lines, at one time there were TWO such
> places), then you just have to add
>
>     INIT_LIST_HEAD(&request->list);
>
> after the test-for-NULL, or anywhere inline thereafter. For example:
>
>     request = kzalloc(sizeof(*request), GFP_KERNEL);
>     if (request == NULL)
>         return -ENOMEM;
>
>     ret = i915_gem_get_seqno(ring->dev, &request->seqno);
>     if (ret) {
>         kfree(request);
>         return ret;
>     }
>
> +   INIT_LIST_HEAD(&request->list);
>     kref_init(&request->ref);
>     request->ring = ring;
>     request->uniq = dev_private->request_uniq++;
>
> HTH,
> .Dave.
>

[-- Attachment #1.2: Type: text/html, Size: 4474 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: Kernel Oops on 3.14.66

[Dave Gordon]

On 28/04/16 09:58, Andreas Lampersperger wrote:
> Hi Dave,
>
> thank you again.
>
> I searched, where the memory for the request came from, it is set in line 2158
> of i915_gem.c
> <https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/gpu/drm/i915/i915_gem.c?id=refs/tags/v3.14.67#n2158>
> and the kmalloc is in intel_ring_alloc_seqno() in intel_ringbuffer.c
> <https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/gpu/drm/i915/intel_ringbuffer.c?id=refs/tags/v3.14.67#n1600>,
> right?
>
> So I will test the following patch for intel_ringbuffer.c:
>
>
> intel_ring_alloc_seqno(struct intel_ring_buffer *ring)
> {
>       if (ring->outstanding_lazy_seqno)
>           return 0;
>
>       if (ring->preallocated_lazy_request == NULL) {
>           struct drm_i915_gem_request *request;
>
>           request = kmalloc(sizeof(*request), GFP_KERNEL);
>           if (request == NULL)
>               return -ENOMEM;
> +
> +       INIT_LIST_HEAD(&request->list);
>           ring->preallocated_lazy_request = request;
>       }
>
>       return i915_gem_get_seqno(ring->dev, &ring->outstanding_lazy_seqno);
> }
> :

Yes, that looks sensible. Interestingly, that version uses kmalloc() 
rather than k*z*alloc, so the uninitialised list head will be random 
rather than zeroed. Anything could happen!

.Dave.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx