[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 2370 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

            Bug ID: 93264
           Summary: Tonga VM Faults since llvm ScheduleDAGInstrs: Rework
                    schedule graph builder.
           Product: DRI
           Version: DRI git
          Hardware: x86-64 (AMD64)
                OS: Linux (All)
            Status: NEW
          Severity: normal
          Priority: medium
         Component: DRM/AMDgpu
          Assignee: dri-devel@lists.freedesktop.org
          Reporter: adf.lists@gmail.com

R9285 using Unreal ElementalDemo to trigger this.

It doesn't start till well into the demo at the same place that triggered an
older resolved issue.

https://bugs.freedesktop.org/show_bug.cgi?id=93015
(so maybe Nicolai knows what happens at this point in demo)

bisecting llvm came up with

c0a189c3792865257c1383f176e5401373ed2270 is the first bad commit
commit c0a189c3792865257c1383f176e5401373ed2270
Author: Matthias Braun <matze@braunis.de>
Date:   Thu Dec 3 02:05:27 2015 +0000

    ScheduleDAGInstrs: Rework schedule graph builder.

    The new algorithm remembers the uses encountered while walking backwards
    until a matching def is found. Contrary to the previous version this:
    - Works without LiveIntervals being available
    - Allows to increase the precision to subregisters/lanemasks
      (not used for now)

    The changes in the AMDGPU tests are necessary because the R600 scheduler
    is not stable with respect to the order of nodes in the ready queues.

    Differential Revision: http://reviews.llvm.org/D9068


The demo continues to run/render OK, but I get thousands of -

amdgpu 0000:01:00.0: GPU fault detected: 147 0x07d04401
amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x092D80FA
amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0A044001
VM fault (0x01, vmid 5) at page 153977082, read from 'TC7' (0x54433700) (68)
amdgpu 0000:01:00.0: GPU fault detected: 147 0x07d00401
amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x0022D16B
amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0A0C4002
VM fault (0x02, vmid 5) at page 2281835, read from 'TC4' (0x54433400) (196)
amdgpu 0000:01:00.0: GPU fault detected: 147 0x07d04001
amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x0022D163

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 3938 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 492 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #1 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Thanks for the report and the bisection! Interestingly, I was able to reproduce
this, but only with the very latest Mesa. So it might actually be some kind of
interaction between Mesa and LLVM - or possibly a red herring because it
doesn't reproduce reliably? I'll investigate more tomorrow.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1333 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 910 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #2 from Andy Furniss <adf.lists@gmail.com> ---
Maybe there is some mesa that doesn't do it I suppose. I don't update llvm as
often as mesa so could have missed that.

When I noticed this I reset mesa back to where the old bug fixes went in, as I
knew that used to be good. It was still bad, then I tried it on older kernels,
still bad so I got back on head and started testing llvm.

On mesa head the llvm bisect does seem good - I haven't had much time, but a
few runs with llvm sitting on the bad were all bad and a few on the one before
all good. It was only a quick test - I didn't throw cpufreq into the mix. When
I saw the result of the bisect I did think red herring as it wasn't AMD - but
then was slightly relieved when AMD got a mention in the commit message.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1729 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 1649 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #3 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Time to document some information I've gathered.

I can now confirm that Mesa has nothing to do with it. Something must have gone
wrong with my builds initially, sorry for having caused confusion.

I captured an apitrace for better reproducibility[0] and ran it with shader
dumps enabled and flushing after each draw call in the "interesting" region.

I am going to attach the R600_DEBUG=check_vm dump which I've cross-referenced
with R600_DEBUG=vm to obtain the shaders that were active during the draw call
(file names with prefix llvm-c0a189c.mesa-caf12bebd). I then matched the
shaders to those dumped by a run with a good version of LLVM (commit just
before the bad one, file names with prefix llvm-26ddca1.mesa-caf12bebd).

Clearly, the LLVM changes caused some significant re-ordering of the
instruction schedule, and that somehow, surprisingly, seems to be responsible
for the VM faults.

Another aspect to note is that the shaders are compiled before draw call
174000, while the VM faults happen shortly after draw call 178000. This seems
to suggest that the shaders alone only cause VM faults in conjunction with some
other state. However, the VM faults have always happened in exactly the same
point so far, so it does appear to be deterministic.

[0] The demo always causes VM faults, so I'm not going to upload the giant
trace; however, the timing varies between runs, so it's cleaner to reproduce
using a trace.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2500 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 311 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #4 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120421
  --> https://bugs.freedesktop.org/attachment.cgi?id=120421&action=edit
check_vm dump

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 467 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

Nicolai Hähnle <nhaehnle@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #120421|check_vm dump               |llvm-c0a189c.mesa-caf12bebd
        description|                            |.run008.check_vm-dump

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1174 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 375 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #5 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120422
  --> https://bugs.freedesktop.org/attachment.cgi?id=120422&action=edit
llvm-c0a189c.mesa-caf12bebd.run008.shader.113a9b

This is the vertex shader

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1404 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 365 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #6 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120423
  --> https://bugs.freedesktop.org/attachment.cgi?id=120423&action=edit
llvm-c0a189c.mesa-caf12bebd.run008.shader.16f00d

Fragment shader

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 344 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #7 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120424
  --> https://bugs.freedesktop.org/attachment.cgi?id=120424&action=edit
llvm-26ddca1.mesa-caf12bebd.run009.shader.vert

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1369 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 344 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #8 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120425
  --> https://bugs.freedesktop.org/attachment.cgi?id=120425&action=edit
llvm-26ddca1.mesa-caf12bebd.run009.shader.frag

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1369 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 638 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #9 from Nicolai Hähnle <nhaehnle@gmail.com> ---
Created attachment 120426
  --> https://bugs.freedesktop.org/attachment.cgi?id=120426&action=edit
dmesg.faults

Sample extractions of the first four reported VM faults across different runs.

Note how something always wants to access page 0x092D80FA, and later accesses
look like they could originate from something being very confused about the
memory layout of textures, e.g. 0x00126AAC and 0x001A6A8C differing only in two
bits.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1595 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 629 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #10 from Michel Dänzer <michel@daenzer.net> ---
(In reply to Nicolai Hähnle from comment #9)
> Note how something always wants to access page 0x092D80FA, and later
> accesses look like they could originate from something being very confused
> about the memory layout of textures, e.g. 0x00126AAC and 0x001A6A8C
> differing only in two bits.

Those symptoms could be the result of incorrect spilling/restoring of register
values, causing corrupted VM addresses to be used.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1549 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 343 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #11 from Vedran Miletić <rivanvx@gmail.com> ---
Not sure if it's related, but I get precisely the same error (GPU fault
detected: 147) when running a hello world OpenCL program using PyOpenCL.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1184 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 335 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

--- Comment #12 from Vedran Miletić <rivanvx@gmail.com> ---
Probably not, because LLVM version just one commit prior the commit mentioned
in comment #1 did not help. I opened a new bug, bug 93374.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1397 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 542 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

Nicolai Hähnle <nhaehnle@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #13 from Nicolai Hähnle <nhaehnle@gmail.com> ---
This is fixed in LLVM master as of r256072.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2132 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 93264] Tonga VM Faults since llvm ScheduleDAGInstrs: Rework schedule graph builder.

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 504 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=93264

Shawn Starr <shawn.starr@rogers.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |shawn.starr@rogers.com

--- Comment #14 from Shawn Starr <shawn.starr@rogers.com> ---
*** Bug 93436 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2196 bytes --]

[-- Attachment #2: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel