| Bug ID | 111077 |
|---|---|
| Summary | link_shader and deserialize_glsl_program suddenly consume hug= e amount of RAM |
| Product | Mesa |
| Version | 18.3 |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | blocker |
| Priority | medium |
| Component | Drivers/Gallium/radeonsi |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | roland@rptd.ch |
| QA Contact | dri-devel@lists.freedesktop.org |
Created attachment 144715 [details]
Valgrind massif log
Since a recent update a few days ago an application which barely consumes 2G
RAM at full load is very slow to load and compiling shaders causes over 16G=
RAM
to be consumed when the app eventually crashes.
I don't know what exactly in the update caused problems but certainly Mesa,=
the
amdgpu driver and LLVM did get updates.
I also tried using Mesa 19.x but the problem is the same.
Driver is xf86-video-amdgpu-19.0.1 . LLVM is 7.0.x .
I've already deleted the mesa shader cache and all caches the application
creates. I've totally recompiled the system (GenToo) to make sure no strange
problems can be around. I've also tried with a completely fresh user to run=
the
app.
Using valgrind --tool=3Dmassif the culprit seems to be ralloc_size which is
called by the two above mentioned methods. I've attached a massif log of a
couple of seconds running of the application and shutting it down before me=
mory
skyrockets even more. The app in question shows at that point of time only =
an
empty scene with a simple shader doing a sky-box. The rest is Non OpenGL UI
stuff.
Classified this as blocker since as soon as you try loading more shaders not
even 32G seems to be enough to cope with the rampaging glsl compiler.
I've started the application now also in a debugger and went t= o loading a simple model which causes tons of RAM to be consumes by the shader compiler= . I interrupted with GDB and made a trace: #0 0x00007f650ee794e7 in __memcpy_ssse3 () from /lib64/libc.so.6 #1 0x00007f650776a390 in blob_write_bytes () from /usr/lib64/dri/radeonsi_dri.so #2 0x00007f650776a4e8 in blob_write_uint32 () from /usr/lib64/dri/radeonsi_dri.so #3 0x00007f6507636421 in serialize_glsl_program () from /usr/lib64/dri/radeonsi_dri.so #4 0x00007f6507638132 in shader_cache_write_program_metadata(gl_context*, gl_shader_program*) () from /usr/lib64/dri/radeonsi_dri.so #5 0x00007f65074a9a38 in link_program_error () from /usr/lib64/dri/radeonsi_dri.so #6 0x00007f6509d85a3d in deoglShaderLanguage::pLinkShader (this=3D0x7f65004= 360c0, handle=3D298) at src/modules/graphic/opengl/src/shaders/deoglShaderLanguage.cpp:1272 #7 0x00007f6509d86537 in deoglShaderLanguage::CompileShader (this=3D0x7f65004360c0, program=3D...) at src/modules/graphic/opengl/src/shaders/deoglShaderLanguage.cpp:530 Mesa gets stuck inside "link_program_error" =3D> "shader_cache_write_program_metadata" =3D> "serialize_gls= l_program" . Most probably serialize_glsl_program goes rampage there but I have no idea if th= is is the real reason. According to the massif logs though ralloc_size is call= ed with GB of data multiple time in there somewhere.
I don't know what other information can help so I collected in= formation about the state that worked (before the update) and the state that does not work anymore (after the update): before update (working state): media-libs/mesa-18.2.8 - x11-drivers/xf86-video-amdgpu-18.1.0 - x11-libs/libdrm-2.4.96 - sys-devel/llvm-6.0.1 - sys-devel/llvmgold-6 - sys-devel/llvm-common-6.0.1 after update (memory consumption bug present): - media-libs/mesa-18.3.6 (I also tested media-libs/mesa-19.0.6 and media-libs/mesa-19.1.1 with same result) - x11-drivers/xf86-video-amdgpu-19.0.1 - x11-libs/libdrm-2.4.97 - sys-devel/llvm-7.1.0 - sys-devel/llvmgold-7 - sys-devel/llvm-common-7.1.0 Is there anything else that can help?
Are you able to build mesa from git and do a git bisect to fin= d the problem commit?
I tried compiling from source but it does not work. Seems to h= ave troubles with libdrm. configure: error: Package requirements (libdrm >=3D 2.4.75 libdrm_intel = >=3D 2.4.75) were not met: Can't seem to get past this one.
(In reply to roland@= rptd.ch from comment #4) > I tried compiling from source but it does not wo= rk. Seems to have troubles > with libdrm. >=20 > configure: error: Package requirements (libdrm >=3D 2.4.75 libdrm_i= ntel >=3D > 2.4.75) were not met: >=20 > Can't seem to get past this one. Are you sure you have the libdrm-devel package installed?
I'm on GenToo. It's a source based distro thus every package i= nstalled has headers and libraries installed too otherwise the distro won't work. So I'm positive everything required for compiling mesa is there (as I've mesa "emerged").
I get two config attempts. This is the second one. Do you see =
anything out of
place here?
meson --buildtype plain --libdir lib64 --localstatedir /var/lib --prefix /u=
sr
--sysconfdir /etc --wrap-mode nodownload
-Dplatforms=3Dx11,surfaceless,wayland,drm -Dllvm=3Dtrue -Dlmsensors=3Dtrue
-Dlibunwind=3Dfalse -Dgallium-nine=3Dfalse -Dgallium-va=3Dfalse -Dgallium-v=
dpau=3Dfalse
-Dgallium-xa=3Dfalse -Dgallium-xvmc=3Dfalse -Dgallium-opencl=3Dicd -Dosmesa=
=3Dnone
-Dbuild-tests=3Dfalse -Dglx=3Ddri -Dshared-glapi=3Dtrue -Ddri3=3Dtrue -Degl=
=3Dtrue
-Dgbm=3Dtrue -Dgles1=3Dfalse -Dgles2=3Dtrue -Dglvnd=3Dfalse -Dselinux=3Dfal=
se
-Dvalgrind=3Dfalse -Ddri-drivers=3Dr100,r200
-Dgallium-drivers=3Dr300,r600,radeonsi,swrast -Dvulkan-drivers=3Damd --buil=
dtype
plain -Db_ndebug=3Dtrue /var/tmp/portage/media-libs/mesa-19.0.8/work/mesa-1=
9.0.8
/var/tmp/portage/media-libs/mesa-19.0.8/work/mesa-19.0.8-abi_x86_64.amd64
(In reply to roland@= rptd.ch from comment #9) > I get two config attempts. This is the second on= e. Do you see anything out > of place here? >=20 > meson --buildtype plain --libdir lib64 --localstatedir /var/lib --pref= ix > /usr --sysconfdir /etc --wrap-mode nodownload > -Dplatforms=3Dx11,surfaceless,wayland,drm -Dllvm=3Dtrue -Dlmsensors=3D= true > -Dlibunwind=3Dfalse -Dgallium-nine=3Dfalse -Dgallium-va=3Dfalse > -Dgallium-vdpau=3Dfalse -Dgallium-xa=3Dfalse -Dgallium-xvmc=3Dfalse > -Dgallium-opencl=3Dicd -Dosmesa=3Dnone -Dbuild-tests=3Dfalse -Dglx=3Dd= ri > -Dshared-glapi=3Dtrue -Ddri3=3Dtrue -Degl=3Dtrue -Dgbm=3Dtrue -Dgles1= =3Dfalse > -Dgles2=3Dtrue -Dglvnd=3Dfalse -Dselinux=3Dfalse -Dvalgrind=3Dfalse > -Ddri-drivers=3Dr100,r200 -Dgallium-drivers=3Dr300,r600,radeonsi,swrast > -Dvulkan-drivers=3Damd --buildtype plain -Db_ndebug=3Dtrue > /var/tmp/portage/media-libs/mesa-19.0.8/work/mesa-19.0.8 > /var/tmp/portage/media-libs/mesa-19.0.8/work/mesa-19.0.8-abi_x86_64.am= d64 Just change the /var/tmp/portage/... path to your own build directory and y= ou should be good :)
Unfortunately it does not compile like this: FAILED: src/amd/common/2a96a08@@amd_common@sta/ac_nir_to_llvm.c= .o=20 cc -Isrc/amd/common/2a96a08@@amd_common@sta -Isrc/amd/common -I../src/amd/common -Isrc/../include -I../src/../include -Isrc -I../src -Isrc/mapi -I../src/mapi -Isrc/mesa -I../src/mesa -I../src/gallium/include -Isrc/gallium/auxiliary -I../src/gallium/auxiliary -Isrc/compiler -I../src/compiler -Isrc/amd -I../src/amd -Isrc/compiler/nir -I../src/compiler/nir -I/usr/lib64/llvm/7/include -I/usr/include/libdrm -fdiagnostics-color=3Dalways -DNDEBUG -pipe -D_FILE_OFFSET_BITS=3D64 -std= =3Dc99 '-DVERSION=3D"18.0.0-rc2"' -DPACKAGE_VERSION=3DVERSION '-DPACKAGE_BUGREPORT=3D"https://bugs.freedesktop.org/enter_bug.cgi?product= =3DMesa"' -DGLX_USE_TLS -DHAVE_X11_PLATFORM -DGLX_INDIRECT_RENDERING -DGLX_DIRECT_RENDERING -DGLX_USE_DRM -DHAVE_DRM_PLATFORM -DHAVE_SURFACELESS_PLATFORM -DENABLE_SHADER_CACHE -DHAVE___BUILTIN_BSWAP32 -DHAVE___BUILTIN_BSWAP64 -DHAVE___BUILTIN_CLZ -DHAVE___BUILTIN_CLZLL -DHAVE___BUILTIN_CTZ -DHAVE___BUILTIN_EXPECT -DHAVE___BUILTIN_FFS -DHAVE___BUILTIN_FFSLL -DHAVE___BUILTIN_POPCOUNT -DHAVE___BUILTIN_POPCOUNTLL -DHAVE___BUILTIN_UNREACHABLE -DHAVE_FUNC_ATTRIBUTE_CONST -DHAVE_FUNC_ATTRIBUTE_FLATTEN -DHAVE_FUNC_ATTRIBUTE_MALLOC -DHAVE_FUNC_ATTRIBUTE_PURE -DHAVE_FUNC_ATTRIBUTE_UNUSED -DHAVE_FUNC_ATTRIBUTE_WARN_UNUSED_RESULT -DHAVE_FUNC_ATTRIBUTE_WEAK -DHAVE_FUNC_ATTRIBUTE_FORMAT -DHAVE_FUNC_ATTRIBUTE_PACKED -DHAVE_FUNC_ATTRIBUTE_RETURNS_NONNULL -DHAVE_FUNC_ATTRIBUTE_VISIBILITY -DHAVE_FUNC_ATTRIBUTE_ALIAS -DHAVE_FUNC_ATTRIBUTE_NORETURN -DUSE_SSE41 -DUSE_GCC_ATOMIC_BUILTINS -DUSE_X86_64_ASM -DMAJOR_IN_SYSMACROS -DHAVE_SYS_SYSCTL_H -DHAVE_LINUX_FUTEX_H -DHAVE_STRTOF -DHAVE_MKOSTEMP -DHAVE_POSIX_MEMALIGN -DHAVE_TIMESPEC_GET -DHAVE_MEMFD_CREATE -DHAVE_STRTOD= _L -DHAVE_DLADDR -DHAVE_DL_ITERATE_PHDR -DHAVE_LIBDRM -DHAVE_ZLIB -DHAVE_PTHRE= AD -DHAVE_LLVM=3D0x0710 -DMESA_LLVM_VERSION_PATCH=3D0 -DHAVE_WAYLAND_PLATFORM -DWL_HIDE_DEPRECATED -DHAVE_DRI3 -DHAVE_LIBSENSORS=3D1 -Wall -Werror=3Dimplicit-function-declaration -Werror=3Dmissing-prototypes -fno-math-errno -fno-trapping-math -fPIC -pthread -D__STDC_LIMIT_MACROS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -fvisibility=3Dhidden -MD -MQ 'src/amd/common/2a96a08@@amd_common@sta/ac_nir_to_llvm.c.o' -MF 'src/amd/common/2a96a08@@amd_common@sta/ac_nir_to_llvm.c.o.d' -o 'src/amd/common/2a96a08@@amd_common@sta/ac_nir_to_llvm.c.o' -c ../src/amd/common/ac_nir_to_llvm.c ../src/amd/common/ac_nir_to_llvm.c: In function =E2=80=98ac_llvm_finalize_m= odule=E2=80=99: ../src/amd/common/ac_nir_to_llvm.c:6614:2: error: implicit declaration of function =E2=80=98LLVMAddPromoteMemoryToRegisterPass=E2=80=99; did you mean =E2=80=98LLVMAddDemoteMemoryToRegisterPass=E2=80=99? [-Werror=3Dimplicit-fu= nction-declaration] LLVMAddPromoteMemoryToRegisterPass(passmgr); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LLVMAddDemoteMemoryToRegisterPass cc1: some warnings being treated as errors [425/1549] Compiling C object 'src/mesa/bbe4a73@@mesa_gallium@sta/main_format_utils.c.o'. ninja: build stopped: subcommand failed.
(In reply to roland@= rptd.ch from comment #11) > ../src/amd/common/ac_nir_to_llvm.c > ../src/amd/common/ac_nir_to_llvm.c: In function =E2=80=98ac_llvm_final= ize_module=E2=80=99: > ../src/amd/common/ac_nir_to_llvm.c:6614:2: error: implicit declaration= of > function =E2=80=98LLVMAddPromoteMemoryToRegisterPass=E2=80=99; did you= mean > =E2=80=98LLVMAddDemoteMemoryToRegisterPass=E2=80=99? [-Werror=3Dimplic= it-function-declaration] > LLVMAddPromoteMemoryToRegisterPass(passmgr); > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > LLVMAddDemoteMemoryToRegisterPass > cc1: some warnings being treated as errors > [425/1549] Compiling C object > 'src/mesa/bbe4a73@@mesa_gallium@sta/main_format_utils.c.o'. > ninja: build stopped: subcommand failed. Is this during the bisect, or when? Try adding=20 #if HAVE_LLVM >=3D 0x0700 #include <llvm-c/Transforms/Utils.h> #endif to the #include section of src/amd/common/ac_nir_to_llvm.c
I checked out the 18.2 branch which I assume should work (if t= he theory is correct). Modifying files won't work with bisecting, right?
(In reply to roland@= rptd.ch from comment #13) > I checked out the 18.2 branch which I assume sho= uld work (if the theory is > correct). >=20 > Modifying files won't work with bisecting, right? You'll probably have to 'git apply' the patch after each step, and 'git checkout -f' before each time you do 'git bisect good/bad'
I've tried now the patching approach. I had to patch in total = three files: src/amd/common/ac_nir_to_llvm.c src/gallium/auxiliary/gallivm/lp_bld_init.c src/gallium/drivers/radeonsi/si_shader_tgsi_setup.c Nevertheless the build still fails at linking stage (see fail_log attachmen= t).
Created attachment 145083 <=
a href=3D"attachment.cgi?id=3D145083&action=3Dedit" title=3D"Linking fa=
il log">[details]
Linking fail log
The build failure is in Clover, the OpenCL implementation. If = the application that triggers the huge amount of RAM problem is not using OpenCL, disable OpenCL in meson configure and try to get past that.
| What | Removed | Added |
|---|---|---|
| CC | mattst88@gmail.com |
Cc'ing myself in case more help is needed bisecting.
Need to shift this back to GenToo since compiling (aka bisecti= ng) is not the problem right now but GenToo (which actually is the "only" distro= I've seen this problem on so far).
| What | Removed | Added |
|---|---|---|
| CC | maraeo@gmail.com | |
| Keywords | bisected |
Great. We've got a bisect, and reverting the commit from 19.0.= 8 fixes the issue. commit 9176703788c66de8287c6224650b1ff8d4238126 Author: Marek Ol=C5=A1=C3=A1k <marek.olsak@amd.com> Date: Wed Aug 8 15:37:21 2018 -0400 radeonsi: increase the maximum UBO size to 2 GB Same as the closed driver. This causes a failure in GL45-CTS.compute_shader.max, which has a trivi= al bug. Tested-by: Dieter N=C3=BCtzel <Dieter@nuetzel-hh.de> (see https://bugs.gentoo.org= /690066#c33)
Hi Marek, This is going to be complicated. The application is not yet free to use by others (working on getting it to release shape). I would have to figure out first how to break this down into a reproducible test case since I don't kn= ow myself what triggers the bug. If you can think of some corner values to narrow down in what direction to search I can fully mess with the source code over here. The faulty commit t= alks about UBO maximum size so this might be a start. The OpenGL Capabilities fr= om the GPU is this: - UBO Maximum Block Size =3D 65536 - UBO Buffer Offset Alignment =3D 4 So the maximum size used by the application is 65536 bytes. UBOs are used as shared buffers so blocks of data are placed next to each o= ther respecting alignment and updated. UBOs are created like this: glBindBuffer(GL_UNIFORM_BUFFER, pUBO) // <=3D done once glBufferData(GL_UNIFORM_BUFFER, bufferSize, NULL, GL_DYNAMIC_DRAW) // <= =3D done once glMapBufferRange(GL_UNIFORM_BUFFER, stride * elementCount, elementStride, GL_WRITE_ONLY | GL_MAP_INVALIDATE_RANGE_BIT) // <=3D done for each data= block written Data then written and unmapped In particular this means a larger UBO is created once then individual blocks are written to it using ranged mapping. Just a wield guess but could the problem be related to this kind of usage pattern?