[PATCH] zstd: reducing inlining for KASAN_STACK

[PATCH] zstd: reducing inlining for KASAN_STACK

[Arnd Bergmann]

From: Arnd Bergmann <arnd@arndb.de>

The zstd code forces inlining to improve code generation with most
compiler versions. However, when CONFIG_KASAN_STACK is enabled, it has
the opposite effect by causing excessive spilling of local variables to
the stack, to the point where the stack warning limit is exceeded:

lib/zstd/decompress/huf_decompress.c: In function 'HUF_decompress4X2_usingDTable_internal_default':
lib/zstd/decompress/huf_decompress.c:1512:1: error: the frame size of 1616 bytes is larger than 1536 bytes [-Werror=frame-larger-than=]
cc1: all warnings being treated as errors
lib/zstd/decompress/zstd_decompress_block.c: In function 'ZSTD_decompressSequencesLong_body.constprop':
lib/zstd/decompress/zstd_decompress_block.c:1889:1: error: the frame size of 1584 bytes is larger than 1536 bytes [-Werror=frame-larger-than=]

Disable the extra inlining in this configuration and rely on
the normal compiler heuristics instead.

Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
 lib/zstd/Makefile | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/zstd/Makefile b/lib/zstd/Makefile
index be218b5e0ed5..3909df12116b 100644
--- a/lib/zstd/Makefile
+++ b/lib/zstd/Makefile
@@ -8,6 +8,13 @@
 # in the COPYING file in the root directory of this source tree).
 # You may select, at your option, one of the above-listed licenses.
 # ################################################################
+
+ifdef CONFIG_KASAN_STACK
+# with address sanitizer, the forced inlining produces worse
+# code and and risks stack overflow
+subdir-ccflags-y += -DZSTD_NO_INLINE
+endif
+
 obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o
 obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o
 obj-$(CONFIG_ZSTD_COMMON) += zstd_common.o
-- 
2.39.5