* Re: [PATCH net-next v6 0/4] bpf: add two helpers to read perf event enabled/running time
From: David Miller @ 2017-10-04 23:00 UTC (permalink / raw)
To: yhs; +Cc: peterz, rostedt, ast, daniel, netdev, kernel-team
In-Reply-To: <20171002224218.3181418-1-yhs@fb.com>
From: Yonghong Song <yhs@fb.com>
Date: Mon, 2 Oct 2017 15:42:14 -0700
> [Dave, Peter,
>
> Previous communcation shows that this patch may potentially have
> merge conflict with upcoming tip changes in the next merge window.
>
> Could you advise how this patch should proceed?
>
> Thanks!
> ]
Indeed, Peter how do you want to handle this?
Thanks.
^ permalink raw reply
* Re: Kernel 4.13 is broken on m68k
From: John Paul Adrian Glaubitz @ 2017-10-04 23:00 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Linux/m68k, Debian m68k
In-Reply-To: <63aaef75-5504-a8b9-485f-746f25391cfa@physik.fu-berlin.de>
On 10/05/2017 12:49 AM, John Paul Adrian Glaubitz wrote:
> busybox has gotten new maintainers in Debian and they have stripped
> "-O2" from CFLAGS so that it's building with "-Os" now [1].
>
> Maybe that caused the code to be miscompiled. I hope I won't have
> to bisect busybox.
Reverting the commit from [1] fixes the problem for me:
root@pacman:~# busybox ash -c 'echo ${x}'
root@pacman:~#
Adrian
>> [1] https://anonscm.debian.org/cgit/d-i/busybox.git/commit/?id=8a115a910fb3cd7a525972e150a32f8aeb1da49
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
^ permalink raw reply
* [PATCH v6 3/3] efd: run-time dispatch over x86 EFD functions
From: Xiaoyun Li @ 2017-10-04 22:58 UTC (permalink / raw)
To: konstantin.ananyev, bruce.richardson
Cc: wenzhuo.lu, helin.zhang, dev, Xiaoyun Li
In-Reply-To: <1507157911-8702-1-git-send-email-xiaoyun.li@intel.com>
This patch enables x86 EFD file be compiled only if the compiler
supports AVX2 since it is already chosen at run-time.
Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
---
lib/librte_efd/Makefile | 6 +++
lib/librte_efd/rte_efd_x86.c | 87 ++++++++++++++++++++++++++++++++++++++++++++
lib/librte_efd/rte_efd_x86.h | 48 +-----------------------
3 files changed, 95 insertions(+), 46 deletions(-)
create mode 100644 lib/librte_efd/rte_efd_x86.c
diff --git a/lib/librte_efd/Makefile b/lib/librte_efd/Makefile
index b9277bc..35bb2bd 100644
--- a/lib/librte_efd/Makefile
+++ b/lib/librte_efd/Makefile
@@ -44,6 +44,12 @@ LIBABIVER := 1
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_EFD) := rte_efd.c
+# if the compiler supports AVX2, add efd x86 file
+ifneq ($(findstring CC_SUPPORT_AVX2,$(MACHINE_CFLAGS)),)
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_efd_x86.c
+CFLAGS_rte_efd_x86.o += -mavx2
+endif
+
# install this header file
SYMLINK-$(CONFIG_RTE_LIBRTE_EFD)-include := rte_efd.h
diff --git a/lib/librte_efd/rte_efd_x86.c b/lib/librte_efd/rte_efd_x86.c
new file mode 100644
index 0000000..d2d1ac5
--- /dev/null
+++ b/lib/librte_efd/rte_efd_x86.c
@@ -0,0 +1,87 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* rte_efd_x86.c
+ * This file holds all x86 specific EFD functions
+ */
+#include <rte_efd.h>
+#include <rte_efd_x86.h>
+
+#if (RTE_EFD_VALUE_NUM_BITS == 8 || RTE_EFD_VALUE_NUM_BITS == 16 || \
+ RTE_EFD_VALUE_NUM_BITS == 24 || RTE_EFD_VALUE_NUM_BITS == 32)
+#define EFD_LOAD_SI128(val) _mm_load_si128(val)
+#else
+#define EFD_LOAD_SI128(val) _mm_lddqu_si128(val)
+#endif
+
+efd_value_t
+efd_lookup_internal_avx2(const efd_hashfunc_t *group_hash_idx,
+ const efd_lookuptbl_t *group_lookup_table,
+ const uint32_t hash_val_a, const uint32_t hash_val_b)
+{
+#ifdef CC_SUPPORT_AVX2
+ efd_value_t value = 0;
+ uint32_t i = 0;
+ __m256i vhash_val_a = _mm256_set1_epi32(hash_val_a);
+ __m256i vhash_val_b = _mm256_set1_epi32(hash_val_b);
+
+ for (; i < RTE_EFD_VALUE_NUM_BITS; i += 8) {
+ __m256i vhash_idx =
+ _mm256_cvtepu16_epi32(EFD_LOAD_SI128(
+ (__m128i const *) &group_hash_idx[i]));
+ __m256i vlookup_table = _mm256_cvtepu16_epi32(
+ EFD_LOAD_SI128((__m128i const *)
+ &group_lookup_table[i]));
+ __m256i vhash = _mm256_add_epi32(vhash_val_a,
+ _mm256_mullo_epi32(vhash_idx, vhash_val_b));
+ __m256i vbucket_idx = _mm256_srli_epi32(vhash,
+ EFD_LOOKUPTBL_SHIFT);
+ __m256i vresult = _mm256_srlv_epi32(vlookup_table,
+ vbucket_idx);
+
+ value |= (_mm256_movemask_ps(
+ (__m256) _mm256_slli_epi32(vresult, 31))
+ & ((1 << (RTE_EFD_VALUE_NUM_BITS - i)) - 1)) << i;
+ }
+
+ return value;
+#else
+ RTE_SET_USED(group_hash_idx);
+ RTE_SET_USED(group_lookup_table);
+ RTE_SET_USED(hash_val_a);
+ RTE_SET_USED(hash_val_b);
+ /* Return dummy value, only to avoid compilation breakage */
+ return 0;
+#endif
+
+}
diff --git a/lib/librte_efd/rte_efd_x86.h b/lib/librte_efd/rte_efd_x86.h
index 34f37d7..7a082aa 100644
--- a/lib/librte_efd/rte_efd_x86.h
+++ b/lib/librte_efd/rte_efd_x86.h
@@ -36,51 +36,7 @@
*/
#include <immintrin.h>
-#if (RTE_EFD_VALUE_NUM_BITS == 8 || RTE_EFD_VALUE_NUM_BITS == 16 || \
- RTE_EFD_VALUE_NUM_BITS == 24 || RTE_EFD_VALUE_NUM_BITS == 32)
-#define EFD_LOAD_SI128(val) _mm_load_si128(val)
-#else
-#define EFD_LOAD_SI128(val) _mm_lddqu_si128(val)
-#endif
-
-static inline efd_value_t
+extern efd_value_t
efd_lookup_internal_avx2(const efd_hashfunc_t *group_hash_idx,
const efd_lookuptbl_t *group_lookup_table,
- const uint32_t hash_val_a, const uint32_t hash_val_b)
-{
-#ifdef RTE_MACHINE_CPUFLAG_AVX2
- efd_value_t value = 0;
- uint32_t i = 0;
- __m256i vhash_val_a = _mm256_set1_epi32(hash_val_a);
- __m256i vhash_val_b = _mm256_set1_epi32(hash_val_b);
-
- for (; i < RTE_EFD_VALUE_NUM_BITS; i += 8) {
- __m256i vhash_idx =
- _mm256_cvtepu16_epi32(EFD_LOAD_SI128(
- (__m128i const *) &group_hash_idx[i]));
- __m256i vlookup_table = _mm256_cvtepu16_epi32(
- EFD_LOAD_SI128((__m128i const *)
- &group_lookup_table[i]));
- __m256i vhash = _mm256_add_epi32(vhash_val_a,
- _mm256_mullo_epi32(vhash_idx, vhash_val_b));
- __m256i vbucket_idx = _mm256_srli_epi32(vhash,
- EFD_LOOKUPTBL_SHIFT);
- __m256i vresult = _mm256_srlv_epi32(vlookup_table,
- vbucket_idx);
-
- value |= (_mm256_movemask_ps(
- (__m256) _mm256_slli_epi32(vresult, 31))
- & ((1 << (RTE_EFD_VALUE_NUM_BITS - i)) - 1)) << i;
- }
-
- return value;
-#else
- RTE_SET_USED(group_hash_idx);
- RTE_SET_USED(group_lookup_table);
- RTE_SET_USED(hash_val_a);
- RTE_SET_USED(hash_val_b);
- /* Return dummy value, only to avoid compilation breakage */
- return 0;
-#endif
-
-}
+ const uint32_t hash_val_a, const uint32_t hash_val_b);
--
2.7.4
^ permalink raw reply related
* [PATCH v6 2/3] app/test: run-time dispatch over memcpy perf test
From: Xiaoyun Li @ 2017-10-04 22:58 UTC (permalink / raw)
To: konstantin.ananyev, bruce.richardson
Cc: wenzhuo.lu, helin.zhang, dev, Xiaoyun Li
In-Reply-To: <1507157911-8702-1-git-send-email-xiaoyun.li@intel.com>
This patch modifies assignment of alignment unit from build-time
to run-time based on CPU flags that machine supports.
Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
---
test/test/test_memcpy_perf.c | 40 +++++++++++++++++++++++++++-------------
1 file changed, 27 insertions(+), 13 deletions(-)
diff --git a/test/test/test_memcpy_perf.c b/test/test/test_memcpy_perf.c
index ff3aaaa..33def3b 100644
--- a/test/test/test_memcpy_perf.c
+++ b/test/test/test_memcpy_perf.c
@@ -79,13 +79,7 @@ static size_t buf_sizes[TEST_VALUE_RANGE];
#define TEST_BATCH_SIZE 100
/* Data is aligned on this many bytes (power of 2) */
-#ifdef RTE_MACHINE_CPUFLAG_AVX512F
-#define ALIGNMENT_UNIT 64
-#elif defined RTE_MACHINE_CPUFLAG_AVX2
-#define ALIGNMENT_UNIT 32
-#else /* RTE_MACHINE_CPUFLAG */
-#define ALIGNMENT_UNIT 16
-#endif /* RTE_MACHINE_CPUFLAG */
+static uint8_t alignment_unit = 16;
/*
* Pointers used in performance tests. The two large buffers are for uncached
@@ -100,20 +94,39 @@ static int
init_buffers(void)
{
unsigned i;
+#ifdef CC_SUPPORT_AVX512
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F))
+ alignment_unit = 64;
+ else
+#endif
+#ifdef CC_SUPPORT_AVX2
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2))
+ alignment_unit = 32;
+ else
+#endif
+ alignment_unit = 16;
- large_buf_read = rte_malloc("memcpy", LARGE_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
+ large_buf_read = rte_malloc("memcpy",
+ LARGE_BUFFER_SIZE + alignment_unit,
+ alignment_unit);
if (large_buf_read == NULL)
goto error_large_buf_read;
- large_buf_write = rte_malloc("memcpy", LARGE_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
+ large_buf_write = rte_malloc("memcpy",
+ LARGE_BUFFER_SIZE + alignment_unit,
+ alignment_unit);
if (large_buf_write == NULL)
goto error_large_buf_write;
- small_buf_read = rte_malloc("memcpy", SMALL_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
+ small_buf_read = rte_malloc("memcpy",
+ SMALL_BUFFER_SIZE + alignment_unit,
+ alignment_unit);
if (small_buf_read == NULL)
goto error_small_buf_read;
- small_buf_write = rte_malloc("memcpy", SMALL_BUFFER_SIZE + ALIGNMENT_UNIT, ALIGNMENT_UNIT);
+ small_buf_write = rte_malloc("memcpy",
+ SMALL_BUFFER_SIZE + alignment_unit,
+ alignment_unit);
if (small_buf_write == NULL)
goto error_small_buf_write;
@@ -153,7 +166,7 @@ static inline size_t
get_rand_offset(size_t uoffset)
{
return ((rte_rand() % (LARGE_BUFFER_SIZE - SMALL_BUFFER_SIZE)) &
- ~(ALIGNMENT_UNIT - 1)) + uoffset;
+ ~(alignment_unit - 1)) + uoffset;
}
/* Fill in source and destination addresses. */
@@ -321,7 +334,8 @@ perf_test(void)
"(bytes) (ticks) (ticks) (ticks) (ticks)\n"
"------- -------------- -------------- -------------- --------------");
- printf("\n========================== %2dB aligned ============================", ALIGNMENT_UNIT);
+ printf("\n========================= %2dB aligned ============================",
+ alignment_unit);
/* Do aligned tests where size is a variable */
perf_test_variable_aligned();
printf("\n------- -------------- -------------- -------------- --------------");
--
2.7.4
^ permalink raw reply related
* [PATCH v6 1/3] eal/x86: run-time dispatch over memcpy
From: Xiaoyun Li @ 2017-10-04 22:58 UTC (permalink / raw)
To: konstantin.ananyev, bruce.richardson
Cc: wenzhuo.lu, helin.zhang, dev, Xiaoyun Li
In-Reply-To: <1507157911-8702-1-git-send-email-xiaoyun.li@intel.com>
This patch dynamically selects functions of memcpy at run-time based
on CPU flags that current machine supports. This patch uses function
pointers which are bind to the relative functions at constrctor time.
In addition, AVX512 instructions set would be compiled only if users
config it enabled and the compiler supports it.
Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
---
lib/librte_eal/bsdapp/eal/Makefile | 19 +
lib/librte_eal/bsdapp/eal/rte_eal_version.map | 1 +
.../common/include/arch/x86/rte_memcpy.c | 59 ++
.../common/include/arch/x86/rte_memcpy.h | 861 +------------------
.../common/include/arch/x86/rte_memcpy_avx2.c | 44 +
.../common/include/arch/x86/rte_memcpy_avx512f.c | 44 +
.../common/include/arch/x86/rte_memcpy_internal.h | 909 +++++++++++++++++++++
.../common/include/arch/x86/rte_memcpy_sse.c | 40 +
lib/librte_eal/linuxapp/eal/Makefile | 19 +
lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 +
mk/rte.cpuflags.mk | 14 +
11 files changed, 1165 insertions(+), 846 deletions(-)
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c
diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 005019e..1dcd2e3 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -36,6 +36,7 @@ LIB = librte_eal.a
ARCH_DIR ?= $(RTE_ARCH)
VPATH += $(RTE_SDK)/lib/librte_eal/common
VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)
CFLAGS += -I$(SRCDIR)/include
CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
@@ -93,6 +94,24 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_service.c
SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_cpuflags.c
SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
+# for run-time dispatch of memcpy
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy.c
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_sse.c
+
+# if the compiler supports AVX512, add avx512 file
+ifneq ($(findstring CC_SUPPORT_AVX512F,$(MACHINE_CFLAGS)),)
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_avx512f.c
+CFLAGS_rte_memcpy_avx512f.o += -mavx512f
+CFLAGS_rte_memcpy_avx512f.o += -DRTE_MACHINE_CPUFLAG_AVX512F
+endif
+
+# if the compiler supports AVX2, add avx2 file
+ifneq ($(findstring CC_SUPPORT_AVX2,$(MACHINE_CFLAGS)),)
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_avx2.c
+CFLAGS_rte_memcpy_avx2.o += -mavx2
+CFLAGS_rte_memcpy_avx2.o += -DRTE_MACHINE_CPUFLAG_AVX2
+endif
+
CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
CFLAGS_eal.o := -D_GNU_SOURCE
diff --git a/lib/librte_eal/bsdapp/eal/rte_eal_version.map b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
index 47a09ea..e46c3e5 100644
--- a/lib/librte_eal/bsdapp/eal/rte_eal_version.map
+++ b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
@@ -236,5 +236,6 @@ EXPERIMENTAL {
rte_service_runstate_set;
rte_service_set_stats_enable;
rte_service_start_with_defaults;
+ rte_memcpy_ptr;
} DPDK_17.08;
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.c b/lib/librte_eal/common/include/arch/x86/rte_memcpy.c
new file mode 100644
index 0000000..74ae702
--- /dev/null
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.c
@@ -0,0 +1,59 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_memcpy.h>
+#include <rte_cpuflags.h>
+#include <rte_log.h>
+
+void *(*rte_memcpy_ptr)(void *dst, const void *src, size_t n) = NULL;
+
+static void __attribute__((constructor))
+rte_memcpy_init(void)
+{
+#ifdef CC_SUPPORT_AVX512F
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) {
+ rte_memcpy_ptr = rte_memcpy_avx512f;
+ RTE_LOG(DEBUG, EAL, "AVX512 memcpy is using!\n");
+ return;
+ }
+#endif
+#ifdef CC_SUPPORT_AVX2
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) {
+ rte_memcpy_ptr = rte_memcpy_avx2;
+ RTE_LOG(DEBUG, EAL, "AVX2 memcpy is using!\n");
+ return;
+ }
+#endif
+ rte_memcpy_ptr = rte_memcpy_sse;
+ RTE_LOG(DEBUG, EAL, "Default SSE/AVX memcpy is using!\n");
+}
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
index 74c280c..460dcdb 100644
--- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h
@@ -1,7 +1,7 @@
/*-
* BSD LICENSE
*
- * Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -34,867 +34,36 @@
#ifndef _RTE_MEMCPY_X86_64_H_
#define _RTE_MEMCPY_X86_64_H_
-/**
- * @file
- *
- * Functions for SSE/AVX/AVX2/AVX512 implementation of memcpy().
- */
-
-#include <stdio.h>
-#include <stdint.h>
-#include <string.h>
-#include <rte_vect.h>
-#include <rte_common.h>
+#include <rte_memcpy_internal.h>
#ifdef __cplusplus
extern "C" {
#endif
-/**
- * Copy bytes from one location to another. The locations must not overlap.
- *
- * @note This is implemented as a macro, so it's address should not be taken
- * and care is needed as parameter expressions may be evaluated multiple times.
- *
- * @param dst
- * Pointer to the destination of the data.
- * @param src
- * Pointer to the source data.
- * @param n
- * Number of bytes to copy.
- * @return
- * Pointer to the destination data.
- */
-static __rte_always_inline void *
-rte_memcpy(void *dst, const void *src, size_t n);
-
-#ifdef RTE_MACHINE_CPUFLAG_AVX512F
+#define RTE_X86_MEMCPY_THRESH 128
-#define ALIGNMENT_MASK 0x3F
+extern void *
+(*rte_memcpy_ptr)(void *dst, const void *src, size_t n);
/**
- * AVX512 implementation below
+ * Different implementations of memcpy.
*/
+extern void*
+rte_memcpy_avx512f(void *dst, const void *src, size_t n);
-/**
- * Copy 16 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov16(uint8_t *dst, const uint8_t *src)
-{
- __m128i xmm0;
-
- xmm0 = _mm_loadu_si128((const __m128i *)src);
- _mm_storeu_si128((__m128i *)dst, xmm0);
-}
-
-/**
- * Copy 32 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov32(uint8_t *dst, const uint8_t *src)
-{
- __m256i ymm0;
+extern void *
+rte_memcpy_avx2(void *dst, const void *src, size_t n);
- ymm0 = _mm256_loadu_si256((const __m256i *)src);
- _mm256_storeu_si256((__m256i *)dst, ymm0);
-}
-
-/**
- * Copy 64 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov64(uint8_t *dst, const uint8_t *src)
-{
- __m512i zmm0;
-
- zmm0 = _mm512_loadu_si512((const void *)src);
- _mm512_storeu_si512((void *)dst, zmm0);
-}
-
-/**
- * Copy 128 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov128(uint8_t *dst, const uint8_t *src)
-{
- rte_mov64(dst + 0 * 64, src + 0 * 64);
- rte_mov64(dst + 1 * 64, src + 1 * 64);
-}
-
-/**
- * Copy 256 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov256(uint8_t *dst, const uint8_t *src)
-{
- rte_mov64(dst + 0 * 64, src + 0 * 64);
- rte_mov64(dst + 1 * 64, src + 1 * 64);
- rte_mov64(dst + 2 * 64, src + 2 * 64);
- rte_mov64(dst + 3 * 64, src + 3 * 64);
-}
-
-/**
- * Copy 128-byte blocks from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n)
-{
- __m512i zmm0, zmm1;
-
- while (n >= 128) {
- zmm0 = _mm512_loadu_si512((const void *)(src + 0 * 64));
- n -= 128;
- zmm1 = _mm512_loadu_si512((const void *)(src + 1 * 64));
- src = src + 128;
- _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0);
- _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1);
- dst = dst + 128;
- }
-}
-
-/**
- * Copy 512-byte blocks from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov512blocks(uint8_t *dst, const uint8_t *src, size_t n)
-{
- __m512i zmm0, zmm1, zmm2, zmm3, zmm4, zmm5, zmm6, zmm7;
-
- while (n >= 512) {
- zmm0 = _mm512_loadu_si512((const void *)(src + 0 * 64));
- n -= 512;
- zmm1 = _mm512_loadu_si512((const void *)(src + 1 * 64));
- zmm2 = _mm512_loadu_si512((const void *)(src + 2 * 64));
- zmm3 = _mm512_loadu_si512((const void *)(src + 3 * 64));
- zmm4 = _mm512_loadu_si512((const void *)(src + 4 * 64));
- zmm5 = _mm512_loadu_si512((const void *)(src + 5 * 64));
- zmm6 = _mm512_loadu_si512((const void *)(src + 6 * 64));
- zmm7 = _mm512_loadu_si512((const void *)(src + 7 * 64));
- src = src + 512;
- _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0);
- _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1);
- _mm512_storeu_si512((void *)(dst + 2 * 64), zmm2);
- _mm512_storeu_si512((void *)(dst + 3 * 64), zmm3);
- _mm512_storeu_si512((void *)(dst + 4 * 64), zmm4);
- _mm512_storeu_si512((void *)(dst + 5 * 64), zmm5);
- _mm512_storeu_si512((void *)(dst + 6 * 64), zmm6);
- _mm512_storeu_si512((void *)(dst + 7 * 64), zmm7);
- dst = dst + 512;
- }
-}
-
-static inline void *
-rte_memcpy_generic(void *dst, const void *src, size_t n)
-{
- uintptr_t dstu = (uintptr_t)dst;
- uintptr_t srcu = (uintptr_t)src;
- void *ret = dst;
- size_t dstofss;
- size_t bits;
-
- /**
- * Copy less than 16 bytes
- */
- if (n < 16) {
- if (n & 0x01) {
- *(uint8_t *)dstu = *(const uint8_t *)srcu;
- srcu = (uintptr_t)((const uint8_t *)srcu + 1);
- dstu = (uintptr_t)((uint8_t *)dstu + 1);
- }
- if (n & 0x02) {
- *(uint16_t *)dstu = *(const uint16_t *)srcu;
- srcu = (uintptr_t)((const uint16_t *)srcu + 1);
- dstu = (uintptr_t)((uint16_t *)dstu + 1);
- }
- if (n & 0x04) {
- *(uint32_t *)dstu = *(const uint32_t *)srcu;
- srcu = (uintptr_t)((const uint32_t *)srcu + 1);
- dstu = (uintptr_t)((uint32_t *)dstu + 1);
- }
- if (n & 0x08)
- *(uint64_t *)dstu = *(const uint64_t *)srcu;
- return ret;
- }
-
- /**
- * Fast way when copy size doesn't exceed 512 bytes
- */
- if (n <= 32) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n,
- (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 64) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov32((uint8_t *)dst - 32 + n,
- (const uint8_t *)src - 32 + n);
- return ret;
- }
- if (n <= 512) {
- if (n >= 256) {
- n -= 256;
- rte_mov256((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 256;
- dst = (uint8_t *)dst + 256;
- }
- if (n >= 128) {
- n -= 128;
- rte_mov128((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 128;
- dst = (uint8_t *)dst + 128;
- }
-COPY_BLOCK_128_BACK63:
- if (n > 64) {
- rte_mov64((uint8_t *)dst, (const uint8_t *)src);
- rte_mov64((uint8_t *)dst - 64 + n,
- (const uint8_t *)src - 64 + n);
- return ret;
- }
- if (n > 0)
- rte_mov64((uint8_t *)dst - 64 + n,
- (const uint8_t *)src - 64 + n);
- return ret;
- }
-
- /**
- * Make store aligned when copy size exceeds 512 bytes
- */
- dstofss = ((uintptr_t)dst & 0x3F);
- if (dstofss > 0) {
- dstofss = 64 - dstofss;
- n -= dstofss;
- rte_mov64((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + dstofss;
- dst = (uint8_t *)dst + dstofss;
- }
-
- /**
- * Copy 512-byte blocks.
- * Use copy block function for better instruction order control,
- * which is important when load is unaligned.
- */
- rte_mov512blocks((uint8_t *)dst, (const uint8_t *)src, n);
- bits = n;
- n = n & 511;
- bits -= n;
- src = (const uint8_t *)src + bits;
- dst = (uint8_t *)dst + bits;
-
- /**
- * Copy 128-byte blocks.
- * Use copy block function for better instruction order control,
- * which is important when load is unaligned.
- */
- if (n >= 128) {
- rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n);
- bits = n;
- n = n & 127;
- bits -= n;
- src = (const uint8_t *)src + bits;
- dst = (uint8_t *)dst + bits;
- }
-
- /**
- * Copy whatever left
- */
- goto COPY_BLOCK_128_BACK63;
-}
-
-#elif defined RTE_MACHINE_CPUFLAG_AVX2
-
-#define ALIGNMENT_MASK 0x1F
-
-/**
- * AVX2 implementation below
- */
-
-/**
- * Copy 16 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov16(uint8_t *dst, const uint8_t *src)
-{
- __m128i xmm0;
-
- xmm0 = _mm_loadu_si128((const __m128i *)src);
- _mm_storeu_si128((__m128i *)dst, xmm0);
-}
-
-/**
- * Copy 32 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov32(uint8_t *dst, const uint8_t *src)
-{
- __m256i ymm0;
-
- ymm0 = _mm256_loadu_si256((const __m256i *)src);
- _mm256_storeu_si256((__m256i *)dst, ymm0);
-}
-
-/**
- * Copy 64 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov64(uint8_t *dst, const uint8_t *src)
-{
- rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
- rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
-}
-
-/**
- * Copy 128 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov128(uint8_t *dst, const uint8_t *src)
-{
- rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
- rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
- rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
- rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
-}
-
-/**
- * Copy 128-byte blocks from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n)
-{
- __m256i ymm0, ymm1, ymm2, ymm3;
-
- while (n >= 128) {
- ymm0 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 0 * 32));
- n -= 128;
- ymm1 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 1 * 32));
- ymm2 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 2 * 32));
- ymm3 = _mm256_loadu_si256((const __m256i *)((const uint8_t *)src + 3 * 32));
- src = (const uint8_t *)src + 128;
- _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
- _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
- _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2);
- _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3);
- dst = (uint8_t *)dst + 128;
- }
-}
-
-static inline void *
-rte_memcpy_generic(void *dst, const void *src, size_t n)
-{
- uintptr_t dstu = (uintptr_t)dst;
- uintptr_t srcu = (uintptr_t)src;
- void *ret = dst;
- size_t dstofss;
- size_t bits;
-
- /**
- * Copy less than 16 bytes
- */
- if (n < 16) {
- if (n & 0x01) {
- *(uint8_t *)dstu = *(const uint8_t *)srcu;
- srcu = (uintptr_t)((const uint8_t *)srcu + 1);
- dstu = (uintptr_t)((uint8_t *)dstu + 1);
- }
- if (n & 0x02) {
- *(uint16_t *)dstu = *(const uint16_t *)srcu;
- srcu = (uintptr_t)((const uint16_t *)srcu + 1);
- dstu = (uintptr_t)((uint16_t *)dstu + 1);
- }
- if (n & 0x04) {
- *(uint32_t *)dstu = *(const uint32_t *)srcu;
- srcu = (uintptr_t)((const uint32_t *)srcu + 1);
- dstu = (uintptr_t)((uint32_t *)dstu + 1);
- }
- if (n & 0x08) {
- *(uint64_t *)dstu = *(const uint64_t *)srcu;
- }
- return ret;
- }
-
- /**
- * Fast way when copy size doesn't exceed 256 bytes
- */
- if (n <= 32) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n,
- (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 48) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst + 16, (const uint8_t *)src + 16);
- rte_mov16((uint8_t *)dst - 16 + n,
- (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 64) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov32((uint8_t *)dst - 32 + n,
- (const uint8_t *)src - 32 + n);
- return ret;
- }
- if (n <= 256) {
- if (n >= 128) {
- n -= 128;
- rte_mov128((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 128;
- dst = (uint8_t *)dst + 128;
- }
-COPY_BLOCK_128_BACK31:
- if (n >= 64) {
- n -= 64;
- rte_mov64((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 64;
- dst = (uint8_t *)dst + 64;
- }
- if (n > 32) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov32((uint8_t *)dst - 32 + n,
- (const uint8_t *)src - 32 + n);
- return ret;
- }
- if (n > 0) {
- rte_mov32((uint8_t *)dst - 32 + n,
- (const uint8_t *)src - 32 + n);
- }
- return ret;
- }
-
- /**
- * Make store aligned when copy size exceeds 256 bytes
- */
- dstofss = (uintptr_t)dst & 0x1F;
- if (dstofss > 0) {
- dstofss = 32 - dstofss;
- n -= dstofss;
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + dstofss;
- dst = (uint8_t *)dst + dstofss;
- }
-
- /**
- * Copy 128-byte blocks
- */
- rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n);
- bits = n;
- n = n & 127;
- bits -= n;
- src = (const uint8_t *)src + bits;
- dst = (uint8_t *)dst + bits;
-
- /**
- * Copy whatever left
- */
- goto COPY_BLOCK_128_BACK31;
-}
-
-#else /* RTE_MACHINE_CPUFLAG */
-
-#define ALIGNMENT_MASK 0x0F
-
-/**
- * SSE & AVX implementation below
- */
-
-/**
- * Copy 16 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov16(uint8_t *dst, const uint8_t *src)
-{
- __m128i xmm0;
-
- xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
- _mm_storeu_si128((__m128i *)dst, xmm0);
-}
-
-/**
- * Copy 32 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov32(uint8_t *dst, const uint8_t *src)
-{
- rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
- rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
-}
-
-/**
- * Copy 64 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov64(uint8_t *dst, const uint8_t *src)
-{
- rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
- rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
- rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
- rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
-}
-
-/**
- * Copy 128 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov128(uint8_t *dst, const uint8_t *src)
-{
- rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
- rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
- rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
- rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
- rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
- rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
- rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
- rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
-}
-
-/**
- * Copy 256 bytes from one location to another,
- * locations should not overlap.
- */
-static inline void
-rte_mov256(uint8_t *dst, const uint8_t *src)
-{
- rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
- rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
- rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
- rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
- rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
- rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
- rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
- rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
- rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
- rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
- rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
- rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
- rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
- rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
- rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
- rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
-}
-
-/**
- * Macro for copying unaligned block from one location to another with constant load offset,
- * 47 bytes leftover maximum,
- * locations should not overlap.
- * Requirements:
- * - Store is aligned
- * - Load offset is <offset>, which must be immediate value within [1, 15]
- * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
- * - <dst>, <src>, <len> must be variables
- * - __m128i <xmm0> ~ <xmm8> must be pre-defined
- */
-#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) \
-__extension__ ({ \
- int tmp; \
- while (len >= 128 + 16 - offset) { \
- xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16)); \
- len -= 128; \
- xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16)); \
- xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16)); \
- xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 3 * 16)); \
- xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 4 * 16)); \
- xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 5 * 16)); \
- xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 6 * 16)); \
- xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7 * 16)); \
- xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8 * 16)); \
- src = (const uint8_t *)src + 128; \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_alignr_epi8(xmm3, xmm2, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_alignr_epi8(xmm4, xmm3, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_alignr_epi8(xmm5, xmm4, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_alignr_epi8(xmm6, xmm5, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_alignr_epi8(xmm7, xmm6, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_alignr_epi8(xmm8, xmm7, offset)); \
- dst = (uint8_t *)dst + 128; \
- } \
- tmp = len; \
- len = ((len - 16 + offset) & 127) + 16 - offset; \
- tmp -= len; \
- src = (const uint8_t *)src + tmp; \
- dst = (uint8_t *)dst + tmp; \
- if (len >= 32 + 16 - offset) { \
- while (len >= 32 + 16 - offset) { \
- xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16)); \
- len -= 32; \
- xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16)); \
- xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16)); \
- src = (const uint8_t *)src + 32; \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset)); \
- _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset)); \
- dst = (uint8_t *)dst + 32; \
- } \
- tmp = len; \
- len = ((len - 16 + offset) & 31) + 16 - offset; \
- tmp -= len; \
- src = (const uint8_t *)src + tmp; \
- dst = (uint8_t *)dst + tmp; \
- } \
-})
-
-/**
- * Macro for copying unaligned block from one location to another,
- * 47 bytes leftover maximum,
- * locations should not overlap.
- * Use switch here because the aligning instruction requires immediate value for shift count.
- * Requirements:
- * - Store is aligned
- * - Load offset is <offset>, which must be within [1, 15]
- * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
- * - <dst>, <src>, <len> must be variables
- * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM must be pre-defined
- */
-#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) \
-__extension__ ({ \
- switch (offset) { \
- case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; \
- case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; \
- case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; \
- case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; \
- case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; \
- case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; \
- case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; \
- case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; \
- case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; \
- case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; \
- case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; \
- case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; \
- case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; \
- case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; \
- case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; \
- default:; \
- } \
-})
-
-static inline void *
-rte_memcpy_generic(void *dst, const void *src, size_t n)
-{
- __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8;
- uintptr_t dstu = (uintptr_t)dst;
- uintptr_t srcu = (uintptr_t)src;
- void *ret = dst;
- size_t dstofss;
- size_t srcofs;
-
- /**
- * Copy less than 16 bytes
- */
- if (n < 16) {
- if (n & 0x01) {
- *(uint8_t *)dstu = *(const uint8_t *)srcu;
- srcu = (uintptr_t)((const uint8_t *)srcu + 1);
- dstu = (uintptr_t)((uint8_t *)dstu + 1);
- }
- if (n & 0x02) {
- *(uint16_t *)dstu = *(const uint16_t *)srcu;
- srcu = (uintptr_t)((const uint16_t *)srcu + 1);
- dstu = (uintptr_t)((uint16_t *)dstu + 1);
- }
- if (n & 0x04) {
- *(uint32_t *)dstu = *(const uint32_t *)srcu;
- srcu = (uintptr_t)((const uint32_t *)srcu + 1);
- dstu = (uintptr_t)((uint32_t *)dstu + 1);
- }
- if (n & 0x08) {
- *(uint64_t *)dstu = *(const uint64_t *)srcu;
- }
- return ret;
- }
-
- /**
- * Fast way when copy size doesn't exceed 512 bytes
- */
- if (n <= 32) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 48) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 64) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
- rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n <= 128) {
- goto COPY_BLOCK_128_BACK15;
- }
- if (n <= 512) {
- if (n >= 256) {
- n -= 256;
- rte_mov128((uint8_t *)dst, (const uint8_t *)src);
- rte_mov128((uint8_t *)dst + 128, (const uint8_t *)src + 128);
- src = (const uint8_t *)src + 256;
- dst = (uint8_t *)dst + 256;
- }
-COPY_BLOCK_255_BACK15:
- if (n >= 128) {
- n -= 128;
- rte_mov128((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 128;
- dst = (uint8_t *)dst + 128;
- }
-COPY_BLOCK_128_BACK15:
- if (n >= 64) {
- n -= 64;
- rte_mov64((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 64;
- dst = (uint8_t *)dst + 64;
- }
-COPY_BLOCK_64_BACK15:
- if (n >= 32) {
- n -= 32;
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + 32;
- dst = (uint8_t *)dst + 32;
- }
- if (n > 16) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
- return ret;
- }
- if (n > 0) {
- rte_mov16((uint8_t *)dst - 16 + n, (const uint8_t *)src - 16 + n);
- }
- return ret;
- }
-
- /**
- * Make store aligned when copy size exceeds 512 bytes,
- * and make sure the first 15 bytes are copied, because
- * unaligned copy functions require up to 15 bytes
- * backwards access.
- */
- dstofss = (uintptr_t)dst & 0x0F;
- if (dstofss > 0) {
- dstofss = 16 - dstofss + 16;
- n -= dstofss;
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- src = (const uint8_t *)src + dstofss;
- dst = (uint8_t *)dst + dstofss;
- }
- srcofs = ((uintptr_t)src & 0x0F);
-
- /**
- * For aligned copy
- */
- if (srcofs == 0) {
- /**
- * Copy 256-byte blocks
- */
- for (; n >= 256; n -= 256) {
- rte_mov256((uint8_t *)dst, (const uint8_t *)src);
- dst = (uint8_t *)dst + 256;
- src = (const uint8_t *)src + 256;
- }
-
- /**
- * Copy whatever left
- */
- goto COPY_BLOCK_255_BACK15;
- }
-
- /**
- * For copy with unaligned load
- */
- MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
-
- /**
- * Copy whatever left
- */
- goto COPY_BLOCK_64_BACK15;
-}
-
-#endif /* RTE_MACHINE_CPUFLAG */
-
-static inline void *
-rte_memcpy_aligned(void *dst, const void *src, size_t n)
-{
- void *ret = dst;
-
- /* Copy size <= 16 bytes */
- if (n < 16) {
- if (n & 0x01) {
- *(uint8_t *)dst = *(const uint8_t *)src;
- src = (const uint8_t *)src + 1;
- dst = (uint8_t *)dst + 1;
- }
- if (n & 0x02) {
- *(uint16_t *)dst = *(const uint16_t *)src;
- src = (const uint16_t *)src + 1;
- dst = (uint16_t *)dst + 1;
- }
- if (n & 0x04) {
- *(uint32_t *)dst = *(const uint32_t *)src;
- src = (const uint32_t *)src + 1;
- dst = (uint32_t *)dst + 1;
- }
- if (n & 0x08)
- *(uint64_t *)dst = *(const uint64_t *)src;
-
- return ret;
- }
-
- /* Copy 16 <= size <= 32 bytes */
- if (n <= 32) {
- rte_mov16((uint8_t *)dst, (const uint8_t *)src);
- rte_mov16((uint8_t *)dst - 16 + n,
- (const uint8_t *)src - 16 + n);
-
- return ret;
- }
-
- /* Copy 32 < size <= 64 bytes */
- if (n <= 64) {
- rte_mov32((uint8_t *)dst, (const uint8_t *)src);
- rte_mov32((uint8_t *)dst - 32 + n,
- (const uint8_t *)src - 32 + n);
-
- return ret;
- }
-
- /* Copy 64 bytes blocks */
- for (; n >= 64; n -= 64) {
- rte_mov64((uint8_t *)dst, (const uint8_t *)src);
- dst = (uint8_t *)dst + 64;
- src = (const uint8_t *)src + 64;
- }
-
- /* Copy whatever left */
- rte_mov64((uint8_t *)dst - 64 + n,
- (const uint8_t *)src - 64 + n);
-
- return ret;
-}
+extern void *
+rte_memcpy_sse(void *dst, const void *src, size_t n);
static inline void *
rte_memcpy(void *dst, const void *src, size_t n)
{
- if (!(((uintptr_t)dst | (uintptr_t)src) & ALIGNMENT_MASK))
- return rte_memcpy_aligned(dst, src, n);
+ if (n <= RTE_X86_MEMCPY_THRESH)
+ return rte_memcpy_internal(dst, src, n);
else
- return rte_memcpy_generic(dst, src, n);
+ return (*rte_memcpy_ptr)(dst, src, n);
}
#ifdef __cplusplus
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c
new file mode 100644
index 0000000..3ad229c
--- /dev/null
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c
@@ -0,0 +1,44 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_memcpy.h>
+
+#ifndef RTE_MACHINE_CPUFLAG_AVX2
+#error RTE_MACHINE_CPUFLAG_AVX2 not defined
+#endif
+
+void *
+rte_memcpy_avx2(void *dst, const void *src, size_t n)
+{
+ return rte_memcpy_internal(dst, src, n);
+}
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c
new file mode 100644
index 0000000..be8d964
--- /dev/null
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c
@@ -0,0 +1,44 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_memcpy.h>
+
+#ifndef RTE_MACHINE_CPUFLAG_AVX512F
+#error RTE_MACHINE_CPUFLAG_AVX512F not defined
+#endif
+
+void *
+rte_memcpy_avx512f(void *dst, const void *src, size_t n)
+{
+ return rte_memcpy_internal(dst, src, n);
+}
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h b/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h
new file mode 100644
index 0000000..d17fb5b
--- /dev/null
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h
@@ -0,0 +1,909 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMCPY_INTERNAL_X86_64_H_
+#define _RTE_MEMCPY_INTERNAL_X86_64_H_
+
+/**
+ * @file
+ *
+ * Functions for SSE/AVX/AVX2/AVX512 implementation of memcpy().
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <rte_vect.h>
+#include <rte_common.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Copy bytes from one location to another. The locations must not overlap.
+ *
+ * @note This is implemented as a macro, so it's address should not be taken
+ * and care is needed as parameter expressions may be evaluated multiple times.
+ *
+ * @param dst
+ * Pointer to the destination of the data.
+ * @param src
+ * Pointer to the source data.
+ * @param n
+ * Number of bytes to copy.
+ * @return
+ * Pointer to the destination data.
+ */
+
+#ifdef RTE_MACHINE_CPUFLAG_AVX512F
+
+#define ALIGNMENT_MASK 0x3F
+
+/**
+ * AVX512 implementation below
+ */
+
+/**
+ * Copy 16 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov16(uint8_t *dst, const uint8_t *src)
+{
+ __m128i xmm0;
+
+ xmm0 = _mm_loadu_si128((const __m128i *)src);
+ _mm_storeu_si128((__m128i *)dst, xmm0);
+}
+
+/**
+ * Copy 32 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov32(uint8_t *dst, const uint8_t *src)
+{
+ __m256i ymm0;
+
+ ymm0 = _mm256_loadu_si256((const __m256i *)src);
+ _mm256_storeu_si256((__m256i *)dst, ymm0);
+}
+
+/**
+ * Copy 64 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov64(uint8_t *dst, const uint8_t *src)
+{
+ __m512i zmm0;
+
+ zmm0 = _mm512_loadu_si512((const void *)src);
+ _mm512_storeu_si512((void *)dst, zmm0);
+}
+
+/**
+ * Copy 128 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov64(dst + 0 * 64, src + 0 * 64);
+ rte_mov64(dst + 1 * 64, src + 1 * 64);
+}
+
+/**
+ * Copy 256 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov256(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov64(dst + 0 * 64, src + 0 * 64);
+ rte_mov64(dst + 1 * 64, src + 1 * 64);
+ rte_mov64(dst + 2 * 64, src + 2 * 64);
+ rte_mov64(dst + 3 * 64, src + 3 * 64);
+}
+
+/**
+ * Copy 128-byte blocks from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n)
+{
+ __m512i zmm0, zmm1;
+
+ while (n >= 128) {
+ zmm0 = _mm512_loadu_si512((const void *)(src + 0 * 64));
+ n -= 128;
+ zmm1 = _mm512_loadu_si512((const void *)(src + 1 * 64));
+ src = src + 128;
+ _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0);
+ _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1);
+ dst = dst + 128;
+ }
+}
+
+/**
+ * Copy 512-byte blocks from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov512blocks(uint8_t *dst, const uint8_t *src, size_t n)
+{
+ __m512i zmm0, zmm1, zmm2, zmm3, zmm4, zmm5, zmm6, zmm7;
+
+ while (n >= 512) {
+ zmm0 = _mm512_loadu_si512((const void *)(src + 0 * 64));
+ n -= 512;
+ zmm1 = _mm512_loadu_si512((const void *)(src + 1 * 64));
+ zmm2 = _mm512_loadu_si512((const void *)(src + 2 * 64));
+ zmm3 = _mm512_loadu_si512((const void *)(src + 3 * 64));
+ zmm4 = _mm512_loadu_si512((const void *)(src + 4 * 64));
+ zmm5 = _mm512_loadu_si512((const void *)(src + 5 * 64));
+ zmm6 = _mm512_loadu_si512((const void *)(src + 6 * 64));
+ zmm7 = _mm512_loadu_si512((const void *)(src + 7 * 64));
+ src = src + 512;
+ _mm512_storeu_si512((void *)(dst + 0 * 64), zmm0);
+ _mm512_storeu_si512((void *)(dst + 1 * 64), zmm1);
+ _mm512_storeu_si512((void *)(dst + 2 * 64), zmm2);
+ _mm512_storeu_si512((void *)(dst + 3 * 64), zmm3);
+ _mm512_storeu_si512((void *)(dst + 4 * 64), zmm4);
+ _mm512_storeu_si512((void *)(dst + 5 * 64), zmm5);
+ _mm512_storeu_si512((void *)(dst + 6 * 64), zmm6);
+ _mm512_storeu_si512((void *)(dst + 7 * 64), zmm7);
+ dst = dst + 512;
+ }
+}
+
+static inline void *
+rte_memcpy_generic(void *dst, const void *src, size_t n)
+{
+ uintptr_t dstu = (uintptr_t)dst;
+ uintptr_t srcu = (uintptr_t)src;
+ void *ret = dst;
+ size_t dstofss;
+ size_t bits;
+
+ /**
+ * Copy less than 16 bytes
+ */
+ if (n < 16) {
+ if (n & 0x01) {
+ *(uint8_t *)dstu = *(const uint8_t *)srcu;
+ srcu = (uintptr_t)((const uint8_t *)srcu + 1);
+ dstu = (uintptr_t)((uint8_t *)dstu + 1);
+ }
+ if (n & 0x02) {
+ *(uint16_t *)dstu = *(const uint16_t *)srcu;
+ srcu = (uintptr_t)((const uint16_t *)srcu + 1);
+ dstu = (uintptr_t)((uint16_t *)dstu + 1);
+ }
+ if (n & 0x04) {
+ *(uint32_t *)dstu = *(const uint32_t *)srcu;
+ srcu = (uintptr_t)((const uint32_t *)srcu + 1);
+ dstu = (uintptr_t)((uint32_t *)dstu + 1);
+ }
+ if (n & 0x08)
+ *(uint64_t *)dstu = *(const uint64_t *)srcu;
+ return ret;
+ }
+
+ /**
+ * Fast way when copy size doesn't exceed 512 bytes
+ */
+ if (n <= 32) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 64) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov32((uint8_t *)dst - 32 + n,
+ (const uint8_t *)src - 32 + n);
+ return ret;
+ }
+ if (n <= 512) {
+ if (n >= 256) {
+ n -= 256;
+ rte_mov256((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 256;
+ dst = (uint8_t *)dst + 256;
+ }
+ if (n >= 128) {
+ n -= 128;
+ rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 128;
+ dst = (uint8_t *)dst + 128;
+ }
+COPY_BLOCK_128_BACK63:
+ if (n > 64) {
+ rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov64((uint8_t *)dst - 64 + n,
+ (const uint8_t *)src - 64 + n);
+ return ret;
+ }
+ if (n > 0)
+ rte_mov64((uint8_t *)dst - 64 + n,
+ (const uint8_t *)src - 64 + n);
+ return ret;
+ }
+
+ /**
+ * Make store aligned when copy size exceeds 512 bytes
+ */
+ dstofss = ((uintptr_t)dst & 0x3F);
+ if (dstofss > 0) {
+ dstofss = 64 - dstofss;
+ n -= dstofss;
+ rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + dstofss;
+ dst = (uint8_t *)dst + dstofss;
+ }
+
+ /**
+ * Copy 512-byte blocks.
+ * Use copy block function for better instruction order control,
+ * which is important when load is unaligned.
+ */
+ rte_mov512blocks((uint8_t *)dst, (const uint8_t *)src, n);
+ bits = n;
+ n = n & 511;
+ bits -= n;
+ src = (const uint8_t *)src + bits;
+ dst = (uint8_t *)dst + bits;
+
+ /**
+ * Copy 128-byte blocks.
+ * Use copy block function for better instruction order control,
+ * which is important when load is unaligned.
+ */
+ if (n >= 128) {
+ rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n);
+ bits = n;
+ n = n & 127;
+ bits -= n;
+ src = (const uint8_t *)src + bits;
+ dst = (uint8_t *)dst + bits;
+ }
+
+ /**
+ * Copy whatever left
+ */
+ goto COPY_BLOCK_128_BACK63;
+}
+
+#elif defined RTE_MACHINE_CPUFLAG_AVX2
+
+#define ALIGNMENT_MASK 0x1F
+
+/**
+ * AVX2 implementation below
+ */
+
+/**
+ * Copy 16 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov16(uint8_t *dst, const uint8_t *src)
+{
+ __m128i xmm0;
+
+ xmm0 = _mm_loadu_si128((const __m128i *)src);
+ _mm_storeu_si128((__m128i *)dst, xmm0);
+}
+
+/**
+ * Copy 32 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov32(uint8_t *dst, const uint8_t *src)
+{
+ __m256i ymm0;
+
+ ymm0 = _mm256_loadu_si256((const __m256i *)src);
+ _mm256_storeu_si256((__m256i *)dst, ymm0);
+}
+
+/**
+ * Copy 64 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov64(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
+ rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
+}
+
+/**
+ * Copy 128 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov32((uint8_t *)dst + 0 * 32, (const uint8_t *)src + 0 * 32);
+ rte_mov32((uint8_t *)dst + 1 * 32, (const uint8_t *)src + 1 * 32);
+ rte_mov32((uint8_t *)dst + 2 * 32, (const uint8_t *)src + 2 * 32);
+ rte_mov32((uint8_t *)dst + 3 * 32, (const uint8_t *)src + 3 * 32);
+}
+
+/**
+ * Copy 128-byte blocks from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128blocks(uint8_t *dst, const uint8_t *src, size_t n)
+{
+ __m256i ymm0, ymm1, ymm2, ymm3;
+
+ while (n >= 128) {
+ ymm0 = _mm256_loadu_si256((const __m256i *)
+ ((const uint8_t *)src + 0 * 32));
+ n -= 128;
+ ymm1 = _mm256_loadu_si256((const __m256i *)
+ ((const uint8_t *)src + 1 * 32));
+ ymm2 = _mm256_loadu_si256((const __m256i *)
+ ((const uint8_t *)src + 2 * 32));
+ ymm3 = _mm256_loadu_si256((const __m256i *)
+ ((const uint8_t *)src + 3 * 32));
+ src = (const uint8_t *)src + 128;
+ _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 0 * 32), ymm0);
+ _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 1 * 32), ymm1);
+ _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 2 * 32), ymm2);
+ _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 3 * 32), ymm3);
+ dst = (uint8_t *)dst + 128;
+ }
+}
+
+static inline void *
+rte_memcpy_generic(void *dst, const void *src, size_t n)
+{
+ uintptr_t dstu = (uintptr_t)dst;
+ uintptr_t srcu = (uintptr_t)src;
+ void *ret = dst;
+ size_t dstofss;
+ size_t bits;
+
+ /**
+ * Copy less than 16 bytes
+ */
+ if (n < 16) {
+ if (n & 0x01) {
+ *(uint8_t *)dstu = *(const uint8_t *)srcu;
+ srcu = (uintptr_t)((const uint8_t *)srcu + 1);
+ dstu = (uintptr_t)((uint8_t *)dstu + 1);
+ }
+ if (n & 0x02) {
+ *(uint16_t *)dstu = *(const uint16_t *)srcu;
+ srcu = (uintptr_t)((const uint16_t *)srcu + 1);
+ dstu = (uintptr_t)((uint16_t *)dstu + 1);
+ }
+ if (n & 0x04) {
+ *(uint32_t *)dstu = *(const uint32_t *)srcu;
+ srcu = (uintptr_t)((const uint32_t *)srcu + 1);
+ dstu = (uintptr_t)((uint32_t *)dstu + 1);
+ }
+ if (n & 0x08)
+ *(uint64_t *)dstu = *(const uint64_t *)srcu;
+ return ret;
+ }
+
+ /**
+ * Fast way when copy size doesn't exceed 256 bytes
+ */
+ if (n <= 32) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 48) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst + 16, (const uint8_t *)src + 16);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 64) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov32((uint8_t *)dst - 32 + n,
+ (const uint8_t *)src - 32 + n);
+ return ret;
+ }
+ if (n <= 256) {
+ if (n >= 128) {
+ n -= 128;
+ rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 128;
+ dst = (uint8_t *)dst + 128;
+ }
+COPY_BLOCK_128_BACK31:
+ if (n >= 64) {
+ n -= 64;
+ rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 64;
+ dst = (uint8_t *)dst + 64;
+ }
+ if (n > 32) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov32((uint8_t *)dst - 32 + n,
+ (const uint8_t *)src - 32 + n);
+ return ret;
+ }
+ if (n > 0) {
+ rte_mov32((uint8_t *)dst - 32 + n,
+ (const uint8_t *)src - 32 + n);
+ }
+ return ret;
+ }
+
+ /**
+ * Make store aligned when copy size exceeds 256 bytes
+ */
+ dstofss = (uintptr_t)dst & 0x1F;
+ if (dstofss > 0) {
+ dstofss = 32 - dstofss;
+ n -= dstofss;
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + dstofss;
+ dst = (uint8_t *)dst + dstofss;
+ }
+
+ /**
+ * Copy 128-byte blocks
+ */
+ rte_mov128blocks((uint8_t *)dst, (const uint8_t *)src, n);
+ bits = n;
+ n = n & 127;
+ bits -= n;
+ src = (const uint8_t *)src + bits;
+ dst = (uint8_t *)dst + bits;
+
+ /**
+ * Copy whatever left
+ */
+ goto COPY_BLOCK_128_BACK31;
+}
+
+#else /* RTE_MACHINE_CPUFLAG */
+
+#define ALIGNMENT_MASK 0x0F
+
+/**
+ * SSE & AVX implementation below
+ */
+
+/**
+ * Copy 16 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov16(uint8_t *dst, const uint8_t *src)
+{
+ __m128i xmm0;
+
+ xmm0 = _mm_loadu_si128((const __m128i *)(const __m128i *)src);
+ _mm_storeu_si128((__m128i *)dst, xmm0);
+}
+
+/**
+ * Copy 32 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov32(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+ rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+}
+
+/**
+ * Copy 64 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov64(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+ rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+ rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+ rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+}
+
+/**
+ * Copy 128 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov128(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+ rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+ rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+ rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+ rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
+ rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
+ rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
+ rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
+}
+
+/**
+ * Copy 256 bytes from one location to another,
+ * locations should not overlap.
+ */
+static inline void
+rte_mov256(uint8_t *dst, const uint8_t *src)
+{
+ rte_mov16((uint8_t *)dst + 0 * 16, (const uint8_t *)src + 0 * 16);
+ rte_mov16((uint8_t *)dst + 1 * 16, (const uint8_t *)src + 1 * 16);
+ rte_mov16((uint8_t *)dst + 2 * 16, (const uint8_t *)src + 2 * 16);
+ rte_mov16((uint8_t *)dst + 3 * 16, (const uint8_t *)src + 3 * 16);
+ rte_mov16((uint8_t *)dst + 4 * 16, (const uint8_t *)src + 4 * 16);
+ rte_mov16((uint8_t *)dst + 5 * 16, (const uint8_t *)src + 5 * 16);
+ rte_mov16((uint8_t *)dst + 6 * 16, (const uint8_t *)src + 6 * 16);
+ rte_mov16((uint8_t *)dst + 7 * 16, (const uint8_t *)src + 7 * 16);
+ rte_mov16((uint8_t *)dst + 8 * 16, (const uint8_t *)src + 8 * 16);
+ rte_mov16((uint8_t *)dst + 9 * 16, (const uint8_t *)src + 9 * 16);
+ rte_mov16((uint8_t *)dst + 10 * 16, (const uint8_t *)src + 10 * 16);
+ rte_mov16((uint8_t *)dst + 11 * 16, (const uint8_t *)src + 11 * 16);
+ rte_mov16((uint8_t *)dst + 12 * 16, (const uint8_t *)src + 12 * 16);
+ rte_mov16((uint8_t *)dst + 13 * 16, (const uint8_t *)src + 13 * 16);
+ rte_mov16((uint8_t *)dst + 14 * 16, (const uint8_t *)src + 14 * 16);
+ rte_mov16((uint8_t *)dst + 15 * 16, (const uint8_t *)src + 15 * 16);
+}
+
+/**
+ * Macro for copying unaligned block from one location to another with constant load offset,
+ * 47 bytes leftover maximum,
+ * locations should not overlap.
+ * Requirements:
+ * - Store is aligned
+ * - Load offset is <offset>, which must be immediate value within [1, 15]
+ * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
+ * - <dst>, <src>, <len> must be variables
+ * - __m128i <xmm0> ~ <xmm8> must be pre-defined
+ */
+#define MOVEUNALIGNED_LEFT47_IMM(dst, src, len, offset) \
+__extension__ ({ \
+ int tmp; \
+ while (len >= 128 + 16 - offset) { \
+ xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16)); \
+ len -= 128; \
+ xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16)); \
+ xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16)); \
+ xmm3 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 3 * 16)); \
+ xmm4 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 4 * 16)); \
+ xmm5 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 5 * 16)); \
+ xmm6 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 6 * 16)); \
+ xmm7 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 7 * 16)); \
+ xmm8 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 8 * 16)); \
+ src = (const uint8_t *)src + 128; \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 2 * 16), _mm_alignr_epi8(xmm3, xmm2, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 3 * 16), _mm_alignr_epi8(xmm4, xmm3, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 4 * 16), _mm_alignr_epi8(xmm5, xmm4, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 5 * 16), _mm_alignr_epi8(xmm6, xmm5, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 6 * 16), _mm_alignr_epi8(xmm7, xmm6, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 7 * 16), _mm_alignr_epi8(xmm8, xmm7, offset)); \
+ dst = (uint8_t *)dst + 128; \
+ } \
+ tmp = len; \
+ len = ((len - 16 + offset) & 127) + 16 - offset; \
+ tmp -= len; \
+ src = (const uint8_t *)src + tmp; \
+ dst = (uint8_t *)dst + tmp; \
+ if (len >= 32 + 16 - offset) { \
+ while (len >= 32 + 16 - offset) { \
+ xmm0 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 0 * 16)); \
+ len -= 32; \
+ xmm1 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 1 * 16)); \
+ xmm2 = _mm_loadu_si128((const __m128i *)((const uint8_t *)src - offset + 2 * 16)); \
+ src = (const uint8_t *)src + 32; \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 0 * 16), _mm_alignr_epi8(xmm1, xmm0, offset)); \
+ _mm_storeu_si128((__m128i *)((uint8_t *)dst + 1 * 16), _mm_alignr_epi8(xmm2, xmm1, offset)); \
+ dst = (uint8_t *)dst + 32; \
+ } \
+ tmp = len; \
+ len = ((len - 16 + offset) & 31) + 16 - offset; \
+ tmp -= len; \
+ src = (const uint8_t *)src + tmp; \
+ dst = (uint8_t *)dst + tmp; \
+ } \
+})
+
+/**
+ * Macro for copying unaligned block from one location to another,
+ * 47 bytes leftover maximum,
+ * locations should not overlap.
+ * Use switch here because the aligning instruction requires immediate value for shift count.
+ * Requirements:
+ * - Store is aligned
+ * - Load offset is <offset>, which must be within [1, 15]
+ * - For <src>, make sure <offset> bit backwards & <16 - offset> bit forwards are available for loading
+ * - <dst>, <src>, <len> must be variables
+ * - __m128i <xmm0> ~ <xmm8> used in MOVEUNALIGNED_LEFT47_IMM must be pre-defined
+ */
+#define MOVEUNALIGNED_LEFT47(dst, src, len, offset) \
+__extension__ ({ \
+ switch (offset) { \
+ case 0x01: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x01); break; \
+ case 0x02: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x02); break; \
+ case 0x03: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x03); break; \
+ case 0x04: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x04); break; \
+ case 0x05: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x05); break; \
+ case 0x06: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x06); break; \
+ case 0x07: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x07); break; \
+ case 0x08: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x08); break; \
+ case 0x09: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x09); break; \
+ case 0x0A: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0A); break; \
+ case 0x0B: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0B); break; \
+ case 0x0C: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0C); break; \
+ case 0x0D: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0D); break; \
+ case 0x0E: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0E); break; \
+ case 0x0F: MOVEUNALIGNED_LEFT47_IMM(dst, src, n, 0x0F); break; \
+ default:; \
+ } \
+})
+
+static inline void *
+rte_memcpy_generic(void *dst, const void *src, size_t n)
+{
+ __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8;
+ uintptr_t dstu = (uintptr_t)dst;
+ uintptr_t srcu = (uintptr_t)src;
+ void *ret = dst;
+ size_t dstofss;
+ size_t srcofs;
+
+ /**
+ * Copy less than 16 bytes
+ */
+ if (n < 16) {
+ if (n & 0x01) {
+ *(uint8_t *)dstu = *(const uint8_t *)srcu;
+ srcu = (uintptr_t)((const uint8_t *)srcu + 1);
+ dstu = (uintptr_t)((uint8_t *)dstu + 1);
+ }
+ if (n & 0x02) {
+ *(uint16_t *)dstu = *(const uint16_t *)srcu;
+ srcu = (uintptr_t)((const uint16_t *)srcu + 1);
+ dstu = (uintptr_t)((uint16_t *)dstu + 1);
+ }
+ if (n & 0x04) {
+ *(uint32_t *)dstu = *(const uint32_t *)srcu;
+ srcu = (uintptr_t)((const uint32_t *)srcu + 1);
+ dstu = (uintptr_t)((uint32_t *)dstu + 1);
+ }
+ if (n & 0x08)
+ *(uint64_t *)dstu = *(const uint64_t *)srcu;
+ return ret;
+ }
+
+ /**
+ * Fast way when copy size doesn't exceed 512 bytes
+ */
+ if (n <= 32) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 48) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 64) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst + 32, (const uint8_t *)src + 32);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n <= 128)
+ goto COPY_BLOCK_128_BACK15;
+ if (n <= 512) {
+ if (n >= 256) {
+ n -= 256;
+ rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov128((uint8_t *)dst + 128,
+ (const uint8_t *)src + 128);
+ src = (const uint8_t *)src + 256;
+ dst = (uint8_t *)dst + 256;
+ }
+COPY_BLOCK_255_BACK15:
+ if (n >= 128) {
+ n -= 128;
+ rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 128;
+ dst = (uint8_t *)dst + 128;
+ }
+COPY_BLOCK_128_BACK15:
+ if (n >= 64) {
+ n -= 64;
+ rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 64;
+ dst = (uint8_t *)dst + 64;
+ }
+COPY_BLOCK_64_BACK15:
+ if (n >= 32) {
+ n -= 32;
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + 32;
+ dst = (uint8_t *)dst + 32;
+ }
+ if (n > 16) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ return ret;
+ }
+ if (n > 0) {
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+ }
+ return ret;
+ }
+
+ /**
+ * Make store aligned when copy size exceeds 512 bytes,
+ * and make sure the first 15 bytes are copied, because
+ * unaligned copy functions require up to 15 bytes
+ * backwards access.
+ */
+ dstofss = (uintptr_t)dst & 0x0F;
+ if (dstofss > 0) {
+ dstofss = 16 - dstofss + 16;
+ n -= dstofss;
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ src = (const uint8_t *)src + dstofss;
+ dst = (uint8_t *)dst + dstofss;
+ }
+ srcofs = ((uintptr_t)src & 0x0F);
+
+ /**
+ * For aligned copy
+ */
+ if (srcofs == 0) {
+ /**
+ * Copy 256-byte blocks
+ */
+ for (; n >= 256; n -= 256) {
+ rte_mov256((uint8_t *)dst, (const uint8_t *)src);
+ dst = (uint8_t *)dst + 256;
+ src = (const uint8_t *)src + 256;
+ }
+
+ /**
+ * Copy whatever left
+ */
+ goto COPY_BLOCK_255_BACK15;
+ }
+
+ /**
+ * For copy with unaligned load
+ */
+ MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
+
+ /**
+ * Copy whatever left
+ */
+ goto COPY_BLOCK_64_BACK15;
+}
+
+#endif /* RTE_MACHINE_CPUFLAG */
+
+static inline void *
+rte_memcpy_aligned(void *dst, const void *src, size_t n)
+{
+ void *ret = dst;
+
+ /* Copy size <= 16 bytes */
+ if (n < 16) {
+ if (n & 0x01) {
+ *(uint8_t *)dst = *(const uint8_t *)src;
+ src = (const uint8_t *)src + 1;
+ dst = (uint8_t *)dst + 1;
+ }
+ if (n & 0x02) {
+ *(uint16_t *)dst = *(const uint16_t *)src;
+ src = (const uint16_t *)src + 1;
+ dst = (uint16_t *)dst + 1;
+ }
+ if (n & 0x04) {
+ *(uint32_t *)dst = *(const uint32_t *)src;
+ src = (const uint32_t *)src + 1;
+ dst = (uint32_t *)dst + 1;
+ }
+ if (n & 0x08)
+ *(uint64_t *)dst = *(const uint64_t *)src;
+
+ return ret;
+ }
+
+ /* Copy 16 <= size <= 32 bytes */
+ if (n <= 32) {
+ rte_mov16((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov16((uint8_t *)dst - 16 + n,
+ (const uint8_t *)src - 16 + n);
+
+ return ret;
+ }
+
+ /* Copy 32 < size <= 64 bytes */
+ if (n <= 64) {
+ rte_mov32((uint8_t *)dst, (const uint8_t *)src);
+ rte_mov32((uint8_t *)dst - 32 + n,
+ (const uint8_t *)src - 32 + n);
+
+ return ret;
+ }
+
+ /* Copy 64 bytes blocks */
+ for (; n >= 64; n -= 64) {
+ rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+ dst = (uint8_t *)dst + 64;
+ src = (const uint8_t *)src + 64;
+ }
+
+ /* Copy whatever left */
+ rte_mov64((uint8_t *)dst - 64 + n,
+ (const uint8_t *)src - 64 + n);
+
+ return ret;
+}
+
+static inline void *
+rte_memcpy_internal(void *dst, const void *src, size_t n)
+{
+ if (!(((uintptr_t)dst | (uintptr_t)src) & ALIGNMENT_MASK))
+ return rte_memcpy_aligned(dst, src, n);
+ else
+ return rte_memcpy_generic(dst, src, n);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMCPY_INTERNAL_X86_64_H_ */
diff --git a/lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c b/lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c
new file mode 100644
index 0000000..55d6b41
--- /dev/null
+++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c
@@ -0,0 +1,40 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_memcpy.h>
+
+void *
+rte_memcpy_sse(void *dst, const void *src, size_t n)
+{
+ return rte_memcpy_internal(dst, src, n);
+}
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 90bca4d..c8bdac0 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -40,6 +40,7 @@ VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR)
LIBABIVER := 5
VPATH += $(RTE_SDK)/lib/librte_eal/common
+VPATH += $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)
CFLAGS += -I$(SRCDIR)/include
CFLAGS += -I$(RTE_SDK)/lib/librte_eal/common
@@ -105,6 +106,24 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_service.c
SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_cpuflags.c
SRCS-$(CONFIG_RTE_ARCH_X86) += rte_spinlock.c
+# for run-time dispatch of memcpy
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy.c
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_sse.c
+
+# if the compiler supports AVX512, add avx512 file
+ifneq ($(findstring CC_SUPPORT_AVX512F,$(MACHINE_CFLAGS)),)
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_avx512f.c
+CFLAGS_rte_memcpy_avx512f.o += -mavx512f
+CFLAGS_rte_memcpy_avx512f.o += -DRTE_MACHINE_CPUFLAG_AVX512F
+endif
+
+# if the compiler supports AVX2, add avx2 file
+ifneq ($(findstring CC_SUPPORT_AVX2,$(MACHINE_CFLAGS)),)
+SRCS-$(CONFIG_RTE_ARCH_X86) += rte_memcpy_avx2.c
+CFLAGS_rte_memcpy_avx2.o += -mavx2
+CFLAGS_rte_memcpy_avx2.o += -DRTE_MACHINE_CPUFLAG_AVX2
+endif
+
CFLAGS_eal_common_cpuflags.o := $(CPUFLAGS_LIST)
CFLAGS_eal.o := -D_GNU_SOURCE
diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
index 8c08b8d..15a2fe9 100644
--- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map
+++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
@@ -241,5 +241,6 @@ EXPERIMENTAL {
rte_service_runstate_set;
rte_service_set_stats_enable;
rte_service_start_with_defaults;
+ rte_memcpy_ptr;
} DPDK_17.08;
diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk
index a813c91..8a7a1e7 100644
--- a/mk/rte.cpuflags.mk
+++ b/mk/rte.cpuflags.mk
@@ -134,6 +134,20 @@ endif
MACHINE_CFLAGS += $(addprefix -DRTE_MACHINE_CPUFLAG_,$(CPUFLAGS))
+# Check if the compiler suppoerts AVX512
+CC_SUPPORT_AVX512F := $(shell $(CC) -mavx512f -dM -E - < /dev/null 2>&1 | grep -q AVX512 && echo 1)
+ifeq ($(CC_SUPPORT_AVX512F),1)
+ifeq ($(CONFIG_RTE_ENABLE_AVX512),y)
+MACHINE_CFLAGS += -DCC_SUPPORT_AVX512F
+endif
+endif
+
+# Check if the compiler supports AVX2
+CC_SUPPORT_AVX2 := $(shell $(CC) -mavx2 -dM -E - < /dev/null 2>&1 | grep -q AVX2 && echo 1)
+ifeq ($(CC_SUPPORT_AVX2),1)
+MACHINE_CFLAGS += -DCC_SUPPORT_AVX2
+endif
+
# To strip whitespace
comma:= ,
empty:=
--
2.7.4
^ permalink raw reply related
* [PATCH v6 0/3] run-time Linking support
From: Xiaoyun Li @ 2017-10-04 22:58 UTC (permalink / raw)
To: konstantin.ananyev, bruce.richardson
Cc: wenzhuo.lu, helin.zhang, dev, Xiaoyun Li
In-Reply-To: <1507042796-86318-1-git-send-email-xiaoyun.li@intel.com>
This patchset dynamically selects functions at run-time based on CPU flags
that current machine supports.This patchset modifies mempcy, memcpy perf
test and x86 EFD, using function pointers and bind them at constructor time.
Then in the cloud environment, users can compiler once for the minimum target
such as 'haswell'(not 'native') and run on different platforms (equal or above
haswell) and can get ISA optimization based on running CPU.
Xiaoyun Li (3):
eal/x86: run-time dispatch over memcpy
app/test: run-time dispatch over memcpy perf test
efd: run-time dispatch over x86 EFD functions
---
v2
* Use gcc function multi-versioning to avoid compilation issues.
* Add macros for AVX512 and AVX2. Only if users enable AVX512 and the compiler
supports it, the AVX512 codes would be compiled. Only if the compiler supports
AVX2, the AVX2 codes would be compiled.
v3
* Reduce function calls via only keep rte_memcpy_xxx.
* Add conditions that when copy size is small, use inline code path.
Otherwise, use dynamic code path.
* To support attribute target, clang version must be greater than 3.7.
Otherwise, would choose SSE/AVX code path, the same as before.
* Move two mocro functions to the top of the code since they would be used in
inline SSE/AVX and dynamic SSE/AVX codes.
v4
* Modify rte_memcpy.h to several .c files and modify makefiles to compile
AVX2 and AVX512 files.
v5
* Delete redundant repeated codes of rte_memcpy_xxx.
* Modify makefiles to enable reuse of existing rte_memcpy.
* Delete redundant codes of rte_efd_x86.h in v4. Move it into .c file and enable
compilation -mavx2 for it in makefile since it is already chosen at run-time.
v6
* Fix shared target build failure.
* Safely remove redundant efd x86 avx2 codes since the file is compiled
with -mavx2.
lib/librte_eal/bsdapp/eal/Makefile | 19 +
lib/librte_eal/bsdapp/eal/rte_eal_version.map | 1 +
.../common/include/arch/x86/rte_memcpy.c | 59 ++
.../common/include/arch/x86/rte_memcpy.h | 861 +------------------
.../common/include/arch/x86/rte_memcpy_avx2.c | 44 +
.../common/include/arch/x86/rte_memcpy_avx512f.c | 44 +
.../common/include/arch/x86/rte_memcpy_internal.h | 909 +++++++++++++++++++++
.../common/include/arch/x86/rte_memcpy_sse.c | 40 +
lib/librte_eal/linuxapp/eal/Makefile | 19 +
lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 +
lib/librte_efd/Makefile | 6 +
lib/librte_efd/rte_efd_x86.c | 87 ++
lib/librte_efd/rte_efd_x86.h | 48 +-
mk/rte.cpuflags.mk | 14 +
test/test/test_memcpy_perf.c | 40 +-
15 files changed, 1287 insertions(+), 905 deletions(-)
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx2.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_avx512f.c
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_internal.h
create mode 100644 lib/librte_eal/common/include/arch/x86/rte_memcpy_sse.c
create mode 100644 lib/librte_efd/rte_efd_x86.c
--
2.7.4
^ permalink raw reply
* Re: [PATCH 09/13] drm/i915/cnl: Invert dvfs default level.
From: Manasi Navare @ 2017-10-04 23:03 UTC (permalink / raw)
To: Rodrigo Vivi; +Cc: intel-gfx, Paulo Zanoni
In-Reply-To: <20171004224011.GH15590@intel.com>
On Wed, Oct 04, 2017 at 03:40:11PM -0700, Manasi Navare wrote:
> On Wed, Oct 04, 2017 at 12:36:42PM -0700, Rodrigo Vivi wrote:
> > On Wed, Oct 04, 2017 at 09:46:41AM +0000, Mika Kahola wrote:
> > > On Tue, 2017-10-03 at 00:06 -0700, Rodrigo Vivi wrote:
> > > > According to spec "If voltage is set too low,
> > > > it will break functionality. If voltage is set too high,
> > > > it will waste power."
> > > >
> > > > So, let's prefer the waste of power instead of breaking
> > > > functionality.
> > > >
> > > > But also the logic of deciding the level on spec
> > > > tells "Else, use level 2."
> > > > So, default is actually "2", not "0".
> > > The spec also says
> > >
> > > "If CD clock = 168 MHz AND DDI clock ≤ 594 MHz, use level 0.
> > > Else If CD clock = 336 MHz AND DDI clock ≤ 594 MHz, use level 1.
> > > Else, use level 2."
> > >
> > > Should we add check for DDI clock rate as well here?
> >
> > By this time dpll are disabled and not associated to any
> > port yet. So portclock is 0. So if we invert the default
> > we do respect the same algorightm you pasted.
> > Also if you notice I'm just inverting this in a separeted
> > patch to make it clear and if needed to bisect we end up
> > on this single point of change. But on a later patch on
> > this series I change to use your function with this algoright
> > there, but giving portclock = 0.
> >
> > So by the end of this series the flow is:
> >
> > - enable cdclk
> > - request dvfs level enough for cdclk in question.
> > - enable pll
> > - adjust dvfs level only if needed taking the port clock into account.
> > - disable pll
> > - put back level to cdclock level if needed.
> >
> > >
> > > >
> > > > v2: Rebase moving it up to avoid some temporary code
> > > > duplication.
> > > >
> > > > Cc: Mika Kahola <mika.kahola@intel.com>
> > > > Cc: Paulo Zanoni <paulo.r.zanoni@intel.com>
> > > > Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > > > ---
> > > > drivers/gpu/drm/i915/intel_cdclk.c | 8 ++++----
> > > > 1 file changed, 4 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/drivers/gpu/drm/i915/intel_cdclk.c
> > > > b/drivers/gpu/drm/i915/intel_cdclk.c
> > > > index af8411c2a6b9..7e9c4444c844 100644
> > > > --- a/drivers/gpu/drm/i915/intel_cdclk.c
> > > > +++ b/drivers/gpu/drm/i915/intel_cdclk.c
> > > > @@ -1562,15 +1562,15 @@ static void cnl_set_cdclk(struct
> > > > drm_i915_private *dev_priv,
> > > > }
> > > >
> > > > switch (cdclk) {
> > > > - case 528000:
> > > > - pcu_ack = 2;
> > > > + case 168000:
> > > > + pcu_ack = 0;
> > > > break;
> > > > case 336000:
> > > > pcu_ack = 1;
> > > > break;
> > > > - case 168000:
> > > > + case 528000:
> > > > default:
> > > > - pcu_ack = 0;
> > > > + pcu_ack = 2;
> > > > break;
>
> The spec says "Else If CD clock ≤ 556.8 AND DDI clock > 594 MHz, use level 1"
> So in case of cdclock = 528000, it is still < 556.8Mhz and the level should be 1
>
> Manasi
>
Never mind, I was looking at a different section which had this logic.
Manasi
> > > > }
> > > >
> > > --
> > > Mika Kahola - Intel OTC
> > >
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/intel-gfx
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* local fetcher and devtool don't work well together
From: Aaron_Wright @ 2017-10-04 22:54 UTC (permalink / raw)
To: yocto
[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]
I have a recipe with:
SRC_URI = "file://a/b/c/d;subdir=src"
S = "${WORKDIR}/src/a/b/c/d"
First off, ${S} must be set to the full path because the basepath uri
parameter[1] doesn't work. So this doesn't work:
SRC_URI = "file://a/b/c/d;subdir=src;basepath=a/b/c/d"
S = "${WORKDIR}/src"
I wish it would, but that's ok. I can deal with the long ${S} path as it
compiles and works fine with bitbake. However, it doesn't work with
devtool:
$ devtool modify -n d ~/mysrc/a/b/c/d
NOTE: Creating workspace layer in <... snip
...>/yocto/poky/build/workspace
NOTE: Enabling workspace layer in bblayers.conf
Parsing recipes..done.
NOTE: Recipe d now set up to build from ~/mysrc/a/b/c/d/b/c/d
Where is the extra 'b/c/d' coming from in where it will be built from?
This directory is reflected in the bbappends created the workspace:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
FILESPATH_prepend := "~/mysrc/a/b/c/d/b/c/d/oe-local-files:"
inherit externalsrc
# NOTE: We use pn- overrides here to avoid affecting multiple variants in
the case where the recipe uses BBCLASSEXTEND
EXTERNALSRC_pn-backtrace = "~/mysrc/a/b/c/d/b/c/d"
If I fix the path in the workspace bbappends, it does compile and work
file.
Is it possible to have a ${S} of "${WORKDIR}/src" in this situation?
How can I get devtool to behave and not duplicate the path?
PS - This also doesn't work with a normal 'devtool modify d'.
[1] -
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.yoctoproject.org_docs_latest_bitbake-2Duser-2Dmanual_bitbake-2Duser-2Dmanual.html-23bb-2Dthe-2Dunpack&d=DwIBAg&c=zVFQZQ67ypsA9mYKSCqWmQHiVkCCaN-Gb60_N6TVnLk&r=IZ3mUDA8X9CZR4hNctKngDDYs_BhK3qVNLG-nWGbW2E&m=wH_YZj_cv9gBOhJqSKTr4fiuM5UCTphWx9zfY2ceudg&s=2z9LBCu2xDjBhB51eToRNH7-wsLrFY4NPKy4_cwuRI0&e=
[-- Attachment #2: Type: text/html, Size: 3127 bytes --]
^ permalink raw reply
* Re: [PATCH v2 4/5] VSOCK: add sock_diag interface
From: David Miller @ 2017-10-04 22:58 UTC (permalink / raw)
To: stefanha; +Cc: netdev, jhansen, decui
In-Reply-To: <20171004163716.3964-5-stefanha@redhat.com>
From: Stefan Hajnoczi <stefanha@redhat.com>
Date: Wed, 4 Oct 2017 12:37:15 -0400
> This patch adds the sock_diag interface for querying sockets from
> userspace. Tools like ss(8) and netstat(8) can use this interface to
> list open sockets.
>
> The userspace ABI is defined in <linux/vm_sockets_diag.h> and includes
> netlink request and response structs. The request can query sockets
> based on their sk_state (e.g. listening sockets only) and the response
> contains socket information fields including the local/remote addresses,
> inode number, etc.
>
> This patch does not dump VMCI pending sockets because I have only tested
> the virtio transport, which does not use pending sockets. Support can
> be added later by extending vsock_diag_dump() if needed by VMCI users.
>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Please post new feature patches against net-next.
> diff --git a/net/vmw_vsock/Makefile b/net/vmw_vsock/Makefile
> index 09fc2eb29dc8..e5dbf153aff0 100644
> --- a/net/vmw_vsock/Makefile
> +++ b/net/vmw_vsock/Makefile
> @@ -1,10 +1,13 @@
> obj-$(CONFIG_VSOCKETS) += vsock.o
> +obj-$(CONFIG_VSOCKETS_DIAG) += vsock_diag.o
> obj-$(CONFIG_VMWARE_VMCI_VSOCKETS) += vmw_vsock_vmci_transport.o
> obj-$(CONFIG_VIRTIO_VSOCKETS) += vmw_vsock_virtio_transport.o
> obj-$(CONFIG_VIRTIO_VSOCKETS_COMMON) += vmw_vsock_virtio_transport_common.o
>
This hunk fails to apply to the net-next tree, the context looks
different.
Thanks.
^ permalink raw reply
* [kernel-hardening] [PATCH RFC v4 3/3] doc: self-protection: Add information about STACKLEAK feature
From: Alexander Popov @ 2017-10-04 22:55 UTC (permalink / raw)
To: kernel-hardening, keescook, pageexec, spender, tycho,
Laura Abbott, Mark Rutland, Ard Biesheuvel, Andy Lutomirski, x86,
alex.popov
In-Reply-To: <1507157703-14972-1-git-send-email-alex.popov@linux.com>
Add information about STACKLEAK feature to "Stack depth overflow" and
"Memory poisoning" sections of self-protection.rst.
Signed-off-by: Alexander Popov <alex.popov@linux.com>
---
Documentation/security/self-protection.rst | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/Documentation/security/self-protection.rst b/Documentation/security/self-protection.rst
index 60c8bd8..9693a90 100644
--- a/Documentation/security/self-protection.rst
+++ b/Documentation/security/self-protection.rst
@@ -165,10 +165,15 @@ Stack depth overflow
A less well understood attack is using a bug that triggers the
kernel to consume stack memory with deep function calls or large stack
allocations. With this attack it is possible to write beyond the end of
-the kernel's preallocated stack space and into sensitive structures. Two
-important changes need to be made for better protections: moving the
-sensitive thread_info structure elsewhere, and adding a faulting memory
-hole at the bottom of the stack to catch these overflows.
+the kernel's preallocated stack space and into sensitive structures.
+The combination of the following measures gives better protection:
+
+* moving the sensitive thread_info structure off the stack
+ (``CONFIG_THREAD_INFO_IN_TASK``);
+* adding a faulting memory hole at the bottom of the stack to catch
+ these overflows (``CONFIG_VMAP_STACK``);
+* runtime checking that alloca() calls don't overstep the stack boundary
+ (``CONFIG_GCC_PLUGIN_STACKLEAK``).
Heap memory integrity
---------------------
@@ -287,11 +292,11 @@ sure structure holes are cleared.
Memory poisoning
----------------
-When releasing memory, it is best to poison the contents (clear stack on
-syscall return, wipe heap memory on a free), to avoid reuse attacks that
-rely on the old contents of memory. This frustrates many uninitialized
-variable attacks, stack content exposures, heap content exposures, and
-use-after-free attacks.
+When releasing memory, it is best to poison the contents, to avoid reuse
+attacks that rely on the old contents of memory. E.g., clear stack on a
+syscall return (``CONFIG_GCC_PLUGIN_STACKLEAK``), wipe heap memory on a
+free. This frustrates many uninitialized variable attacks, stack content
+exposures, heap content exposures, and use-after-free attacks.
Destination tracking
--------------------
--
2.7.4
^ permalink raw reply related
* [kernel-hardening] [PATCH RFC v4 2/3] lkdtm: Add a test for STACKLEAK
From: Alexander Popov @ 2017-10-04 22:55 UTC (permalink / raw)
To: kernel-hardening, keescook, pageexec, spender, tycho,
Laura Abbott, Mark Rutland, Ard Biesheuvel, Andy Lutomirski, x86,
alex.popov
In-Reply-To: <1507157703-14972-1-git-send-email-alex.popov@linux.com>
Introduce two lkdtm tests for the STACKLEAK feature: STACKLEAK_CHECK_ALLOCA
and STACKLEAK_TRACK_STACK. Both of them check that the current task stack
is properly erased (filled with STACKLEAK_POISON).
In addition, STACKLEAK_CHECK_ALLOCA tests that:
- check_alloca() allows alloca calls which don't exhaust the kernel stack;
- alloca calls which exhaust/overflow the kernel stack hit BUG() in
check_alloca().
And STACKLEAK_TRACK_STACK checks that exhausting the thread stack with a
recursion is detected:
- it hits the BUG() in track_stack() if CONFIG_VMAP_STACK is not enabled,
- or hits the guard page otherwise.
Signed-off-by: Tycho Andersen <tycho@docker.com>
Signed-off-by: Alexander Popov <alex.popov@linux.com>
---
drivers/misc/Makefile | 3 +
drivers/misc/lkdtm.h | 4 ++
drivers/misc/lkdtm_core.c | 2 +
drivers/misc/lkdtm_stackleak.c | 139 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 148 insertions(+)
create mode 100644 drivers/misc/lkdtm_stackleak.c
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index d84819d..105e16e 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -63,6 +63,9 @@ lkdtm-$(CONFIG_LKDTM) += lkdtm_perms.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_refcount.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_rodata_objcopy.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_usercopy.o
+lkdtm-$(CONFIG_LKDTM) += lkdtm_stackleak.o
+
+KASAN_SANITIZE_lkdtm_stackleak.o := n
KCOV_INSTRUMENT_lkdtm_rodata.o := n
diff --git a/drivers/misc/lkdtm.h b/drivers/misc/lkdtm.h
index bfb6c45..5952339 100644
--- a/drivers/misc/lkdtm.h
+++ b/drivers/misc/lkdtm.h
@@ -82,4 +82,8 @@ void lkdtm_USERCOPY_STACK_FRAME_FROM(void);
void lkdtm_USERCOPY_STACK_BEYOND(void);
void lkdtm_USERCOPY_KERNEL(void);
+/* lkdtm_stackleak.c */
+void lkdtm_STACKLEAK_CHECK_ALLOCA(void);
+void lkdtm_STACKLEAK_TRACK_STACK(void);
+
#endif
diff --git a/drivers/misc/lkdtm_core.c b/drivers/misc/lkdtm_core.c
index 981b3ef..264d948 100644
--- a/drivers/misc/lkdtm_core.c
+++ b/drivers/misc/lkdtm_core.c
@@ -251,6 +251,8 @@ struct crashtype crashtypes[] = {
CRASHTYPE(USERCOPY_STACK_FRAME_FROM),
CRASHTYPE(USERCOPY_STACK_BEYOND),
CRASHTYPE(USERCOPY_KERNEL),
+ CRASHTYPE(STACKLEAK_CHECK_ALLOCA),
+ CRASHTYPE(STACKLEAK_TRACK_STACK),
};
diff --git a/drivers/misc/lkdtm_stackleak.c b/drivers/misc/lkdtm_stackleak.c
new file mode 100644
index 0000000..1f95ee3
--- /dev/null
+++ b/drivers/misc/lkdtm_stackleak.c
@@ -0,0 +1,139 @@
+/*
+ * This file tests a few aspects of the STACKLEAK feature:
+ * - the current task stack is properly erased (filled with STACKLEAK_POISON);
+ * - check_alloca() allows alloca calls which don't exhaust the kernel stack;
+ * - alloca calls which exhaust/overflow the kernel stack hit BUG() in
+ * check_alloca();
+ * - exhausting the thread stack with a recursion is detected: it hits the
+ * BUG() in track_stack() if CONFIG_VMAP_STACK is not enabled, or hits the
+ * guard page otherwise.
+ *
+ * Copyright (C) Docker, Inc. 2017
+ *
+ * Authors:
+ * Tycho Andersen <tycho@docker.com>
+ * Alexander Popov <alex.popov@linux.com>
+ */
+
+#include "lkdtm.h"
+#include <linux/sched.h>
+#include <linux/compiler.h>
+
+#ifndef CONFIG_GCC_PLUGIN_STACKLEAK
+# define STACKLEAK_POISON -0xBEEF
+# define CONFIG_STACKLEAK_TRACK_MIN_SIZE 100
+#endif
+
+static noinline bool stack_is_erased(void)
+{
+ unsigned long *sp, left, found, i;
+
+ /*
+ * For the details about the alignment of the poison values, see
+ * the comment in track_stack().
+ */
+ sp = PTR_ALIGN(&i, sizeof(unsigned long));
+
+ left = ((unsigned long)sp & (THREAD_SIZE - 1)) / sizeof(unsigned long);
+ sp--;
+
+ /*
+ * Two unsigned long ints at the bottom of the thread stack are
+ * reserved and not poisoned.
+ */
+ if (left <= 2)
+ return false;
+
+ left -= 2;
+ pr_info("checking unused part of the thread stack (%lu bytes)...\n",
+ left * sizeof(unsigned long));
+
+ /* Search for 17 poison values in a row (like erase_kstack() does) */
+ for (i = 0, found = 0; i < left && found < 17; i++) {
+ if (*(sp - i) == STACKLEAK_POISON)
+ found++;
+ else
+ found = 0;
+ }
+
+ if (found < 17) {
+ pr_err("FAIL: thread stack is not erased (checked %lu bytes)\n",
+ i * sizeof(unsigned long));
+ return false;
+ }
+
+ pr_info("first %lu bytes are unpoisoned\n",
+ (i - found) * sizeof(unsigned long));
+
+ /* The rest of thread stack should be erased */
+ for (; i < left; i++) {
+ if (*(sp - i) != STACKLEAK_POISON) {
+ pr_err("FAIL: thread stack is NOT properly erased\n");
+ return false;
+ }
+ }
+
+ pr_info("the rest of the thread stack is properly erased\n");
+ return true;
+}
+
+static noinline void do_alloca(unsigned long size)
+{
+ char buf[size];
+
+ /* So this doesn't get inlined or optimized out */
+ snprintf(buf, size, "hello world\n");
+}
+
+void lkdtm_STACKLEAK_CHECK_ALLOCA(void)
+{
+ unsigned long left = (unsigned long)&left & (THREAD_SIZE - 1);
+
+ if (!stack_is_erased())
+ return;
+
+ /* Try a small alloca to see if it works */
+ pr_info("try a small alloca of 16 bytes...\n");
+ do_alloca(16);
+ pr_info("small alloca is successful\n");
+
+ /* Try to hit the BUG() in check_alloca() */
+ pr_info("try a large alloca of %lu bytes (stack overflow)...\n", left);
+ do_alloca(left);
+ pr_err("FAIL: large alloca overstepped the thread stack boundary\n");
+}
+
+/*
+ * The stack frame size of recursion() is bigger than the
+ * CONFIG_STACKLEAK_TRACK_MIN_SIZE, hence that function is instrumented
+ * by the STACKLEAK gcc plugin and it calls track_stack() at the beginning.
+ */
+static noinline unsigned long recursion(unsigned long prev_sp)
+{
+ char buf[CONFIG_STACKLEAK_TRACK_MIN_SIZE + 42];
+ unsigned long sp = (unsigned long)&sp;
+
+ snprintf(buf, sizeof(buf), "hello world\n");
+
+ if (prev_sp < sp + THREAD_SIZE)
+ sp = recursion(prev_sp);
+
+ return sp;
+}
+
+void lkdtm_STACKLEAK_TRACK_STACK(void)
+{
+ unsigned long sp = (unsigned long)&sp;
+
+ if (!stack_is_erased())
+ return;
+
+ /*
+ * Exhaust the thread stack with a recursion. It should hit the
+ * BUG() in track_stack() if CONFIG_VMAP_STACK is not enabled, or
+ * access the guard page otherwise.
+ */
+ pr_info("try to exhaust the thread stack with the recursion...\n");
+ pr_err("FAIL: thread stack exhaustion (%lu bytes) is not detected\n",
+ sp - recursion(sp));
+}
--
2.7.4
^ permalink raw reply related
* [kernel-hardening] [PATCH RFC v4 1/3] gcc-plugins: Add STACKLEAK erasing the kernel stack at the end of syscalls
From: Alexander Popov @ 2017-10-04 22:55 UTC (permalink / raw)
To: kernel-hardening, keescook, pageexec, spender, tycho,
Laura Abbott, Mark Rutland, Ard Biesheuvel, Andy Lutomirski, x86,
alex.popov
In-Reply-To: <1507157703-14972-1-git-send-email-alex.popov@linux.com>
The STACKLEAK feature erases the kernel stack before returning from
syscalls. That reduces the information which a kernel stack leak bug can
reveal and blocks some uninitialized stack variable attacks. Moreover,
STACKLEAK provides runtime checks for kernel stack overflow detection.
This feature consists of:
- the architecture-specific code filling the used part of the kernel
stack with a poison value before returning to the userspace;
- the STACKLEAK gcc plugin. It instruments the kernel code inserting
the track_stack() call for tracking the lowest border of the kernel
stack and check_alloca() call for checking alloca size.
The STACKLEAK feature is ported from grsecurity/PaX. More information at:
https://grsecurity.net/
https://pax.grsecurity.net/
This code is modified from Brad Spengler/PaX Team's code in the last
public patch of grsecurity/PaX based on our understanding of the code.
Changes or omissions from the original code are ours and don't reflect
the original grsecurity/PaX code.
Signed-off-by: Alexander Popov <alex.popov@linux.com>
---
arch/Kconfig | 39 ++++
arch/x86/Kconfig | 1 +
arch/x86/entry/common.c | 17 +-
arch/x86/entry/entry_32.S | 69 ++++++
arch/x86/entry/entry_64.S | 95 ++++++++
arch/x86/entry/entry_64_compat.S | 8 +
arch/x86/include/asm/processor.h | 4 +
arch/x86/kernel/asm-offsets.c | 9 +
arch/x86/kernel/dumpstack_32.c | 12 +
arch/x86/kernel/dumpstack_64.c | 15 ++
arch/x86/kernel/process_32.c | 5 +
arch/x86/kernel/process_64.c | 5 +
fs/exec.c | 30 +++
include/linux/compiler.h | 4 +
scripts/Makefile.gcc-plugins | 3 +
scripts/gcc-plugins/stackleak_plugin.c | 397 +++++++++++++++++++++++++++++++++
16 files changed, 710 insertions(+), 3 deletions(-)
create mode 100644 scripts/gcc-plugins/stackleak_plugin.c
diff --git a/arch/Kconfig b/arch/Kconfig
index 1aafb4e..f5a30cc 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -386,6 +386,13 @@ config SECCOMP_FILTER
See Documentation/prctl/seccomp_filter.txt for details.
+config HAVE_ARCH_STACKLEAK
+ bool
+ help
+ An architecture should select this if it has the code which
+ fills the used part of the kernel stack with the STACKLEAK_POISON
+ value before returning from system calls.
+
config HAVE_GCC_PLUGINS
bool
help
@@ -516,6 +523,38 @@ config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
in structures. This reduces the performance hit of RANDSTRUCT
at the cost of weakened randomization.
+config GCC_PLUGIN_STACKLEAK
+ bool "Erase the kernel stack before returning from syscalls"
+ depends on GCC_PLUGINS
+ depends on HAVE_ARCH_STACKLEAK
+ help
+ This option makes the kernel erase the kernel stack before it
+ returns from a system call. That reduces the information which
+ a kernel stack leak bug can reveal and blocks some uninitialized
+ stack variable attacks. This option also provides runtime checks
+ for kernel stack overflow detection.
+
+ The tradeoff is the performance impact: on a single CPU system kernel
+ compilation sees a 1% slowdown, other systems and workloads may vary
+ and you are advised to test this feature on your expected workload
+ before deploying it.
+
+ This plugin was ported from grsecurity/PaX. More information at:
+ * https://grsecurity.net/
+ * https://pax.grsecurity.net/
+
+config STACKLEAK_TRACK_MIN_SIZE
+ int "Minimum stack frame size of functions tracked by STACKLEAK"
+ default 100
+ range 0 4096
+ depends on GCC_PLUGIN_STACKLEAK
+ help
+ The STACKLEAK gcc plugin instruments the kernel code for tracking
+ the lowest border of the kernel stack (and for some other purposes).
+ It inserts the track_stack() call for the functions with a stack
+ frame size greater than or equal to this parameter. If unsure,
+ leave the default value 100.
+
config HAVE_CC_STACKPROTECTOR
bool
help
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 971feac..b7da58f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -114,6 +114,7 @@ config X86
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT
select HAVE_ARCH_COMPAT_MMAP_BASES if MMU && COMPAT
select HAVE_ARCH_SECCOMP_FILTER
+ select HAVE_ARCH_STACKLEAK
select HAVE_ARCH_TRACEHOOK
select HAVE_ARCH_TRANSPARENT_HUGEPAGE
select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 03505ff..075487e 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -45,6 +45,12 @@ __visible inline void enter_from_user_mode(void)
static inline void enter_from_user_mode(void) {}
#endif
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+asmlinkage void erase_kstack(void);
+#else
+static void erase_kstack(void) {}
+#endif
+
static void do_audit_syscall_entry(struct pt_regs *regs, u32 arch)
{
#ifdef CONFIG_X86_64
@@ -81,8 +87,10 @@ static long syscall_trace_enter(struct pt_regs *regs)
emulated = true;
if ((emulated || (work & _TIF_SYSCALL_TRACE)) &&
- tracehook_report_syscall_entry(regs))
+ tracehook_report_syscall_entry(regs)) {
+ erase_kstack();
return -1L;
+ }
if (emulated)
return -1L;
@@ -116,9 +124,11 @@ static long syscall_trace_enter(struct pt_regs *regs)
sd.args[5] = regs->bp;
}
- ret = __secure_computing(&sd);
- if (ret == -1)
+ ret = secure_computing(&sd);
+ if (ret == -1) {
+ erase_kstack();
return ret;
+ }
}
#endif
@@ -127,6 +137,7 @@ static long syscall_trace_enter(struct pt_regs *regs)
do_audit_syscall_entry(regs, arch);
+ erase_kstack();
return ret ?: regs->orig_ax;
}
diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index 8a13d46..06bc57b 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -75,6 +75,71 @@
#endif
.endm
+.macro erase_kstack
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ call erase_kstack
+#endif
+.endm
+
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+/* For the detailed comments, see erase_kstack in entry_64.S */
+ENTRY(erase_kstack)
+ pushl %edi
+ pushl %ecx
+ pushl %eax
+ pushl %ebp
+
+ movl PER_CPU_VAR(current_task), %ebp
+ mov TASK_lowest_stack(%ebp), %edi
+ mov $STACKLEAK_POISON, %eax
+ std
+
+1:
+ mov %edi, %ecx
+ and $THREAD_SIZE_asm - 1, %ecx
+ shr $2, %ecx
+ repne scasl
+ jecxz 2f
+
+ cmp $2*16, %ecx
+ jc 2f
+
+ mov $2*16, %ecx
+ repe scasl
+ jecxz 2f
+ jne 1b
+
+2:
+ cld
+ or $2*4, %edi
+ mov %esp, %ecx
+ sub %edi, %ecx
+
+ cmp $THREAD_SIZE_asm, %ecx
+ jb 3f
+ ud2
+
+3:
+ shr $2, %ecx
+ rep stosl
+
+ /*
+ * TODO: sp0 on x86_32 is not reliable, right?
+ * Doubt because of the definition of cpu_current_top_of_stack
+ * in arch/x86/kernel/cpu/common.c.
+ */
+ mov TASK_thread_sp0(%ebp), %edi
+ sub $128, %edi
+ mov %edi, TASK_lowest_stack(%ebp)
+
+ popl %ebp
+ popl %eax
+ popl %ecx
+ popl %edi
+ ret
+ENDPROC(erase_kstack)
+#endif
+
/*
* User gs save/restore
*
@@ -445,6 +510,8 @@ ENTRY(entry_SYSENTER_32)
ALTERNATIVE "testl %eax, %eax; jz .Lsyscall_32_done", \
"jmp .Lsyscall_32_done", X86_FEATURE_XENPV
+ erase_kstack
+
/* Opportunistic SYSEXIT */
TRACE_IRQS_ON /* User mode traces as IRQs on. */
movl PT_EIP(%esp), %edx /* pt_regs->ip */
@@ -531,6 +598,8 @@ ENTRY(entry_INT80_32)
call do_int80_syscall_32
.Lsyscall_32_done:
+ erase_kstack
+
restore_all:
TRACE_IRQS_IRET
.Lrestore_all_notrace:
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 4916725..189d843 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -59,6 +59,90 @@ END(native_usergs_sysret64)
#endif
.endm
+.macro erase_kstack
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ call erase_kstack
+#endif
+.endm
+
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ENTRY(erase_kstack)
+ pushq %rdi
+ pushq %rcx
+ pushq %rax
+ pushq %r11
+
+ movq PER_CPU_VAR(current_task), %r11
+ mov TASK_lowest_stack(%r11), %rdi
+ mov $STACKLEAK_POISON, %rax
+ std
+
+ /*
+ * Let's search for the poison value in the stack.
+ * Start from the lowest_stack and go to the bottom (see std above).
+ */
+1:
+ mov %edi, %ecx
+ and $THREAD_SIZE_asm - 1, %ecx
+ shr $3, %ecx
+ repne scasq
+ jecxz 2f /* Didn't find it. Go to poisoning. */
+
+ /*
+ * Found the poison value in the stack. Go to poisoning if there are
+ * less than 16 qwords left.
+ */
+ cmp $2*8, %ecx
+ jc 2f
+
+ /*
+ * Check that 16 further qwords contain poison (avoid false positives).
+ * If so, the part of the stack below the address in %rdi is likely
+ * to be poisoned. Otherwise we need to search deeper.
+ */
+ mov $2*8, %ecx
+ repe scasq
+ jecxz 2f /* Poison the upper part of the stack. */
+ jne 1b /* Search deeper. */
+
+2:
+ /*
+ * Prepare the counter for poisoning the kernel stack between
+ * %rdi and %rsp. Two qwords at the bottom of the stack are reserved
+ * and should not be poisoned (see CONFIG_SCHED_STACK_END_CHECK).
+ */
+ cld
+ or $2*8, %rdi
+ mov %esp, %ecx
+ sub %edi, %ecx
+
+ /* Check that the counter value is sane. */
+ cmp $THREAD_SIZE_asm, %rcx
+ jb 3f
+ ud2
+
+3:
+ /*
+ * So let's write the poison value to the kernel stack. Start from the
+ * address in %rdi and move up (see cld above) to the address in %rsp
+ * (not included, used memory).
+ */
+ shr $3, %ecx
+ rep stosq
+
+ /* Set the lowest_stack value to the top_of_stack - 256. */
+ mov TASK_thread_sp0(%r11), %rdi
+ sub $256, %rdi
+ mov %rdi, TASK_lowest_stack(%r11)
+
+ popq %r11
+ popq %rax
+ popq %rcx
+ popq %rdi
+ ret
+ENDPROC(erase_kstack)
+#endif
+
/*
* When dynamic function tracer is enabled it will add a breakpoint
* to all locations that it is about to modify, sync CPUs, update
@@ -216,6 +300,8 @@ entry_SYSCALL_64_fastpath:
testl $_TIF_ALLWORK_MASK, TASK_TI_flags(%r11)
jnz 1f
+ erase_kstack
+
LOCKDEP_SYS_EXIT
TRACE_IRQS_ON /* user mode is traced as IRQs on */
movq RIP(%rsp), %rcx
@@ -245,6 +331,8 @@ entry_SYSCALL64_slow_path:
call do_syscall_64 /* returns with IRQs disabled */
return_from_SYSCALL_64:
+ erase_kstack
+
RESTORE_EXTRA_REGS
TRACE_IRQS_IRETQ /* we're about to change IF */
@@ -421,6 +509,7 @@ ENTRY(ret_from_fork)
UNWIND_HINT_REGS
movq %rsp, %rdi
call syscall_return_slowpath /* returns with IRQs disabled */
+ erase_kstack
TRACE_IRQS_ON /* user mode is traced as IRQS on */
SWAPGS
jmp restore_regs_and_iret
@@ -610,6 +699,12 @@ ret_from_intr:
GLOBAL(retint_user)
mov %rsp,%rdi
call prepare_exit_to_usermode
+
+ /*
+ * TODO: Do we need to call erase_kstack here?
+ * The PaX patch has it here commented out.
+ */
+
TRACE_IRQS_IRETQ
SWAPGS
jmp restore_regs_and_iret
diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S
index e26c25c..f79cbf4 100644
--- a/arch/x86/entry/entry_64_compat.S
+++ b/arch/x86/entry/entry_64_compat.S
@@ -18,6 +18,12 @@
.section .entry.text, "ax"
+ .macro erase_kstack
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ call erase_kstack
+#endif
+ .endm
+
/*
* 32-bit SYSENTER entry.
*
@@ -228,6 +234,7 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe)
/* Opportunistic SYSRET */
sysret32_from_system_call:
+ erase_kstack
TRACE_IRQS_ON /* User mode traces as IRQs on. */
movq RBX(%rsp), %rbx /* pt_regs->rbx */
movq RBP(%rsp), %rbp /* pt_regs->rbp */
@@ -335,6 +342,7 @@ ENTRY(entry_INT80_compat)
.Lsyscall_32_done:
/* Go back to user mode. */
+ erase_kstack
TRACE_IRQS_ON
SWAPGS
jmp restore_regs_and_iret
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index b390ff7..c6eaf2d 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -477,6 +477,10 @@ struct thread_struct {
mm_segment_t addr_limit;
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ unsigned long lowest_stack;
+#endif
+
unsigned int sig_on_uaccess_err:1;
unsigned int uaccess_err:1; /* uaccess failed */
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index de827d6..4ed7451 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -37,6 +37,10 @@ void common(void) {
BLANK();
OFFSET(TASK_TI_flags, task_struct, thread_info.flags);
OFFSET(TASK_addr_limit, task_struct, thread.addr_limit);
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ OFFSET(TASK_lowest_stack, task_struct, thread.lowest_stack);
+ OFFSET(TASK_thread_sp0, task_struct, thread.sp0);
+#endif
BLANK();
OFFSET(crypto_tfm_ctx_offset, crypto_tfm, __crt_ctx);
@@ -73,6 +77,11 @@ void common(void) {
OFFSET(PV_MMU_read_cr2, pv_mmu_ops, read_cr2);
#endif
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ BLANK();
+ DEFINE(THREAD_SIZE_asm, THREAD_SIZE);
+#endif
+
#ifdef CONFIG_XEN
BLANK();
OFFSET(XEN_vcpu_info_mask, vcpu_info, evtchn_upcall_mask);
diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c
index 4f04814..4627543 100644
--- a/arch/x86/kernel/dumpstack_32.c
+++ b/arch/x86/kernel/dumpstack_32.c
@@ -162,3 +162,15 @@ void show_regs(struct pt_regs *regs)
}
pr_cont("\n");
}
+
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+void __used check_alloca(unsigned long size)
+{
+ unsigned long sp = (unsigned long)&sp, stack_left;
+
+ /* all kernel stacks are of the same size */
+ stack_left = sp & (THREAD_SIZE - 1);
+ BUG_ON(stack_left < 256 || size >= stack_left - 256);
+}
+EXPORT_SYMBOL(check_alloca);
+#endif
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 225af41..ba82699 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -178,3 +178,18 @@ void show_regs(struct pt_regs *regs)
}
pr_cont("\n");
}
+
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+void __used check_alloca(unsigned long size)
+{
+ unsigned long sp = (unsigned long)&sp;
+ struct stack_info stack_info = {0};
+ unsigned long visit_mask = 0;
+ unsigned long stack_left;
+
+ BUG_ON(get_stack_info(&sp, current, &stack_info, &visit_mask));
+ stack_left = sp - (unsigned long)stack_info.begin;
+ BUG_ON(stack_left < 256 || size >= stack_left - 256);
+}
+EXPORT_SYMBOL(check_alloca);
+#endif
diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index 1196625..c7345d2 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -136,6 +136,11 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp,
p->thread.sp0 = (unsigned long) (childregs+1);
memset(p->thread.ptrace_bps, 0, sizeof(p->thread.ptrace_bps));
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ p->thread.lowest_stack = (unsigned long)task_stack_page(p) +
+ 2 * sizeof(unsigned long);
+#endif
+
if (unlikely(p->flags & PF_KTHREAD)) {
/* kernel thread */
memset(childregs, 0, sizeof(struct pt_regs));
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index 302e7b2..65ba73f 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -283,6 +283,11 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp,
p->thread.sp = (unsigned long) fork_frame;
p->thread.io_bitmap_ptr = NULL;
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+ p->thread.lowest_stack = (unsigned long)task_stack_page(p) +
+ 2 * sizeof(unsigned long);
+#endif
+
savesegment(gs, p->thread.gsindex);
p->thread.gsbase = p->thread.gsindex ? 0 : me->thread.gsbase;
savesegment(fs, p->thread.fsindex);
diff --git a/fs/exec.c b/fs/exec.c
index ac34d97..fb215e3 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1957,3 +1957,33 @@ COMPAT_SYSCALL_DEFINE5(execveat, int, fd,
argv, envp, flags);
}
#endif
+
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+void __used track_stack(void)
+{
+ /*
+ * N.B. The arch-specific part of the STACKLEAK feature fills the
+ * kernel stack with the poison value, which has the register width.
+ * That code assumes that the value of thread.lowest_stack is aligned
+ * on the register width boundary.
+ *
+ * That is true for x86 and x86_64 because of the kernel stack
+ * alignment on these platforms (for details, see cc_stack_align in
+ * arch/x86/Makefile). Take care of that when you port STACKLEAK to
+ * new platforms.
+ */
+ unsigned long sp = (unsigned long)&sp;
+
+ if (sp < current->thread.lowest_stack &&
+ sp >= (unsigned long)task_stack_page(current) +
+ 2 * sizeof(unsigned long)) {
+ current->thread.lowest_stack = sp;
+ }
+
+#ifndef CONFIG_VMAP_STACK
+ if (unlikely((sp & (THREAD_SIZE - 1)) < (THREAD_SIZE / 16)))
+ BUG();
+#endif /* !CONFIG_VMAP_STACK */
+}
+EXPORT_SYMBOL(track_stack);
+#endif /* CONFIG_GCC_PLUGIN_STACKLEAK */
diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index e95a263..916f02d 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -624,4 +624,8 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s
(_________p1); \
})
+#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
+# define STACKLEAK_POISON -0xBEEF
+#endif
+
#endif /* __LINUX_COMPILER_H */
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index d1f7b0d..3793c41 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -34,6 +34,9 @@ ifdef CONFIG_GCC_PLUGINS
gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT) += -DRANDSTRUCT_PLUGIN
gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE) += -fplugin-arg-randomize_layout_plugin-performance-mode
+ gcc-plugin-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak_plugin.so
+ gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STACKLEAK) += -DSTACKLEAK_PLUGIN -fplugin-arg-stackleak_plugin-track-min-size=$(CONFIG_STACKLEAK_TRACK_MIN_SIZE)
+
GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
export PLUGINCC GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
diff --git a/scripts/gcc-plugins/stackleak_plugin.c b/scripts/gcc-plugins/stackleak_plugin.c
new file mode 100644
index 0000000..321ca80
--- /dev/null
+++ b/scripts/gcc-plugins/stackleak_plugin.c
@@ -0,0 +1,397 @@
+/*
+ * Copyright 2011-2017 by the PaX Team <pageexec@freemail.hu>
+ * Licensed under the GPL v2
+ *
+ * Note: the choice of the license means that the compilation process is
+ * NOT 'eligible' as defined by gcc's library exception to the GPL v3,
+ * but for the kernel it doesn't matter since it doesn't link against
+ * any of the gcc libraries
+ *
+ * This gcc plugin is needed for tracking the lowest border of the kernel stack
+ * and checking that alloca calls don't cause stack overflow. It instruments
+ * the kernel code inserting:
+ * - the check_alloca() call before alloca and track_stack() call after it;
+ * - the track_stack() call for the functions with a stack frame size greater
+ * than or equal to the "track-min-size" plugin parameter.
+ *
+ * This plugin is ported from grsecurity/PaX. For more information see:
+ * https://grsecurity.net/
+ * https://pax.grsecurity.net/
+ *
+ * Debugging:
+ * - use fprintf() to stderr, debug_generic_expr(), debug_gimple_stmt()
+ * and print_rtl();
+ * - add "-fdump-tree-all -fdump-rtl-all" to the plugin CFLAGS in
+ * Makefile.gcc-plugins to see the verbose dumps of the gcc passes;
+ * - use gcc -E to understand the preprocessing shenanigans;
+ * - use gcc with enabled CFG/GIMPLE/SSA verification (--enable-checking).
+ */
+
+#include "gcc-common.h"
+
+__visible int plugin_is_GPL_compatible;
+
+static int track_frame_size = -1;
+static const char track_function[] = "track_stack";
+static const char check_function[] = "check_alloca";
+
+// Mark these global variables (roots) for gcc garbage collector since
+// they point to the garbage-collected memory.
+static GTY(()) tree track_function_decl;
+static GTY(()) tree check_function_decl;
+
+static struct plugin_info stackleak_plugin_info = {
+ .version = "201707101337",
+ .help = "track-min-size=nn\ttrack stack for functions with a stack frame size >= nn bytes\n"
+};
+
+static void stackleak_check_alloca(gimple_stmt_iterator *gsi)
+{
+ gimple stmt;
+ gcall *check_alloca;
+ tree alloca_size;
+ cgraph_node_ptr node;
+ int frequency;
+ basic_block bb;
+
+ // insert call to void check_alloca(unsigned long size)
+ alloca_size = gimple_call_arg(gsi_stmt(*gsi), 0);
+ stmt = gimple_build_call(check_function_decl, 1, alloca_size);
+ check_alloca = as_a_gcall(stmt);
+ gsi_insert_before(gsi, check_alloca, GSI_SAME_STMT);
+
+ // update the cgraph
+ bb = gimple_bb(check_alloca);
+ node = cgraph_get_create_node(check_function_decl);
+ gcc_assert(node);
+ frequency = compute_call_stmt_bb_frequency(current_function_decl, bb);
+ cgraph_create_edge(cgraph_get_node(current_function_decl), node, check_alloca, bb->count, frequency, bb->loop_depth);
+}
+
+static void stackleak_add_instrumentation(gimple_stmt_iterator *gsi, bool after)
+{
+ gimple stmt;
+ gcall *track_stack;
+ cgraph_node_ptr node;
+ int frequency;
+ basic_block bb;
+
+ // insert call to void track_stack(void)
+ stmt = gimple_build_call(track_function_decl, 0);
+ track_stack = as_a_gcall(stmt);
+ if (after)
+ gsi_insert_after(gsi, track_stack, GSI_CONTINUE_LINKING);
+ else
+ gsi_insert_before(gsi, track_stack, GSI_SAME_STMT);
+
+ // update the cgraph
+ bb = gimple_bb(track_stack);
+ node = cgraph_get_create_node(track_function_decl);
+ gcc_assert(node);
+ frequency = compute_call_stmt_bb_frequency(current_function_decl, bb);
+ cgraph_create_edge(cgraph_get_node(current_function_decl), node, track_stack, bb->count, frequency, bb->loop_depth);
+}
+
+static bool is_alloca(gimple stmt)
+{
+ if (gimple_call_builtin_p(stmt, BUILT_IN_ALLOCA))
+ return true;
+
+#if BUILDING_GCC_VERSION >= 4007
+ if (gimple_call_builtin_p(stmt, BUILT_IN_ALLOCA_WITH_ALIGN))
+ return true;
+#endif
+
+ return false;
+}
+
+// Work with the GIMPLE representation of the code.
+// Insert the check_alloca() call before alloca and track_stack() call after
+// it. Also insert track_stack() call into the beginning of the function
+// if it is not instrumented.
+static unsigned int stackleak_tree_instrument_execute(void)
+{
+ basic_block bb, entry_bb;
+ bool prologue_instrumented = false, is_leaf = true;
+
+ // ENTRY_BLOCK_PTR is a basic block which represents possible entry
+ // point of a function. This block does not contain any code and
+ // has a CFG edge to its successor.
+ gcc_assert(single_succ_p(ENTRY_BLOCK_PTR_FOR_FN(cfun)));
+ entry_bb = single_succ(ENTRY_BLOCK_PTR_FOR_FN(cfun));
+
+ // 1. Loop through the GIMPLE statements in each of cfun basic blocks.
+ // cfun is a global variable which represents the function that is
+ // currently processed.
+ FOR_EACH_BB_FN(bb, cfun) {
+ gimple_stmt_iterator gsi;
+
+ for (gsi = gsi_start_bb(bb); !gsi_end_p(gsi); gsi_next(&gsi)) {
+ gimple stmt;
+
+ stmt = gsi_stmt(gsi);
+
+ // Leaf function is a function which makes no calls
+ if (is_gimple_call(stmt))
+ is_leaf = false;
+
+ // gimple match: align 8 built-in BUILT_IN_NORMAL:BUILT_IN_ALLOCA attributes <tree_list 0xb7576450>
+ if (!is_alloca(stmt))
+ continue;
+
+ // 2. insert stack overflow check before each alloca call
+ stackleak_check_alloca(&gsi);
+
+ // 3. insert track_stack() call after each alloca call
+ stackleak_add_instrumentation(&gsi, true);
+ if (bb == entry_bb)
+ prologue_instrumented = true;
+ }
+ }
+
+ // special cases for some Linux code: taking the address of static inline functions will materialize them
+ // but we mustn't instrument some of them as the resulting stack alignment required by the function call ABI
+ // will break other assumptions regarding the expected (but not otherwise enforced) register clobbering ABI.
+ // case in point: native_save_fl on amd64 when optimized for size clobbers rdx if it were instrumented here.
+ //
+ // TODO: any more special cases?
+ if (is_leaf && !TREE_PUBLIC(current_function_decl) && DECL_DECLARED_INLINE_P(current_function_decl))
+ return 0;
+ if (is_leaf && !strncmp(IDENTIFIER_POINTER(DECL_NAME(current_function_decl)), "_paravirt_", 10))
+ return 0;
+
+ if (!prologue_instrumented) {
+ // 4. insert track_stack() call at the beginning of the function
+ gimple_stmt_iterator gsi;
+
+ bb = entry_bb;
+ if (!single_pred_p(bb)) {
+// gcc_assert(bb_loop_depth(bb) || (bb->flags & BB_IRREDUCIBLE_LOOP));
+ split_edge(single_succ_edge(ENTRY_BLOCK_PTR_FOR_FN(cfun)));
+ gcc_assert(single_succ_p(ENTRY_BLOCK_PTR_FOR_FN(cfun)));
+ bb = single_succ(ENTRY_BLOCK_PTR_FOR_FN(cfun));
+ }
+ gsi = gsi_after_labels(bb);
+ stackleak_add_instrumentation(&gsi, false);
+ }
+
+ return 0;
+}
+
+// Work with the RTL representation of the code.
+// Remove the unneeded track_stack() calls from the functions which
+// don't call alloca and have the stack frame size less than track_frame_size.
+static unsigned int stackleak_final_execute(void)
+{
+ rtx_insn *insn, *next;
+
+ if (cfun->calls_alloca)
+ return 0;
+
+ if (get_frame_size() >= track_frame_size)
+ return 0;
+
+ // 1. Find track_stack() calls. Loop through the chain of insns,
+ // which is an RTL representation of the code for a function.
+ for (insn = get_insns(); insn; insn = next) {
+ // rtl match: (call_insn 8 7 9 3 (call (mem (symbol_ref ("track_stack") [flags 0x41] <function_decl 0xb7470e80 track_stack>) [0 S1 A8]) (4)) -1 (nil) (nil))
+ rtx body;
+
+ next = NEXT_INSN(insn);
+
+ // Check the expression code of the insn.
+ if (!CALL_P(insn))
+ continue;
+
+ // Check the expression code of the insn body, which is an RTL
+ // Expression (RTX) describing the side effect performed by
+ // that insn.
+ body = PATTERN(insn);
+ if (GET_CODE(body) != CALL)
+ continue;
+
+ // Check the first operand of the call expression. It should
+ // be a mem RTX describing the needed subroutine with a
+ // symbol_ref RTX.
+ body = XEXP(body, 0);
+ if (GET_CODE(body) != MEM)
+ continue;
+
+ body = XEXP(body, 0);
+ if (GET_CODE(body) != SYMBOL_REF)
+ continue;
+
+ if (SYMBOL_REF_DECL(body) != track_function_decl)
+ continue;
+
+ // 2. delete call
+ delete_insn_and_edges(insn);
+#if BUILDING_GCC_VERSION >= 4007
+ if (GET_CODE(next) == NOTE && NOTE_KIND(next) == NOTE_INSN_CALL_ARG_LOCATION) {
+ insn = next;
+ next = NEXT_INSN(insn);
+ delete_insn_and_edges(insn);
+ }
+#endif
+ }
+
+ // Uncomment the following to see the code which was cleaned in this
+ // function. It should not contain check_alloca() and track_stack() calls.
+ // The stack frame size should be less than track_frame_size.
+ //
+ // warning(0, "cleaned from check_alloca and track_stack calls, stack frame size: %ld", get_frame_size());
+ // print_simple_rtl(stderr, get_insns());
+
+ return 0;
+}
+
+static bool stackleak_track_stack_gate(void)
+{
+ tree section;
+
+ section = lookup_attribute("section", DECL_ATTRIBUTES(current_function_decl));
+ if (section && TREE_VALUE(section)) {
+ section = TREE_VALUE(TREE_VALUE(section));
+
+ if (!strncmp(TREE_STRING_POINTER(section), ".init.text", 10))
+ return false;
+ if (!strncmp(TREE_STRING_POINTER(section), ".devinit.text", 13))
+ return false;
+ if (!strncmp(TREE_STRING_POINTER(section), ".cpuinit.text", 13))
+ return false;
+ if (!strncmp(TREE_STRING_POINTER(section), ".meminit.text", 13))
+ return false;
+ }
+
+ return track_frame_size >= 0;
+}
+
+// Build function declarations for track_stack() and check_alloca()
+static void stackleak_start_unit(void *gcc_data __unused, void *user_data __unused)
+{
+ tree fntype;
+
+ // void track_stack(void)
+ fntype = build_function_type_list(void_type_node, NULL_TREE);
+ track_function_decl = build_fn_decl(track_function, fntype);
+ DECL_ASSEMBLER_NAME(track_function_decl); // for LTO
+ TREE_PUBLIC(track_function_decl) = 1;
+ TREE_USED(track_function_decl) = 1;
+ DECL_EXTERNAL(track_function_decl) = 1;
+ DECL_ARTIFICIAL(track_function_decl) = 1;
+ DECL_PRESERVE_P(track_function_decl) = 1;
+
+ // void check_alloca(unsigned long)
+ fntype = build_function_type_list(void_type_node, long_unsigned_type_node, NULL_TREE);
+ check_function_decl = build_fn_decl(check_function, fntype);
+ DECL_ASSEMBLER_NAME(check_function_decl); // for LTO
+ TREE_PUBLIC(check_function_decl) = 1;
+ TREE_USED(check_function_decl) = 1;
+ DECL_EXTERNAL(check_function_decl) = 1;
+ DECL_ARTIFICIAL(check_function_decl) = 1;
+ DECL_PRESERVE_P(check_function_decl) = 1;
+}
+
+// Pass gate function is a predicate function that gets executed before the
+// corresponding pass. If the return value is 'true' the pass gets executed,
+// otherwise, it is skipped.
+static bool stackleak_tree_instrument_gate(void)
+{
+ return stackleak_track_stack_gate();
+}
+
+#define PASS_NAME stackleak_tree_instrument
+#define PROPERTIES_REQUIRED PROP_gimple_leh | PROP_cfg
+#define TODO_FLAGS_START TODO_verify_ssa | TODO_verify_flow | TODO_verify_stmts
+#define TODO_FLAGS_FINISH TODO_verify_ssa | TODO_verify_stmts | TODO_dump_func | TODO_update_ssa | TODO_rebuild_cgraph_edges
+#include "gcc-generate-gimple-pass.h"
+
+static bool stackleak_final_gate(void)
+{
+ return stackleak_track_stack_gate();
+}
+
+#define PASS_NAME stackleak_final
+#define TODO_FLAGS_FINISH TODO_dump_func
+#include "gcc-generate-rtl-pass.h"
+
+// Every gcc plugin exports a plugin_init() function that is called right
+// after the plugin is loaded. This function is responsible for registering
+// the plugin callbacks and doing other required initialization.
+__visible int plugin_init(struct plugin_name_args *plugin_info, struct plugin_gcc_version *version)
+{
+ const char * const plugin_name = plugin_info->base_name;
+ const int argc = plugin_info->argc;
+ const struct plugin_argument * const argv = plugin_info->argv;
+ int i;
+
+ // Extra GGC root tables describing our GTY-ed data.
+ static const struct ggc_root_tab gt_ggc_r_gt_stackleak[] = {
+ {
+ .base = &track_function_decl,
+ .nelt = 1,
+ .stride = sizeof(track_function_decl),
+ .cb = >_ggc_mx_tree_node,
+ .pchw = >_pch_nx_tree_node
+ },
+ {
+ .base = &check_function_decl,
+ .nelt = 1,
+ .stride = sizeof(check_function_decl),
+ .cb = >_ggc_mx_tree_node,
+ .pchw = >_pch_nx_tree_node
+ },
+ LAST_GGC_ROOT_TAB
+ };
+
+ // The stackleak_tree_instrument pass should be executed before the
+ // "optimized" pass, which is the control flow graph cleanup that is
+ // performed just before expanding gcc trees to the RTL. In former
+ // versions of the plugin this new pass was inserted before the
+ // "tree_profile" pass, which is currently called "profile".
+ PASS_INFO(stackleak_tree_instrument, "optimized", 1, PASS_POS_INSERT_BEFORE);
+
+ // The stackleak_final pass should be executed before the "final" pass,
+ // which turns the RTL (Register Transfer Language) into assembly.
+ PASS_INFO(stackleak_final, "final", 1, PASS_POS_INSERT_BEFORE);
+
+ if (!plugin_default_version_check(version, &gcc_version)) {
+ error(G_("incompatible gcc/plugin versions"));
+ return 1;
+ }
+
+ // Give the information about the plugin
+ register_callback(plugin_name, PLUGIN_INFO, NULL, &stackleak_plugin_info);
+
+ // Parse the plugin arguments
+ for (i = 0; i < argc; ++i) {
+ if (!strcmp(argv[i].key, "track-min-size")) {
+ if (!argv[i].value) {
+ error(G_("no value supplied for option '-fplugin-arg-%s-%s'"), plugin_name, argv[i].key);
+ continue;
+ }
+ track_frame_size = atoi(argv[i].value);
+ if (argv[i].value[0] < '0' || argv[i].value[0] > '9' || track_frame_size < 0)
+ error(G_("invalid option argument '-fplugin-arg-%s-%s=%s'"), plugin_name, argv[i].key, argv[i].value);
+ continue;
+ }
+ error(G_("unknown option '-fplugin-arg-%s-%s'"), plugin_name, argv[i].key);
+ }
+
+ // Register to be called before processing a translation unit
+ register_callback(plugin_name, PLUGIN_START_UNIT, &stackleak_start_unit, NULL);
+
+ // Register an extra GCC garbage collector (GGC) root table
+ register_callback(plugin_name, PLUGIN_REGISTER_GGC_ROOTS, NULL, (void *)>_ggc_r_gt_stackleak);
+
+ // Hook into the Pass Manager to register new gcc passes.
+ //
+ // The stack frame size info is available only at the last RTL pass,
+ // when it's too late to insert complex code like a function call.
+ // So we register two gcc passes to instrument every function at first
+ // and remove the unneeded instrumentation later.
+ register_callback(plugin_name, PLUGIN_PASS_MANAGER_SETUP, NULL, &stackleak_tree_instrument_pass_info);
+ register_callback(plugin_name, PLUGIN_PASS_MANAGER_SETUP, NULL, &stackleak_final_pass_info);
+
+ return 0;
+}
--
2.7.4
^ permalink raw reply related
* [kernel-hardening] [PATCH RFC v4 0/3] Introduce the STACKLEAK feature and a test for it
From: Alexander Popov @ 2017-10-04 22:55 UTC (permalink / raw)
To: kernel-hardening, keescook, pageexec, spender, tycho,
Laura Abbott, Mark Rutland, Ard Biesheuvel, Andy Lutomirski, x86,
alex.popov
This is the 4th version of the patch introducing STACKLEAK to the mainline
kernel. STACKLEAK is a security feature developed by Grsecurity/PaX (kudos
to them), which:
- reduces the information that can be revealed by some kernel stack leak bugs
(e.g. CVE-2016-4569 and CVE-2016-4578);
- blocks some uninitialized stack variable attacks (e.g. CVE-2010-2963);
- introduces some runtime checks for kernel stack overflow detection.
Further work:
- think of erasing the kernel stack after invoking EFI runtime services;
- try to port STACKLEAK to arm64 (Laura Abbott is working on it).
Changes in v4
1. Introduced the CONFIG_STACKLEAK_TRACK_MIN_SIZE parameter instead of
hard-coded track-lowest-sp.
2. Carefully looked into the assertions in track_stack() and check_alloca().
- Fixed the incorrect BUG() condition in track_stack(), thanks to Tycho
Andersen. Also disabled that check if CONFIG_VMAP_STACK is enabled.
- Fixed the surplus and erroneous code for calculating stack_left in
check_alloca() on x86_64. That code repeats the work which is already
done in get_stack_info() and it misses the fact that different
exception stacks on x86_64 have different size.
3. Introduced two lkdtm tests for the STACKLEAK feature (developed together
with Tycho).
4. Added info about STACKLEAK to Documentation/security/self-protection.rst.
5. Improved the comments.
6. Decided not to change "unsigned long sp = (unsigned long)&sp" to
current_stack_pointer. The original variant is more platform independent
since current_stack_pointer has different type on x86 and arm.
Changes in v3
1. Added the detailed comments describing the STACKLEAK gcc plugin.
Read the plugin from bottom up, like you do for Linux kernel drivers.
Additional information:
- gcc internals documentation, which is available from gcc sources;
- wonderful slides by Diego Novillo
https://www.airs.com/dnovillo/200711-GCC-Internals/
- nice introduction to gcc plugins at LWN
https://lwn.net/Articles/457543/
2. Improved the commit message and Kconfig description according the
feedback from Kees Cook. Also added the original notice describing
the performance impact of the STACKLEAK feature.
3. Removed arch-specific ix86_cmodel check from stackleak_track_stack_gate().
It caused skipping the kernel code instrumentation for i386.
4. Fixed a minor mistake in stackleak_tree_instrument_execute().
First versions of the plugin used ENTRY_BLOCK_PTR_FOR_FN(cfun)->next_bb
to get the basic block with the function prologue. That was not correct
since the control flow graph edge from the ENTRY_BLOCK_PTR doesn't always
go to that basic block.
So later it was changed it to single_succ(ENTRY_BLOCK_PTR_FOR_FN(cfun)),
but not completely. next_bb was still used for entry_bb assignment,
which could cause the wrong value of the prologue_instrumented variable.
I've reported this issue to Grsecurity and proposed the fix for it, but
unfortunately didn't get any reply.
5. Introduced the STACKLEAK_POISON macro and renamed the config option
according the feedback from Kees Cook.
Alexander Popov (3):
gcc-plugins: Add STACKLEAK erasing the kernel stack at the end of
syscalls
lkdtm: Add a test for STACKLEAK
doc: self-protection: Add information about STACKLEAK feature
Documentation/security/self-protection.rst | 23 +-
arch/Kconfig | 39 +++
arch/x86/Kconfig | 1 +
arch/x86/entry/common.c | 17 +-
arch/x86/entry/entry_32.S | 69 +++++
arch/x86/entry/entry_64.S | 95 +++++++
arch/x86/entry/entry_64_compat.S | 8 +
arch/x86/include/asm/processor.h | 4 +
arch/x86/kernel/asm-offsets.c | 9 +
arch/x86/kernel/dumpstack_32.c | 12 +
arch/x86/kernel/dumpstack_64.c | 15 ++
arch/x86/kernel/process_32.c | 5 +
arch/x86/kernel/process_64.c | 5 +
drivers/misc/Makefile | 3 +
drivers/misc/lkdtm.h | 4 +
drivers/misc/lkdtm_core.c | 2 +
drivers/misc/lkdtm_stackleak.c | 139 ++++++++++
fs/exec.c | 30 +++
include/linux/compiler.h | 4 +
scripts/Makefile.gcc-plugins | 3 +
scripts/gcc-plugins/stackleak_plugin.c | 397 +++++++++++++++++++++++++++++
21 files changed, 872 insertions(+), 12 deletions(-)
create mode 100644 drivers/misc/lkdtm_stackleak.c
create mode 100644 scripts/gcc-plugins/stackleak_plugin.c
--
2.7.4
^ permalink raw reply
* Re: [PATCH v2 2/2] drm/i915/bdw: Fix DP_AUX_CH_CTL_TIME_OUT setting
From: Runyan, Arthur J @ 2017-10-04 22:54 UTC (permalink / raw)
To: Vivi, Rodrigo, Ausmus, James
Cc: intel-gfx@lists.freedesktop.org, ben@bwidawsk.net
In-Reply-To: <20171004202452.4ph6hisncthcbkdh@intel.com>
I think the failure was with one particularly slow eDP panel, but it is safer to apply this to all ports.
-----Original Message-----
From: Vivi, Rodrigo
Sent: Wednesday, 4 October, 2017 1:25 PM
To: Ausmus, James <james.ausmus@intel.com>
Cc: intel-gfx@lists.freedesktop.org; ville.syrjala@linux.intel.com; jani.nikula@linux.intel.com; Runyan, Arthur J <arthur.j.runyan@intel.com>; ben@bwidawsk.net
Subject: Re: [PATCH v2 2/2] drm/i915/bdw: Fix DP_AUX_CH_CTL_TIME_OUT setting
On Wed, Oct 04, 2017 at 08:09:22PM +0000, James Ausmus wrote:
> Per BSpec, 400us is "BDW+ Do not use this setting." - not just PORT_A.
> Set BDW to 600us unconditionally.
Besides that statement I also found on BSpec:
"
Workaround
Project
BDW, EXCLUDE(CHV)
Set the Timeout timer value to at least 600us before initiating a transaction.
"
Also I tracked this on the log and arrived to commit 'a81a507d487c
("drm/i915/bdw: Change dp aux timeout to 600us on DDIA")'
It seems during BDW enabling HW team found that need but only for port A
and later they might have extended it and we never noticed.
Ccin't Art and Ben here to see if they can comment on that.
But I believe we should add this so
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
>
> v2:
> -Split in to two patches (Rodrigo)
>
> Cc: Jani Nikula <jani.nikula@linux.intel.com>
> Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
> Signed-off-by: James Ausmus <james.ausmus@intel.com>
> ---
> drivers/gpu/drm/i915/intel_dp.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/intel_dp.c b/drivers/gpu/drm/i915/intel_dp.c
> index 5b4c9484575b..df301e00d9d9 100644
> --- a/drivers/gpu/drm/i915/intel_dp.c
> +++ b/drivers/gpu/drm/i915/intel_dp.c
> @@ -1019,7 +1019,7 @@ static uint32_t g4x_get_aux_send_ctl(struct intel_dp *intel_dp,
> else
> precharge = 5;
>
> - if (IS_BROADWELL(dev_priv) && intel_dig_port->port == PORT_A)
> + if (IS_BROADWELL(dev_priv))
> timeout = DP_AUX_CH_CTL_TIME_OUT_600us;
> else
> timeout = DP_AUX_CH_CTL_TIME_OUT_400us;
> --
> 2.14.1
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* ✓ Fi.CI.IGT: success for drm/i915: Guc code reorg cont'd
From: Patchwork @ 2017-10-04 22:54 UTC (permalink / raw)
To: Michal Wajdeczko; +Cc: intel-gfx
In-Reply-To: <20171004181343.66348-1-michal.wajdeczko@intel.com>
== Series Details ==
Series: drm/i915: Guc code reorg cont'd
URL : https://patchwork.freedesktop.org/series/31401/
State : success
== Summary ==
Test perf:
Subgroup blocking:
fail -> PASS (shard-hsw) fdo#102252
fdo#102252 https://bugs.freedesktop.org/show_bug.cgi?id=102252
shard-hsw total:2430 pass:1331 dwarn:7 dfail:0 fail:8 skip:1084 time:10232s
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_5903/shards.html
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* Re: [PATCH net-next v4 1/3] bridge: add new BR_NEIGH_SUPPRESS port flag to suppress arp and nd flood
From: David Miller @ 2017-10-04 22:52 UTC (permalink / raw)
To: roopa; +Cc: netdev, nikolay, stephen, bridge
In-Reply-To: <1507093953-59929-2-git-send-email-roopa@cumulusnetworks.com>
From: Roopa Prabhu <roopa@cumulusnetworks.com>
Date: Tue, 3 Oct 2017 22:12:31 -0700
> BR_ARP_PROXY flag but has a few semantic differences to conform
This should be "BR_PROXYARP".
Otherwise this series looks fine to me, but I see there will be
a v5 respin.
^ permalink raw reply
* Re: [Bridge] [PATCH net-next v4 1/3] bridge: add new BR_NEIGH_SUPPRESS port flag to suppress arp and nd flood
From: David Miller @ 2017-10-04 22:52 UTC (permalink / raw)
To: roopa; +Cc: nikolay, netdev, bridge
In-Reply-To: <1507093953-59929-2-git-send-email-roopa@cumulusnetworks.com>
From: Roopa Prabhu <roopa@cumulusnetworks.com>
Date: Tue, 3 Oct 2017 22:12:31 -0700
> BR_ARP_PROXY flag but has a few semantic differences to conform
This should be "BR_PROXYARP".
Otherwise this series looks fine to me, but I see there will be
a v5 respin.
^ permalink raw reply
* libnftables, next steps
From: Phil Sutter @ 2017-10-04 22:51 UTC (permalink / raw)
To: netfilter-devel; +Cc: Eric Leblond, Pablo Neira Ayuso, Florian Westphal
Hi!
I rebased Eric's libnftables patch series onto current master to get an
overview of what's still missing (and what I could work on :). Here's
what I collected:
* Implement application accessible batch support.
-> This basically splits nft_run() into stages.
-> I would change nft_run_cmd_from_*() to use this internally.
-> Do we want this in the early library version or is this going to be
part of the 'advanced API' to add later?
* Add erec_free_list().
-> This becomes handy if the application wants to drop erec list
without printing it (erec_print_list() clears the list while
traversing it).
-> No use for this if we only export nft_run_cmd_from_*() functions.
* Create src/nftables_common.c and include/nftables_common.h to hold
nft_run() and nft_netlink().
-> Is this meant as the (not exported) high-level library backend?
-> If batch support is implemented, these could be removed after
changing nft_run_cmd_from_*() and cli_complete() to use it.
* Move library routines from src/main.c into src/libnftables.c and
create include/nftables/nftables.h to hold the signatures.
* Introduce the library (i.e., generate libnftables.so).
Some additional thoughts:
* Should we support different output streams for debug and/or error
messages?
* Should we reuse src/erec.c for regular output as well? (This probably
needs a 'print immediately' switch for monitor mode, though.)
Feedback highly appreciated, of course! Should I start with moving the
library stuff into libnftables.{c,h} so we get an impression of what the
API will look like?
Cheers, Phil
^ permalink raw reply
* [PATCH] x86/intel_rdt: Remove redundant assignment
From: Jithu Joseph @ 2017-10-04 22:48 UTC (permalink / raw)
To: tglx
Cc: fenghua.yu, tony.luck, vikas.shivappa, mingo, hpa,
reinette.chatre, x86, linux-kernel, Jithu Joseph
The assignment to the 'files' variable is immediately overwritten
in the following line. Remove the older assignment, which was meant
specifially for creating control groups files.
Fixes: c7d9aac61311 ("x86/intel_rdt/cqm: Add mkdir support for RDT monitoring")
Reported-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Jithu Joseph <jithu.joseph@intel.com>
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
---
arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index a869d4a..dc22064 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -1560,7 +1560,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn,
if (ret)
goto out_destroy;
- files = RFTYPE_BASE | RFTYPE_CTRL;
files = RFTYPE_BASE | BIT(RF_CTRLSHIFT + rtype);
ret = rdtgroup_add_files(kn, files);
if (ret)
--
1.9.1
^ permalink raw reply related
* Re: Kernel 4.13 is broken on m68k
From: John Paul Adrian Glaubitz @ 2017-10-04 22:49 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Linux/m68k, Debian m68k
In-Reply-To: <874lre7cvf.fsf@linux-m68k.org>
On 10/05/2017 12:43 AM, Andreas Schwab wrote:
> On Okt 04 2017, John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote:
>
>> Ok, switching just the initrd to the one from 4.13 while keeping the 4.12 kernel
>> image results in the same crash. Maybe the glibc is broken?
>
> It's busybox. Try setting BUSYBOX=n in
> /etc/initramfs-tools/initramfs.conf (or deinstall busybox).
Indeed. I also just found that out.
> $ busybox ash -c 'echo ${x}'
> ash: syntax error: missing '}'
Thanks. That's a very handy testcase.
busybox has gotten new maintainers in Debian and they have stripped
"-O2" from CFLAGS so that it's building with "-Os" now [1].
Maybe that caused the code to be miscompiled. I hope I won't have
to bisect busybox.
Adrian
> [1] https://anonscm.debian.org/cgit/d-i/busybox.git/commit/?id=8a115a910fb3cd7a525972e150a32f8aeb1da492
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
^ permalink raw reply
* [PATCH v4 13/14] platform/x86: dell-smbios-wmi: introduce userspace interface
From: Mario Limonciello @ 2017-10-04 22:48 UTC (permalink / raw)
To: dvhart, Andy Shevchenko
Cc: LKML, platform-driver-x86, Andy Lutomirski, quasisec, pali.rohar,
rjw, mjg59, hch, Greg KH, Mario Limonciello
In-Reply-To: <cover.1507156392.git.mario.limonciello@dell.com>
This userspace character device will be used to perform SMBIOS calls
from any applications.
It provides an ioctl that will allow passing the 32k WMI calling
interface buffer between userspace and kernel space.
This character device is intended to deprecate the dcdbas kernel module
and the interface that it provides to userspace.
It's important for the driver to provide a R/W ioctl to ensure that
two competing userspace processes don't race to provide or read each
others data.
The API for interacting with this interface is defined in documentation
as well as a uapi header provides the format of the structures.
Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
---
Documentation/ABI/testing/dell-smbios-wmi | 43 ++++++++++
.../ABI/testing/sysfs-platform-dell-smbios-wmi | 10 +++
MAINTAINERS | 1 +
drivers/platform/x86/dell-smbios-wmi.c | 98 ++++++++++++++++++++++
drivers/platform/x86/dell-smbios-wmi.h | 10 ---
include/uapi/linux/dell-smbios-wmi.h | 25 ++++++
6 files changed, 177 insertions(+), 10 deletions(-)
create mode 100644 Documentation/ABI/testing/dell-smbios-wmi
create mode 100644 Documentation/ABI/testing/sysfs-platform-dell-smbios-wmi
create mode 100644 include/uapi/linux/dell-smbios-wmi.h
diff --git a/Documentation/ABI/testing/dell-smbios-wmi b/Documentation/ABI/testing/dell-smbios-wmi
new file mode 100644
index 000000000000..86ded18b41f7
--- /dev/null
+++ b/Documentation/ABI/testing/dell-smbios-wmi
@@ -0,0 +1,43 @@
+What: /dev/wmi/dell-smbios
+Date: November 2017
+KernelVersion: 4.15
+Contact: "Mario Limonciello" <mario.limonciello@dell.com>
+Description:
+ Perform SMBIOS calls on supported Dell machines.
+ through the Dell ACPI-WMI interface.
+
+ IOCTL's and buffer formats are defined in:
+ <uapi/linux/dell-wmi-smbios.h>
+
+ 1) To perform a call from userspace, you'll need to first
+ determine the size of the buffer for your machine.
+ Platforms that contain larger buffers can return larger
+ objects from the system firmware.
+ Commonly this size is either 4k or 32k.
+
+ To determine the size of the buffer, refer to:
+ sysfs-platform-dell-smbios-wmi
+
+ 2) After you've determined the size of the calling interface
+ buffer, you can allocate a structure that represents the
+ ioctl struct documented above.
+
+ 3) In the 'length' object store the size of the buffer you
+ determined above.
+
+ 4) Then use this size to allocate an appropriately sized
+ calling interface buffer to assign to 'buf' object.
+
+ 5) In this buf object, prepare as necessary for the SMBIOS
+ call you're interested in. Typically SMBIOS buffers have
+ "class", "select", and "input" defined to values that coincide
+ with the data you are interested in.
+ Documenting class/select/input values is outside of the scope
+ of this documentation. Check with the libsmbios project for
+ further documentation on these values.
+
+ 6) Run the call by using ioctl() as described in the header.
+
+ 7) The output will be returned in the buf object.
+
+ 8) Be sure to free up both of your allocated objects.
diff --git a/Documentation/ABI/testing/sysfs-platform-dell-smbios-wmi b/Documentation/ABI/testing/sysfs-platform-dell-smbios-wmi
new file mode 100644
index 000000000000..6a0513703a3c
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-dell-smbios-wmi
@@ -0,0 +1,10 @@
+What: /sys/devices/platform/<platform>/buffer_size
+Date: November 2017
+KernelVersion: 4.15
+Contact: "Mario Limonciello" <mario.limonciello@dell.com>
+Description:
+ A read-only description of the size of a calling
+ interface buffer that can be passed to Dell
+ firmware.
+
+ Commonly this size is either 4k or 32k.
diff --git a/MAINTAINERS b/MAINTAINERS
index 6db1d84999bc..0dd373cf7280 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3986,6 +3986,7 @@ M: Mario Limonciello <mario.limonciello@dell.com>
L: platform-driver-x86@vger.kernel.org
S: Maintained
F: drivers/platform/x86/dell-smbios-wmi.*
+F: include/uapi/linux/dell-smbios-wmi.h
DELL LAPTOP DRIVER
M: Matthew Garrett <mjg59@srcf.ucam.org>
diff --git a/drivers/platform/x86/dell-smbios-wmi.c b/drivers/platform/x86/dell-smbios-wmi.c
index 32e4e7dbf575..3156864e65e0 100644
--- a/drivers/platform/x86/dell-smbios-wmi.c
+++ b/drivers/platform/x86/dell-smbios-wmi.c
@@ -15,6 +15,7 @@
#include <linux/mutex.h>
#include <linux/uaccess.h>
#include <linux/wmi.h>
+#include <uapi/linux/dell-smbios-wmi.h>
#include "dell-smbios-wmi.h"
#include "dell-wmi-descriptor.h"
static DEFINE_MUTEX(wmi_mutex);
@@ -107,6 +108,87 @@ void dell_smbios_wmi_call(struct calling_interface_buffer *buffer)
}
EXPORT_SYMBOL_GPL(dell_smbios_wmi_call);
+static long dell_smbios_wmi_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ void __user *p = (void __user *) arg;
+ struct wmi_smbios_ioctl *input;
+ struct wmi_smbios_priv *priv;
+ struct wmi_device *wdev;
+ size_t ioctl_size;
+ int ret = 0;
+
+ switch (cmd) {
+ /* we only operate on first instance */
+ case DELL_WMI_SMBIOS_CMD:
+ wdev = get_first_wmi_device();
+ if (!wdev) {
+ pr_err("No WMI devices bound\n");
+ return -ENODEV;
+ }
+ ioctl_size = sizeof(struct wmi_smbios_ioctl);
+ priv = dev_get_drvdata(&wdev->dev);
+ input = kmalloc(ioctl_size, GFP_KERNEL);
+ if (!input)
+ return -ENOMEM;
+ mutex_lock(&wmi_mutex);
+ if (!access_ok(VERIFY_READ, p, ioctl_size)) {
+ pr_err("Unsafe userspace pointer passed\n");
+ return -EFAULT;
+ }
+ if (copy_from_user(input, p, ioctl_size)) {
+ ret = -EFAULT;
+ goto fail_smbios_cmd;
+ }
+ if (input->length != priv->buffer_size) {
+ pr_err("Got buffer size %d expected %d\n",
+ input->length, priv->buffer_size);
+ ret = -EINVAL;
+ goto fail_smbios_cmd;
+ }
+ if (!access_ok(VERIFY_WRITE, input->buf, priv->buffer_size)) {
+ pr_err("Unsafe userspace pointer passed\n");
+ ret = -EFAULT;
+ goto fail_smbios_cmd;
+ }
+ if (copy_from_user(priv->buf, input->buf, priv->buffer_size)) {
+ ret = -EFAULT;
+ goto fail_smbios_cmd;
+ }
+ ret = run_smbios_call(wdev);
+ if (ret != 0)
+ goto fail_smbios_cmd;
+ if (copy_to_user(input->buf, priv->buf, priv->buffer_size))
+ ret = -EFAULT;
+fail_smbios_cmd:
+ kfree(input);
+ mutex_unlock(&wmi_mutex);
+ break;
+ default:
+ pr_err("unsupported ioctl: %d.\n", cmd);
+ ret = -ENOIOCTLCMD;
+ }
+ return ret;
+}
+
+static ssize_t buffer_size_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct wmi_smbios_priv *priv = dev_get_drvdata(dev);
+
+ return sprintf(buf, "%d\n", priv->buffer_size);
+}
+static DEVICE_ATTR_RO(buffer_size);
+
+static struct attribute *smbios_wmi_attrs[] = {
+ &dev_attr_buffer_size.attr,
+ NULL
+};
+
+static const struct attribute_group smbios_wmi_attribute_group = {
+ .attrs = smbios_wmi_attrs,
+};
+
static int dell_smbios_wmi_probe(struct wmi_device *wdev)
{
struct wmi_smbios_priv *priv;
@@ -127,6 +209,11 @@ static int dell_smbios_wmi_probe(struct wmi_device *wdev)
if (!priv->buf)
return -ENOMEM;
+ ret = sysfs_create_group(&wdev->dev.kobj,
+ &smbios_wmi_attribute_group);
+ if (ret)
+ goto fail_create_group;
+
/* ID is used by dell-smbios to set priority of drivers */
wdev->dev.id = 1;
ret = dell_smbios_register_device(&wdev->dev, &dell_smbios_wmi_call);
@@ -140,6 +227,10 @@ static int dell_smbios_wmi_probe(struct wmi_device *wdev)
return 0;
fail_register:
+ sysfs_remove_group(&wdev->dev.kobj,
+ &smbios_wmi_attribute_group);
+
+fail_create_group:
free_pages((unsigned long)priv->buf, count);
return ret;
}
@@ -151,6 +242,7 @@ static int dell_smbios_wmi_remove(struct wmi_device *wdev)
list_del(&priv->list);
dell_smbios_unregister_device(&wdev->dev);
+ sysfs_remove_group(&wdev->dev.kobj, &smbios_wmi_attribute_group);
count = get_order(priv->buffer_size);
free_pages((unsigned long)priv->buf, count);
return 0;
@@ -161,6 +253,11 @@ static const struct wmi_device_id dell_smbios_wmi_id_table[] = {
{ },
};
+static const struct file_operations dell_smbios_wmi_fops = {
+ .owner = THIS_MODULE,
+ .unlocked_ioctl = dell_smbios_wmi_ioctl,
+};
+
static void __init parse_b1_table(const struct dmi_header *dm)
{
struct misc_bios_flags_structure *flags =
@@ -189,6 +286,7 @@ static struct wmi_driver dell_smbios_wmi_driver = {
.probe = dell_smbios_wmi_probe,
.remove = dell_smbios_wmi_remove,
.id_table = dell_smbios_wmi_id_table,
+ .file_operations = &dell_smbios_wmi_fops,
};
static int __init init_dell_smbios_wmi(void)
diff --git a/drivers/platform/x86/dell-smbios-wmi.h b/drivers/platform/x86/dell-smbios-wmi.h
index 5dd91d8ff3d8..40a3666ea87a 100644
--- a/drivers/platform/x86/dell-smbios-wmi.h
+++ b/drivers/platform/x86/dell-smbios-wmi.h
@@ -13,16 +13,6 @@
#include "dell-smbios.h"
-struct wmi_calling_interface_buffer {
- u16 class;
- u16 select;
- u32 input[4];
- u32 output[4];
- u32 argattrib;
- u32 blength;
- u8 *data;
-} __packed;
-
void dell_smbios_wmi_call(struct calling_interface_buffer *buffer);
#endif
diff --git a/include/uapi/linux/dell-smbios-wmi.h b/include/uapi/linux/dell-smbios-wmi.h
new file mode 100644
index 000000000000..0d0d09b04021
--- /dev/null
+++ b/include/uapi/linux/dell-smbios-wmi.h
@@ -0,0 +1,25 @@
+#ifndef _UAPI_DELL_SMBIOS_WMI_H_
+#define _UAPI_DELL_SMBIOS_WMI_H_
+
+#include <linux/ioctl.h>
+#include <linux/wmi.h>
+
+struct wmi_calling_interface_buffer {
+ u16 class;
+ u16 select;
+ u32 input[4];
+ u32 output[4];
+ u32 argattrib;
+ u32 blength;
+ u8 *data;
+} __packed;
+
+struct wmi_smbios_ioctl {
+ u32 length;
+ struct wmi_calling_interface_buffer *buf;
+};
+
+/* only offers on the single instance */
+#define DELL_WMI_SMBIOS_CMD WMI_IOWR(0)
+
+#endif /* _UAPI_DELL_WMI_SMBIOS_H_ */
--
2.14.1
^ permalink raw reply related
* [PATCH v4 14/14] platform/x86: Kconfig: Set default for dell-smbios to ACPI_WMI
From: Mario Limonciello @ 2017-10-04 22:48 UTC (permalink / raw)
To: dvhart, Andy Shevchenko
Cc: LKML, platform-driver-x86, Andy Lutomirski, quasisec, pali.rohar,
rjw, mjg59, hch, Greg KH, Mario Limonciello
In-Reply-To: <cover.1507156392.git.mario.limonciello@dell.com>
The dell-smbios driver no longer depends on dcdbas, and it should
now be offered with the preferred WMI backend when the WMI bus is
enabled.
Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
---
drivers/platform/x86/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/platform/x86/Kconfig b/drivers/platform/x86/Kconfig
index ef597f440d2e..cd9399572156 100644
--- a/drivers/platform/x86/Kconfig
+++ b/drivers/platform/x86/Kconfig
@@ -94,6 +94,7 @@ config ASUS_LAPTOP
config DELL_SMBIOS
tristate "Dell SMBIOS calling interface"
depends on DELL_SMBIOS_WMI || DELL_SMBIOS_SMM
+ default ACPI_WMI
---help---
This module provides common functions for kernel modules using
Dell SMBIOS.
--
2.14.1
^ permalink raw reply related
* [PATCH v4 01/14] platform/x86: wmi: Add new method wmidev_evaluate_method
From: Mario Limonciello @ 2017-10-04 22:48 UTC (permalink / raw)
To: dvhart, Andy Shevchenko
Cc: LKML, platform-driver-x86, Andy Lutomirski, quasisec, pali.rohar,
rjw, mjg59, hch, Greg KH, Mario Limonciello
In-Reply-To: <cover.1507156392.git.mario.limonciello@dell.com>
Drivers properly using the wmibus can pass their wmi_device
pointer rather than the GUID back to the WMI bus to evaluate
the proper methods.
Any "new" drivers added that use the WMI bus should use this
rather than the old wmi_evaluate_method that would take the
GUID.
Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
---
drivers/platform/x86/wmi.c | 28 ++++++++++++++++++++++++----
include/linux/wmi.h | 6 ++++++
2 files changed, 30 insertions(+), 4 deletions(-)
diff --git a/drivers/platform/x86/wmi.c b/drivers/platform/x86/wmi.c
index 7a05843aff19..4d73a87c2ddf 100644
--- a/drivers/platform/x86/wmi.c
+++ b/drivers/platform/x86/wmi.c
@@ -200,6 +200,28 @@ static acpi_status wmi_method_enable(struct wmi_block *wblock, int enable)
*/
acpi_status wmi_evaluate_method(const char *guid_string, u8 instance,
u32 method_id, const struct acpi_buffer *in, struct acpi_buffer *out)
+{
+ struct wmi_block *wblock = NULL;
+
+ if (!find_guid(guid_string, &wblock))
+ return AE_ERROR;
+ return wmidev_evaluate_method(&wblock->dev, instance, method_id,
+ in, out);
+}
+EXPORT_SYMBOL_GPL(wmi_evaluate_method);
+
+/**
+ * wmidev_evaluate_method - Evaluate a WMI method
+ * @wdev: A wmi bus device from a driver
+ * @instance: Instance index
+ * @method_id: Method ID to call
+ * &in: Buffer containing input for the method call
+ * &out: Empty buffer to return the method results
+ *
+ * Call an ACPI-WMI method
+ */
+acpi_status wmidev_evaluate_method(struct wmi_device *wdev, u8 instance,
+ u32 method_id, const struct acpi_buffer *in, struct acpi_buffer *out)
{
struct guid_block *block = NULL;
struct wmi_block *wblock = NULL;
@@ -209,9 +231,7 @@ u32 method_id, const struct acpi_buffer *in, struct acpi_buffer *out)
union acpi_object params[3];
char method[5] = "WM";
- if (!find_guid(guid_string, &wblock))
- return AE_ERROR;
-
+ wblock = container_of(wdev, struct wmi_block, dev);
block = &wblock->gblock;
handle = wblock->acpi_device->handle;
@@ -246,7 +266,7 @@ u32 method_id, const struct acpi_buffer *in, struct acpi_buffer *out)
return status;
}
-EXPORT_SYMBOL_GPL(wmi_evaluate_method);
+EXPORT_SYMBOL_GPL(wmidev_evaluate_method);
static acpi_status __query_block(struct wmi_block *wblock, u8 instance,
struct acpi_buffer *out)
diff --git a/include/linux/wmi.h b/include/linux/wmi.h
index cd0d7734dc49..2cd10c3b89e9 100644
--- a/include/linux/wmi.h
+++ b/include/linux/wmi.h
@@ -26,6 +26,12 @@ struct wmi_device {
bool setable;
};
+/* evaluate the ACPI method associated with this device */
+extern acpi_status wmidev_evaluate_method(struct wmi_device *wdev,
+ u8 instance, u32 method_id,
+ const struct acpi_buffer *in,
+ struct acpi_buffer *out);
+
/* Caller must kfree the result. */
extern union acpi_object *wmidev_block_query(struct wmi_device *wdev,
u8 instance);
--
2.14.1
^ permalink raw reply related
* [PATCH v4 03/14] platform/x86: dell-wmi: allow 32k return size in the descriptor
From: Mario Limonciello @ 2017-10-04 22:48 UTC (permalink / raw)
To: dvhart, Andy Shevchenko
Cc: LKML, platform-driver-x86, Andy Lutomirski, quasisec, pali.rohar,
rjw, mjg59, hch, Greg KH, Mario Limonciello
In-Reply-To: <cover.1507156392.git.mario.limonciello@dell.com>
Some platforms this year will be adopting 32k WMI buffer, so don't
complain when encountering those.
Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
---
drivers/platform/x86/dell-wmi.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/platform/x86/dell-wmi.c b/drivers/platform/x86/dell-wmi.c
index 5973aa20f10e..4807ec4a2e8f 100644
--- a/drivers/platform/x86/dell-wmi.c
+++ b/drivers/platform/x86/dell-wmi.c
@@ -624,7 +624,7 @@ static void dell_wmi_input_destroy(struct wmi_device *wdev)
* Vendor Signature 0 4 "DELL"
* Object Signature 4 4 " WMI"
* WMI Interface Version 8 4 <version>
- * WMI buffer length 12 4 4096
+ * WMI buffer length 12 4 4096 or 32768
*/
static int dell_wmi_check_descriptor_buffer(struct wmi_device *wdev)
{
@@ -673,7 +673,7 @@ static int dell_wmi_check_descriptor_buffer(struct wmi_device *wdev)
dev_warn(&wdev->dev, "Dell descriptor buffer has unknown version (%u)\n",
buffer[2]);
- if (buffer[3] != 4096)
+ if (desc_buffer[3] != 4096 && desc_buffer[3] != 32768)
dev_warn(&wdev->dev, "Dell descriptor buffer has invalid buffer length (%u)\n",
buffer[3]);
--
2.14.1
^ permalink raw reply related
* [PATCH v4 02/14] platform/x86: dell-wmi: clean up wmi descriptor check
From: Mario Limonciello @ 2017-10-04 22:48 UTC (permalink / raw)
To: dvhart, Andy Shevchenko
Cc: LKML, platform-driver-x86, Andy Lutomirski, quasisec, pali.rohar,
rjw, mjg59, hch, Greg KH, Mario Limonciello
In-Reply-To: <cover.1507156392.git.mario.limonciello@dell.com>
Some cases the wrong type was used for errors and checks can be
done more cleanly.
Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
Reviewed-by: Edward O'Callaghan <quasisec@google.com>
Suggested-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
drivers/platform/x86/dell-wmi.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/platform/x86/dell-wmi.c b/drivers/platform/x86/dell-wmi.c
index 1fbef560ca67..5973aa20f10e 100644
--- a/drivers/platform/x86/dell-wmi.c
+++ b/drivers/platform/x86/dell-wmi.c
@@ -665,16 +665,16 @@ static int dell_wmi_check_descriptor_buffer(struct wmi_device *wdev)
buffer = (u32 *)obj->buffer.pointer;
- if (buffer[0] != 0x4C4C4544 && buffer[1] != 0x494D5720)
- dev_warn(&wdev->dev, "Dell descriptor buffer has invalid signature (%*ph)\n",
- 8, buffer);
+ if (strncmp(obj->string.pointer, "DELL WMI", 8) != 0)
+ dev_warn(&wdev->dev, "Dell descriptor buffer has invalid signature (%8ph)\n",
+ buffer);
if (buffer[2] != 0 && buffer[2] != 1)
- dev_warn(&wdev->dev, "Dell descriptor buffer has unknown version (%d)\n",
+ dev_warn(&wdev->dev, "Dell descriptor buffer has unknown version (%u)\n",
buffer[2]);
if (buffer[3] != 4096)
- dev_warn(&wdev->dev, "Dell descriptor buffer has invalid buffer length (%d)\n",
+ dev_warn(&wdev->dev, "Dell descriptor buffer has invalid buffer length (%u)\n",
buffer[3]);
priv->interface_version = buffer[2];
--
2.14.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.