* Re: [net-next PATCH] net: dummy: Introduce dummy virtual functions
From: kbuild test robot @ 2016-11-11 19:23 UTC (permalink / raw)
To: Phil Sutter; +Cc: kbuild-all, David Miller, netdev, Sabrina Dubroca
In-Reply-To: <20161111173333.12603-1-phil@nwl.cc>
[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]
Hi Phil,
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Phil-Sutter/net-dummy-Introduce-dummy-virtual-functions/20161112-013558
config: m68k-sun3_defconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 4.9.0
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=m68k
All errors (new ones prefixed by >>):
drivers/net/dummy.c:53:2: error: unknown field 'sriov' specified in initializer
.sriov = &pdev_sriov,
^
drivers/net/dummy.c:53:2: warning: initialization makes integer from pointer without a cast
drivers/net/dummy.c:53:2: warning: (near initialization for 'pci_pdev.is_virtfn')
drivers/net/dummy.c:53:2: error: initializer element is not computable at load time
drivers/net/dummy.c:53:2: error: (near initialization for 'pci_pdev.is_virtfn')
>> drivers/net/dummy.c:54:14: error: 'pci_bus_type' undeclared here (not in a function)
.dev.bus = &pci_bus_type,
^
vim +/pci_bus_type +54 drivers/net/dummy.c
47 static int num_vfs;
48
49 static struct pci_sriov pdev_sriov;
50
51 static struct pci_dev pci_pdev = {
52 .is_physfn = 1,
> 53 .sriov = &pdev_sriov,
> 54 .dev.bus = &pci_bus_type,
55 };
56
57 struct vf_data_storage {
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 11553 bytes --]
^ permalink raw reply
* Re: [net-next PATCH] net: dummy: Introduce dummy virtual functions
From: kbuild test robot @ 2016-11-11 19:23 UTC (permalink / raw)
To: Phil Sutter; +Cc: kbuild-all, David Miller, netdev, Sabrina Dubroca
In-Reply-To: <20161111173333.12603-1-phil@nwl.cc>
[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]
Hi Phil,
[auto build test ERROR on net-next/master]
url: https://github.com/0day-ci/linux/commits/Phil-Sutter/net-dummy-Introduce-dummy-virtual-functions/20161112-013558
config: xtensa-common_defconfig (attached as .config)
compiler: xtensa-linux-gcc (GCC) 4.9.0
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=xtensa
All error/warnings (new ones prefixed by >>):
>> drivers/net/dummy.c:53:2: error: unknown field 'sriov' specified in initializer
.sriov = &pdev_sriov,
^
>> drivers/net/dummy.c:53:2: warning: initialization makes integer from pointer without a cast
drivers/net/dummy.c:53:2: warning: (near initialization for 'pci_pdev.is_virtfn')
>> drivers/net/dummy.c:53:2: error: initializer element is not computable at load time
drivers/net/dummy.c:53:2: error: (near initialization for 'pci_pdev.is_virtfn')
vim +/sriov +53 drivers/net/dummy.c
47 static int num_vfs;
48
49 static struct pci_sriov pdev_sriov;
50
51 static struct pci_dev pci_pdev = {
52 .is_physfn = 1,
> 53 .sriov = &pdev_sriov,
54 .dev.bus = &pci_bus_type,
55 };
56
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 10114 bytes --]
^ permalink raw reply
* Re: [PATCH] bpf: fix range arithmetic for bpf map access
From: Josef Bacik @ 2016-11-11 19:09 UTC (permalink / raw)
To: Jann Horn; +Cc: Daniel Borkmann, Alexei Starovoitov, David S. Miller, netdev
In-Reply-To: <CAG48ez1c_s-S8dh=37Pj5XOCUbQ1C3SyJGWd7dXhEA2D2+ps9g@mail.gmail.com>
On 11/11/2016 11:36 AM, Jann Horn wrote:
> On Fri, Nov 11, 2016 at 1:18 AM, Josef Bacik <jbacik@fb.com> wrote:
>> ---
>> Sorry Jann, I saw your response last night and then promptly forgot about it,
>> here's the git-send-email version.
>> ---
>
> A note: This doesn't seem to apply cleanly to current net-next (or I'm
> too stupid to
> use "git am"), so I'm applying it on f41cd11d64b2b21012eb4abffbe579bc0b90467f,
> which is net-next from a few days ago.
>
Yeah Dave pulled in a cleanup fix like right after I rebased onto net-next, I'll
rebase again before I send the next patch.
>
>> I made some invalid assumptions with BPF_AND and BPF_MOD that could result in
>> invalid accesses to bpf map entries. Fix this up by doing a few things
>>
>> 1) Kill BPF_MOD support. This doesn't actually get used by the compiler in real
>> life and just adds extra complexity.
>
> Yay! As a security person, I am very much in favor of killing unused features.
>
I almost dropped AND but thought better of it ;).
>
>> 2) Fix the logic for BPF_AND. If the min value is negative then that is the new
>> minimum, otherwise it is unconditionally 0.
>>
>> 3) Don't do operations on the ranges if they are set to the limits, as they are
>> by definition undefined, and allowing arithmetic operations on those values
>> could make them appear valid when they really aren't.
>>
>> This fixes the testcase provided by Jann as well as a few other theoretical
>> problems.
>>
>> Reported-by: Jann Horn <jannh@google.com>
>> Signed-off-by: Josef Bacik <jbacik@fb.com>
>
> A nit: check_mem_access() still has an explicit cast of reg->min_value to s64, I
> think that's not necessary anymore?
Yup just missed that, I'll fix it.
>
>> case BPF_AND:
>> - /* & is special since it could end up with 0 bits set. */
>> - dst_reg->min_value &= min_val;
>> + /* & is special since it's could be any value within our range,
>> + * including 0. But if the thing we're AND'ing against is
>> + * negative and we're negative then that's the minimum value,
>> + * otherwise the minimum will always be 0.
>> + */
>> + if (min_val < 0 && dst_reg->min_value < 0)
>> + dst_reg->min_value = min_t(s64, dst_reg->min_value,
>> + min_val);
>> + else
>> + dst_reg->min_value = 0;
>> dst_reg->max_value = max_val;
>
> I'm not sure whether this is correct when dealing with signed numbers.
> Let's say I have -2 and -3 (as u32: 0xfffffffe and 0xfffffffd) and AND them
> together. The result is 0xfffffffc, or -4, right? So if I just compute
> the AND of
> constant numbers -2 and -3 (known to the verifier), the verifier would
> compute minimum -3 while the actual value is -4, right?
>
> If I am correct about this, I think it might make sense to just reset
> the state to
> unknown in the `min_val < 0 && dst_reg->min_value < 0` case. That shouldn't
> occur in legitimate programs, right?
>
Yeah actually I think you are right, we'll just assume that AND'ing negative
values means you did something wrong and set it to the RANGE_MIN value. Thanks!
Josef
^ permalink raw reply
* [PATCH iproute2 v1 1/2] iproute2: avoid exit in case of error.
From: David Decotigny @ 2016-11-11 18:55 UTC (permalink / raw)
To: Stephen Hemminger, Alexey Kuznetsov; +Cc: netdev, David Decotigny
Be consistent with how non-0 print_route() return values are handled
elesewhere: return -1.
---
ip/iproute.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ip/iproute.c b/ip/iproute.c
index 98bfad6..dae793b 100644
--- a/ip/iproute.c
+++ b/ip/iproute.c
@@ -1743,7 +1743,7 @@ static int iproute_get(int argc, char **argv)
if (print_route(NULL, &req.n, (void *)stdout) < 0) {
fprintf(stderr, "An error :-)\n");
- exit(1);
+ return -1;
}
if (req.n.nlmsg_type != RTM_NEWROUTE) {
--
2.8.0.rc3.226.g39d4020
^ permalink raw reply related
* [PATCH iproute2 v1 2/2] iproute2: a non-expected rtnl message is an error
From: David Decotigny @ 2016-11-11 18:55 UTC (permalink / raw)
To: Stephen Hemminger, Alexey Kuznetsov; +Cc: netdev, David Decotigny
In-Reply-To: <1478890537-88715-1-git-send-email-ddecotig@gmail.com>
---
ip/iproute.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ip/iproute.c b/ip/iproute.c
index dae793b..10d0afe 100644
--- a/ip/iproute.c
+++ b/ip/iproute.c
@@ -320,7 +320,7 @@ int print_route(const struct sockaddr_nl *who, struct nlmsghdr *n, void *arg)
if (n->nlmsg_type != RTM_NEWROUTE && n->nlmsg_type != RTM_DELROUTE) {
fprintf(stderr, "Not a route: %08x %08x %08x\n",
n->nlmsg_len, n->nlmsg_type, n->nlmsg_flags);
- return 0;
+ return -1;
}
if (filter.flushb && n->nlmsg_type != RTM_NEWROUTE)
return 0;
--
2.8.0.rc3.226.g39d4020
^ permalink raw reply related
* [PATCH v2 net-next 6/6] bpf: Add tests for the LRU bpf_htab
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
This patch has some unit tests and a test_lru_dist.
The test_lru_dist reads in the numeric keys from a file.
The files used here are generated by a modified fio-genzipf tool
originated from the fio test suit. The sample data file can be
found here: https://github.com/iamkafai/bpf-lru
The zipf.* data files have 100k numeric keys and the key is also
ranged from 1 to 100k.
The test_lru_dist outputs the number of unique keys (nr_unique).
F.e. The following means, 61239 of them is unique out of 100k keys.
nr_misses means it cannot be found in the LRU map, so nr_misses
must be >= nr_unique. test_lru_dist also simulates a perfect LRU
map as a comparison:
[root@arch-fb-vm1 ~]# ~/devshare/fb-kernel/linux/samples/bpf/test_lru_dist \
/root/zipf.100k.a1_01.out 4000 1
...
test_parallel_lru_dist (map_type:9 map_flags:0x0):
task:0 BPF LRU: nr_unique:23093(/100000) nr_misses:31603(/100000)
task:0 Perfect LRU: nr_unique:23093(/100000 nr_misses:34328(/100000)
....
test_parallel_lru_dist (map_type:9 map_flags:0x2):
task:0 BPF LRU: nr_unique:23093(/100000) nr_misses:31710(/100000)
task:0 Perfect LRU: nr_unique:23093(/100000 nr_misses:34328(/100000)
[root@arch-fb-vm1 ~]# ~/devshare/fb-kernel/linux/samples/bpf/test_lru_dist \
/root/zipf.100k.a0_01.out 40000 1
...
test_parallel_lru_dist (map_type:9 map_flags:0x0):
task:0 BPF LRU: nr_unique:61239(/100000) nr_misses:67054(/100000)
task:0 Perfect LRU: nr_unique:61239(/100000 nr_misses:66993(/100000)
...
test_parallel_lru_dist (map_type:9 map_flags:0x2):
task:0 BPF LRU: nr_unique:61239(/100000) nr_misses:67068(/100000)
task:0 Perfect LRU: nr_unique:61239(/100000 nr_misses:66993(/100000)
LRU map has also been added to map_perf_test:
/* Global LRU */
[root@kerneltest003.31.prn1 ~]# for i in 1 4 8; do echo -n "$i cpus: "; \
./map_perf_test 16 $i | awk '{r += $3}END{print r " updates"}'; done
1 cpus: 2934082 updates
4 cpus: 7391434 updates
8 cpus: 6500576 updates
/* Percpu LRU */
[root@kerneltest003.31.prn1 ~]# for i in 1 4 8; do echo -n "$i cpus: "; \
./map_perf_test 32 $i | awk '{r += $3}END{print r " updates"}'; done
1 cpus: 2896553 updates
4 cpus: 9766395 updates
8 cpus: 17460553 updates
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
samples/bpf/Makefile | 2 +
samples/bpf/map_perf_test_kern.c | 39 ++
samples/bpf/map_perf_test_user.c | 32 ++
samples/bpf/test_lru_dist.c | 538 ++++++++++++++++++++++++++
tools/testing/selftests/bpf/Makefile | 6 +-
tools/testing/selftests/bpf/test_lru_map.c | 583 +++++++++++++++++++++++++++++
6 files changed, 1197 insertions(+), 3 deletions(-)
create mode 100644 samples/bpf/test_lru_dist.c
create mode 100644 tools/testing/selftests/bpf/test_lru_map.c
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 5c53fdb..efd08c0 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -2,6 +2,7 @@
obj- := dummy.o
# List of programs to build
+hostprogs-y := test_lru_dist
hostprogs-y += sock_example
hostprogs-y += fds_example
hostprogs-y += sockex1
@@ -27,6 +28,7 @@ hostprogs-y += test_current_task_under_cgroup
hostprogs-y += trace_event
hostprogs-y += sampleip
+test_lru_dist-objs := test_lru_dist.o libbpf.o
sock_example-objs := sock_example.o libbpf.o
fds_example-objs := bpf_load.o libbpf.o fds_example.o
sockex1-objs := bpf_load.o libbpf.o sockex1_user.o
diff --git a/samples/bpf/map_perf_test_kern.c b/samples/bpf/map_perf_test_kern.c
index 311538e..7ee1574 100644
--- a/samples/bpf/map_perf_test_kern.c
+++ b/samples/bpf/map_perf_test_kern.c
@@ -19,6 +19,21 @@ struct bpf_map_def SEC("maps") hash_map = {
.max_entries = MAX_ENTRIES,
};
+struct bpf_map_def SEC("maps") lru_hash_map = {
+ .type = BPF_MAP_TYPE_LRU_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = 10000,
+};
+
+struct bpf_map_def SEC("maps") percpu_lru_hash_map = {
+ .type = BPF_MAP_TYPE_LRU_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = 10000,
+ .map_flags = BPF_F_NO_COMMON_LRU,
+};
+
struct bpf_map_def SEC("maps") percpu_hash_map = {
.type = BPF_MAP_TYPE_PERCPU_HASH,
.key_size = sizeof(u32),
@@ -53,6 +68,7 @@ int stress_hmap(struct pt_regs *ctx)
value = bpf_map_lookup_elem(&hash_map, &key);
if (value)
bpf_map_delete_elem(&hash_map, &key);
+
return 0;
}
@@ -96,5 +112,28 @@ int stress_percpu_hmap_alloc(struct pt_regs *ctx)
bpf_map_delete_elem(&percpu_hash_map_alloc, &key);
return 0;
}
+
+SEC("kprobe/sys_getpid")
+int stress_lru_hmap_alloc(struct pt_regs *ctx)
+{
+ u32 key = bpf_get_prandom_u32();
+ long val = 1;
+
+ bpf_map_update_elem(&lru_hash_map, &key, &val, BPF_ANY);
+
+ return 0;
+}
+
+SEC("kprobe/sys_getppid")
+int stress_percpu_lru_hmap_alloc(struct pt_regs *ctx)
+{
+ u32 key = bpf_get_prandom_u32();
+ long val = 1;
+
+ bpf_map_update_elem(&percpu_lru_hash_map, &key, &val, BPF_ANY);
+
+ return 0;
+}
+
char _license[] SEC("license") = "GPL";
u32 _version SEC("version") = LINUX_VERSION_CODE;
diff --git a/samples/bpf/map_perf_test_user.c b/samples/bpf/map_perf_test_user.c
index 3147377..9505b4d 100644
--- a/samples/bpf/map_perf_test_user.c
+++ b/samples/bpf/map_perf_test_user.c
@@ -35,6 +35,8 @@ static __u64 time_get_ns(void)
#define PERCPU_HASH_PREALLOC (1 << 1)
#define HASH_KMALLOC (1 << 2)
#define PERCPU_HASH_KMALLOC (1 << 3)
+#define LRU_HASH_PREALLOC (1 << 4)
+#define PERCPU_LRU_HASH_PREALLOC (1 << 5)
static int test_flags = ~0;
@@ -50,6 +52,30 @@ static void test_hash_prealloc(int cpu)
cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
}
+static void test_lru_hash_prealloc(int cpu)
+{
+ __u64 start_time;
+ int i;
+
+ start_time = time_get_ns();
+ for (i = 0; i < MAX_CNT; i++)
+ syscall(__NR_getpid);
+ printf("%d:lru_hash_map_perf pre-alloc %lld events per sec\n",
+ cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+}
+
+static void test_percpu_lru_hash_prealloc(int cpu)
+{
+ __u64 start_time;
+ int i;
+
+ start_time = time_get_ns();
+ for (i = 0; i < MAX_CNT; i++)
+ syscall(__NR_getppid);
+ printf("%d:lru_hash_map_perf pre-alloc %lld events per sec\n",
+ cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+}
+
static void test_percpu_hash_prealloc(int cpu)
{
__u64 start_time;
@@ -105,6 +131,12 @@ static void loop(int cpu)
if (test_flags & PERCPU_HASH_KMALLOC)
test_percpu_hash_kmalloc(cpu);
+
+ if (test_flags & LRU_HASH_PREALLOC)
+ test_lru_hash_prealloc(cpu);
+
+ if (test_flags & PERCPU_LRU_HASH_PREALLOC)
+ test_percpu_lru_hash_prealloc(cpu);
}
static void run_perf_test(int tasks)
diff --git a/samples/bpf/test_lru_dist.c b/samples/bpf/test_lru_dist.c
new file mode 100644
index 0000000..2859977
--- /dev/null
+++ b/samples/bpf/test_lru_dist.c
@@ -0,0 +1,538 @@
+/*
+ * Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#define _GNU_SOURCE
+#include <linux/types.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <linux/bpf.h>
+#include <errno.h>
+#include <string.h>
+#include <assert.h>
+#include <sched.h>
+#include <sys/wait.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <time.h>
+#include "libbpf.h"
+
+#define min(a, b) ((a) < (b) ? (a) : (b))
+#define offsetof(TYPE, MEMBER) ((size_t)&((TYPE *)0)->MEMBER)
+#define container_of(ptr, type, member) ({ \
+ const typeof( ((type *)0)->member ) *__mptr = (ptr); \
+ (type *)( (char *)__mptr - offsetof(type,member) );})
+
+static int nr_cpus;
+static unsigned long long *dist_keys;
+static unsigned int dist_key_counts;
+
+struct list_head {
+ struct list_head *next, *prev;
+};
+
+static inline void INIT_LIST_HEAD(struct list_head *list)
+{
+ list->next = list;
+ list->prev = list;
+}
+
+static inline int list_empty(const struct list_head *head)
+{
+ return head->next == head;
+}
+
+static inline void __list_add(struct list_head *new,
+ struct list_head *prev,
+ struct list_head *next)
+{
+ next->prev = new;
+ new->next = next;
+ new->prev = prev;
+ prev->next = new;
+}
+
+static inline void list_add(struct list_head *new, struct list_head *head)
+{
+ __list_add(new, head, head->next);
+}
+
+static inline void __list_del(struct list_head *prev, struct list_head *next)
+{
+ next->prev = prev;
+ prev->next = next;
+}
+
+static inline void __list_del_entry(struct list_head *entry)
+{
+ __list_del(entry->prev, entry->next);
+}
+
+static inline void list_move(struct list_head *list, struct list_head *head)
+{
+ __list_del_entry(list);
+ list_add(list, head);
+}
+
+#define list_entry(ptr, type, member) \
+ container_of(ptr, type, member)
+
+#define list_last_entry(ptr, type, member) \
+ list_entry((ptr)->prev, type, member)
+
+struct pfect_lru_node {
+ struct list_head list;
+ unsigned long long key;
+};
+
+struct pfect_lru {
+ struct list_head list;
+ struct pfect_lru_node *free_nodes;
+ unsigned int cur_size;
+ unsigned int lru_size;
+ unsigned int nr_unique;
+ unsigned int nr_misses;
+ unsigned int total;
+ int map_fd;
+};
+
+static void pfect_lru_init(struct pfect_lru *lru, unsigned int lru_size,
+ unsigned int nr_possible_elems)
+{
+ lru->map_fd = bpf_create_map(BPF_MAP_TYPE_HASH,
+ sizeof(unsigned long long),
+ sizeof(struct pfect_lru_node *),
+ nr_possible_elems, 0);
+ assert(lru->map_fd != -1);
+
+ lru->free_nodes = malloc(lru_size * sizeof(struct pfect_lru_node));
+ assert(lru->free_nodes);
+
+ INIT_LIST_HEAD(&lru->list);
+ lru->cur_size = 0;
+ lru->lru_size = lru_size;
+ lru->nr_unique = lru->nr_misses = lru->total = 0;
+}
+
+static void pfect_lru_destroy(struct pfect_lru *lru)
+{
+ close(lru->map_fd);
+ free(lru->free_nodes);
+}
+
+static int pfect_lru_lookup_or_insert(struct pfect_lru *lru,
+ unsigned long long key)
+{
+ struct pfect_lru_node *node = NULL;
+ int seen = 0;
+
+ lru->total++;
+ if (!bpf_lookup_elem(lru->map_fd, &key, &node)) {
+ if (node) {
+ list_move(&node->list, &lru->list);
+ return 1;
+ }
+ seen = 1;
+ }
+
+ if (lru->cur_size < lru->lru_size) {
+ node = &lru->free_nodes[lru->cur_size++];
+ INIT_LIST_HEAD(&node->list);
+ } else {
+ struct pfect_lru_node *null_node = NULL;
+
+ node = list_last_entry(&lru->list,
+ struct pfect_lru_node,
+ list);
+ bpf_update_elem(lru->map_fd, &node->key, &null_node, BPF_EXIST);
+ }
+
+ node->key = key;
+ list_move(&node->list, &lru->list);
+
+ lru->nr_misses++;
+ if (seen) {
+ assert(!bpf_update_elem(lru->map_fd, &key, &node, BPF_EXIST));
+ } else {
+ lru->nr_unique++;
+ assert(!bpf_update_elem(lru->map_fd, &key, &node, BPF_NOEXIST));
+ }
+
+ return seen;
+}
+
+static unsigned int read_keys(const char *dist_file,
+ unsigned long long **keys)
+{
+ struct stat fst;
+ unsigned long long *retkeys;
+ unsigned int counts = 0;
+ int dist_fd;
+ char *b, *l;
+ int i;
+
+ dist_fd = open(dist_file, 0);
+ assert(dist_fd != -1);
+
+ assert(fstat(dist_fd, &fst) == 0);
+ b = malloc(fst.st_size);
+ assert(b);
+
+ assert(read(dist_fd, b, fst.st_size) == fst.st_size);
+ close(dist_fd);
+ for (i = 0; i < fst.st_size; i++) {
+ if (b[i] == '\n')
+ counts++;
+ }
+ counts++; /* in case the last line has no \n */
+
+ retkeys = malloc(counts * sizeof(unsigned long long));
+ assert(retkeys);
+
+ counts = 0;
+ for (l = strtok(b, "\n"); l; l = strtok(NULL, "\n"))
+ retkeys[counts++] = strtoull(l, NULL, 10);
+ free(b);
+
+ *keys = retkeys;
+
+ return counts;
+}
+
+static int create_map(int map_type, int map_flags, unsigned int size)
+{
+ int map_fd;
+
+ map_fd = bpf_create_map(map_type, sizeof(unsigned long long),
+ sizeof(unsigned long long), size, map_flags);
+
+ if (map_fd == -1)
+ perror("bpf_create_map");
+
+ return map_fd;
+}
+
+static int sched_next_online(int pid, int next_to_try)
+{
+ cpu_set_t cpuset;
+
+ if (next_to_try == nr_cpus)
+ return -1;
+
+ while (next_to_try < nr_cpus) {
+ CPU_ZERO(&cpuset);
+ CPU_SET(next_to_try++, &cpuset);
+ if (!sched_setaffinity(pid, sizeof(cpuset), &cpuset))
+ break;
+ }
+
+ return next_to_try;
+}
+
+static void run_parallel(unsigned int tasks, void (*fn)(int i, void *data),
+ void *data)
+{
+ int next_sched_cpu = 0;
+ pid_t pid[tasks];
+ int i;
+
+ for (i = 0; i < tasks; i++) {
+ pid[i] = fork();
+ if (pid[i] == 0) {
+ next_sched_cpu = sched_next_online(0, next_sched_cpu);
+ fn(i, data);
+ exit(0);
+ } else if (pid[i] == -1) {
+ printf("couldn't spawn #%d process\n", i);
+ exit(1);
+ }
+ /* It is mostly redundant and just allow the parent
+ * process to update next_shced_cpu for the next child
+ * process
+ */
+ next_sched_cpu = sched_next_online(pid[i], next_sched_cpu);
+ }
+ for (i = 0; i < tasks; i++) {
+ int status;
+
+ assert(waitpid(pid[i], &status, 0) == pid[i]);
+ assert(status == 0);
+ }
+}
+
+static void do_test_lru_dist(int task, void *data)
+{
+ unsigned int nr_misses = 0;
+ struct pfect_lru pfect_lru;
+ unsigned long long key, value = 1234;
+ unsigned int i;
+
+ unsigned int lru_map_fd = ((unsigned int *)data)[0];
+ unsigned int lru_size = ((unsigned int *)data)[1];
+ unsigned long long key_offset = task * dist_key_counts;
+
+ pfect_lru_init(&pfect_lru, lru_size, dist_key_counts);
+
+ for (i = 0; i < dist_key_counts; i++) {
+ key = dist_keys[i] + key_offset;
+
+ pfect_lru_lookup_or_insert(&pfect_lru, key);
+
+ if (!bpf_lookup_elem(lru_map_fd, &key, &value))
+ continue;
+
+ if (bpf_update_elem(lru_map_fd, &key, &value, BPF_NOEXIST)) {
+ printf("bpf_update_elem(lru_map_fd, %llu): errno:%d\n",
+ key, errno);
+ assert(0);
+ }
+
+ nr_misses++;
+ }
+
+ printf(" task:%d BPF LRU: nr_unique:%u(/%u) nr_misses:%u(/%u)\n",
+ task, pfect_lru.nr_unique, dist_key_counts, nr_misses,
+ dist_key_counts);
+ printf(" task:%d Perfect LRU: nr_unique:%u(/%u) nr_misses:%u(/%u)\n",
+ task, pfect_lru.nr_unique, pfect_lru.total,
+ pfect_lru.nr_misses, pfect_lru.total);
+
+ pfect_lru_destroy(&pfect_lru);
+ close(lru_map_fd);
+}
+
+static void test_parallel_lru_dist(int map_type, int map_flags,
+ int nr_tasks, unsigned int lru_size)
+{
+ int child_data[2];
+ int lru_map_fd;
+
+ printf("%s (map_type:%d map_flags:0x%X):\n", __func__, map_type,
+ map_flags);
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ lru_map_fd = create_map(map_type, map_flags,
+ nr_cpus * lru_size);
+ else
+ lru_map_fd = create_map(map_type, map_flags,
+ nr_tasks * lru_size);
+ assert(lru_map_fd != -1);
+
+ child_data[0] = lru_map_fd;
+ child_data[1] = lru_size;
+
+ run_parallel(nr_tasks, do_test_lru_dist, child_data);
+
+ close(lru_map_fd);
+}
+
+static void test_lru_loss0(int map_type, int map_flags)
+{
+ unsigned long long key, value[nr_cpus];
+ unsigned int old_unused_losses = 0;
+ unsigned int new_unused_losses = 0;
+ unsigned int used_losses = 0;
+ int map_fd;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ map_fd = create_map(map_type, map_flags, 900 * nr_cpus);
+ else
+ map_fd = create_map(map_type, map_flags, 900);
+
+ assert(map_fd != -1);
+
+ value[0] = 1234;
+
+ for (key = 1; key <= 1000; key++) {
+ int start_key, end_key;
+
+ assert(bpf_update_elem(map_fd, &key, value, BPF_NOEXIST) == 0);
+
+ start_key = 101;
+ end_key = min(key, 900);
+
+ while (start_key <= end_key) {
+ bpf_lookup_elem(map_fd, &start_key, value);
+ start_key++;
+ }
+ }
+
+ for (key = 1; key <= 1000; key++) {
+ if (bpf_lookup_elem(map_fd, &key, value)) {
+ if (key <= 100)
+ old_unused_losses++;
+ else if (key <= 900)
+ used_losses++;
+ else
+ new_unused_losses++;
+ }
+ }
+
+ close(map_fd);
+
+ printf("older-elem-losses:%d(/100) active-elem-losses:%d(/800) "
+ "newer-elem-losses:%d(/100)\n",
+ old_unused_losses, used_losses, new_unused_losses);
+}
+
+static void test_lru_loss1(int map_type, int map_flags)
+{
+ unsigned long long key, value[nr_cpus];
+ int map_fd;
+ unsigned int nr_losses = 0;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ map_fd = create_map(map_type, map_flags, 1000 * nr_cpus);
+ else
+ map_fd = create_map(map_type, map_flags, 1000);
+
+ assert(map_fd != -1);
+
+ value[0] = 1234;
+
+ for (key = 1; key <= 1000; key++)
+ assert(!bpf_update_elem(map_fd, &key, value, BPF_NOEXIST));
+
+ for (key = 1; key <= 1000; key++) {
+ if (bpf_lookup_elem(map_fd, &key, value))
+ nr_losses++;
+ }
+
+ close(map_fd);
+
+ printf("nr_losses:%d(/1000)\n", nr_losses);
+}
+
+static void do_test_parallel_lru_loss(int task, void *data)
+{
+ const unsigned int nr_stable_elems = 1000;
+ const unsigned int nr_repeats = 100000;
+
+ int map_fd = *(int *)data;
+ unsigned long long stable_base;
+ unsigned long long key, value[nr_cpus];
+ unsigned long long next_ins_key;
+ unsigned int nr_losses = 0;
+ unsigned int i;
+
+ stable_base = task * nr_repeats * 2 + 1;
+ next_ins_key = stable_base;
+ value[0] = 1234;
+ for (i = 0; i < nr_stable_elems; i++) {
+ assert(bpf_update_elem(map_fd, &next_ins_key, value,
+ BPF_NOEXIST) == 0);
+ next_ins_key++;
+ }
+
+ for (i = 0; i < nr_repeats; i++) {
+ int rn;
+
+ rn = rand();
+
+ if (rn % 10) {
+ key = rn % nr_stable_elems + stable_base;
+ bpf_lookup_elem(map_fd, &key, value);
+ } else {
+ bpf_update_elem(map_fd, &next_ins_key, value,
+ BPF_NOEXIST);
+ next_ins_key++;
+ }
+ }
+
+ key = stable_base;
+ for (i = 0; i < nr_stable_elems; i++) {
+ if (bpf_lookup_elem(map_fd, &key, value))
+ nr_losses++;
+ key++;
+ }
+
+ printf(" task:%d nr_losses:%u\n", task, nr_losses);
+}
+
+static void test_parallel_lru_loss(int map_type, int map_flags, int nr_tasks)
+{
+ int map_fd;
+
+ printf("%s (map_type:%d map_flags:0x%X):\n", __func__, map_type,
+ map_flags);
+
+ /* Give 20% more than the active working set */
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ map_fd = create_map(map_type, map_flags,
+ nr_cpus * (1000 + 200));
+ else
+ map_fd = create_map(map_type, map_flags,
+ nr_tasks * (1000 + 200));
+
+ assert(map_fd != -1);
+
+ run_parallel(nr_tasks, do_test_parallel_lru_loss, &map_fd);
+
+ close(map_fd);
+}
+
+int main(int argc, char **argv)
+{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+ int map_flags[] = {0, BPF_F_NO_COMMON_LRU};
+ const char *dist_file;
+ int nr_tasks = 1;
+ int lru_size;
+ int f;
+
+ if (argc < 4) {
+ printf("Usage: %s <dist-file> <lru-size> <nr-tasks>\n",
+ argv[0]);
+ return -1;
+ }
+
+ dist_file = argv[1];
+ lru_size = atoi(argv[2]);
+ nr_tasks = atoi(argv[3]);
+
+ setbuf(stdout, NULL);
+
+ assert(!setrlimit(RLIMIT_MEMLOCK, &r));
+
+ srand(time(NULL));
+
+ nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+ assert(nr_cpus != -1);
+ printf("nr_cpus:%d\n\n", nr_cpus);
+
+ nr_tasks = min(nr_tasks, nr_cpus);
+
+ dist_key_counts = read_keys(dist_file, &dist_keys);
+ if (!dist_key_counts) {
+ printf("%s has no key\n", dist_file);
+ return -1;
+ }
+
+ for (f = 0; f < sizeof(map_flags) / sizeof(*map_flags); f++) {
+ test_lru_loss0(BPF_MAP_TYPE_LRU_HASH, map_flags[f]);
+ test_lru_loss1(BPF_MAP_TYPE_LRU_HASH, map_flags[f]);
+ test_parallel_lru_loss(BPF_MAP_TYPE_LRU_HASH, map_flags[f],
+ nr_tasks);
+ test_parallel_lru_dist(BPF_MAP_TYPE_LRU_HASH, map_flags[f],
+ nr_tasks, lru_size);
+ printf("\n");
+ }
+
+ free(dist_keys);
+
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 4761e2d6..7a5f245 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -1,8 +1,8 @@
-CFLAGS += -Wall -O2
+CFLAGS += -Wall -O2 -I../../../../usr/include
-test_objs = test_verifier test_maps
+test_objs = test_verifier test_maps test_lru_map
-TEST_PROGS := test_verifier test_maps test_kmod.sh
+TEST_PROGS := test_verifier test_maps test_lru_map test_kmod.sh
TEST_FILES := $(test_objs)
all: $(test_objs)
diff --git a/tools/testing/selftests/bpf/test_lru_map.c b/tools/testing/selftests/bpf/test_lru_map.c
new file mode 100644
index 0000000..627757e
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_lru_map.c
@@ -0,0 +1,583 @@
+/*
+ * Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <unistd.h>
+#include <errno.h>
+#include <string.h>
+#include <assert.h>
+#include <sched.h>
+#include <sys/wait.h>
+#include <stdlib.h>
+#include <time.h>
+#include "bpf_sys.h"
+
+#define LOCAL_FREE_TARGET (128)
+#define PERCPU_FREE_TARGET (16)
+
+static int nr_cpus;
+
+static int create_map(int map_type, int map_flags, unsigned int size)
+{
+ int map_fd;
+
+ map_fd = bpf_map_create(map_type, sizeof(unsigned long long),
+ sizeof(unsigned long long), size, map_flags);
+
+ if (map_fd == -1)
+ perror("bpf_map_create");
+
+ return map_fd;
+}
+
+static int map_subset(int map0, int map1)
+{
+ unsigned long long next_key = 0;
+ unsigned long long value0[nr_cpus], value1[nr_cpus];
+ int ret;
+
+ while (!bpf_map_next_key(map1, &next_key, &next_key)) {
+ assert(!bpf_map_lookup(map1, &next_key, value1));
+ ret = bpf_map_lookup(map0, &next_key, value0);
+ if (ret) {
+ printf("key:%llu not found from map. %s(%d)\n",
+ next_key, strerror(errno), errno);
+ return 0;
+ }
+ if (value0[0] != value1[0]) {
+ printf("key:%llu value0:%llu != value1:%llu\n",
+ next_key, value0[0], value1[0]);
+ return 0;
+ }
+ }
+ return 1;
+}
+
+static int map_equal(int lru_map, int expected)
+{
+ return map_subset(lru_map, expected) && map_subset(expected, lru_map);
+}
+
+static int sched_next_online(int pid, int next_to_try)
+{
+ cpu_set_t cpuset;
+
+ if (next_to_try == nr_cpus)
+ return -1;
+
+ while (next_to_try < nr_cpus) {
+ CPU_ZERO(&cpuset);
+ CPU_SET(next_to_try++, &cpuset);
+ if (!sched_setaffinity(pid, sizeof(cpuset), &cpuset))
+ break;
+ }
+
+ return next_to_try;
+}
+
+/* Size of the LRU amp is 2
+ * Add key=1 (+1 key)
+ * Add key=2 (+1 key)
+ * Lookup Key=1
+ * Add Key=3
+ * => Key=2 will be removed by LRU
+ * Iterate map. Only found key=1 and key=3
+ */
+static void test_lru_sanity0(int map_type, int map_flags)
+{
+ unsigned long long key, value[nr_cpus];
+ int lru_map_fd, expected_map_fd;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ lru_map_fd = create_map(map_type, map_flags, 2 * nr_cpus);
+ else
+ lru_map_fd = create_map(map_type, map_flags, 2);
+ assert(lru_map_fd != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, 2);
+ assert(expected_map_fd != -1);
+
+ value[0] = 1234;
+
+ /* insert key=1 element */
+
+ key = 1;
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value, BPF_NOEXIST));
+
+ /* BPF_NOEXIST means: add new element if it doesn't exist */
+ assert(bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST) == -1 &&
+ /* key=1 already exists */
+ errno == EEXIST);
+
+ assert(bpf_map_update(lru_map_fd, &key, value, -1) == -1 &&
+ errno == EINVAL);
+
+ /* insert key=2 element */
+
+ /* check that key=2 is not found */
+ key = 2;
+ assert(bpf_map_lookup(lru_map_fd, &key, value) == -1 &&
+ errno == ENOENT);
+
+ /* BPF_EXIST means: update existing element */
+ assert(bpf_map_update(lru_map_fd, &key, value, BPF_EXIST) == -1 &&
+ /* key=2 is not there */
+ errno == ENOENT);
+
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ /* insert key=3 element */
+
+ /* check that key=3 is not found */
+ key = 3;
+ assert(bpf_map_lookup(lru_map_fd, &key, value) == -1 &&
+ errno == ENOENT);
+
+ /* check that key=1 can be found and mark the ref bit to
+ * stop LRU from removing key=1
+ */
+ key = 1;
+ assert(!bpf_map_lookup(lru_map_fd, &key, value));
+ assert(value[0] == 1234);
+
+ key = 3;
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value, BPF_NOEXIST));
+
+ /* key=2 has been removed from the LRU */
+ key = 2;
+ assert(bpf_map_lookup(lru_map_fd, &key, value) == -1);
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
+/* Size of the LRU map is 1.5*tgt_free
+ * Insert 1 to tgt_free (+tgt_free keys)
+ * Lookup 1 to tgt_free/2
+ * Insert 1+tgt_free to 2*tgt_free (+tgt_free keys)
+ * => 1+tgt_free/2 to LOCALFREE_TARGET will be removed by LRU
+ */
+static void test_lru_sanity1(int map_type, int map_flags, unsigned int tgt_free)
+{
+ unsigned long long key, end_key, value[nr_cpus];
+ int lru_map_fd, expected_map_fd;
+ unsigned int batch_size;
+ unsigned int map_size;
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ /* Ther percpu lru list (i.e each cpu has its own LRU
+ * list) does not have a local free list. Hence,
+ * it will only free old nodes till there is no free
+ * from the LRU list. Hence, this test does not apply
+ * to BPF_F_NO_COMMON_LRU
+ */
+ return;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ batch_size = tgt_free / 2;
+ assert(batch_size * 2 == tgt_free);
+
+ map_size = tgt_free + batch_size;
+ lru_map_fd = create_map(map_type, map_flags, map_size);
+ assert(lru_map_fd != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
+ assert(expected_map_fd != -1);
+
+ value[0] = 1234;
+
+ /* Insert 1 to tgt_free (+tgt_free keys) */
+ end_key = 1 + tgt_free;
+ for (key = 1; key < end_key; key++)
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ /* Lookup 1 to tgt_free/2 */
+ end_key = 1 + batch_size;
+ for (key = 1; key < end_key; key++) {
+ assert(!bpf_map_lookup(lru_map_fd, &key, value));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ /* Insert 1+tgt_free to 2*tgt_free
+ * => 1+tgt_free/2 to LOCALFREE_TARGET will be
+ * removed by LRU
+ */
+ key = 1 + tgt_free;
+ end_key = key + tgt_free;
+ for (; key < end_key; key++) {
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
+/* Size of the LRU map 1.5 * tgt_free
+ * Insert 1 to tgt_free (+tgt_free keys)
+ * Update 1 to tgt_free/2
+ * => The original 1 to tgt_free/2 will be removed due to
+ * the LRU shrink process
+ * Re-insert 1 to tgt_free/2 again and do a lookup immeidately
+ * Insert 1+tgt_free to tgt_free*3/2
+ * Insert 1+tgt_free*3/2 to tgt_free*5/2
+ * => Key 1+tgt_free to tgt_free*3/2
+ * will be removed from LRU because it has never
+ * been lookup and ref bit is not set
+ */
+static void test_lru_sanity2(int map_type, int map_flags, unsigned int tgt_free)
+{
+ unsigned long long key, value[nr_cpus];
+ unsigned long long end_key;
+ int lru_map_fd, expected_map_fd;
+ unsigned int batch_size;
+ unsigned int map_size;
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ /* Ther percpu lru list (i.e each cpu has its own LRU
+ * list) does not have a local free list. Hence,
+ * it will only free old nodes till there is no free
+ * from the LRU list. Hence, this test does not apply
+ * to BPF_F_NO_COMMON_LRU
+ */
+ return;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ batch_size = tgt_free / 2;
+ assert(batch_size * 2 == tgt_free);
+
+ map_size = tgt_free + batch_size;
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ lru_map_fd = create_map(map_type, map_flags,
+ map_size * nr_cpus);
+ else
+ lru_map_fd = create_map(map_type, map_flags, map_size);
+ assert(lru_map_fd != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
+ assert(expected_map_fd != -1);
+
+ value[0] = 1234;
+
+ /* Insert 1 to tgt_free (+tgt_free keys) */
+ end_key = 1 + tgt_free;
+ for (key = 1; key < end_key; key++)
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ /* Any bpf_map_update will require to acquire a new node
+ * from LRU first.
+ *
+ * The local list is running out of free nodes.
+ * It gets from the global LRU list which tries to
+ * shrink the inactive list to get tgt_free
+ * number of free nodes.
+ *
+ * Hence, the oldest key 1 to tgt_free/2
+ * are removed from the LRU list.
+ */
+ key = 1;
+ if (map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_delete(lru_map_fd, &key));
+ } else {
+ assert(bpf_map_update(lru_map_fd, &key, value, BPF_EXIST));
+ }
+
+ /* Re-insert 1 to tgt_free/2 again and do a lookup
+ * immeidately.
+ */
+ end_key = 1 + batch_size;
+ value[0] = 4321;
+ for (key = 1; key < end_key; key++) {
+ assert(bpf_map_lookup(lru_map_fd, &key, value));
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_lookup(lru_map_fd, &key, value));
+ assert(value[0] == 4321);
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ value[0] = 1234;
+
+ /* Insert 1+tgt_free to tgt_free*3/2 */
+ end_key = 1 + tgt_free + batch_size;
+ for (key = 1 + tgt_free; key < end_key; key++)
+ /* These newly added but not referenced keys will be
+ * gone during the next LRU shrink.
+ */
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ /* Insert 1+tgt_free*3/2 to tgt_free*5/2 */
+ end_key = key + tgt_free;
+ for (; key < end_key; key++) {
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
+/* Size of the LRU map is 2*tgt_free
+ * It is to test the active/inactive list rotation
+ * Insert 1 to 2*tgt_free (+2*tgt_free keys)
+ * Lookup key 1 to tgt_free*3/2
+ * Add 1+2*tgt_free to tgt_free*5/2 (+tgt_free/2 keys)
+ * => key 1+tgt_free*3/2 to 2*tgt_free are removed from LRU
+ */
+static void test_lru_sanity3(int map_type, int map_flags, unsigned int tgt_free)
+{
+ unsigned long long key, end_key, value[nr_cpus];
+ int lru_map_fd, expected_map_fd;
+ unsigned int batch_size;
+ unsigned int map_size;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ batch_size = tgt_free / 2;
+ assert(batch_size * 2 == tgt_free);
+
+ map_size = tgt_free * 2;
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ lru_map_fd = create_map(map_type, map_flags,
+ map_size * nr_cpus);
+ else
+ lru_map_fd = create_map(map_type, map_flags, map_size);
+ assert(lru_map_fd != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
+ assert(expected_map_fd != -1);
+
+ value[0] = 1234;
+
+ /* Insert 1 to 2*tgt_free (+2*tgt_free keys) */
+ end_key = 1 + (2 * tgt_free);
+ for (key = 1; key < end_key; key++)
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ /* Lookup key 1 to tgt_free*3/2 */
+ end_key = tgt_free + batch_size;
+ for (key = 1; key < end_key; key++) {
+ assert(!bpf_map_lookup(lru_map_fd, &key, value));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ /* Add 1+2*tgt_free to tgt_free*5/2
+ * (+tgt_free/2 keys)
+ */
+ key = 2 * tgt_free + 1;
+ end_key = key + batch_size;
+ for (; key < end_key; key++) {
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
+/* Test deletion */
+static void test_lru_sanity4(int map_type, int map_flags, unsigned int tgt_free)
+{
+ int lru_map_fd, expected_map_fd;
+ unsigned long long key, value[nr_cpus];
+ unsigned long long end_key;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, 0) != -1);
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ lru_map_fd = create_map(map_type, map_flags,
+ 3 * tgt_free * nr_cpus);
+ else
+ lru_map_fd = create_map(map_type, map_flags, 3 * tgt_free);
+ assert(lru_map_fd != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0,
+ 3 * tgt_free);
+ assert(expected_map_fd != -1);
+
+ value[0] = 1234;
+
+ for (key = 1; key <= 2 * tgt_free; key++)
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ key = 1;
+ assert(bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+
+ for (key = 1; key <= tgt_free; key++) {
+ assert(!bpf_map_lookup(lru_map_fd, &key, value));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ for (; key <= 2 * tgt_free; key++) {
+ assert(!bpf_map_delete(lru_map_fd, &key));
+ assert(bpf_map_delete(lru_map_fd, &key));
+ }
+
+ end_key = key + 2 * tgt_free;
+ for (; key < end_key; key++) {
+ assert(!bpf_map_update(lru_map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_update(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
+static void do_test_lru_sanity5(unsigned long long last_key, int map_fd)
+{
+ unsigned long long key, value[nr_cpus];
+
+ /* Ensure the last key inserted by previous CPU can be found */
+ assert(!bpf_map_lookup(map_fd, &last_key, value));
+
+ value[0] = 1234;
+
+ key = last_key + 1;
+ assert(!bpf_map_update(map_fd, &key, value, BPF_NOEXIST));
+ assert(!bpf_map_lookup(map_fd, &key, value));
+
+ /* Cannot find the last key because it was removed by LRU */
+ assert(bpf_map_lookup(map_fd, &last_key, value));
+}
+
+/* Test map with only one element */
+static void test_lru_sanity5(int map_type, int map_flags)
+{
+ unsigned long long key, value[nr_cpus];
+ int next_sched_cpu = 0;
+ int map_fd;
+ int i;
+
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ return;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ map_fd = create_map(map_type, map_flags, 1);
+ assert(map_fd != -1);
+
+ value[0] = 1234;
+ key = 0;
+ assert(!bpf_map_update(map_fd, &key, value, BPF_NOEXIST));
+
+ for (i = 0; i < nr_cpus; i++) {
+ pid_t pid;
+
+ pid = fork();
+ if (pid == 0) {
+ next_sched_cpu = sched_next_online(0, next_sched_cpu);
+ if (next_sched_cpu != -1)
+ do_test_lru_sanity5(key, map_fd);
+ exit(0);
+ } else if (pid == -1) {
+ printf("couldn't spawn #%d process\n", i);
+ exit(1);
+ } else {
+ int status;
+
+ /* It is mostly redundant and just allow the parent
+ * process to update next_shced_cpu for the next child
+ * process
+ */
+ next_sched_cpu = sched_next_online(pid, next_sched_cpu);
+
+ assert(waitpid(pid, &status, 0) == pid);
+ assert(status == 0);
+ key++;
+ }
+ }
+
+ close(map_fd);
+
+ printf("Pass\n");
+}
+
+int main(int argc, char **argv)
+{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+ int map_types[] = {BPF_MAP_TYPE_LRU_HASH,
+ BPF_MAP_TYPE_LRU_PERCPU_HASH};
+ int map_flags[] = {0, BPF_F_NO_COMMON_LRU};
+ int t, f;
+
+ setbuf(stdout, NULL);
+
+ assert(!setrlimit(RLIMIT_MEMLOCK, &r));
+
+ nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+ assert(nr_cpus != -1);
+ printf("nr_cpus:%d\n\n", nr_cpus);
+
+ for (f = 0; f < sizeof(map_flags) / sizeof(*map_flags); f++) {
+ unsigned int tgt_free = (map_flags[f] & BPF_F_NO_COMMON_LRU) ?
+ PERCPU_FREE_TARGET : LOCAL_FREE_TARGET;
+
+ for (t = 0; t < sizeof(map_types) / sizeof(*map_types); t++) {
+ test_lru_sanity0(map_types[t], map_flags[f]);
+ test_lru_sanity1(map_types[t], map_flags[f], tgt_free);
+ test_lru_sanity2(map_types[t], map_flags[f], tgt_free);
+ test_lru_sanity3(map_types[t], map_flags[f], tgt_free);
+ test_lru_sanity4(map_types[t], map_flags[f], tgt_free);
+ test_lru_sanity5(map_types[t], map_flags[f]);
+
+ printf("\n");
+ }
+ }
+
+ return 0;
+}
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 4/6] bpf: Add BPF_MAP_TYPE_LRU_HASH
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
Provide a LRU version of the existing BPF_MAP_TYPE_HASH.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
include/uapi/linux/bpf.h | 8 ++
kernel/bpf/hashtab.c | 266 ++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 260 insertions(+), 14 deletions(-)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index e2f38e0..ed8c679 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -85,6 +85,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_PERCPU_ARRAY,
BPF_MAP_TYPE_STACK_TRACE,
BPF_MAP_TYPE_CGROUP_ARRAY,
+ BPF_MAP_TYPE_LRU_HASH,
};
enum bpf_prog_type {
@@ -106,6 +107,13 @@ enum bpf_prog_type {
#define BPF_EXIST 2 /* update existing element */
#define BPF_F_NO_PREALLOC (1U << 0)
+/* Instead of having one common LRU list in the
+ * BPF_MAP_TYPE_LRU_HASH map, use a percpu LRU list
+ * which can scale and perform better.
+ * Note, the LRU nodes (including free nodes) cannot be moved
+ * across different LRU lists.
+ */
+#define BPF_F_NO_COMMON_LRU (1U << 1)
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index a5e3915..60e9e85 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -15,6 +15,7 @@
#include <linux/filter.h>
#include <linux/vmalloc.h>
#include "percpu_freelist.h"
+#include "bpf_lru_list.h"
struct bucket {
struct hlist_head head;
@@ -25,7 +26,10 @@ struct bpf_htab {
struct bpf_map map;
struct bucket *buckets;
void *elems;
- struct pcpu_freelist freelist;
+ union {
+ struct pcpu_freelist freelist;
+ struct bpf_lru lru;
+ };
void __percpu *extra_elems;
atomic_t count; /* number of elements in this hashtable */
u32 n_buckets; /* number of hash buckets */
@@ -48,11 +52,19 @@ struct htab_elem {
union {
struct rcu_head rcu;
enum extra_elem_state state;
+ struct bpf_lru_node lru_node;
};
u32 hash;
char key[0] __aligned(8);
};
+static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node);
+
+static bool htab_is_lru(const struct bpf_htab *htab)
+{
+ return htab->map.map_type == BPF_MAP_TYPE_LRU_HASH;
+}
+
static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size,
void __percpu *pptr)
{
@@ -87,7 +99,22 @@ static void htab_free_elems(struct bpf_htab *htab)
vfree(htab->elems);
}
-static int prealloc_elems_and_freelist(struct bpf_htab *htab)
+static struct htab_elem *prealloc_lru_pop(struct bpf_htab *htab, void *key,
+ u32 hash)
+{
+ struct bpf_lru_node *node = bpf_lru_pop_free(&htab->lru, hash);
+ struct htab_elem *l;
+
+ if (node) {
+ l = container_of(node, struct htab_elem, lru_node);
+ memcpy(l->key, key, htab->map.key_size);
+ return l;
+ }
+
+ return NULL;
+}
+
+static int prealloc_init(struct bpf_htab *htab)
{
int err = -ENOMEM, i;
@@ -110,12 +137,27 @@ static int prealloc_elems_and_freelist(struct bpf_htab *htab)
}
skip_percpu_elems:
- err = pcpu_freelist_init(&htab->freelist);
+ if (htab_is_lru(htab))
+ err = bpf_lru_init(&htab->lru,
+ htab->map.map_flags & BPF_F_NO_COMMON_LRU,
+ offsetof(struct htab_elem, hash) -
+ offsetof(struct htab_elem, lru_node),
+ htab_lru_map_delete_node,
+ htab);
+ else
+ err = pcpu_freelist_init(&htab->freelist);
+
if (err)
goto free_elems;
- pcpu_freelist_populate(&htab->freelist, htab->elems, htab->elem_size,
- htab->map.max_entries);
+ if (htab_is_lru(htab))
+ bpf_lru_populate(&htab->lru, htab->elems,
+ offsetof(struct htab_elem, lru_node),
+ htab->elem_size, htab->map.max_entries);
+ else
+ pcpu_freelist_populate(&htab->freelist, htab->elems,
+ htab->elem_size, htab->map.max_entries);
+
return 0;
free_elems:
@@ -123,6 +165,16 @@ static int prealloc_elems_and_freelist(struct bpf_htab *htab)
return err;
}
+static void prealloc_destroy(struct bpf_htab *htab)
+{
+ htab_free_elems(htab);
+
+ if (htab_is_lru(htab))
+ bpf_lru_destroy(&htab->lru);
+ else
+ pcpu_freelist_destroy(&htab->freelist);
+}
+
static int alloc_extra_elems(struct bpf_htab *htab)
{
void __percpu *pptr;
@@ -144,14 +196,34 @@ static int alloc_extra_elems(struct bpf_htab *htab)
static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
{
bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_HASH;
+ bool lru = attr->map_type == BPF_MAP_TYPE_LRU_HASH;
+ /* percpu_lru means each cpu has its own LRU list.
+ * it is different from BPF_MAP_TYPE_PERCPU_HASH where
+ * the map's value itself is percpu. percpu_lru has
+ * nothing to do with the map's value.
+ */
+ bool percpu_lru = (attr->map_flags & BPF_F_NO_COMMON_LRU);
+ bool prealloc = !(attr->map_flags & BPF_F_NO_PREALLOC);
struct bpf_htab *htab;
int err, i;
u64 cost;
- if (attr->map_flags & ~BPF_F_NO_PREALLOC)
+ if (lru && !capable(CAP_SYS_ADMIN))
+ /* LRU implementation is much complicated than other
+ * maps. Hence, limit to CAP_SYS_ADMIN for now.
+ */
+ return ERR_PTR(-EPERM);
+
+ if (attr->map_flags & ~(BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU))
/* reserved bits should not be used */
return ERR_PTR(-EINVAL);
+ if (!lru && percpu_lru)
+ return ERR_PTR(-EINVAL);
+
+ if (lru && !prealloc)
+ return ERR_PTR(-ENOTSUPP);
+
htab = kzalloc(sizeof(*htab), GFP_USER);
if (!htab)
return ERR_PTR(-ENOMEM);
@@ -171,6 +243,18 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
htab->map.value_size == 0)
goto free_htab;
+ if (percpu_lru) {
+ /* ensure each CPU's lru list has >=1 elements.
+ * since we are at it, make each lru list has the same
+ * number of elements.
+ */
+ htab->map.max_entries = roundup(attr->max_entries,
+ num_possible_cpus());
+ if (htab->map.max_entries < attr->max_entries)
+ htab->map.max_entries = rounddown(attr->max_entries,
+ num_possible_cpus());
+ }
+
/* hash table size must be power of 2 */
htab->n_buckets = roundup_pow_of_two(htab->map.max_entries);
@@ -241,14 +325,17 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
raw_spin_lock_init(&htab->buckets[i].lock);
}
- if (!percpu) {
+ if (!percpu && !lru) {
+ /* lru itself can remove the least used element, so
+ * there is no need for an extra elem during map_update.
+ */
err = alloc_extra_elems(htab);
if (err)
goto free_buckets;
}
- if (!(attr->map_flags & BPF_F_NO_PREALLOC)) {
- err = prealloc_elems_and_freelist(htab);
+ if (prealloc) {
+ err = prealloc_init(htab);
if (err)
goto free_extra_elems;
}
@@ -323,6 +410,46 @@ static void *htab_map_lookup_elem(struct bpf_map *map, void *key)
return NULL;
}
+static void *htab_lru_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct htab_elem *l = __htab_map_lookup_elem(map, key);
+
+ if (l) {
+ bpf_lru_node_set_ref(&l->lru_node);
+ return l->key + round_up(map->key_size, 8);
+ }
+
+ return NULL;
+}
+
+/* It is called from the bpf_lru_list when the LRU needs to delete
+ * older elements from the htab.
+ */
+static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)
+{
+ struct bpf_htab *htab = (struct bpf_htab *)arg;
+ struct htab_elem *l, *tgt_l;
+ struct hlist_head *head;
+ unsigned long flags;
+ struct bucket *b;
+
+ tgt_l = container_of(node, struct htab_elem, lru_node);
+ b = __select_bucket(htab, tgt_l->hash);
+ head = &b->head;
+
+ raw_spin_lock_irqsave(&b->lock, flags);
+
+ hlist_for_each_entry_rcu(l, head, hash_node)
+ if (l == tgt_l) {
+ hlist_del_rcu(&l->hash_node);
+ break;
+ }
+
+ raw_spin_unlock_irqrestore(&b->lock, flags);
+
+ return l == tgt_l;
+}
+
/* Called from syscall */
static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
{
@@ -579,6 +706,70 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
return ret;
}
+static int htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value,
+ u64 map_flags)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct htab_elem *l_new, *l_old = NULL;
+ struct hlist_head *head;
+ unsigned long flags;
+ struct bucket *b;
+ u32 key_size, hash;
+ int ret;
+
+ if (unlikely(map_flags > BPF_EXIST))
+ /* unknown flags */
+ return -EINVAL;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+
+ key_size = map->key_size;
+
+ hash = htab_map_hash(key, key_size);
+
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ /* For LRU, we need to alloc before taking bucket's
+ * spinlock because getting free nodes from LRU may need
+ * to remove older elements from htab and this removal
+ * operation will need a bucket lock.
+ */
+ l_new = prealloc_lru_pop(htab, key, hash);
+ if (!l_new)
+ return -ENOMEM;
+ memcpy(l_new->key + round_up(map->key_size, 8), value, map->value_size);
+
+ /* bpf_map_update_elem() can be called in_irq() */
+ raw_spin_lock_irqsave(&b->lock, flags);
+
+ l_old = lookup_elem_raw(head, hash, key, key_size);
+
+ ret = check_flags(htab, l_old, map_flags);
+ if (ret)
+ goto err;
+
+ /* add new element to the head of the list, so that
+ * concurrent search will find it before old elem
+ */
+ hlist_add_head_rcu(&l_new->hash_node, head);
+ if (l_old) {
+ bpf_lru_node_set_ref(&l_new->lru_node);
+ hlist_del_rcu(&l_old->hash_node);
+ }
+ ret = 0;
+
+err:
+ raw_spin_unlock_irqrestore(&b->lock, flags);
+
+ if (ret)
+ bpf_lru_push_free(&htab->lru, &l_new->lru_node);
+ else if (l_old)
+ bpf_lru_push_free(&htab->lru, &l_old->lru_node);
+
+ return ret;
+}
+
static int __htab_percpu_map_update_elem(struct bpf_map *map, void *key,
void *value, u64 map_flags,
bool onallcpus)
@@ -671,6 +862,39 @@ static int htab_map_delete_elem(struct bpf_map *map, void *key)
return ret;
}
+static int htab_lru_map_delete_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct hlist_head *head;
+ struct bucket *b;
+ struct htab_elem *l;
+ unsigned long flags;
+ u32 hash, key_size;
+ int ret = -ENOENT;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+
+ key_size = map->key_size;
+
+ hash = htab_map_hash(key, key_size);
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ raw_spin_lock_irqsave(&b->lock, flags);
+
+ l = lookup_elem_raw(head, hash, key, key_size);
+
+ if (l) {
+ hlist_del_rcu(&l->hash_node);
+ ret = 0;
+ }
+
+ raw_spin_unlock_irqrestore(&b->lock, flags);
+ if (l)
+ bpf_lru_push_free(&htab->lru, &l->lru_node);
+ return ret;
+}
+
static void delete_all_elements(struct bpf_htab *htab)
{
int i;
@@ -702,12 +926,11 @@ static void htab_map_free(struct bpf_map *map)
* not have executed. Wait for them.
*/
rcu_barrier();
- if (htab->map.map_flags & BPF_F_NO_PREALLOC) {
+ if (htab->map.map_flags & BPF_F_NO_PREALLOC)
delete_all_elements(htab);
- } else {
- htab_free_elems(htab);
- pcpu_freelist_destroy(&htab->freelist);
- }
+ else
+ prealloc_destroy(htab);
+
free_percpu(htab->extra_elems);
kvfree(htab->buckets);
kfree(htab);
@@ -727,6 +950,20 @@ static struct bpf_map_type_list htab_type __read_mostly = {
.type = BPF_MAP_TYPE_HASH,
};
+static const struct bpf_map_ops htab_lru_ops = {
+ .map_alloc = htab_map_alloc,
+ .map_free = htab_map_free,
+ .map_get_next_key = htab_map_get_next_key,
+ .map_lookup_elem = htab_lru_map_lookup_elem,
+ .map_update_elem = htab_lru_map_update_elem,
+ .map_delete_elem = htab_lru_map_delete_elem,
+};
+
+static struct bpf_map_type_list htab_lru_type __read_mostly = {
+ .ops = &htab_lru_ops,
+ .type = BPF_MAP_TYPE_LRU_HASH,
+};
+
/* Called from eBPF program */
static void *htab_percpu_map_lookup_elem(struct bpf_map *map, void *key)
{
@@ -797,6 +1034,7 @@ static int __init register_htab_map(void)
{
bpf_register_map_type(&htab_type);
bpf_register_map_type(&htab_percpu_type);
+ bpf_register_map_type(&htab_lru_type);
return 0;
}
late_initcall(register_htab_map);
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 5/6] bpf: Add BPF_MAP_TYPE_LRU_PERCPU_HASH
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
Provide a LRU version of the existing BPF_MAP_TYPE_PERCPU_HASH
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
include/uapi/linux/bpf.h | 3 +-
kernel/bpf/hashtab.c | 129 ++++++++++++++++++++++++++++++++++++++++++++---
kernel/bpf/syscall.c | 8 ++-
3 files changed, 131 insertions(+), 9 deletions(-)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index ed8c679..7d9b283 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -86,6 +86,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_STACK_TRACE,
BPF_MAP_TYPE_CGROUP_ARRAY,
BPF_MAP_TYPE_LRU_HASH,
+ BPF_MAP_TYPE_LRU_PERCPU_HASH,
};
enum bpf_prog_type {
@@ -108,7 +109,7 @@ enum bpf_prog_type {
#define BPF_F_NO_PREALLOC (1U << 0)
/* Instead of having one common LRU list in the
- * BPF_MAP_TYPE_LRU_HASH map, use a percpu LRU list
+ * BPF_MAP_TYPE_LRU_[PERCPU_]HASH map, use a percpu LRU list
* which can scale and perform better.
* Note, the LRU nodes (including free nodes) cannot be moved
* across different LRU lists.
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 60e9e85..60e76fc 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -62,7 +62,14 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node);
static bool htab_is_lru(const struct bpf_htab *htab)
{
- return htab->map.map_type == BPF_MAP_TYPE_LRU_HASH;
+ return htab->map.map_type == BPF_MAP_TYPE_LRU_HASH ||
+ htab->map.map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
+}
+
+static bool htab_is_percpu(const struct bpf_htab *htab)
+{
+ return htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ htab->map.map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
}
static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size,
@@ -85,7 +92,7 @@ static void htab_free_elems(struct bpf_htab *htab)
{
int i;
- if (htab->map.map_type != BPF_MAP_TYPE_PERCPU_HASH)
+ if (!htab_is_percpu(htab))
goto free_elems;
for (i = 0; i < htab->map.max_entries; i++) {
@@ -122,7 +129,7 @@ static int prealloc_init(struct bpf_htab *htab)
if (!htab->elems)
return -ENOMEM;
- if (htab->map.map_type != BPF_MAP_TYPE_PERCPU_HASH)
+ if (!htab_is_percpu(htab))
goto skip_percpu_elems;
for (i = 0; i < htab->map.max_entries; i++) {
@@ -195,8 +202,10 @@ static int alloc_extra_elems(struct bpf_htab *htab)
/* Called from syscall */
static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
{
- bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_HASH;
- bool lru = attr->map_type == BPF_MAP_TYPE_LRU_HASH;
+ bool percpu = (attr->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ attr->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH);
+ bool lru = (attr->map_type == BPF_MAP_TYPE_LRU_HASH ||
+ attr->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH);
/* percpu_lru means each cpu has its own LRU list.
* it is different from BPF_MAP_TYPE_PERCPU_HASH where
* the map's value itself is percpu. percpu_lru has
@@ -823,12 +832,84 @@ static int __htab_percpu_map_update_elem(struct bpf_map *map, void *key,
return ret;
}
+static int __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
+ void *value, u64 map_flags,
+ bool onallcpus)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct htab_elem *l_new = NULL, *l_old;
+ struct hlist_head *head;
+ unsigned long flags;
+ struct bucket *b;
+ u32 key_size, hash;
+ int ret;
+
+ if (unlikely(map_flags > BPF_EXIST))
+ /* unknown flags */
+ return -EINVAL;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+
+ key_size = map->key_size;
+
+ hash = htab_map_hash(key, key_size);
+
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ /* For LRU, we need to alloc before taking bucket's
+ * spinlock because LRU's elem alloc may need
+ * to remove older elem from htab and this removal
+ * operation will need a bucket lock.
+ */
+ if (map_flags != BPF_EXIST) {
+ l_new = prealloc_lru_pop(htab, key, hash);
+ if (!l_new)
+ return -ENOMEM;
+ }
+
+ /* bpf_map_update_elem() can be called in_irq() */
+ raw_spin_lock_irqsave(&b->lock, flags);
+
+ l_old = lookup_elem_raw(head, hash, key, key_size);
+
+ ret = check_flags(htab, l_old, map_flags);
+ if (ret)
+ goto err;
+
+ if (l_old) {
+ bpf_lru_node_set_ref(&l_old->lru_node);
+
+ /* per-cpu hash map can update value in-place */
+ pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size),
+ value, onallcpus);
+ } else {
+ pcpu_copy_value(htab, htab_elem_get_ptr(l_new, key_size),
+ value, onallcpus);
+ hlist_add_head_rcu(&l_new->hash_node, head);
+ l_new = NULL;
+ }
+ ret = 0;
+err:
+ raw_spin_unlock_irqrestore(&b->lock, flags);
+ if (l_new)
+ bpf_lru_push_free(&htab->lru, &l_new->lru_node);
+ return ret;
+}
+
static int htab_percpu_map_update_elem(struct bpf_map *map, void *key,
void *value, u64 map_flags)
{
return __htab_percpu_map_update_elem(map, key, value, map_flags, false);
}
+static int htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
+ void *value, u64 map_flags)
+{
+ return __htab_lru_percpu_map_update_elem(map, key, value, map_flags,
+ false);
+}
+
/* Called from syscall or from eBPF program */
static int htab_map_delete_elem(struct bpf_map *map, void *key)
{
@@ -975,8 +1056,21 @@ static void *htab_percpu_map_lookup_elem(struct bpf_map *map, void *key)
return NULL;
}
+static void *htab_lru_percpu_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct htab_elem *l = __htab_map_lookup_elem(map, key);
+
+ if (l) {
+ bpf_lru_node_set_ref(&l->lru_node);
+ return this_cpu_ptr(htab_elem_get_ptr(l, map->key_size));
+ }
+
+ return NULL;
+}
+
int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
struct htab_elem *l;
void __percpu *pptr;
int ret = -ENOENT;
@@ -992,6 +1086,8 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
l = __htab_map_lookup_elem(map, key);
if (!l)
goto out;
+ if (htab_is_lru(htab))
+ bpf_lru_node_set_ref(&l->lru_node);
pptr = htab_elem_get_ptr(l, map->key_size);
for_each_possible_cpu(cpu) {
bpf_long_memcpy(value + off,
@@ -1007,10 +1103,16 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
u64 map_flags)
{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
int ret;
rcu_read_lock();
- ret = __htab_percpu_map_update_elem(map, key, value, map_flags, true);
+ if (htab_is_lru(htab))
+ ret = __htab_lru_percpu_map_update_elem(map, key, value,
+ map_flags, true);
+ else
+ ret = __htab_percpu_map_update_elem(map, key, value, map_flags,
+ true);
rcu_read_unlock();
return ret;
@@ -1030,11 +1132,26 @@ static struct bpf_map_type_list htab_percpu_type __read_mostly = {
.type = BPF_MAP_TYPE_PERCPU_HASH,
};
+static const struct bpf_map_ops htab_lru_percpu_ops = {
+ .map_alloc = htab_map_alloc,
+ .map_free = htab_map_free,
+ .map_get_next_key = htab_map_get_next_key,
+ .map_lookup_elem = htab_lru_percpu_map_lookup_elem,
+ .map_update_elem = htab_lru_percpu_map_update_elem,
+ .map_delete_elem = htab_lru_map_delete_elem,
+};
+
+static struct bpf_map_type_list htab_lru_percpu_type __read_mostly = {
+ .ops = &htab_lru_percpu_ops,
+ .type = BPF_MAP_TYPE_LRU_PERCPU_HASH,
+};
+
static int __init register_htab_map(void)
{
bpf_register_map_type(&htab_type);
bpf_register_map_type(&htab_percpu_type);
bpf_register_map_type(&htab_lru_type);
+ bpf_register_map_type(&htab_lru_percpu_type);
return 0;
}
late_initcall(register_htab_map);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 228f962..bc69eb4 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -295,6 +295,7 @@ static int map_lookup_elem(union bpf_attr *attr)
goto free_key;
if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
value_size = round_up(map->value_size, 8) * num_possible_cpus();
else
@@ -305,7 +306,8 @@ static int map_lookup_elem(union bpf_attr *attr)
if (!value)
goto free_key;
- if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH) {
+ if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
err = bpf_percpu_hash_copy(map, key, value);
} else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
err = bpf_percpu_array_copy(map, key, value);
@@ -369,6 +371,7 @@ static int map_update_elem(union bpf_attr *attr)
goto free_key;
if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
value_size = round_up(map->value_size, 8) * num_possible_cpus();
else
@@ -388,7 +391,8 @@ static int map_update_elem(union bpf_attr *attr)
*/
preempt_disable();
__this_cpu_inc(bpf_prog_active);
- if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH) {
+ if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+ map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) {
err = bpf_percpu_hash_update(map, key, value, attr->flags);
} else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
err = bpf_percpu_array_update(map, key, value, attr->flags);
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 3/6] bpf: Refactor codes handling percpu map
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
Refactor the codes that populate the value
of a htab_elem in a BPF_MAP_TYPE_PERCPU_HASH
typed bpf_map.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
kernel/bpf/hashtab.c | 47 +++++++++++++++++++++--------------------------
1 file changed, 21 insertions(+), 26 deletions(-)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 570eeca..a5e3915 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -420,6 +420,24 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
}
}
+static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
+ void *value, bool onallcpus)
+{
+ if (!onallcpus) {
+ /* copy true value_size bytes */
+ memcpy(this_cpu_ptr(pptr), value, htab->map.value_size);
+ } else {
+ u32 size = round_up(htab->map.value_size, 8);
+ int off = 0, cpu;
+
+ for_each_possible_cpu(cpu) {
+ bpf_long_memcpy(per_cpu_ptr(pptr, cpu),
+ value + off, size);
+ off += size;
+ }
+ }
+}
+
static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
void *value, u32 key_size, u32 hash,
bool percpu, bool onallcpus,
@@ -479,18 +497,8 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
}
}
- if (!onallcpus) {
- /* copy true value_size bytes */
- memcpy(this_cpu_ptr(pptr), value, htab->map.value_size);
- } else {
- int off = 0, cpu;
+ pcpu_copy_value(htab, pptr, value, onallcpus);
- for_each_possible_cpu(cpu) {
- bpf_long_memcpy(per_cpu_ptr(pptr, cpu),
- value + off, size);
- off += size;
- }
- }
if (!prealloc)
htab_elem_set_ptr(l_new, key_size, pptr);
} else {
@@ -606,22 +614,9 @@ static int __htab_percpu_map_update_elem(struct bpf_map *map, void *key,
goto err;
if (l_old) {
- void __percpu *pptr = htab_elem_get_ptr(l_old, key_size);
- u32 size = htab->map.value_size;
-
/* per-cpu hash map can update value in-place */
- if (!onallcpus) {
- memcpy(this_cpu_ptr(pptr), value, size);
- } else {
- int off = 0, cpu;
-
- size = round_up(size, 8);
- for_each_possible_cpu(cpu) {
- bpf_long_memcpy(per_cpu_ptr(pptr, cpu),
- value + off, size);
- off += size;
- }
- }
+ pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size),
+ value, onallcpus);
} else {
l_new = alloc_htab_elem(htab, key, value, key_size,
hash, true, onallcpus, false);
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 2/6] bpf: Add percpu LRU list
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
Instead of having a common LRU list, this patch allows a
percpu LRU list which can be selected by specifying a map
attribute. The map attribute will be added in the later
patch.
While the common use case for LRU is #reads >> #updates,
percpu LRU list allows bpf prog to absorb unusual #updates
under pathological case (e.g. external traffic facing machine which
could be under attack).
Each percpu LRU is isolated from each other. The LRU nodes (including
free nodes) cannot be moved across different LRU Lists.
Here are the update performance comparison between
common LRU list and percpu LRU list (the test code is
at the last patch):
[root@kerneltest003.31.prn1 ~]# for i in 1 4 8; do echo -n "$i cpus: "; \
./map_perf_test 16 $i | awk '{r += $3}END{print r " updates"}'; done
1 cpus: 2934082 updates
4 cpus: 7391434 updates
8 cpus: 6500576 updates
[root@kerneltest003.31.prn1 ~]# for i in 1 4 8; do echo -n "$i cpus: "; \
./map_perf_test 32 $i | awk '{r += $3}END{printr " updates"}'; done
1 cpus: 2896553 updates
4 cpus: 9766395 updates
8 cpus: 17460553 updates
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
kernel/bpf/bpf_lru_list.c | 162 +++++++++++++++++++++++++++++++++++++++++-----
kernel/bpf/bpf_lru_list.h | 8 ++-
2 files changed, 151 insertions(+), 19 deletions(-)
diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c
index 73f6709..bfebff0 100644
--- a/kernel/bpf/bpf_lru_list.c
+++ b/kernel/bpf/bpf_lru_list.c
@@ -13,6 +13,9 @@
#define LOCAL_FREE_TARGET (128)
#define LOCAL_NR_SCANS LOCAL_FREE_TARGET
+#define PERCPU_FREE_TARGET (16)
+#define PERCPU_NR_SCANS PERCPU_FREE_TARGET
+
/* Helpers to get the local list index */
#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET)
#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
@@ -396,7 +399,40 @@ struct bpf_lru_node *__local_list_pop_pending(struct bpf_lru *lru,
return NULL;
}
-struct bpf_lru_node *bpf_lru_pop_free(struct bpf_lru *lru, u32 hash)
+static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru,
+ u32 hash)
+{
+ struct list_head *free_list;
+ struct bpf_lru_node *node = NULL;
+ struct bpf_lru_list *l;
+ unsigned long flags;
+ int cpu = raw_smp_processor_id();
+
+ l = per_cpu_ptr(lru->percpu_lru, cpu);
+
+ raw_spin_lock_irqsave(&l->lock, flags);
+
+ __bpf_lru_list_rotate(lru, l);
+
+ free_list = &l->lists[BPF_LRU_LIST_T_FREE];
+ if (list_empty(free_list))
+ __bpf_lru_list_shrink(lru, l, PERCPU_FREE_TARGET, free_list,
+ BPF_LRU_LIST_T_FREE);
+
+ if (!list_empty(free_list)) {
+ node = list_first_entry(free_list, struct bpf_lru_node, list);
+ *(u32 *)((void *)node + lru->hash_offset) = hash;
+ node->ref = 0;
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
+ }
+
+ raw_spin_unlock_irqrestore(&l->lock, flags);
+
+ return node;
+}
+
+static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
+ u32 hash)
{
struct bpf_lru_locallist *loc_l, *steal_loc_l;
struct bpf_common_lru *clru = &lru->common_lru;
@@ -458,7 +494,16 @@ struct bpf_lru_node *bpf_lru_pop_free(struct bpf_lru *lru, u32 hash)
return node;
}
-void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
+struct bpf_lru_node *bpf_lru_pop_free(struct bpf_lru *lru, u32 hash)
+{
+ if (lru->percpu)
+ return bpf_percpu_lru_pop_free(lru, hash);
+ else
+ return bpf_common_lru_pop_free(lru, hash);
+}
+
+static void bpf_common_lru_push_free(struct bpf_lru *lru,
+ struct bpf_lru_node *node)
{
unsigned long flags;
@@ -490,8 +535,31 @@ void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
bpf_lru_list_push_free(&lru->common_lru.lru_list, node);
}
-void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
- u32 elem_size, u32 nr_elems)
+static void bpf_percpu_lru_push_free(struct bpf_lru *lru,
+ struct bpf_lru_node *node)
+{
+ struct bpf_lru_list *l;
+ unsigned long flags;
+
+ l = per_cpu_ptr(lru->percpu_lru, node->cpu);
+
+ raw_spin_lock_irqsave(&l->lock, flags);
+
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
+
+ raw_spin_unlock_irqrestore(&l->lock, flags);
+}
+
+void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
+{
+ if (lru->percpu)
+ bpf_percpu_lru_push_free(lru, node);
+ else
+ bpf_common_lru_push_free(lru, node);
+}
+
+void bpf_common_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
+ u32 elem_size, u32 nr_elems)
{
struct bpf_lru_list *l = &lru->common_lru.lru_list;
u32 i;
@@ -507,6 +575,47 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
}
}
+void bpf_percpu_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
+ u32 elem_size, u32 nr_elems)
+{
+ u32 i, pcpu_entries;
+ int cpu;
+ struct bpf_lru_list *l;
+
+ pcpu_entries = nr_elems / num_possible_cpus();
+
+ i = 0;
+
+ for_each_possible_cpu(cpu) {
+ struct bpf_lru_node *node;
+
+ l = per_cpu_ptr(lru->percpu_lru, cpu);
+again:
+ node = (struct bpf_lru_node *)(buf + node_offset);
+ node->cpu = cpu;
+ node->type = BPF_LRU_LIST_T_FREE;
+ node->ref = 0;
+ list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
+ i++;
+ buf += elem_size;
+ if (i == nr_elems)
+ break;
+ if (i % pcpu_entries)
+ goto again;
+ }
+}
+
+void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
+ u32 elem_size, u32 nr_elems)
+{
+ if (lru->percpu)
+ bpf_percpu_lru_populate(lru, buf, node_offset, elem_size,
+ nr_elems);
+ else
+ bpf_common_lru_populate(lru, buf, node_offset, elem_size,
+ nr_elems);
+}
+
static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
{
int i;
@@ -534,26 +643,42 @@ static void bpf_lru_list_init(struct bpf_lru_list *l)
raw_spin_lock_init(&l->lock);
}
-int bpf_lru_init(struct bpf_lru *lru, u32 hash_offset,
+int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,
del_from_htab_func del_from_htab, void *del_arg)
{
int cpu;
- struct bpf_common_lru *clru = &lru->common_lru;
- clru->local_list = alloc_percpu(struct bpf_lru_locallist);
- if (!clru->local_list)
- return -ENOMEM;
+ if (percpu) {
+ lru->percpu_lru = alloc_percpu(struct bpf_lru_list);
+ if (!lru->percpu_lru)
+ return -ENOMEM;
- for_each_possible_cpu(cpu) {
- struct bpf_lru_locallist *loc_l;
+ for_each_possible_cpu(cpu) {
+ struct bpf_lru_list *l;
- loc_l = per_cpu_ptr(clru->local_list, cpu);
- bpf_lru_locallist_init(loc_l, cpu);
- }
+ l = per_cpu_ptr(lru->percpu_lru, cpu);
+ bpf_lru_list_init(l);
+ }
+ lru->nr_scans = PERCPU_NR_SCANS;
+ } else {
+ struct bpf_common_lru *clru = &lru->common_lru;
- bpf_lru_list_init(&clru->lru_list);
- lru->nr_scans = LOCAL_NR_SCANS;
+ clru->local_list = alloc_percpu(struct bpf_lru_locallist);
+ if (!clru->local_list)
+ return -ENOMEM;
+ for_each_possible_cpu(cpu) {
+ struct bpf_lru_locallist *loc_l;
+
+ loc_l = per_cpu_ptr(clru->local_list, cpu);
+ bpf_lru_locallist_init(loc_l, cpu);
+ }
+
+ bpf_lru_list_init(&clru->lru_list);
+ lru->nr_scans = LOCAL_NR_SCANS;
+ }
+
+ lru->percpu = percpu;
lru->del_from_htab = del_from_htab;
lru->del_arg = del_arg;
lru->hash_offset = hash_offset;
@@ -563,5 +688,8 @@ int bpf_lru_init(struct bpf_lru *lru, u32 hash_offset,
void bpf_lru_destroy(struct bpf_lru *lru)
{
- free_percpu(lru->common_lru.local_list);
+ if (lru->percpu)
+ free_percpu(lru->percpu_lru);
+ else
+ free_percpu(lru->common_lru.local_list);
}
diff --git a/kernel/bpf/bpf_lru_list.h b/kernel/bpf/bpf_lru_list.h
index aaa2445..5c35a98 100644
--- a/kernel/bpf/bpf_lru_list.h
+++ b/kernel/bpf/bpf_lru_list.h
@@ -53,11 +53,15 @@ struct bpf_common_lru {
typedef bool (*del_from_htab_func)(void *arg, struct bpf_lru_node *node);
struct bpf_lru {
- struct bpf_common_lru common_lru;
+ union {
+ struct bpf_common_lru common_lru;
+ struct bpf_lru_list __percpu *percpu_lru;
+ };
del_from_htab_func del_from_htab;
void *del_arg;
unsigned int hash_offset;
unsigned int nr_scans;
+ bool percpu;
};
static inline void bpf_lru_node_set_ref(struct bpf_lru_node *node)
@@ -68,7 +72,7 @@ static inline void bpf_lru_node_set_ref(struct bpf_lru_node *node)
node->ref = 1;
}
-int bpf_lru_init(struct bpf_lru *lru, u32 hash_offset,
+int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,
del_from_htab_func del_from_htab, void *delete_arg);
void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
u32 elem_size, u32 nr_elems);
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 1/6] bpf: LRU List
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
In-Reply-To: <1478890511-1346984-1-git-send-email-kafai@fb.com>
Introduce bpf_lru_list which will provide LRU capability to
the bpf_htab in the later patch.
* General Thoughts:
1. Target use case. Read is more often than update.
(i.e. bpf_lookup_elem() is more often than bpf_update_elem()).
If bpf_prog does a bpf_lookup_elem() first and then an in-place
update, it still counts as a read operation to the LRU list concern.
2. It may be useful to think of it as a LRU cache
3. Optimize the read case
3.1 No lock in read case
3.2 The LRU maintenance is only done during bpf_update_elem()
4. If there is a percpu LRU list, it will lose the system-wise LRU
property. A completely isolated percpu LRU list has the best
performance but the memory utilization is not ideal considering
the work load may be imbalance.
5. Hence, this patch starts the LRU implementation with a global LRU
list with batched operations before accessing the global LRU list.
As a LRU cache, #read >> #update/#insert operations, it will work well.
6. There is a local list (for each cpu) which is named
'struct bpf_lru_locallist'. This local list is not used to sort
the LRU property. Instead, the local list is to batch enough
operations before acquiring the lock of the global LRU list. More
details on this later.
7. In the later patch, it allows a percpu LRU list by specifying a
map-attribute for scalability reason and for use cases that need to
prepare for the worst (and pathological) case like DoS attack.
The percpu LRU list is completely isolated from each other and the
LRU nodes (including free nodes) cannot be moved across the list. The
following description is for the global LRU list but mostly applicable
to the percpu LRU list also.
* Global LRU List:
1. It has three sub-lists: active-list, inactive-list and free-list.
2. The two list idea, active and inactive, is borrowed from the
page cache.
3. All nodes are pre-allocated and all sit at the free-list (of the
global LRU list) at the beginning. The pre-allocation reasoning
is similar to the existing BPF_MAP_TYPE_HASH. However,
opting-out prealloc (BPF_F_NO_PREALLOC) is not supported in
the LRU map.
* Active/Inactive List (of the global LRU list):
1. The active list, as its name says it, maintains the active set of
the nodes. We can think of it as the working set or more frequently
accessed nodes. The access frequency is approximated by a ref-bit.
The ref-bit is set during the bpf_lookup_elem().
2. The inactive list, as its name also says it, maintains a less
active set of nodes. They are the candidates to be removed
from the bpf_htab when we are running out of free nodes.
3. The ordering of these two lists is acting as a rough clock.
The tail of the inactive list is the older nodes and
should be released first if the bpf_htab needs free element.
* Rotating the Active/Inactive List (of the global LRU list):
1. It is the basic operation to maintain the LRU property of
the global list.
2. The active list is only rotated when the inactive list is running
low. This idea is similar to the current page cache.
Inactive running low is currently defined as
"# of inactive < # of active".
3. The active list rotation always starts from the tail. It moves
node without ref-bit set to the head of the inactive list.
It moves node with ref-bit set back to the head of the active
list and then clears its ref-bit.
4. The inactive rotation is pretty simply.
It walks the inactive list and moves the nodes back to the head of
active list if its ref-bit is set. The ref-bit is cleared after moving
to the active list.
If the node does not have ref-bit set, it just leave it as it is
because it is already in the inactive list.
* Shrinking the Inactive List (of the global LRU list):
1. Shrinking is the operation to get free nodes when the bpf_htab is
full.
2. It usually only shrinks the inactive list to get free nodes.
3. During shrinking, it will walk the inactive list from the tail,
delete the nodes without ref-bit set from bpf_htab.
4. If no free node found after step (3), it will forcefully get
one node from the tail of inactive or active list. Forcefully is
in the sense that it ignores the ref-bit.
* Local List:
1. Each CPU has a 'struct bpf_lru_locallist'. The purpose is to
batch enough operations before acquiring the lock of the
global LRU.
2. A local list has two sub-lists, free-list and pending-list.
3. During bpf_update_elem(), it will try to get from the free-list
of (the current CPU local list).
4. If the local free-list is empty, it will acquire from the
global LRU list. The global LRU list can either satisfy it
by its global free-list or by shrinking the global inactive
list. Since we have acquired the global LRU list lock,
it will try to get at most LOCAL_FREE_TARGET elements
to the local free list.
5. When a new element is added to the bpf_htab, it will
first sit at the pending-list (of the local list) first.
The pending-list will be flushed to the global LRU list
when it needs to acquire free nodes from the global list
next time.
* Lock Consideration:
The LRU list has a lock (lru_lock). Each bucket of htab has a
lock (buck_lock). If both locks need to be acquired together,
the lock order is always lru_lock -> buck_lock and this only
happens in the bpf_lru_list.c logic.
In hashtab.c, both locks are not acquired together (i.e. one
lock is always released first before acquiring another lock).
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
---
kernel/bpf/Makefile | 2 +-
kernel/bpf/bpf_lru_list.c | 567 ++++++++++++++++++++++++++++++++++++++++++++++
kernel/bpf/bpf_lru_list.h | 80 +++++++
3 files changed, 648 insertions(+), 1 deletion(-)
create mode 100644 kernel/bpf/bpf_lru_list.c
create mode 100644 kernel/bpf/bpf_lru_list.h
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index eed911d..c4d89d6 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -1,7 +1,7 @@
obj-y := core.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o
-obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o
+obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o
ifeq ($(CONFIG_PERF_EVENTS),y)
obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
endif
diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c
new file mode 100644
index 0000000..73f6709
--- /dev/null
+++ b/kernel/bpf/bpf_lru_list.c
@@ -0,0 +1,567 @@
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/cpumask.h>
+#include <linux/spinlock.h>
+#include <linux/percpu.h>
+
+#include "bpf_lru_list.h"
+
+#define LOCAL_FREE_TARGET (128)
+#define LOCAL_NR_SCANS LOCAL_FREE_TARGET
+
+/* Helpers to get the local list index */
+#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET)
+#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
+#define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING)
+#define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET)
+
+static int get_next_cpu(int cpu)
+{
+ cpu = cpumask_next(cpu, cpu_possible_mask);
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_first(cpu_possible_mask);
+ return cpu;
+}
+
+/* Local list helpers */
+static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l)
+{
+ return &loc_l->lists[LOCAL_FREE_LIST_IDX];
+}
+
+static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l)
+{
+ return &loc_l->lists[LOCAL_PENDING_LIST_IDX];
+}
+
+/* bpf_lru_node helpers */
+static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)
+{
+ return node->ref;
+}
+
+static void bpf_lru_list_count_inc(struct bpf_lru_list *l,
+ enum bpf_lru_list_type type)
+{
+ if (type < NR_BPF_LRU_LIST_COUNT)
+ l->counts[type]++;
+}
+
+static void bpf_lru_list_count_dec(struct bpf_lru_list *l,
+ enum bpf_lru_list_type type)
+{
+ if (type < NR_BPF_LRU_LIST_COUNT)
+ l->counts[type]--;
+}
+
+static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l,
+ struct bpf_lru_node *node,
+ struct list_head *free_list,
+ enum bpf_lru_list_type tgt_free_type)
+{
+ if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
+ return;
+
+ /* If the removing node is the next_inactive_rotation candidate,
+ * move the next_inactive_rotation pointer also.
+ */
+ if (&node->list == l->next_inactive_rotation)
+ l->next_inactive_rotation = l->next_inactive_rotation->prev;
+
+ bpf_lru_list_count_dec(l, node->type);
+
+ node->type = tgt_free_type;
+ list_move(&node->list, free_list);
+}
+
+/* Move nodes from local list to the LRU list */
+static void __bpf_lru_node_move_in(struct bpf_lru_list *l,
+ struct bpf_lru_node *node,
+ enum bpf_lru_list_type tgt_type)
+{
+ if (WARN_ON_ONCE(!IS_LOCAL_LIST_TYPE(node->type)) ||
+ WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(tgt_type)))
+ return;
+
+ bpf_lru_list_count_inc(l, tgt_type);
+ node->type = tgt_type;
+ node->ref = 0;
+ list_move(&node->list, &l->lists[tgt_type]);
+}
+
+/* Move nodes between or within active and inactive list (like
+ * active to inactive, inactive to active or tail of active back to
+ * the head of active).
+ */
+static void __bpf_lru_node_move(struct bpf_lru_list *l,
+ struct bpf_lru_node *node,
+ enum bpf_lru_list_type tgt_type)
+{
+ if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)) ||
+ WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(tgt_type)))
+ return;
+
+ if (node->type != tgt_type) {
+ bpf_lru_list_count_dec(l, node->type);
+ bpf_lru_list_count_inc(l, tgt_type);
+ node->type = tgt_type;
+ }
+ node->ref = 0;
+
+ /* If the moving node is the next_inactive_rotation candidate,
+ * move the next_inactive_rotation pointer also.
+ */
+ if (&node->list == l->next_inactive_rotation)
+ l->next_inactive_rotation = l->next_inactive_rotation->prev;
+
+ list_move(&node->list, &l->lists[tgt_type]);
+}
+
+static bool bpf_lru_list_inactive_low(const struct bpf_lru_list *l)
+{
+ return l->counts[BPF_LRU_LIST_T_INACTIVE] <
+ l->counts[BPF_LRU_LIST_T_ACTIVE];
+}
+
+/* Rotate the active list:
+ * 1. Start from tail
+ * 2. If the node has the ref bit set, it will be rotated
+ * back to the head of active list with the ref bit cleared.
+ * Give this node one more chance to survive in the active list.
+ * 3. If the ref bit is not set, move it to the head of the
+ * inactive list.
+ * 4. It will at most scan nr_scans nodes
+ */
+static void __bpf_lru_list_rotate_active(struct bpf_lru *lru,
+ struct bpf_lru_list *l)
+{
+ struct list_head *active = &l->lists[BPF_LRU_LIST_T_ACTIVE];
+ struct bpf_lru_node *node, *tmp_node, *first_node;
+ unsigned int i = 0;
+
+ first_node = list_first_entry(active, struct bpf_lru_node, list);
+ list_for_each_entry_safe_reverse(node, tmp_node, active, list) {
+ if (bpf_lru_node_is_ref(node))
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
+ else
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
+
+ if (++i == lru->nr_scans || node == first_node)
+ break;
+ }
+}
+
+/* Rotate the inactive list. It starts from the next_inactive_rotation
+ * 1. If the node has ref bit set, it will be moved to the head
+ * of active list with the ref bit cleared.
+ * 2. If the node does not have ref bit set, it will leave it
+ * at its current location (i.e. do nothing) so that it can
+ * be considered during the next inactive_shrink.
+ * 3. It will at most scan nr_scans nodes
+ */
+static void __bpf_lru_list_rotate_inactive(struct bpf_lru *lru,
+ struct bpf_lru_list *l)
+{
+ struct list_head *inactive = &l->lists[BPF_LRU_LIST_T_INACTIVE];
+ struct list_head *cur, *next, *last;
+ struct bpf_lru_node *node;
+ unsigned int i = 0;
+
+ if (list_empty(inactive))
+ return;
+
+ last = l->next_inactive_rotation->next;
+ if (last == inactive)
+ last = last->next;
+
+ cur = l->next_inactive_rotation;
+ while (i < lru->nr_scans) {
+ if (cur == inactive) {
+ cur = cur->prev;
+ continue;
+ }
+
+ node = list_entry(cur, struct bpf_lru_node, list);
+ next = cur->prev;
+ if (bpf_lru_node_is_ref(node))
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
+ if (cur == last)
+ break;
+ cur = next;
+ i++;
+ }
+
+ l->next_inactive_rotation = next;
+}
+
+/* Shrink the inactive list. It starts from the tail of the
+ * inactive list and only move the nodes without the ref bit
+ * set to the designated free list.
+ */
+static unsigned int
+__bpf_lru_list_shrink_inactive(struct bpf_lru *lru,
+ struct bpf_lru_list *l,
+ unsigned int tgt_nshrink,
+ struct list_head *free_list,
+ enum bpf_lru_list_type tgt_free_type)
+{
+ struct list_head *inactive = &l->lists[BPF_LRU_LIST_T_INACTIVE];
+ struct bpf_lru_node *node, *tmp_node, *first_node;
+ unsigned int nshrinked = 0;
+ unsigned int i = 0;
+
+ first_node = list_first_entry(inactive, struct bpf_lru_node, list);
+ list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {
+ if (bpf_lru_node_is_ref(node)) {
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
+ } else if (lru->del_from_htab(lru->del_arg, node)) {
+ __bpf_lru_node_move_to_free(l, node, free_list,
+ tgt_free_type);
+ if (++nshrinked == tgt_nshrink)
+ break;
+ }
+
+ if (++i == lru->nr_scans)
+ break;
+ }
+
+ return nshrinked;
+}
+
+/* 1. Rotate the active list (if needed)
+ * 2. Always rotate the inactive list
+ */
+static void __bpf_lru_list_rotate(struct bpf_lru *lru, struct bpf_lru_list *l)
+{
+ if (bpf_lru_list_inactive_low(l))
+ __bpf_lru_list_rotate_active(lru, l);
+
+ __bpf_lru_list_rotate_inactive(lru, l);
+}
+
+/* Calls __bpf_lru_list_shrink_inactive() to shrink some
+ * ref-bit-cleared nodes and move them to the designated
+ * free list.
+ *
+ * If it cannot get a free node after calling
+ * __bpf_lru_list_shrink_inactive(). It will just remove
+ * one node from either inactive or active list without
+ * honoring the ref-bit. It prefers inactive list to active
+ * list in this situation.
+ */
+static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru,
+ struct bpf_lru_list *l,
+ unsigned int tgt_nshrink,
+ struct list_head *free_list,
+ enum bpf_lru_list_type tgt_free_type)
+
+{
+ struct bpf_lru_node *node, *tmp_node;
+ struct list_head *force_shrink_list;
+ unsigned int nshrinked;
+
+ nshrinked = __bpf_lru_list_shrink_inactive(lru, l, tgt_nshrink,
+ free_list, tgt_free_type);
+ if (nshrinked)
+ return nshrinked;
+
+ /* Do a force shrink by ignoring the reference bit */
+ if (!list_empty(&l->lists[BPF_LRU_LIST_T_INACTIVE]))
+ force_shrink_list = &l->lists[BPF_LRU_LIST_T_INACTIVE];
+ else
+ force_shrink_list = &l->lists[BPF_LRU_LIST_T_ACTIVE];
+
+ list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,
+ list) {
+ if (lru->del_from_htab(lru->del_arg, node)) {
+ __bpf_lru_node_move_to_free(l, node, free_list,
+ tgt_free_type);
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+/* Flush the nodes from the local pending list to the LRU list */
+static void __local_list_flush(struct bpf_lru_list *l,
+ struct bpf_lru_locallist *loc_l)
+{
+ struct bpf_lru_node *node, *tmp_node;
+
+ list_for_each_entry_safe_reverse(node, tmp_node,
+ local_pending_list(loc_l), list) {
+ if (bpf_lru_node_is_ref(node))
+ __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
+ else
+ __bpf_lru_node_move_in(l, node,
+ BPF_LRU_LIST_T_INACTIVE);
+ }
+}
+
+static void bpf_lru_list_push_free(struct bpf_lru_list *l,
+ struct bpf_lru_node *node)
+{
+ unsigned long flags;
+
+ if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
+ return;
+
+ raw_spin_lock_irqsave(&l->lock, flags);
+ __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
+ raw_spin_unlock_irqrestore(&l->lock, flags);
+}
+
+static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
+ struct bpf_lru_locallist *loc_l)
+{
+ struct bpf_lru_list *l = &lru->common_lru.lru_list;
+ struct bpf_lru_node *node, *tmp_node;
+ unsigned int nfree = 0;
+
+ raw_spin_lock(&l->lock);
+
+ __local_list_flush(l, loc_l);
+
+ __bpf_lru_list_rotate(lru, l);
+
+ list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],
+ list) {
+ __bpf_lru_node_move_to_free(l, node, local_free_list(loc_l),
+ BPF_LRU_LOCAL_LIST_T_FREE);
+ if (++nfree == LOCAL_FREE_TARGET)
+ break;
+ }
+
+ if (nfree < LOCAL_FREE_TARGET)
+ __bpf_lru_list_shrink(lru, l, LOCAL_FREE_TARGET - nfree,
+ local_free_list(loc_l),
+ BPF_LRU_LOCAL_LIST_T_FREE);
+
+ raw_spin_unlock(&l->lock);
+}
+
+static void __local_list_add_pending(struct bpf_lru *lru,
+ struct bpf_lru_locallist *loc_l,
+ int cpu,
+ struct bpf_lru_node *node,
+ u32 hash)
+{
+ *(u32 *)((void *)node + lru->hash_offset) = hash;
+ node->cpu = cpu;
+ node->type = BPF_LRU_LOCAL_LIST_T_PENDING;
+ node->ref = 0;
+ list_add(&node->list, local_pending_list(loc_l));
+}
+
+struct bpf_lru_node *__local_list_pop_free(struct bpf_lru_locallist *loc_l)
+{
+ struct bpf_lru_node *node;
+
+ node = list_first_entry_or_null(local_free_list(loc_l),
+ struct bpf_lru_node,
+ list);
+ if (node)
+ list_del(&node->list);
+
+ return node;
+}
+
+struct bpf_lru_node *__local_list_pop_pending(struct bpf_lru *lru,
+ struct bpf_lru_locallist *loc_l)
+{
+ struct bpf_lru_node *node;
+ bool force = false;
+
+ignore_ref:
+ /* Get from the tail (i.e. older element) of the pending list. */
+ list_for_each_entry_reverse(node, local_pending_list(loc_l),
+ list) {
+ if ((!bpf_lru_node_is_ref(node) || force) &&
+ lru->del_from_htab(lru->del_arg, node)) {
+ list_del(&node->list);
+ return node;
+ }
+ }
+
+ if (!force) {
+ force = true;
+ goto ignore_ref;
+ }
+
+ return NULL;
+}
+
+struct bpf_lru_node *bpf_lru_pop_free(struct bpf_lru *lru, u32 hash)
+{
+ struct bpf_lru_locallist *loc_l, *steal_loc_l;
+ struct bpf_common_lru *clru = &lru->common_lru;
+ struct bpf_lru_node *node;
+ int steal, first_steal;
+ unsigned long flags;
+ int cpu = raw_smp_processor_id();
+
+ loc_l = per_cpu_ptr(clru->local_list, cpu);
+
+ raw_spin_lock_irqsave(&loc_l->lock, flags);
+
+ node = __local_list_pop_free(loc_l);
+ if (!node) {
+ bpf_lru_list_pop_free_to_local(lru, loc_l);
+ node = __local_list_pop_free(loc_l);
+ }
+
+ if (node)
+ __local_list_add_pending(lru, loc_l, cpu, node, hash);
+
+ raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+
+ if (node)
+ return node;
+
+ /* No free nodes found from the local free list and
+ * the global LRU list.
+ *
+ * Steal from the local free/pending list of the
+ * current CPU and remote CPU in RR. It starts
+ * with the loc_l->next_steal CPU.
+ */
+
+ first_steal = loc_l->next_steal;
+ steal = first_steal;
+ do {
+ steal_loc_l = per_cpu_ptr(clru->local_list, steal);
+
+ raw_spin_lock_irqsave(&steal_loc_l->lock, flags);
+
+ node = __local_list_pop_free(steal_loc_l);
+ if (!node)
+ node = __local_list_pop_pending(lru, steal_loc_l);
+
+ raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
+
+ steal = get_next_cpu(steal);
+ } while (!node && steal != first_steal);
+
+ loc_l->next_steal = steal;
+
+ if (node) {
+ raw_spin_lock_irqsave(&loc_l->lock, flags);
+ __local_list_add_pending(lru, loc_l, cpu, node, hash);
+ raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+ }
+
+ return node;
+}
+
+void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
+{
+ unsigned long flags;
+
+ if (WARN_ON_ONCE(node->type == BPF_LRU_LIST_T_FREE) ||
+ WARN_ON_ONCE(node->type == BPF_LRU_LOCAL_LIST_T_FREE))
+ return;
+
+ if (node->type == BPF_LRU_LOCAL_LIST_T_PENDING) {
+ struct bpf_lru_locallist *loc_l;
+
+ loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);
+
+ raw_spin_lock_irqsave(&loc_l->lock, flags);
+
+ if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
+ raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+ goto check_lru_list;
+ }
+
+ node->type = BPF_LRU_LOCAL_LIST_T_FREE;
+ node->ref = 0;
+ list_move(&node->list, local_free_list(loc_l));
+
+ raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+ return;
+ }
+
+check_lru_list:
+ bpf_lru_list_push_free(&lru->common_lru.lru_list, node);
+}
+
+void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
+ u32 elem_size, u32 nr_elems)
+{
+ struct bpf_lru_list *l = &lru->common_lru.lru_list;
+ u32 i;
+
+ for (i = 0; i < nr_elems; i++) {
+ struct bpf_lru_node *node;
+
+ node = (struct bpf_lru_node *)(buf + node_offset);
+ node->type = BPF_LRU_LIST_T_FREE;
+ node->ref = 0;
+ list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
+ buf += elem_size;
+ }
+}
+
+static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
+{
+ int i;
+
+ for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++)
+ INIT_LIST_HEAD(&loc_l->lists[i]);
+
+ loc_l->next_steal = cpu;
+
+ raw_spin_lock_init(&loc_l->lock);
+}
+
+static void bpf_lru_list_init(struct bpf_lru_list *l)
+{
+ int i;
+
+ for (i = 0; i < NR_BPF_LRU_LIST_T; i++)
+ INIT_LIST_HEAD(&l->lists[i]);
+
+ for (i = 0; i < NR_BPF_LRU_LIST_COUNT; i++)
+ l->counts[i] = 0;
+
+ l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];
+
+ raw_spin_lock_init(&l->lock);
+}
+
+int bpf_lru_init(struct bpf_lru *lru, u32 hash_offset,
+ del_from_htab_func del_from_htab, void *del_arg)
+{
+ int cpu;
+ struct bpf_common_lru *clru = &lru->common_lru;
+
+ clru->local_list = alloc_percpu(struct bpf_lru_locallist);
+ if (!clru->local_list)
+ return -ENOMEM;
+
+ for_each_possible_cpu(cpu) {
+ struct bpf_lru_locallist *loc_l;
+
+ loc_l = per_cpu_ptr(clru->local_list, cpu);
+ bpf_lru_locallist_init(loc_l, cpu);
+ }
+
+ bpf_lru_list_init(&clru->lru_list);
+ lru->nr_scans = LOCAL_NR_SCANS;
+
+ lru->del_from_htab = del_from_htab;
+ lru->del_arg = del_arg;
+ lru->hash_offset = hash_offset;
+
+ return 0;
+}
+
+void bpf_lru_destroy(struct bpf_lru *lru)
+{
+ free_percpu(lru->common_lru.local_list);
+}
diff --git a/kernel/bpf/bpf_lru_list.h b/kernel/bpf/bpf_lru_list.h
new file mode 100644
index 0000000..aaa2445
--- /dev/null
+++ b/kernel/bpf/bpf_lru_list.h
@@ -0,0 +1,80 @@
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#ifndef __BPF_LRU_LIST_H_
+#define __BPF_LRU_LIST_H_
+
+#include <linux/list.h>
+#include <linux/spinlock_types.h>
+
+#define NR_BPF_LRU_LIST_T (3)
+#define NR_BPF_LRU_LIST_COUNT (2)
+#define NR_BPF_LRU_LOCAL_LIST_T (2)
+#define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T
+
+enum bpf_lru_list_type {
+ BPF_LRU_LIST_T_ACTIVE,
+ BPF_LRU_LIST_T_INACTIVE,
+ BPF_LRU_LIST_T_FREE,
+ BPF_LRU_LOCAL_LIST_T_FREE,
+ BPF_LRU_LOCAL_LIST_T_PENDING,
+};
+
+struct bpf_lru_node {
+ struct list_head list;
+ u16 cpu;
+ u8 type;
+ u8 ref;
+};
+
+struct bpf_lru_list {
+ struct list_head lists[NR_BPF_LRU_LIST_T];
+ unsigned int counts[NR_BPF_LRU_LIST_COUNT];
+ /* The next inacitve list rotation starts from here */
+ struct list_head *next_inactive_rotation;
+
+ raw_spinlock_t lock ____cacheline_aligned_in_smp;
+};
+
+struct bpf_lru_locallist {
+ struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T];
+ u16 next_steal;
+ raw_spinlock_t lock;
+};
+
+struct bpf_common_lru {
+ struct bpf_lru_list lru_list;
+ struct bpf_lru_locallist __percpu *local_list;
+};
+
+typedef bool (*del_from_htab_func)(void *arg, struct bpf_lru_node *node);
+
+struct bpf_lru {
+ struct bpf_common_lru common_lru;
+ del_from_htab_func del_from_htab;
+ void *del_arg;
+ unsigned int hash_offset;
+ unsigned int nr_scans;
+};
+
+static inline void bpf_lru_node_set_ref(struct bpf_lru_node *node)
+{
+ /* ref is an approximation on access frequency. It does not
+ * have to be very accurate. Hence, no protection is used.
+ */
+ node->ref = 1;
+}
+
+int bpf_lru_init(struct bpf_lru *lru, u32 hash_offset,
+ del_from_htab_func del_from_htab, void *delete_arg);
+void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
+ u32 elem_size, u32 nr_elems);
+void bpf_lru_destroy(struct bpf_lru *lru);
+struct bpf_lru_node *bpf_lru_pop_free(struct bpf_lru *lru, u32 hash);
+void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node);
+void bpf_lru_promote(struct bpf_lru *lru, struct bpf_lru_node *node);
+
+#endif
--
2.5.1
^ permalink raw reply related
* [PATCH v2 net-next 0/6] bpf: LRU map
From: Martin KaFai Lau @ 2016-11-11 18:55 UTC (permalink / raw)
To: netdev, David Miller; +Cc: Alexei Starovoitov, Daniel Borkmann, Kernel Team
Hi,
This patch set adds LRU map implementation to the existing BPF map
family.
The first few patches introduce the basic BPF LRU list
implementation.
The later patches introduce the LRU versions of the
existing BPF_MAP_TYPE_LRU_[PERCPU_]HASH maps by leveraging
the BPF LRU list.
v2:
- Added a percpu LRU list option which can be specified as
a map attribute.
[Note: percpu LRU list has nothing to do with the map's value]
- Removed the cpu variable from the struct bpf_lru_locallist
since it is not needed.
- Changed the __bpf_lru_node_move_out to __bpf_lru_node_move_to_free in
patch 1 to prepare the percpu LRU list in patch 2.
- Moved the test_lru_map under selftests
- Refactored a few things in the test codes
Thanks,
-- Martin
^ permalink raw reply
* [Patch net-next] net: fix sleeping for sk_wait_event()
From: Cong Wang @ 2016-11-11 18:20 UTC (permalink / raw)
To: netdev; +Cc: Cong Wang, Eric Dumazet, Peter Zijlstra
Similar to commit 14135f30e33c ("inet: fix sleeping inside inet_wait_for_connect()"),
sk_wait_event() needs to fix too, because release_sock() is blocking,
it changes the process state back to running after sleep, which breaks
the previous prepare_to_wait().
Switch to the new wait API.
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
---
crypto/algif_aead.c | 9 ++++-----
crypto/algif_skcipher.c | 18 +++++++++---------
include/net/sock.h | 8 +++++---
net/core/sock.c | 8 ++++----
net/core/stream.c | 28 ++++++++++++++--------------
net/decnet/af_decnet.c | 16 ++++++++--------
net/llc/af_llc.c | 24 ++++++++++++------------
net/phonet/pep.c | 9 ++++-----
net/tipc/socket.c | 24 ++++++++++++------------
net/vmw_vsock/virtio_transport_common.c | 10 +++++-----
10 files changed, 77 insertions(+), 77 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 80a0f1a..8948392 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -132,28 +132,27 @@ static void aead_wmem_wakeup(struct sock *sk)
static int aead_wait_for_data(struct sock *sk, unsigned flags)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
long timeout;
- DEFINE_WAIT(wait);
int err = -ERESTARTSYS;
if (flags & MSG_DONTWAIT)
return -EAGAIN;
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-
+ add_wait_queue(sk_sleep(sk), &wait);
for (;;) {
if (signal_pending(current))
break;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
timeout = MAX_SCHEDULE_TIMEOUT;
- if (sk_wait_event(sk, &timeout, !ctx->more)) {
+ if (sk_wait_event(sk, &timeout, !ctx->more, &wait)) {
err = 0;
break;
}
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index 28556fc..1e38aaa 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -199,26 +199,26 @@ static void skcipher_free_sgl(struct sock *sk)
static int skcipher_wait_for_wmem(struct sock *sk, unsigned flags)
{
- long timeout;
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
int err = -ERESTARTSYS;
+ long timeout;
if (flags & MSG_DONTWAIT)
return -EAGAIN;
sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
+ add_wait_queue(sk_sleep(sk), &wait);
for (;;) {
if (signal_pending(current))
break;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
timeout = MAX_SCHEDULE_TIMEOUT;
- if (sk_wait_event(sk, &timeout, skcipher_writable(sk))) {
+ if (sk_wait_event(sk, &timeout, skcipher_writable(sk), &wait)) {
err = 0;
break;
}
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return err;
}
@@ -242,10 +242,10 @@ static void skcipher_wmem_wakeup(struct sock *sk)
static int skcipher_wait_for_data(struct sock *sk, unsigned flags)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
long timeout;
- DEFINE_WAIT(wait);
int err = -ERESTARTSYS;
if (flags & MSG_DONTWAIT) {
@@ -254,17 +254,17 @@ static int skcipher_wait_for_data(struct sock *sk, unsigned flags)
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
+ add_wait_queue(sk_sleep(sk), &wait);
for (;;) {
if (signal_pending(current))
break;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
timeout = MAX_SCHEDULE_TIMEOUT;
- if (sk_wait_event(sk, &timeout, ctx->used)) {
+ if (sk_wait_event(sk, &timeout, ctx->used, &wait)) {
err = 0;
break;
}
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
diff --git a/include/net/sock.h b/include/net/sock.h
index cf617ee..9d905ed 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -915,14 +915,16 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
#endif
}
-#define sk_wait_event(__sk, __timeo, __condition) \
+#define sk_wait_event(__sk, __timeo, __condition, __wait) \
({ int __rc; \
release_sock(__sk); \
__rc = __condition; \
if (!__rc) { \
- *(__timeo) = schedule_timeout(*(__timeo)); \
+ *(__timeo) = wait_woken(__wait, \
+ TASK_INTERRUPTIBLE, \
+ *(__timeo)); \
} \
- sched_annotate_sleep(); \
+ sched_annotate_sleep(); \
lock_sock(__sk); \
__rc = __condition; \
__rc; \
diff --git a/net/core/sock.c b/net/core/sock.c
index 40dbc13..0397928 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2078,14 +2078,14 @@ void __sk_flush_backlog(struct sock *sk)
*/
int sk_wait_data(struct sock *sk, long *timeo, const struct sk_buff *skb)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
int rc;
- DEFINE_WAIT(wait);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
- rc = sk_wait_event(sk, timeo, skb_peek_tail(&sk->sk_receive_queue) != skb);
+ rc = sk_wait_event(sk, timeo, skb_peek_tail(&sk->sk_receive_queue) != skb, &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return rc;
}
EXPORT_SYMBOL(sk_wait_data);
diff --git a/net/core/stream.c b/net/core/stream.c
index 1086c8b..f575bcf 100644
--- a/net/core/stream.c
+++ b/net/core/stream.c
@@ -53,8 +53,8 @@ void sk_stream_write_space(struct sock *sk)
*/
int sk_stream_wait_connect(struct sock *sk, long *timeo_p)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct task_struct *tsk = current;
- DEFINE_WAIT(wait);
int done;
do {
@@ -68,13 +68,13 @@ int sk_stream_wait_connect(struct sock *sk, long *timeo_p)
if (signal_pending(tsk))
return sock_intr_errno(*timeo_p);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
sk->sk_write_pending++;
done = sk_wait_event(sk, timeo_p,
!sk->sk_err &&
!((1 << sk->sk_state) &
- ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)));
- finish_wait(sk_sleep(sk), &wait);
+ ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
sk->sk_write_pending--;
} while (!done);
return 0;
@@ -94,16 +94,16 @@ static inline int sk_stream_closing(struct sock *sk)
void sk_stream_wait_close(struct sock *sk, long timeout)
{
if (timeout) {
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+
+ add_wait_queue(sk_sleep(sk), &wait);
do {
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
- if (sk_wait_event(sk, &timeout, !sk_stream_closing(sk)))
+ if (sk_wait_event(sk, &timeout, !sk_stream_closing(sk), &wait))
break;
} while (!signal_pending(current) && timeout);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
}
}
EXPORT_SYMBOL(sk_stream_wait_close);
@@ -119,16 +119,16 @@ int sk_stream_wait_memory(struct sock *sk, long *timeo_p)
long vm_wait = 0;
long current_timeo = *timeo_p;
bool noblock = (*timeo_p ? false : true);
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
if (sk_stream_memory_free(sk))
current_timeo = vm_wait = (prandom_u32() % (HZ / 5)) + 2;
+ add_wait_queue(sk_sleep(sk), &wait);
+
while (1) {
sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
-
if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN))
goto do_error;
if (!*timeo_p) {
@@ -147,7 +147,7 @@ int sk_stream_wait_memory(struct sock *sk, long *timeo_p)
sk_wait_event(sk, ¤t_timeo, sk->sk_err ||
(sk->sk_shutdown & SEND_SHUTDOWN) ||
(sk_stream_memory_free(sk) &&
- !vm_wait));
+ !vm_wait), &wait);
sk->sk_write_pending--;
if (vm_wait) {
@@ -161,7 +161,7 @@ int sk_stream_wait_memory(struct sock *sk, long *timeo_p)
*timeo_p = current_timeo;
}
out:
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return err;
do_error:
diff --git a/net/decnet/af_decnet.c b/net/decnet/af_decnet.c
index 13d6b1a..a90ed67 100644
--- a/net/decnet/af_decnet.c
+++ b/net/decnet/af_decnet.c
@@ -1718,7 +1718,7 @@ static int dn_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,
* See if there is data ready to read, sleep if there isn't
*/
for(;;) {
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
if (sk->sk_err)
goto out;
@@ -1749,11 +1749,11 @@ static int dn_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,
goto out;
}
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
- sk_wait_event(sk, &timeo, dn_data_ready(sk, queue, flags, target));
+ sk_wait_event(sk, &timeo, dn_data_ready(sk, queue, flags, target), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
}
skb_queue_walk_safe(queue, skb, n) {
@@ -1999,19 +1999,19 @@ static int dn_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
* size.
*/
if (dn_queue_too_long(scp, queue, flags)) {
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
if (flags & MSG_DONTWAIT) {
err = -EWOULDBLOCK;
goto out;
}
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
sk_wait_event(sk, &timeo,
- !dn_queue_too_long(scp, queue, flags));
+ !dn_queue_too_long(scp, queue, flags), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
continue;
}
diff --git a/net/llc/af_llc.c b/net/llc/af_llc.c
index db916cf..5e92963 100644
--- a/net/llc/af_llc.c
+++ b/net/llc/af_llc.c
@@ -532,12 +532,12 @@ static int llc_ui_listen(struct socket *sock, int backlog)
static int llc_ui_wait_for_disc(struct sock *sk, long timeout)
{
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
int rc = 0;
+ add_wait_queue(sk_sleep(sk), &wait);
while (1) {
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
- if (sk_wait_event(sk, &timeout, sk->sk_state == TCP_CLOSE))
+ if (sk_wait_event(sk, &timeout, sk->sk_state == TCP_CLOSE, &wait))
break;
rc = -ERESTARTSYS;
if (signal_pending(current))
@@ -547,39 +547,39 @@ static int llc_ui_wait_for_disc(struct sock *sk, long timeout)
break;
rc = 0;
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return rc;
}
static bool llc_ui_wait_for_conn(struct sock *sk, long timeout)
{
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+ add_wait_queue(sk_sleep(sk), &wait);
while (1) {
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
- if (sk_wait_event(sk, &timeout, sk->sk_state != TCP_SYN_SENT))
+ if (sk_wait_event(sk, &timeout, sk->sk_state != TCP_SYN_SENT, &wait))
break;
if (signal_pending(current) || !timeout)
break;
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return timeout;
}
static int llc_ui_wait_for_busy_core(struct sock *sk, long timeout)
{
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct llc_sock *llc = llc_sk(sk);
int rc;
+ add_wait_queue(sk_sleep(sk), &wait);
while (1) {
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
rc = 0;
if (sk_wait_event(sk, &timeout,
(sk->sk_shutdown & RCV_SHUTDOWN) ||
(!llc_data_accept_state(llc->state) &&
!llc->remote_busy_flag &&
- !llc->p_flag)))
+ !llc->p_flag), &wait))
break;
rc = -ERESTARTSYS;
if (signal_pending(current))
@@ -588,7 +588,7 @@ static int llc_ui_wait_for_busy_core(struct sock *sk, long timeout)
if (!timeout)
break;
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
return rc;
}
diff --git a/net/phonet/pep.c b/net/phonet/pep.c
index 850a86c..8bad562 100644
--- a/net/phonet/pep.c
+++ b/net/phonet/pep.c
@@ -1167,7 +1167,7 @@ static int pep_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
/* Wait until flow control allows TX */
done = atomic_read(&pn->tx_credits);
while (!done) {
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
if (!timeo) {
err = -EAGAIN;
@@ -1178,10 +1178,9 @@ static int pep_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
goto out;
}
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
- done = sk_wait_event(sk, &timeo, atomic_read(&pn->tx_credits));
- finish_wait(sk_sleep(sk), &wait);
+ add_wait_queue(sk_sleep(sk), &wait);
+ done = sk_wait_event(sk, &timeo, atomic_read(&pn->tx_credits), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
if (sk->sk_state != TCP_ESTABLISHED)
goto disabled;
diff --git a/net/tipc/socket.c b/net/tipc/socket.c
index 1493963..22d92f0 100644
--- a/net/tipc/socket.c
+++ b/net/tipc/socket.c
@@ -876,9 +876,9 @@ static void tipc_sk_proto_rcv(struct tipc_sock *tsk, struct sk_buff *skb,
static int tipc_wait_for_sndmsg(struct socket *sock, long *timeo_p)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct sock *sk = sock->sk;
struct tipc_sock *tsk = tipc_sk(sk);
- DEFINE_WAIT(wait);
int done;
do {
@@ -892,9 +892,9 @@ static int tipc_wait_for_sndmsg(struct socket *sock, long *timeo_p)
if (signal_pending(current))
return sock_intr_errno(*timeo_p);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
- done = sk_wait_event(sk, timeo_p, !tsk->link_cong);
- finish_wait(sk_sleep(sk), &wait);
+ add_wait_queue(sk_sleep(sk), &wait);
+ done = sk_wait_event(sk, timeo_p, !tsk->link_cong, &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
} while (!done);
return 0;
}
@@ -1031,9 +1031,9 @@ static int __tipc_sendmsg(struct socket *sock, struct msghdr *m, size_t dsz)
static int tipc_wait_for_sndpkt(struct socket *sock, long *timeo_p)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct sock *sk = sock->sk;
struct tipc_sock *tsk = tipc_sk(sk);
- DEFINE_WAIT(wait);
int done;
do {
@@ -1049,12 +1049,12 @@ static int tipc_wait_for_sndpkt(struct socket *sock, long *timeo_p)
if (signal_pending(current))
return sock_intr_errno(*timeo_p);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
done = sk_wait_event(sk, timeo_p,
(!tsk->link_cong &&
!tsk_conn_cong(tsk)) ||
- !tipc_sk_connected(sk));
- finish_wait(sk_sleep(sk), &wait);
+ !tipc_sk_connected(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
} while (!done);
return 0;
}
@@ -1929,8 +1929,8 @@ void tipc_sk_rcv(struct net *net, struct sk_buff_head *inputq)
static int tipc_wait_for_connect(struct socket *sock, long *timeo_p)
{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
struct sock *sk = sock->sk;
- DEFINE_WAIT(wait);
int done;
do {
@@ -1942,10 +1942,10 @@ static int tipc_wait_for_connect(struct socket *sock, long *timeo_p)
if (signal_pending(current))
return sock_intr_errno(*timeo_p);
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
done = sk_wait_event(sk, timeo_p,
- sk->sk_state != TIPC_CONNECTING);
- finish_wait(sk_sleep(sk), &wait);
+ sk->sk_state != TIPC_CONNECTING, &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
} while (!done);
return 0;
}
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index a53b3a1..687e9fd 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -619,17 +619,17 @@ static int virtio_transport_reset_no_sock(struct virtio_vsock_pkt *pkt)
static void virtio_transport_wait_close(struct sock *sk, long timeout)
{
if (timeout) {
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+
+ add_wait_queue(sk_sleep(sk), &wait);
do {
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
if (sk_wait_event(sk, &timeout,
- sock_flag(sk, SOCK_DONE)))
+ sock_flag(sk, SOCK_DONE), &wait))
break;
} while (!signal_pending(current) && timeout);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
}
}
--
2.1.0
^ permalink raw reply related
* Re: [PATCH net-next] ibmveth: v1 calculate correct gso_size and set gso_type
From: Brian King @ 2016-11-11 18:16 UTC (permalink / raw)
To: Eric Dumazet, Jon Maxwell
Cc: tlfalcon, tom, netdev, jmaxwell, linux-kernel, jarod, paulus,
hofrat, mleitner, linuxppc-dev, davem
In-Reply-To: <1477582016.7065.212.camel@edumazet-glaptop3.roam.corp.google.com>
On 10/27/2016 10:26 AM, Eric Dumazet wrote:
> On Wed, 2016-10-26 at 11:09 +1100, Jon Maxwell wrote:
>> We recently encountered a bug where a few customers using ibmveth on the
>> same LPAR hit an issue where a TCP session hung when large receive was
>> enabled. Closer analysis revealed that the session was stuck because the
>> one side was advertising a zero window repeatedly.
>>
>> We narrowed this down to the fact the ibmveth driver did not set gso_size
>> which is translated by TCP into the MSS later up the stack. The MSS is
>> used to calculate the TCP window size and as that was abnormally large,
>> it was calculating a zero window, even although the sockets receive buffer
>> was completely empty.
>>
>> We were able to reproduce this and worked with IBM to fix this. Thanks Tom
>> and Marcelo for all your help and review on this.
>>
>> The patch fixes both our internal reproduction tests and our customers tests.
>>
>> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
>> ---
>> drivers/net/ethernet/ibm/ibmveth.c | 20 ++++++++++++++++++++
>> 1 file changed, 20 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
>> index 29c05d0..c51717e 100644
>> --- a/drivers/net/ethernet/ibm/ibmveth.c
>> +++ b/drivers/net/ethernet/ibm/ibmveth.c
>> @@ -1182,6 +1182,8 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
>> int frames_processed = 0;
>> unsigned long lpar_rc;
>> struct iphdr *iph;
>> + bool large_packet = 0;
>> + u16 hdr_len = ETH_HLEN + sizeof(struct tcphdr);
>>
>> restart_poll:
>> while (frames_processed < budget) {
>> @@ -1236,10 +1238,28 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
>> iph->check = 0;
>> iph->check = ip_fast_csum((unsigned char *)iph, iph->ihl);
>> adapter->rx_large_packets++;
>> + large_packet = 1;
>> }
>> }
>> }
>>
>> + if (skb->len > netdev->mtu) {
>> + iph = (struct iphdr *)skb->data;
>> + if (be16_to_cpu(skb->protocol) == ETH_P_IP &&
>> + iph->protocol == IPPROTO_TCP) {
>> + hdr_len += sizeof(struct iphdr);
>> + skb_shinfo(skb)->gso_type = SKB_GSO_TCPV4;
>> + skb_shinfo(skb)->gso_size = netdev->mtu - hdr_len;
>> + } else if (be16_to_cpu(skb->protocol) == ETH_P_IPV6 &&
>> + iph->protocol == IPPROTO_TCP) {
>> + hdr_len += sizeof(struct ipv6hdr);
>> + skb_shinfo(skb)->gso_type = SKB_GSO_TCPV6;
>> + skb_shinfo(skb)->gso_size = netdev->mtu - hdr_len;
>> + }
>> + if (!large_packet)
>> + adapter->rx_large_packets++;
>> + }
>> +
>>
>
> This might break forwarding and PMTU discovery.
>
> You force gso_size to device mtu, regardless of real MSS used by the TCP
> sender.
>
> Don't you have the MSS provided in RX descriptor, instead of guessing
> the value ?
Eric,
We are currently pursuing making changes to the Power Virtual I/O Server to provide
the MSS to the ibmveth driver. However, this will take time to go through test
and ultimately get released. Although imperfect, this patch does help a real customer
hitting this issue right now. Would you object to this patch getting merged as is,
with the understanding that when we get the change in the Virtual I/O Server released,
we will revert this interim change and apply the new method?
Thanks,
Brian
--
Brian King
Power Linux I/O
IBM Linux Technology Center
^ permalink raw reply
* Re: [patch net-next 5/8] Introduce sample tc action
From: David Miller @ 2016-11-11 17:47 UTC (permalink / raw)
To: john.fastabend
Cc: simon.horman, yotamg, jiri, netdev, idosch, eladr, nogahf,
ogerlitz, jhs, geert+renesas, stephen, xiyou.wangcong, linux,
roopa
In-Reply-To: <5825DB2F.1090208@gmail.com>
From: John Fastabend <john.fastabend@gmail.com>
Date: Fri, 11 Nov 2016 06:52:31 -0800
> On 16-11-11 04:43 AM, Simon Horman wrote:
>> On Fri, Nov 11, 2016 at 08:28:50AM +0000, Yotam Gigi wrote:
>>
>> ...
>>
>>> John, as a result of your question I realized that our hardware does do
>>> randomized sampling that I was not aware of. I will use the extensibility of
>>> the API and implement a random keyword, that will be offloaded in our
>>> hardware. Those changes will be sent on v2.
>>>
>>> Eventually, your question was very relevant :) Thanks!
>>
>> Perhaps I am missing the point but why not just make random the default and
>> implement the inverse as an extension if it turns out to be needed in
>> future?
>>
>
> +1 just implement the random one.
Agreed.
^ permalink raw reply
* Re: [PATCH 0/2] bnx2: Hard reset bnx2 chip at probe stage
From: Michael Chan @ 2016-11-11 17:37 UTC (permalink / raw)
To: Baoquan He
Cc: Netdev, open list, Dept-GELinuxNICDev, rasesh.mody, harish.patil,
frank, jsr, pmenzel, jroedel, dyoung
In-Reply-To: <20161111140205.GD15325@x1>
On Fri, Nov 11, 2016 at 6:02 AM, Baoquan He <bhe@redhat.com> wrote:
> On 11/11/16 at 09:46pm, Baoquan He wrote:
>> Hi bnx2 experts,
>>
>> In commit 3e1be7a ("bnx2: Reset device during driver initialization"),
>> firmware requesting code was moved from open stage to probe stage.
>> The reason is in kdump kernel hardware iommu need device be reset in
>> driver probe stage, otherwise those in-flight DMA from 1st kernel
>> will continue going and look up into the newly created io-page tables.
>> So we need reset device to stop in-flight DMA as early as possibe.
>>
>> But with commit 3e1be7a merged, people reported their bnx2 driver init
>> failed because of failed firmware loading. After discussion, it's found
>> that they built bnx2 driver into kernel, and that makes probe function
>> bnx2_init_one be called in do_initcalls(). But at this time the initramfs
>> has not been uncompressed yet and mounted, kernel can't detect firmware.
>>
>> So there's only one way to cover both. Try to hard reset the bnx2 device
>> at probe stage, without involving firmware issues. I tried to add function
>> bnx2_hard_reset_chip() to do this and it's only called in kdump kernel.
>> The thing is I am not quite familiar with bnx2 chip spec, just abstract
>> code from bnx2_reset_chip, the testing result is good.
>
> Here I changed to send BNX2_MISC_COMMAND_HD_RESET in BNX2_CHIP_5709
> case.
>
>From my old 5709 Documentation:
Bit 6 HD_RESET: Writing this bit as 1 will cause the chip to do a
hard reset like bit 5 except the sticky bits in the PCI function are
not reset.
Bit 5 POR_RESET: Writing this bit as 1 will cause the chip to do an
internal reset exactly like a power-up reset. There is no protection
for this request and it may cause any current PCI cycle to lock up.
This reset is intended for use under manufacturing conditions only.
So it sounds like doing HD_RESET can potentially cause a PCI bus lock up.
Why not just disable DMA gracefully as done at the beginning of
bnx2_reset_chip()?
^ permalink raw reply
* [net-next PATCH] net: dummy: Introduce dummy virtual functions
From: Phil Sutter @ 2016-11-11 17:33 UTC (permalink / raw)
To: David Miller; +Cc: netdev, Sabrina Dubroca
The idea for this was born when testing VF support in iproute2 which was
impeded by hardware requirements. In fact, not every VF-capable hardware
driver implements all netdev ops, so testing the interface is still hard
to do even with a well-sorted hardware shelf.
To overcome this and allow for testing the user-kernel interface, this
patch allows to turn dummy into a PF with a configurable amount of VFs.
Due to the assumption that all PFs are PCI devices, this implementation
is not completely straightforward: In order to allow for
rtnl_fill_ifinfo() to see the dummy VFs, a fake PCI parent device is
attached to the dummy netdev. This has to happen at the right spot so
register_netdevice() does not get confused. This patch abuses
ndo_fix_features callback for that. In ndo_uninit callback, the fake
parent is removed again for the same purpose.
Joint work with Sabrina Dubroca.
Signed-off-by: Sabrina Dubroca <sd@queasysnail.net>
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
drivers/net/dummy.c | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 193 insertions(+), 2 deletions(-)
diff --git a/drivers/net/dummy.c b/drivers/net/dummy.c
index 69fc8409a9733..39d0d5354414a 100644
--- a/drivers/net/dummy.c
+++ b/drivers/net/dummy.c
@@ -34,6 +34,8 @@
#include <linux/etherdevice.h>
#include <linux/init.h>
#include <linux/moduleparam.h>
+#include <linux/pci.h>
+#include "../pci/pci.h" /* for struct pci_sriov */
#include <linux/rtnetlink.h>
#include <net/rtnetlink.h>
#include <linux/u64_stats_sync.h>
@@ -42,6 +44,33 @@
#define DRV_VERSION "1.0"
static int numdummies = 1;
+static int num_vfs;
+
+static struct pci_sriov pdev_sriov;
+
+static struct pci_dev pci_pdev = {
+ .is_physfn = 1,
+ .sriov = &pdev_sriov,
+ .dev.bus = &pci_bus_type,
+};
+
+struct vf_data_storage {
+ unsigned char vf_mac[ETH_ALEN];
+ u16 pf_vlan; /* When set, guest VLAN config not allowed. */
+ u16 pf_qos;
+ __be16 vlan_proto;
+ u16 min_tx_rate;
+ u16 max_tx_rate;
+ u8 spoofchk_enabled;
+ bool rss_query_enabled;
+ u8 trusted;
+ int link_state;
+};
+
+struct dummy_priv {
+ int num_vfs;
+ struct vf_data_storage *vfinfo;
+};
/* fake multicast ability */
static void set_multicast_list(struct net_device *dev)
@@ -91,15 +120,29 @@ static netdev_tx_t dummy_xmit(struct sk_buff *skb, struct net_device *dev)
static int dummy_dev_init(struct net_device *dev)
{
+ struct dummy_priv *priv = netdev_priv(dev);
+
dev->dstats = netdev_alloc_pcpu_stats(struct pcpu_dstats);
if (!dev->dstats)
return -ENOMEM;
+ priv->num_vfs = num_vfs;
+ priv->vfinfo = NULL;
+
+ if (!num_vfs)
+ return 0;
+
+ priv->vfinfo = kcalloc(num_vfs, sizeof(struct vf_data_storage),
+ GFP_KERNEL);
+ if (!priv->vfinfo)
+ return -ENOMEM;
+
return 0;
}
static void dummy_dev_uninit(struct net_device *dev)
{
+ dev->dev.parent = NULL;
free_percpu(dev->dstats);
}
@@ -112,6 +155,129 @@ static int dummy_change_carrier(struct net_device *dev, bool new_carrier)
return 0;
}
+/* fake, just to set fake PCI parent after netdev_register_kobject() */
+static netdev_features_t dummy_fix_features(struct net_device *dev,
+ netdev_features_t features)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (priv->num_vfs)
+ dev->dev.parent = &pci_pdev.dev;
+
+ return features;
+}
+
+static int dummy_set_vf_mac(struct net_device *dev, int vf, u8 *mac)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (!is_valid_ether_addr(mac) || (vf >= priv->num_vfs))
+ return -EINVAL;
+
+ memcpy(priv->vfinfo[vf].vf_mac, mac, ETH_ALEN);
+
+ return 0;
+}
+
+static int dummy_set_vf_vlan(struct net_device *dev, int vf,
+ u16 vlan, u8 qos, __be16 vlan_proto)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if ((vf >= priv->num_vfs) || (vlan > 4095) || (qos > 7))
+ return -EINVAL;
+
+ priv->vfinfo[vf].pf_vlan = vlan;
+ priv->vfinfo[vf].pf_qos = qos;
+ priv->vfinfo[vf].vlan_proto = vlan_proto;
+
+ return 0;
+}
+
+static int dummy_set_vf_rate(struct net_device *dev, int vf, int min, int max)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ priv->vfinfo[vf].min_tx_rate = min;
+ priv->vfinfo[vf].max_tx_rate = max;
+
+ return 0;
+}
+
+static int dummy_set_vf_spoofchk(struct net_device *dev, int vf, bool val)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ priv->vfinfo[vf].spoofchk_enabled = val;
+
+ return 0;
+}
+
+static int dummy_set_vf_rss_query_en(struct net_device *dev, int vf, bool val)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ priv->vfinfo[vf].rss_query_enabled = val;
+
+ return 0;
+}
+
+static int dummy_set_vf_trust(struct net_device *dev, int vf, bool val)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ priv->vfinfo[vf].trusted = val;
+
+ return 0;
+}
+
+static int dummy_get_vf_config(struct net_device *dev,
+ int vf, struct ifla_vf_info *ivi)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ ivi->vf = vf;
+ memcpy(&ivi->mac, priv->vfinfo[vf].vf_mac, ETH_ALEN);
+ ivi->vlan = priv->vfinfo[vf].pf_vlan;
+ ivi->qos = priv->vfinfo[vf].pf_qos;
+ ivi->spoofchk = priv->vfinfo[vf].spoofchk_enabled;
+ ivi->linkstate = priv->vfinfo[vf].link_state;
+ ivi->min_tx_rate = priv->vfinfo[vf].min_tx_rate;
+ ivi->max_tx_rate = priv->vfinfo[vf].max_tx_rate;
+ ivi->rss_query_en = priv->vfinfo[vf].rss_query_enabled;
+ ivi->trusted = priv->vfinfo[vf].trusted;
+ ivi->vlan_proto = priv->vfinfo[vf].vlan_proto;
+
+ return 0;
+}
+
+static int dummy_set_vf_link_state(struct net_device *dev, int vf, int state)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ if (vf >= priv->num_vfs)
+ return -EINVAL;
+
+ priv->vfinfo[vf].link_state = state;
+
+ return 0;
+}
+
static const struct net_device_ops dummy_netdev_ops = {
.ndo_init = dummy_dev_init,
.ndo_uninit = dummy_dev_uninit,
@@ -121,6 +287,15 @@ static const struct net_device_ops dummy_netdev_ops = {
.ndo_set_mac_address = eth_mac_addr,
.ndo_get_stats64 = dummy_get_stats64,
.ndo_change_carrier = dummy_change_carrier,
+ .ndo_fix_features = dummy_fix_features,
+ .ndo_set_vf_mac = dummy_set_vf_mac,
+ .ndo_set_vf_vlan = dummy_set_vf_vlan,
+ .ndo_set_vf_rate = dummy_set_vf_rate,
+ .ndo_set_vf_spoofchk = dummy_set_vf_spoofchk,
+ .ndo_set_vf_trust = dummy_set_vf_trust,
+ .ndo_get_vf_config = dummy_get_vf_config,
+ .ndo_set_vf_link_state = dummy_set_vf_link_state,
+ .ndo_set_vf_rss_query_en = dummy_set_vf_rss_query_en,
};
static void dummy_get_drvinfo(struct net_device *dev,
@@ -134,6 +309,14 @@ static const struct ethtool_ops dummy_ethtool_ops = {
.get_drvinfo = dummy_get_drvinfo,
};
+static void dummy_free_netdev(struct net_device *dev)
+{
+ struct dummy_priv *priv = netdev_priv(dev);
+
+ kfree(priv->vfinfo);
+ free_netdev(dev);
+}
+
static void dummy_setup(struct net_device *dev)
{
ether_setup(dev);
@@ -141,7 +324,7 @@ static void dummy_setup(struct net_device *dev)
/* Initialize the device structure. */
dev->netdev_ops = &dummy_netdev_ops;
dev->ethtool_ops = &dummy_ethtool_ops;
- dev->destructor = free_netdev;
+ dev->destructor = dummy_free_netdev;
/* Fill in device structure with ethernet-generic values. */
dev->flags |= IFF_NOARP;
@@ -169,6 +352,7 @@ static int dummy_validate(struct nlattr *tb[], struct nlattr *data[])
static struct rtnl_link_ops dummy_link_ops __read_mostly = {
.kind = DRV_NAME,
+ .priv_size = sizeof(struct dummy_priv),
.setup = dummy_setup,
.validate = dummy_validate,
};
@@ -177,12 +361,16 @@ static struct rtnl_link_ops dummy_link_ops __read_mostly = {
module_param(numdummies, int, 0);
MODULE_PARM_DESC(numdummies, "Number of dummy pseudo devices");
+module_param(num_vfs, int, 0);
+MODULE_PARM_DESC(num_vfs, "Number of dummy VFs per dummy device");
+
static int __init dummy_init_one(void)
{
struct net_device *dev_dummy;
int err;
- dev_dummy = alloc_netdev(0, "dummy%d", NET_NAME_UNKNOWN, dummy_setup);
+ dev_dummy = alloc_netdev(sizeof(struct dummy_priv),
+ "dummy%d", NET_NAME_UNKNOWN, dummy_setup);
if (!dev_dummy)
return -ENOMEM;
@@ -190,6 +378,7 @@ static int __init dummy_init_one(void)
err = register_netdevice(dev_dummy);
if (err < 0)
goto err;
+
return 0;
err:
@@ -201,6 +390,8 @@ static int __init dummy_init_module(void)
{
int i, err = 0;
+ pdev_sriov.num_VFs = num_vfs;
+
rtnl_lock();
err = __rtnl_link_register(&dummy_link_ops);
if (err < 0)
--
2.10.0
^ permalink raw reply related
* wl1251 & mac address & calibration data
From: Pali Rohár @ 2016-11-11 17:20 UTC (permalink / raw)
To: Kalle Valo, Pavel Machek, Ivaylo Dimitrov, Sebastian Reichel,
Aaro Koskinen, Tony Lindgren
Cc: linux-wireless, netdev, linux-kernel
[-- Attachment #1: Type: Text/Plain, Size: 1626 bytes --]
Hi! I will open discussion about mac address and calibration data for
wl1251 wireless chip again...
Problem: Mac address & calibration data for wl1251 chip on Nokia N900
are stored on second nand partition (mtd1) in special proprietary format
which is used only for Nokia N900 (probably on N8x0 and N9 too).
Wireless driver wl1251.ko cannot work without mac address and
calibration data.
Absence of mac address cause that driver generates random mac address at
every kernel boot which has couple of problems (unstable identifier of
wireless device due to udev permanent storage rules; unpredictable
behaviour for dhcp mac address assignment, mac address filtering, ...).
Currently there is no way to set (permanent) mac address for network
interface from userspace. And it does not make sense to implement in
linux kernel large parser for proprietary format of second nand
partition where is mac address stored only for one device -- Nokia N900.
Driver wl1251.ko loads calibration data via request_firmware() for file
wl1251-nvs.bin. There are some "example" calibration file in linux-
firmware repository, but it is not suitable for normal usage as real
calibration data are per-device specific.
So questions are:
1) How to set mac address from userspace for that wl1251 interface? In
userspace I can write parser for that proprietary format of nand
partition and extract mac address from it
2) How to send calibration data to wl1251 driver? Those are again stored
in proprietary format and I can write userspace parser for it.
--
Pali Rohár
pali.rohar@gmail.com
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* RE: [patch net-next 5/8] Introduce sample tc action
From: Yotam Gigi @ 2016-11-11 16:34 UTC (permalink / raw)
To: Simon Horman
Cc: John Fastabend, Jiri Pirko, netdev@vger.kernel.org,
davem@davemloft.net, Ido Schimmel, Elad Raz, Nogah Frankel,
Or Gerlitz, jhs@mojatatu.com, geert+renesas@glider.be,
stephen@networkplumber.org, xiyou.wangcong@gmail.com,
linux@roeck-us.net, roopa@cumulusnetworks.com
In-Reply-To: <20161111124353.GA31511@penelope.isobedori.kobe.vergenet.net>
>-----Original Message-----
>From: Simon Horman [mailto:simon.horman@netronome.com]
>Sent: Friday, November 11, 2016 2:44 PM
>To: Yotam Gigi <yotamg@mellanox.com>
>Cc: John Fastabend <john.fastabend@gmail.com>; Jiri Pirko <jiri@resnulli.us>;
>netdev@vger.kernel.org; davem@davemloft.net; Ido Schimmel
><idosch@mellanox.com>; Elad Raz <eladr@mellanox.com>; Nogah Frankel
><nogahf@mellanox.com>; Or Gerlitz <ogerlitz@mellanox.com>;
>jhs@mojatatu.com; geert+renesas@glider.be; stephen@networkplumber.org;
>xiyou.wangcong@gmail.com; linux@roeck-us.net; roopa@cumulusnetworks.com
>Subject: Re: [patch net-next 5/8] Introduce sample tc action
>
>On Fri, Nov 11, 2016 at 08:28:50AM +0000, Yotam Gigi wrote:
>
>...
>
>> John, as a result of your question I realized that our hardware does do
>> randomized sampling that I was not aware of. I will use the extensibility of
>> the API and implement a random keyword, that will be offloaded in our
>> hardware. Those changes will be sent on v2.
>>
>> Eventually, your question was very relevant :) Thanks!
>
>Perhaps I am missing the point but why not just make random the default and
>implement the inverse as an extension if it turns out to be needed in
>future?
It makes sense. It does seem to me that the average user does prefer random
sampling over deterministic one.
We will consider that. Thanks for the comment!
^ permalink raw reply
* Re: [PATCH v2 5/6] qedi: Add support for iSCSI session management.
From: Hannes Reinecke @ 2016-11-11 16:47 UTC (permalink / raw)
To: Manish Rangankar, martin.petersen, lduncan, cleech
Cc: linux-scsi, netdev, QLogic-Storage-Upstream, Yuval.Mintz
In-Reply-To: <1478588223-16183-6-git-send-email-manish.rangankar@cavium.com>
On 11/08/2016 07:57 AM, Manish Rangankar wrote:
> This patch adds support for iscsi_transport LLD Login,
> Logout, NOP-IN/NOP-OUT, Async, Reject PDU processing
> and Firmware async event handling support.
>
> Signed-off-by: Nilesh Javali <nilesh.javali@cavium.com>
> Signed-off-by: Adheer Chandravanshi <adheer.chandravanshi@qlogic.com>
> Signed-off-by: Chad Dupuis <chad.dupuis@cavium.com>
> Signed-off-by: Saurav Kashyap <saurav.kashyap@cavium.com>
> Signed-off-by: Arun Easi <arun.easi@cavium.com>
> Signed-off-by: Manish Rangankar <manish.rangankar@cavium.com>
> ---
> drivers/scsi/qedi/qedi_fw.c | 1106 +++++++++++++++++++++++++++
> drivers/scsi/qedi/qedi_gbl.h | 67 ++
> drivers/scsi/qedi/qedi_iscsi.c | 1611 ++++++++++++++++++++++++++++++++++++++++
> drivers/scsi/qedi/qedi_iscsi.h | 232 ++++++
> drivers/scsi/qedi/qedi_main.c | 166 +++++
> 5 files changed, 3182 insertions(+)
> create mode 100644 drivers/scsi/qedi/qedi_fw.c
> create mode 100644 drivers/scsi/qedi/qedi_gbl.h
> create mode 100644 drivers/scsi/qedi/qedi_iscsi.c
> create mode 100644 drivers/scsi/qedi/qedi_iscsi.h
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply
* [PATCH net 2/2] ibmvnic: Fix size of debugfs name buffer
From: Thomas Falcon @ 2016-11-11 17:00 UTC (permalink / raw)
To: netdev; +Cc: mwb
In-Reply-To: <1478883646-10760-1-git-send-email-tlfalcon@linux.vnet.ibm.com>
This mistake was causing debugfs directory creation
failures when multiple ibmvnic devices were probed.
Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
drivers/net/ethernet/ibm/ibmvnic.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 921c40f..4f3281a 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -3705,7 +3705,7 @@ static int ibmvnic_probe(struct vio_dev *dev, const struct vio_device_id *id)
struct net_device *netdev;
unsigned char *mac_addr_p;
struct dentry *ent;
- char buf[16]; /* debugfs name buf */
+ char buf[17]; /* debugfs name buf */
int rc;
dev_dbg(&dev->dev, "entering ibmvnic_probe for UA 0x%x\n",
--
1.8.3.1
^ permalink raw reply related
* [PATCH net 1/2] ibmvnic: Unmap ibmvnic_statistics structure
From: Thomas Falcon @ 2016-11-11 17:00 UTC (permalink / raw)
To: netdev; +Cc: mwb
This structure was mapped but never subsequently unmapped.
Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
drivers/net/ethernet/ibm/ibmvnic.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index f6c9b6d..921c40f 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -3844,6 +3844,9 @@ static int ibmvnic_remove(struct vio_dev *dev)
if (adapter->debugfs_dir && !IS_ERR(adapter->debugfs_dir))
debugfs_remove_recursive(adapter->debugfs_dir);
+ dma_unmap_single(&dev->dev, adapter->stats_token,
+ sizeof(struct ibmvnic_statistics), DMA_FROM_DEVICE);
+
if (adapter->ras_comps)
dma_free_coherent(&dev->dev,
adapter->ras_comp_num *
--
1.8.3.1
^ permalink raw reply related
* Re: [PATCH v2 6/6] qedi: Add support for data path.
From: Hannes Reinecke @ 2016-11-11 16:49 UTC (permalink / raw)
To: Manish Rangankar, martin.petersen, lduncan, cleech
Cc: linux-scsi, netdev, QLogic-Storage-Upstream, Yuval.Mintz
In-Reply-To: <1478588223-16183-7-git-send-email-manish.rangankar@cavium.com>
On 11/08/2016 07:57 AM, Manish Rangankar wrote:
> This patch adds support for data path and TMF handling.
>
> Signed-off-by: Nilesh Javali <nilesh.javali@cavium.com>
> Signed-off-by: Adheer Chandravanshi <adheer.chandravanshi@qlogic.com>
> Signed-off-by: Chad Dupuis <chad.dupuis@cavium.com>
> Signed-off-by: Saurav Kashyap <saurav.kashyap@cavium.com>
> Signed-off-by: Arun Easi <arun.easi@cavium.com>
> Signed-off-by: Manish Rangankar <manish.rangankar@cavium.com>
> ---
> drivers/scsi/qedi/qedi_fw.c | 1272 ++++++++++++++++++++++++++++++++++++++++
> drivers/scsi/qedi/qedi_gbl.h | 6 +
> drivers/scsi/qedi/qedi_iscsi.c | 13 +
> drivers/scsi/qedi/qedi_main.c | 4 +
> 4 files changed, 1295 insertions(+)
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH v2 4/6] qedi: Add LL2 iSCSI interface for offload iSCSI.
From: Hannes Reinecke @ 2016-11-11 16:44 UTC (permalink / raw)
To: Manish Rangankar, martin.petersen, lduncan, cleech
Cc: linux-scsi, netdev, QLogic-Storage-Upstream, Yuval.Mintz
In-Reply-To: <1478588223-16183-5-git-send-email-manish.rangankar@cavium.com>
On 11/08/2016 07:57 AM, Manish Rangankar wrote:
> This patch adds support for iscsiuio interface using Light L2 (LL2) qed
> interface.
>
> Signed-off-by: Nilesh Javali <nilesh.javali@cavium.com>
> Signed-off-by: Adheer Chandravanshi <adheer.chandravanshi@qlogic.com>
> Signed-off-by: Chad Dupuis <chad.dupuis@cavium.com>
> Signed-off-by: Saurav Kashyap <saurav.kashyap@cavium.com>
> Signed-off-by: Arun Easi <arun.easi@cavium.com>
> Signed-off-by: Manish Rangankar <manish.rangankar@cavium.com>
> ---
> drivers/scsi/qedi/qedi.h | 73 +++++++++
> drivers/scsi/qedi/qedi_main.c | 357 ++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 430 insertions(+)
>
Oh well; and I thought we could do away with the iscsiuio thingie ...
Sigh.
But nevertheless,
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH v2 3/6] qedi: Add QLogic FastLinQ offload iSCSI driver framework.
From: Hannes Reinecke @ 2016-11-11 16:43 UTC (permalink / raw)
To: Manish Rangankar, martin.petersen, lduncan, cleech
Cc: linux-scsi, netdev, QLogic-Storage-Upstream, Yuval.Mintz
In-Reply-To: <1478588223-16183-4-git-send-email-manish.rangankar@cavium.com>
On 11/08/2016 07:57 AM, Manish Rangankar wrote:
> The QLogic FastLinQ Driver for iSCSI (qedi) is the iSCSI specific module
> for 41000 Series Converged Network Adapters by QLogic.
>
> This patch consists of following changes:
> - MAINTAINERS Makefile and Kconfig changes for qedi,
> - PCI driver registration,
> - iSCSI host level initialization,
> - Debugfs and log level infrastructure.
>
> Signed-off-by: Nilesh Javali <nilesh.javali@cavium.com>
> Signed-off-by: Adheer Chandravanshi <adheer.chandravanshi@qlogic.com>
> Signed-off-by: Chad Dupuis <chad.dupuis@cavium.com>
> Signed-off-by: Saurav Kashyap <saurav.kashyap@cavium.com>
> Signed-off-by: Arun Easi <arun.easi@cavium.com>
> Signed-off-by: Manish Rangankar <manish.rangankar@cavium.com>
> ---
> MAINTAINERS | 6 +
> drivers/net/ethernet/qlogic/Kconfig | 12 -
> drivers/scsi/Kconfig | 1 +
> drivers/scsi/Makefile | 1 +
> drivers/scsi/qedi/Kconfig | 10 +
> drivers/scsi/qedi/Makefile | 5 +
> drivers/scsi/qedi/qedi.h | 291 +++++++
> drivers/scsi/qedi/qedi_dbg.c | 143 ++++
> drivers/scsi/qedi/qedi_dbg.h | 144 ++++
> drivers/scsi/qedi/qedi_debugfs.c | 244 ++++++
> drivers/scsi/qedi/qedi_hsi.h | 52 ++
> drivers/scsi/qedi/qedi_main.c | 1616 +++++++++++++++++++++++++++++++++++
> drivers/scsi/qedi/qedi_sysfs.c | 52 ++
> drivers/scsi/qedi/qedi_version.h | 14 +
> 14 files changed, 2579 insertions(+), 12 deletions(-)
> create mode 100644 drivers/scsi/qedi/Kconfig
> create mode 100644 drivers/scsi/qedi/Makefile
> create mode 100644 drivers/scsi/qedi/qedi.h
> create mode 100644 drivers/scsi/qedi/qedi_dbg.c
> create mode 100644 drivers/scsi/qedi/qedi_dbg.h
> create mode 100644 drivers/scsi/qedi/qedi_debugfs.c
> create mode 100644 drivers/scsi/qedi/qedi_hsi.h
> create mode 100644 drivers/scsi/qedi/qedi_main.c
> create mode 100644 drivers/scsi/qedi/qedi_sysfs.c
> create mode 100644 drivers/scsi/qedi/qedi_version.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e5c17a9..04eec14 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9934,6 +9934,12 @@ F: drivers/net/ethernet/qlogic/qed/
> F: include/linux/qed/
> F: drivers/net/ethernet/qlogic/qede/
>
> +QLOGIC QL41xxx ISCSI DRIVER
> +M: QLogic-Storage-Upstream@cavium.com
> +L: linux-scsi@vger.kernel.org
> +S: Supported
> +F: drivers/scsi/qedi/
> +
> QNX4 FILESYSTEM
> M: Anders Larsen <al@alarsen.net>
> W: http://www.alarsen.net/linux/qnx4fs/
> diff --git a/drivers/net/ethernet/qlogic/Kconfig b/drivers/net/ethernet/qlogic/Kconfig
> index 2832570..3cfd105 100644
> --- a/drivers/net/ethernet/qlogic/Kconfig
> +++ b/drivers/net/ethernet/qlogic/Kconfig
> @@ -113,16 +113,4 @@ config QED_RDMA
> config QED_ISCSI
> bool
>
> -config QEDI
> - tristate "QLogic QED 25/40/100Gb iSCSI driver"
> - depends on QED
> - select QED_LL2
> - select QED_ISCSI
> - default n
> - ---help---
> - This provides a temporary node that allows the compilation
> - and logical testing of the hardware offload iSCSI support
> - for QLogic QED. This would be replaced by the 'real' option
> - once the QEDI driver is added [+relocated].
> -
> endif # NET_VENDOR_QLOGIC
> diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
> index 3e2bdb9..5cf03db 100644
> --- a/drivers/scsi/Kconfig
> +++ b/drivers/scsi/Kconfig
> @@ -1254,6 +1254,7 @@ config SCSI_QLOGICPTI
>
> source "drivers/scsi/qla2xxx/Kconfig"
> source "drivers/scsi/qla4xxx/Kconfig"
> +source "drivers/scsi/qedi/Kconfig"
>
> config SCSI_LPFC
> tristate "Emulex LightPulse Fibre Channel Support"
> diff --git a/drivers/scsi/Makefile b/drivers/scsi/Makefile
> index 38d938d..da9e312 100644
> --- a/drivers/scsi/Makefile
> +++ b/drivers/scsi/Makefile
> @@ -132,6 +132,7 @@ obj-$(CONFIG_PS3_ROM) += ps3rom.o
> obj-$(CONFIG_SCSI_CXGB3_ISCSI) += libiscsi.o libiscsi_tcp.o cxgbi/
> obj-$(CONFIG_SCSI_CXGB4_ISCSI) += libiscsi.o libiscsi_tcp.o cxgbi/
> obj-$(CONFIG_SCSI_BNX2_ISCSI) += libiscsi.o bnx2i/
> +obj-$(CONFIG_QEDI) += libiscsi.o qedi/
> obj-$(CONFIG_BE2ISCSI) += libiscsi.o be2iscsi/
> obj-$(CONFIG_SCSI_ESAS2R) += esas2r/
> obj-$(CONFIG_SCSI_PMCRAID) += pmcraid.o
> diff --git a/drivers/scsi/qedi/Kconfig b/drivers/scsi/qedi/Kconfig
> new file mode 100644
> index 0000000..23ca8a2
> --- /dev/null
> +++ b/drivers/scsi/qedi/Kconfig
> @@ -0,0 +1,10 @@
> +config QEDI
> + tristate "QLogic QEDI 25/40/100Gb iSCSI Initiator Driver Support"
> + depends on PCI && SCSI
> + depends on QED
> + select SCSI_ISCSI_ATTRS
> + select QED_LL2
> + select QED_ISCSI
> + ---help---
> + This driver supports iSCSI offload for the QLogic FastLinQ
> + 41000 Series Converged Network Adapters.
> diff --git a/drivers/scsi/qedi/Makefile b/drivers/scsi/qedi/Makefile
> new file mode 100644
> index 0000000..2b3e16b
> --- /dev/null
> +++ b/drivers/scsi/qedi/Makefile
> @@ -0,0 +1,5 @@
> +obj-$(CONFIG_QEDI) := qedi.o
> +qedi-y := qedi_main.o qedi_iscsi.o qedi_fw.o qedi_sysfs.o \
> + qedi_dbg.o
> +
> +qedi-$(CONFIG_DEBUG_FS) += qedi_debugfs.o
> diff --git a/drivers/scsi/qedi/qedi.h b/drivers/scsi/qedi/qedi.h
> new file mode 100644
> index 0000000..92136a3
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi.h
> @@ -0,0 +1,291 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#ifndef _QEDI_H_
> +#define _QEDI_H_
> +
> +#define __PREVENT_QED_HSI__
> +
> +#include <scsi/scsi_transport_iscsi.h>
> +#include <scsi/libiscsi.h>
> +#include <scsi/scsi_host.h>
> +#include <linux/uio_driver.h>
> +
> +#include "qedi_hsi.h"
> +#include <linux/qed/qed_if.h>
> +#include "qedi_dbg.h"
> +#include <linux/qed/qed_iscsi_if.h>
> +#include "qedi_version.h"
> +
> +#define QEDI_MODULE_NAME "qedi"
> +
> +struct qedi_endpoint;
> +
> +/*
> + * PCI function probe defines
> + */
> +#define QEDI_MODE_NORMAL 0
> +#define QEDI_MODE_RECOVERY 1
> +
> +#define ISCSI_WQE_SET_PTU_INVALIDATE 1
> +#define QEDI_MAX_ISCSI_TASK 4096
> +#define QEDI_MAX_TASK_NUM 0x0FFF
> +#define QEDI_MAX_ISCSI_CONNS_PER_HBA 1024
> +#define QEDI_ISCSI_MAX_BDS_PER_CMD 256 /* Firmware max BDs is 256 */
> +#define MAX_OUSTANDING_TASKS_PER_CON 1024
> +
> +#define QEDI_MAX_BD_LEN 0xffff
> +#define QEDI_BD_SPLIT_SZ 0x1000
> +#define QEDI_PAGE_SIZE 4096
> +#define QEDI_FAST_SGE_COUNT 4
> +/* MAX Length for cached SGL */
> +#define MAX_SGLEN_FOR_CACHESGL ((1U << 16) - 1)
> +
> +#define MAX_NUM_MSIX_PF 8
> +#define MIN_NUM_CPUS_MSIX(x) min(x->msix_count, num_online_cpus())
> +
> +#define QEDI_LOCAL_PORT_MIN 60000
> +#define QEDI_LOCAL_PORT_MAX 61024
> +#define QEDI_LOCAL_PORT_RANGE (QEDI_LOCAL_PORT_MAX - QEDI_LOCAL_PORT_MIN)
> +#define QEDI_LOCAL_PORT_INVALID 0xffff
> +
> +/* Queue sizes in number of elements */
> +#define QEDI_SQ_SIZE MAX_OUSTANDING_TASKS_PER_CON
> +#define QEDI_CQ_SIZE 2048
> +#define QEDI_CMDQ_SIZE QEDI_MAX_ISCSI_TASK
> +#define QEDI_PROTO_CQ_PROD_IDX 0
> +
> +struct qedi_glbl_q_params {
> + u64 hw_p_cq; /* Completion queue PBL */
> + u64 hw_p_rq; /* Request queue PBL */
> + u64 hw_p_cmdq; /* Command queue PBL */
> +};
> +
> +struct global_queue {
> + union iscsi_cqe *cq;
> + dma_addr_t cq_dma;
> + u32 cq_mem_size;
> + u32 cq_cons_idx; /* Completion queue consumer index */
> +
> + void *cq_pbl;
> + dma_addr_t cq_pbl_dma;
> + u32 cq_pbl_size;
> +
> +};
> +
> +struct qedi_fastpath {
> + struct qed_sb_info *sb_info;
> + u16 sb_id;
> +#define QEDI_NAME_SIZE 16
> + char name[QEDI_NAME_SIZE];
> + struct qedi_ctx *qedi;
> +};
> +
> +/* Used to pass fastpath information needed to process CQEs */
> +struct qedi_io_work {
> + struct list_head list;
> + struct iscsi_cqe_solicited cqe;
> + u16 que_idx;
> +};
> +
> +/**
> + * struct iscsi_cid_queue - Per adapter iscsi cid queue
> + *
> + * @cid_que_base: queue base memory
> + * @cid_que: queue memory pointer
> + * @cid_q_prod_idx: produce index
> + * @cid_q_cons_idx: consumer index
> + * @cid_q_max_idx: max index. used to detect wrap around condition
> + * @cid_free_cnt: queue size
> + * @conn_cid_tbl: iscsi cid to conn structure mapping table
> + *
> + * Per adapter iSCSI CID Queue
> + */
> +struct iscsi_cid_queue {
> + void *cid_que_base;
> + u32 *cid_que;
> + u32 cid_q_prod_idx;
> + u32 cid_q_cons_idx;
> + u32 cid_q_max_idx;
> + u32 cid_free_cnt;
> + struct qedi_conn **conn_cid_tbl;
> +};
> +
> +struct qedi_portid_tbl {
> + spinlock_t lock; /* Port id lock */
> + u16 start;
> + u16 max;
> + u16 next;
> + unsigned long *table;
> +};
> +
> +struct qedi_itt_map {
> + __le32 itt;
> + struct qedi_cmd *p_cmd;
> +};
> +
> +/* I/O tracing entry */
> +#define QEDI_IO_TRACE_SIZE 2048
> +struct qedi_io_log {
> +#define QEDI_IO_TRACE_REQ 0
> +#define QEDI_IO_TRACE_RSP 1
> + u8 direction;
> + u16 task_id;
> + u32 cid;
> + u32 port_id; /* Remote port fabric ID */
> + int lun;
> + u8 op; /* SCSI CDB */
> + u8 lba[4];
> + unsigned int bufflen; /* SCSI buffer length */
> + unsigned int sg_count; /* Number of SG elements */
> + u8 fast_sgs; /* number of fast sgls */
> + u8 slow_sgs; /* number of slow sgls */
> + u8 cached_sgs; /* number of cached sgls */
> + int result; /* Result passed back to mid-layer */
> + unsigned long jiffies; /* Time stamp when I/O logged */
> + int refcount; /* Reference count for task id */
> + unsigned int blk_req_cpu; /* CPU that the task is queued on by
> + * blk layer
> + */
> + unsigned int req_cpu; /* CPU that the task is queued on */
> + unsigned int intr_cpu; /* Interrupt CPU that the task is received on */
> + unsigned int blk_rsp_cpu;/* CPU that task is actually processed and
> + * returned to blk layer
> + */
> + bool cached_sge;
> + bool slow_sge;
> + bool fast_sge;
> +};
> +
> +/* Number of entries in BDQ */
> +#define QEDI_BDQ_NUM 256
> +#define QEDI_BDQ_BUF_SIZE 256
> +
> +/* DMA coherent buffers for BDQ */
> +struct qedi_bdq_buf {
> + void *buf_addr;
> + dma_addr_t buf_dma;
> +};
> +
> +/* Main port level struct */
> +struct qedi_ctx {
> + struct qedi_dbg_ctx dbg_ctx;
> + struct Scsi_Host *shost;
> + struct pci_dev *pdev;
> + struct qed_dev *cdev;
> + struct qed_dev_iscsi_info dev_info;
> + struct qed_int_info int_info;
> + struct qedi_glbl_q_params *p_cpuq;
> + struct global_queue **global_queues;
> + /* uio declaration */
> + struct qedi_uio_dev *udev;
> + struct list_head ll2_skb_list;
> + spinlock_t ll2_lock; /* Light L2 lock */
> + spinlock_t hba_lock; /* per port lock */
> + struct task_struct *ll2_recv_thread;
> + unsigned long flags;
> +#define UIO_DEV_OPENED 1
> +#define QEDI_IOTHREAD_WAKE 2
> +#define QEDI_IN_RECOVERY 5
> +#define QEDI_IN_OFFLINE 6
> +
> + u8 mac[ETH_ALEN];
> + u32 src_ip[4];
> + u8 ip_type;
> +
> + /* Physical address of above array */
> + u64 hw_p_cpuq;
> +
> + struct qedi_bdq_buf bdq[QEDI_BDQ_NUM];
> + void *bdq_pbl;
> + dma_addr_t bdq_pbl_dma;
> + size_t bdq_pbl_mem_size;
> + void *bdq_pbl_list;
> + dma_addr_t bdq_pbl_list_dma;
> + u8 bdq_pbl_list_num_entries;
> + void __iomem *bdq_primary_prod;
> + void __iomem *bdq_secondary_prod;
> + u16 bdq_prod_idx;
> + u16 rq_num_entries;
> +
> + u32 msix_count;
> + u32 max_sqes;
> + u8 num_queues;
> + u32 max_active_conns;
> +
> + struct iscsi_cid_queue cid_que;
> + struct qedi_endpoint **ep_tbl;
> + struct qedi_portid_tbl lcl_port_tbl;
> +
> + /* Rx fast path intr context */
> + struct qed_sb_info *sb_array;
> + struct qedi_fastpath *fp_array;
> + struct qed_iscsi_tid tasks;
> +
> +#define QEDI_LINK_DOWN 0
> +#define QEDI_LINK_UP 1
> + atomic_t link_state;
> +
> +#define QEDI_RESERVE_TASK_ID 0
> +#define MAX_ISCSI_TASK_ENTRIES 4096
> +#define QEDI_INVALID_TASK_ID (MAX_ISCSI_TASK_ENTRIES + 1)
> + unsigned long task_idx_map[MAX_ISCSI_TASK_ENTRIES / BITS_PER_LONG];
> + struct qedi_itt_map *itt_map;
> + u16 tid_reuse_count[QEDI_MAX_ISCSI_TASK];
> + struct qed_pf_params pf_params;
> +
> + struct workqueue_struct *tmf_thread;
> + struct workqueue_struct *offload_thread;
> +
> + u16 ll2_mtu;
> +
> + struct workqueue_struct *dpc_wq;
> +
> + spinlock_t task_idx_lock; /* To protect gbl context */
> + s32 last_tidx_alloc;
> + s32 last_tidx_clear;
> +
> + struct qedi_io_log io_trace_buf[QEDI_IO_TRACE_SIZE];
> + spinlock_t io_trace_lock; /* prtect trace Log buf */
> + u16 io_trace_idx;
> + unsigned int intr_cpu;
> + u32 cached_sgls;
> + bool use_cached_sge;
> + u32 slow_sgls;
> + bool use_slow_sge;
> + u32 fast_sgls;
> + bool use_fast_sge;
> +
> + atomic_t num_offloads;
> +};
> +
> +struct qedi_work {
> + struct list_head list;
> + struct qedi_ctx *qedi;
> + union iscsi_cqe cqe;
> + u16 que_idx;
> + bool is_solicited;
> +};
> +
> +struct qedi_percpu_s {
> + struct task_struct *iothread;
> + struct list_head work_list;
> + spinlock_t p_work_lock; /* Per cpu worker lock */
> +};
> +
> +static inline void *qedi_get_task_mem(struct qed_iscsi_tid *info, u32 tid)
> +{
> + return (info->blocks[tid / info->num_tids_per_block] +
> + (tid % info->num_tids_per_block) * info->size);
> +}
> +
> +#define QEDI_U64_HI(val) ((u32)(((u64)(val)) >> 32))
> +#define QEDI_U64_LO(val) ((u32)(((u64)(val)) & 0xffffffff))
> +
> +#endif /* _QEDI_H_ */
> diff --git a/drivers/scsi/qedi/qedi_dbg.c b/drivers/scsi/qedi/qedi_dbg.c
> new file mode 100644
> index 0000000..b03d9af
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_dbg.c
> @@ -0,0 +1,143 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#include "qedi_dbg.h"
> +#include <linux/vmalloc.h>
> +
> +void
> +qedi_dbg_err(struct qedi_dbg_ctx *qedi, const char *func, u32 line,
> + const char *fmt, ...)
> +{
> + va_list va;
> + struct va_format vaf;
> + char nfunc[32];
> +
> + memset(nfunc, 0, sizeof(nfunc));
> + memcpy(nfunc, func, sizeof(nfunc) - 1);
> +
> + va_start(va, fmt);
> +
> + vaf.fmt = fmt;
> + vaf.va = &va;
> +
> + if (likely(qedi) && likely(qedi->pdev))
> + pr_err("[%s]:[%s:%d]:%d: %pV", dev_name(&qedi->pdev->dev),
> + nfunc, line, qedi->host_no, &vaf);
> + else
> + pr_err("[0000:00:00.0]:[%s:%d]: %pV", nfunc, line, &vaf);
> +
> + va_end(va);
> +}
> +
> +void
> +qedi_dbg_warn(struct qedi_dbg_ctx *qedi, const char *func, u32 line,
> + const char *fmt, ...)
> +{
> + va_list va;
> + struct va_format vaf;
> + char nfunc[32];
> +
> + memset(nfunc, 0, sizeof(nfunc));
> + memcpy(nfunc, func, sizeof(nfunc) - 1);
> +
> + va_start(va, fmt);
> +
> + vaf.fmt = fmt;
> + vaf.va = &va;
> +
> + if (!(debug & QEDI_LOG_WARN))
> + return;
> +
> + if (likely(qedi) && likely(qedi->pdev))
> + pr_warn("[%s]:[%s:%d]:%d: %pV", dev_name(&qedi->pdev->dev),
> + nfunc, line, qedi->host_no, &vaf);
> + else
> + pr_warn("[0000:00:00.0]:[%s:%d]: %pV", nfunc, line, &vaf);
> +
> + va_end(va);
> +}
> +
> +void
> +qedi_dbg_notice(struct qedi_dbg_ctx *qedi, const char *func, u32 line,
> + const char *fmt, ...)
> +{
> + va_list va;
> + struct va_format vaf;
> + char nfunc[32];
> +
> + memset(nfunc, 0, sizeof(nfunc));
> + memcpy(nfunc, func, sizeof(nfunc) - 1);
> +
> + va_start(va, fmt);
> +
> + vaf.fmt = fmt;
> + vaf.va = &va;
> +
> + if (!(debug & QEDI_LOG_NOTICE))
> + return;
> +
> + if (likely(qedi) && likely(qedi->pdev))
> + pr_notice("[%s]:[%s:%d]:%d: %pV",
> + dev_name(&qedi->pdev->dev), nfunc, line,
> + qedi->host_no, &vaf);
> + else
> + pr_notice("[0000:00:00.0]:[%s:%d]: %pV", nfunc, line, &vaf);
> +
> + va_end(va);
> +}
> +
> +void
> +qedi_dbg_info(struct qedi_dbg_ctx *qedi, const char *func, u32 line,
> + u32 level, const char *fmt, ...)
> +{
> + va_list va;
> + struct va_format vaf;
> + char nfunc[32];
> +
> + memset(nfunc, 0, sizeof(nfunc));
> + memcpy(nfunc, func, sizeof(nfunc) - 1);
> +
> + va_start(va, fmt);
> +
> + vaf.fmt = fmt;
> + vaf.va = &va;
> +
> + if (!(debug & level))
> + return;
> +
> + if (likely(qedi) && likely(qedi->pdev))
> + pr_info("[%s]:[%s:%d]:%d: %pV", dev_name(&qedi->pdev->dev),
> + nfunc, line, qedi->host_no, &vaf);
> + else
> + pr_info("[0000:00:00.0]:[%s:%d]: %pV", nfunc, line, &vaf);
> +
> + va_end(va);
> +}
> +
> +int
> +qedi_create_sysfs_attr(struct Scsi_Host *shost, struct sysfs_bin_attrs *iter)
> +{
> + int ret = 0;
> +
> + for (; iter->name; iter++) {
> + ret = sysfs_create_bin_file(&shost->shost_gendev.kobj,
> + iter->attr);
> + if (ret)
> + pr_err("Unable to create sysfs %s attr, err(%d).\n",
> + iter->name, ret);
> + }
> + return ret;
> +}
> +
> +void
> +qedi_remove_sysfs_attr(struct Scsi_Host *shost, struct sysfs_bin_attrs *iter)
> +{
> + for (; iter->name; iter++)
> + sysfs_remove_bin_file(&shost->shost_gendev.kobj, iter->attr);
> +}
> diff --git a/drivers/scsi/qedi/qedi_dbg.h b/drivers/scsi/qedi/qedi_dbg.h
> new file mode 100644
> index 0000000..5beb3ec
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_dbg.h
> @@ -0,0 +1,144 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#ifndef _QEDI_DBG_H_
> +#define _QEDI_DBG_H_
> +
> +#include <linux/types.h>
> +#include <linux/kernel.h>
> +#include <linux/compiler.h>
> +#include <linux/string.h>
> +#include <linux/version.h>
> +#include <linux/pci.h>
> +#include <linux/delay.h>
> +#include <scsi/scsi_transport.h>
> +#include <scsi/scsi_transport_iscsi.h>
> +#include <linux/fs.h>
> +
> +#define __PREVENT_QED_HSI__
> +#include <linux/qed/common_hsi.h>
> +#include <linux/qed/qed_if.h>
> +
> +extern uint debug;
> +
> +/* Debug print level definitions */
> +#define QEDI_LOG_DEFAULT 0x1 /* Set default logging mask */
> +#define QEDI_LOG_INFO 0x2 /* Informational logs,
> + * MAC address, WWPN, WWNN
> + */
> +#define QEDI_LOG_DISC 0x4 /* Init, discovery, rport */
> +#define QEDI_LOG_LL2 0x8 /* LL2, VLAN logs */
> +#define QEDI_LOG_CONN 0x10 /* Connection setup, cleanup */
> +#define QEDI_LOG_EVT 0x20 /* Events, link, mtu */
> +#define QEDI_LOG_TIMER 0x40 /* Timer events */
> +#define QEDI_LOG_MP_REQ 0x80 /* Middle Path (MP) logs */
> +#define QEDI_LOG_SCSI_TM 0x100 /* SCSI Aborts, Task Mgmt */
> +#define QEDI_LOG_UNSOL 0x200 /* unsolicited event logs */
> +#define QEDI_LOG_IO 0x400 /* scsi cmd, completion */
> +#define QEDI_LOG_MQ 0x800 /* Multi Queue logs */
> +#define QEDI_LOG_BSG 0x1000 /* BSG logs */
> +#define QEDI_LOG_DEBUGFS 0x2000 /* debugFS logs */
> +#define QEDI_LOG_LPORT 0x4000 /* lport logs */
> +#define QEDI_LOG_ELS 0x8000 /* ELS logs */
> +#define QEDI_LOG_NPIV 0x10000 /* NPIV logs */
> +#define QEDI_LOG_SESS 0x20000 /* Conection setup, cleanup */
> +#define QEDI_LOG_UIO 0x40000 /* iSCSI UIO logs */
> +#define QEDI_LOG_TID 0x80000 /* FW TID context acquire,
> + * free
> + */
> +#define QEDI_TRACK_TID 0x100000 /* Track TID state. To be
> + * enabled only at module load
> + * and not run-time.
> + */
> +#define QEDI_TRACK_CMD_LIST 0x300000 /* Track active cmd list nodes,
> + * done with reference to TID,
> + * hence TRACK_TID also enabled.
> + */
> +#define QEDI_LOG_NOTICE 0x40000000 /* Notice logs */
> +#define QEDI_LOG_WARN 0x80000000 /* Warning logs */
> +
> +/* Debug context structure */
> +struct qedi_dbg_ctx {
> + unsigned int host_no;
> + struct pci_dev *pdev;
> +#ifdef CONFIG_DEBUG_FS
> + struct dentry *bdf_dentry;
> +#endif
> +};
> +
> +#define QEDI_ERR(pdev, fmt, ...) \
> + qedi_dbg_err(pdev, __func__, __LINE__, fmt, ## __VA_ARGS__)
> +#define QEDI_WARN(pdev, fmt, ...) \
> + qedi_dbg_warn(pdev, __func__, __LINE__, fmt, ## __VA_ARGS__)
> +#define QEDI_NOTICE(pdev, fmt, ...) \
> + qedi_dbg_notice(pdev, __func__, __LINE__, fmt, ## __VA_ARGS__)
> +#define QEDI_INFO(pdev, level, fmt, ...) \
> + qedi_dbg_info(pdev, __func__, __LINE__, level, fmt, \
> + ## __VA_ARGS__)
> +
> +void qedi_dbg_err(struct qedi_dbg_ctx *, const char *, u32,
> + const char *, ...);
> +void qedi_dbg_warn(struct qedi_dbg_ctx *, const char *, u32,
> + const char *, ...);
> +void qedi_dbg_notice(struct qedi_dbg_ctx *, const char *, u32,
> + const char *, ...);
> +void qedi_dbg_info(struct qedi_dbg_ctx *, const char *, u32, u32,
> + const char *, ...);
> +
> +struct Scsi_Host;
> +
> +struct sysfs_bin_attrs {
> + char *name;
> + struct bin_attribute *attr;
> +};
> +
> +int qedi_create_sysfs_attr(struct Scsi_Host *,
> + struct sysfs_bin_attrs *);
> +void qedi_remove_sysfs_attr(struct Scsi_Host *,
> + struct sysfs_bin_attrs *);
> +
> +#ifdef CONFIG_DEBUG_FS
> +/* DebugFS related code */
> +struct qedi_list_of_funcs {
> + char *oper_str;
> + ssize_t (*oper_func)(struct qedi_dbg_ctx *qedi);
> +};
> +
> +struct qedi_debugfs_ops {
> + char *name;
> + struct qedi_list_of_funcs *qedi_funcs;
> +};
> +
> +#define qedi_dbg_fileops(drv, ops) \
> +{ \
> + .owner = THIS_MODULE, \
> + .open = simple_open, \
> + .read = drv##_dbg_##ops##_cmd_read, \
> + .write = drv##_dbg_##ops##_cmd_write \
> +}
> +
> +/* Used for debugfs sequential files */
> +#define qedi_dbg_fileops_seq(drv, ops) \
> +{ \
> + .owner = THIS_MODULE, \
> + .open = drv##_dbg_##ops##_open, \
> + .read = seq_read, \
> + .llseek = seq_lseek, \
> + .release = single_release, \
> +}
> +
> +void qedi_dbg_host_init(struct qedi_dbg_ctx *,
> + struct qedi_debugfs_ops *,
> + const struct file_operations *);
> +void qedi_dbg_host_exit(struct qedi_dbg_ctx *);
> +void qedi_dbg_init(char *);
> +void qedi_dbg_exit(void);
> +#endif /* CONFIG_DEBUG_FS */
> +
> +#endif /* _QEDI_DBG_H_ */
> diff --git a/drivers/scsi/qedi/qedi_debugfs.c b/drivers/scsi/qedi/qedi_debugfs.c
> new file mode 100644
> index 0000000..9559362
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_debugfs.c
> @@ -0,0 +1,244 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#include "qedi.h"
> +#include "qedi_dbg.h"
> +
> +#include <linux/uaccess.h>
> +#include <linux/debugfs.h>
> +#include <linux/module.h>
> +
> +int do_not_recover;
> +static struct dentry *qedi_dbg_root;
> +
> +void
> +qedi_dbg_host_init(struct qedi_dbg_ctx *qedi,
> + struct qedi_debugfs_ops *dops,
> + const struct file_operations *fops)
> +{
> + char host_dirname[32];
> + struct dentry *file_dentry = NULL;
> +
> + sprintf(host_dirname, "host%u", qedi->host_no);
> + qedi->bdf_dentry = debugfs_create_dir(host_dirname, qedi_dbg_root);
> + if (!qedi->bdf_dentry)
> + return;
> +
> + while (dops) {
> + if (!(dops->name))
> + break;
> +
> + file_dentry = debugfs_create_file(dops->name, 0600,
> + qedi->bdf_dentry, qedi,
> + fops);
> + if (!file_dentry) {
> + QEDI_INFO(qedi, QEDI_LOG_DEBUGFS,
> + "Debugfs entry %s creation failed\n",
> + dops->name);
> + debugfs_remove_recursive(qedi->bdf_dentry);
> + return;
> + }
> + dops++;
> + fops++;
> + }
> +}
> +
> +void
> +qedi_dbg_host_exit(struct qedi_dbg_ctx *qedi)
> +{
> + debugfs_remove_recursive(qedi->bdf_dentry);
> + qedi->bdf_dentry = NULL;
> +}
> +
> +void
> +qedi_dbg_init(char *drv_name)
> +{
> + qedi_dbg_root = debugfs_create_dir(drv_name, NULL);
> + if (!qedi_dbg_root)
> + QEDI_INFO(NULL, QEDI_LOG_DEBUGFS, "Init of debugfs failed\n");
> +}
> +
> +void
> +qedi_dbg_exit(void)
> +{
> + debugfs_remove_recursive(qedi_dbg_root);
> + qedi_dbg_root = NULL;
> +}
> +
> +static ssize_t
> +qedi_dbg_do_not_recover_enable(struct qedi_dbg_ctx *qedi_dbg)
> +{
> + if (!do_not_recover)
> + do_not_recover = 1;
> +
> + QEDI_INFO(qedi_dbg, QEDI_LOG_DEBUGFS, "do_not_recover=%d\n",
> + do_not_recover);
> + return 0;
> +}
> +
> +static ssize_t
> +qedi_dbg_do_not_recover_disable(struct qedi_dbg_ctx *qedi_dbg)
> +{
> + if (do_not_recover)
> + do_not_recover = 0;
> +
> + QEDI_INFO(qedi_dbg, QEDI_LOG_DEBUGFS, "do_not_recover=%d\n",
> + do_not_recover);
> + return 0;
> +}
> +
> +static struct qedi_list_of_funcs qedi_dbg_do_not_recover_ops[] = {
> + { "enable", qedi_dbg_do_not_recover_enable },
> + { "disable", qedi_dbg_do_not_recover_disable },
> + { NULL, NULL }
> +};
> +
> +struct qedi_debugfs_ops qedi_debugfs_ops[] = {
> + { "gbl_ctx", NULL },
> + { "do_not_recover", qedi_dbg_do_not_recover_ops},
> + { "io_trace", NULL },
> + { NULL, NULL }
> +};
> +
> +static ssize_t
> +qedi_dbg_do_not_recover_cmd_write(struct file *filp, const char __user *buffer,
> + size_t count, loff_t *ppos)
> +{
> + size_t cnt = 0;
> + struct qedi_dbg_ctx *qedi_dbg =
> + (struct qedi_dbg_ctx *)filp->private_data;
> + struct qedi_list_of_funcs *lof = qedi_dbg_do_not_recover_ops;
> +
> + if (*ppos)
> + return 0;
> +
> + while (lof) {
> + if (!(lof->oper_str))
> + break;
> +
> + if (!strncmp(lof->oper_str, buffer, strlen(lof->oper_str))) {
> + cnt = lof->oper_func(qedi_dbg);
> + break;
> + }
> +
> + lof++;
> + }
> + return (count - cnt);
> +}
> +
> +static ssize_t
> +qedi_dbg_do_not_recover_cmd_read(struct file *filp, char __user *buffer,
> + size_t count, loff_t *ppos)
> +{
> + size_t cnt = 0;
> +
> + if (*ppos)
> + return 0;
> +
> + cnt = sprintf(buffer, "do_not_recover=%d\n", do_not_recover);
> + cnt = min_t(int, count, cnt - *ppos);
> + *ppos += cnt;
> + return cnt;
> +}
> +
> +static int
> +qedi_gbl_ctx_show(struct seq_file *s, void *unused)
> +{
> + struct qedi_fastpath *fp = NULL;
> + struct qed_sb_info *sb_info = NULL;
> + struct status_block *sb = NULL;
> + struct global_queue *que = NULL;
> + int id;
> + u16 prod_idx;
> + struct qedi_ctx *qedi = s->private;
> + unsigned long flags;
> +
> + seq_puts(s, " DUMP CQ CONTEXT:\n");
> +
> + for (id = 0; id < MIN_NUM_CPUS_MSIX(qedi); id++) {
> + spin_lock_irqsave(&qedi->hba_lock, flags);
> + seq_printf(s, "=========FAST CQ PATH [%d] ==========\n", id);
> + fp = &qedi->fp_array[id];
> + sb_info = fp->sb_info;
> + sb = sb_info->sb_virt;
> + prod_idx = (sb->pi_array[QEDI_PROTO_CQ_PROD_IDX] &
> + STATUS_BLOCK_PROD_INDEX_MASK);
> + seq_printf(s, "SB PROD IDX: %d\n", prod_idx);
> + que = qedi->global_queues[fp->sb_id];
> + seq_printf(s, "DRV CONS IDX: %d\n", que->cq_cons_idx);
> + seq_printf(s, "CQ complete host memory: %d\n", fp->sb_id);
> + seq_puts(s, "=========== END ==================\n\n\n");
> + spin_unlock_irqrestore(&qedi->hba_lock, flags);
> + }
> + return 0;
> +}
> +
> +static int
> +qedi_dbg_gbl_ctx_open(struct inode *inode, struct file *file)
> +{
> + struct qedi_dbg_ctx *qedi_dbg = inode->i_private;
> + struct qedi_ctx *qedi = container_of(qedi_dbg, struct qedi_ctx,
> + dbg_ctx);
> +
> + return single_open(file, qedi_gbl_ctx_show, qedi);
> +}
> +
> +static int
> +qedi_io_trace_show(struct seq_file *s, void *unused)
> +{
> + int id, idx = 0;
> + struct qedi_ctx *qedi = s->private;
> + struct qedi_io_log *io_log;
> + unsigned long flags;
> +
> + seq_puts(s, " DUMP IO LOGS:\n");
> + spin_lock_irqsave(&qedi->io_trace_lock, flags);
> + idx = qedi->io_trace_idx;
> + for (id = 0; id < QEDI_IO_TRACE_SIZE; id++) {
> + io_log = &qedi->io_trace_buf[idx];
> + seq_printf(s, "iodir-%d:", io_log->direction);
> + seq_printf(s, "tid-0x%x:", io_log->task_id);
> + seq_printf(s, "cid-0x%x:", io_log->cid);
> + seq_printf(s, "lun-%d:", io_log->lun);
> + seq_printf(s, "op-0x%02x:", io_log->op);
> + seq_printf(s, "0x%02x%02x%02x%02x:", io_log->lba[0],
> + io_log->lba[1], io_log->lba[2], io_log->lba[3]);
> + seq_printf(s, "buflen-%d:", io_log->bufflen);
> + seq_printf(s, "sgcnt-%d:", io_log->sg_count);
> + seq_printf(s, "res-0x%08x:", io_log->result);
> + seq_printf(s, "jif-%lu:", io_log->jiffies);
> + seq_printf(s, "blk_req_cpu-%d:", io_log->blk_req_cpu);
> + seq_printf(s, "req_cpu-%d:", io_log->req_cpu);
> + seq_printf(s, "intr_cpu-%d:", io_log->intr_cpu);
> + seq_printf(s, "blk_rsp_cpu-%d\n", io_log->blk_rsp_cpu);
> +
> + idx++;
> + if (idx == QEDI_IO_TRACE_SIZE)
> + idx = 0;
> + }
> + spin_unlock_irqrestore(&qedi->io_trace_lock, flags);
> + return 0;
> +}
> +
> +static int
> +qedi_dbg_io_trace_open(struct inode *inode, struct file *file)
> +{
> + struct qedi_dbg_ctx *qedi_dbg = inode->i_private;
> + struct qedi_ctx *qedi = container_of(qedi_dbg, struct qedi_ctx,
> + dbg_ctx);
> +
> + return single_open(file, qedi_io_trace_show, qedi);
> +}
> +
> +const struct file_operations qedi_dbg_fops[] = {
> + qedi_dbg_fileops_seq(qedi, gbl_ctx),
> + qedi_dbg_fileops(qedi, do_not_recover),
> + qedi_dbg_fileops_seq(qedi, io_trace),
> + { NULL, NULL },
> +};
> diff --git a/drivers/scsi/qedi/qedi_hsi.h b/drivers/scsi/qedi/qedi_hsi.h
> new file mode 100644
> index 0000000..8ca44c7
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_hsi.h
> @@ -0,0 +1,52 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +#ifndef __QEDI_HSI__
> +#define __QEDI_HSI__
> +/*
> + * Add include to common target
> + */
> +#include <linux/qed/common_hsi.h>
> +
> +/*
> + * Add include to common storage target
> + */
> +#include <linux/qed/storage_common.h>
> +
> +/*
> + * Add include to common TCP target
> + */
> +#include <linux/qed/tcp_common.h>
> +
> +/*
> + * Add include to common iSCSI target for both eCore and protocol driver
> + */
> +#include <linux/qed/iscsi_common.h>
> +
> +/*
> + * iSCSI CMDQ element
> + */
> +struct iscsi_cmdqe {
> + __le16 conn_id;
> + u8 invalid_command;
> + u8 cmd_hdr_type;
> + __le32 reserved1[2];
> + __le32 cmd_payload[13];
> +};
> +
> +/*
> + * iSCSI CMD header type
> + */
> +enum iscsi_cmd_hdr_type {
> + ISCSI_CMD_HDR_TYPE_BHS_ONLY /* iSCSI BHS with no expected AHS */,
> + ISCSI_CMD_HDR_TYPE_BHS_W_AHS /* iSCSI BHS with expected AHS */,
> + ISCSI_CMD_HDR_TYPE_AHS /* iSCSI AHS */,
> + MAX_ISCSI_CMD_HDR_TYPE
> +};
> +
> +#endif /* __QEDI_HSI__ */
> diff --git a/drivers/scsi/qedi/qedi_main.c b/drivers/scsi/qedi/qedi_main.c
> new file mode 100644
> index 0000000..5845dc9
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_main.c
> @@ -0,0 +1,1616 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/kernel.h>
> +#include <linux/if_arp.h>
> +#include <scsi/iscsi_if.h>
> +#include <linux/inet.h>
> +#include <net/arp.h>
> +#include <linux/list.h>
> +#include <linux/kthread.h>
> +#include <linux/mm.h>
> +#include <linux/if_vlan.h>
> +#include <linux/cpu.h>
> +
> +#include <scsi/scsi_cmnd.h>
> +#include <scsi/scsi_device.h>
> +#include <scsi/scsi_eh.h>
> +#include <scsi/scsi_host.h>
> +#include <scsi/scsi.h>
> +
> +#include "qedi.h"
> +
> +static uint fw_debug;
> +module_param(fw_debug, uint, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(fw_debug, " Firmware debug level 0(default) to 3");
> +
> +static uint int_mode;
> +module_param(int_mode, uint, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(int_mode,
> + " Force interrupt mode other than MSI-X: (1 INT#x; 2 MSI)");
> +
> +uint debug = QEDI_LOG_WARN | QEDI_LOG_SCSI_TM;
> +module_param(debug, uint, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(debug, " Default debug level");
> +
> +const struct qed_iscsi_ops *qedi_ops;
> +static struct scsi_transport_template *qedi_scsi_transport;
> +static struct pci_driver qedi_pci_driver;
> +static DEFINE_PER_CPU(struct qedi_percpu_s, qedi_percpu);
> +/* Static function declaration */
> +static int qedi_alloc_global_queues(struct qedi_ctx *qedi);
> +static void qedi_free_global_queues(struct qedi_ctx *qedi);
> +static struct qedi_cmd *qedi_get_cmd_from_tid(struct qedi_ctx *qedi, u32 tid);
> +
> +static int qedi_iscsi_event_cb(void *context, u8 fw_event_code, void *fw_handle)
> +{
> + struct qedi_ctx *qedi;
> + struct qedi_endpoint *qedi_ep;
> + struct async_data *data;
> + int rval = 0;
> +
> + if (!context || !fw_handle) {
> + QEDI_ERR(NULL, "Recv event with ctx NULL\n");
> + return -EINVAL;
> + }
> +
> + qedi = (struct qedi_ctx *)context;
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_INFO,
> + "Recv Event %d fw_handle %p\n", fw_event_code, fw_handle);
> +
> + data = (struct async_data *)fw_handle;
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_INFO,
> + "cid=0x%x tid=0x%x err-code=0x%x fw-dbg-param=0x%x\n",
> + data->cid, data->itid, data->error_code,
> + data->fw_debug_param);
> +
> + qedi_ep = qedi->ep_tbl[data->cid];
> +
> + if (!qedi_ep) {
> + QEDI_WARN(&qedi->dbg_ctx,
> + "Cannot process event, ep already disconnected, cid=0x%x\n",
> + data->cid);
> + WARN_ON(1);
> + return -ENODEV;
> + }
> +
> + switch (fw_event_code) {
> + case ISCSI_EVENT_TYPE_ASYN_CONNECT_COMPLETE:
> + if (qedi_ep->state == EP_STATE_OFLDCONN_START)
> + qedi_ep->state = EP_STATE_OFLDCONN_COMPL;
> +
> + wake_up_interruptible(&qedi_ep->tcp_ofld_wait);
> + break;
> + case ISCSI_EVENT_TYPE_ASYN_TERMINATE_DONE:
> + qedi_ep->state = EP_STATE_DISCONN_COMPL;
> + wake_up_interruptible(&qedi_ep->tcp_ofld_wait);
> + break;
> + case ISCSI_EVENT_TYPE_ISCSI_CONN_ERROR:
> + qedi_process_iscsi_error(qedi_ep, data);
> + break;
> + case ISCSI_EVENT_TYPE_ASYN_ABORT_RCVD:
> + case ISCSI_EVENT_TYPE_ASYN_SYN_RCVD:
> + case ISCSI_EVENT_TYPE_ASYN_MAX_RT_TIME:
> + case ISCSI_EVENT_TYPE_ASYN_MAX_RT_CNT:
> + case ISCSI_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT:
> + case ISCSI_EVENT_TYPE_ASYN_FIN_WAIT2:
> + case ISCSI_EVENT_TYPE_TCP_CONN_ERROR:
> + qedi_process_tcp_error(qedi_ep, data);
> + break;
> + default:
> + QEDI_ERR(&qedi->dbg_ctx, "Recv Unknown Event %u\n",
> + fw_event_code);
> + }
> +
> + return rval;
> +}
> +
> +static int qedi_alloc_and_init_sb(struct qedi_ctx *qedi,
> + struct qed_sb_info *sb_info, u16 sb_id)
> +{
> + struct status_block *sb_virt;
> + dma_addr_t sb_phys;
> + int ret;
> +
> + sb_virt = dma_alloc_coherent(&qedi->pdev->dev,
> + sizeof(struct status_block), &sb_phys,
> + GFP_KERNEL);
> + if (!sb_virt) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Status block allocation failed for id = %d.\n",
> + sb_id);
> + return -ENOMEM;
> + }
> +
> + ret = qedi_ops->common->sb_init(qedi->cdev, sb_info, sb_virt, sb_phys,
> + sb_id, QED_SB_TYPE_STORAGE);
> + if (ret) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Status block initialization failed for id = %d.\n",
> + sb_id);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static void qedi_free_sb(struct qedi_ctx *qedi)
> +{
> + struct qed_sb_info *sb_info;
> + int id;
> +
> + for (id = 0; id < MIN_NUM_CPUS_MSIX(qedi); id++) {
> + sb_info = &qedi->sb_array[id];
> + if (sb_info->sb_virt)
> + dma_free_coherent(&qedi->pdev->dev,
> + sizeof(*sb_info->sb_virt),
> + (void *)sb_info->sb_virt,
> + sb_info->sb_phys);
> + }
> +}
> +
> +static void qedi_free_fp(struct qedi_ctx *qedi)
> +{
> + kfree(qedi->fp_array);
> + kfree(qedi->sb_array);
> +}
> +
> +static void qedi_destroy_fp(struct qedi_ctx *qedi)
> +{
> + qedi_free_sb(qedi);
> + qedi_free_fp(qedi);
> +}
> +
> +static int qedi_alloc_fp(struct qedi_ctx *qedi)
> +{
> + int ret = 0;
> +
> + qedi->fp_array = kcalloc(MIN_NUM_CPUS_MSIX(qedi),
> + sizeof(struct qedi_fastpath), GFP_KERNEL);
> + if (!qedi->fp_array) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "fastpath fp array allocation failed.\n");
> + return -ENOMEM;
> + }
> +
> + qedi->sb_array = kcalloc(MIN_NUM_CPUS_MSIX(qedi),
> + sizeof(struct qed_sb_info), GFP_KERNEL);
> + if (!qedi->sb_array) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "fastpath sb array allocation failed.\n");
> + ret = -ENOMEM;
> + goto free_fp;
> + }
> +
> + return ret;
> +
> +free_fp:
> + qedi_free_fp(qedi);
> + return ret;
> +}
> +
> +static void qedi_int_fp(struct qedi_ctx *qedi)
> +{
> + struct qedi_fastpath *fp;
> + int id;
> +
> + memset(qedi->fp_array, 0, MIN_NUM_CPUS_MSIX(qedi) *
> + sizeof(*qedi->fp_array));
> + memset(qedi->sb_array, 0, MIN_NUM_CPUS_MSIX(qedi) *
> + sizeof(*qedi->sb_array));
> +
> + for (id = 0; id < MIN_NUM_CPUS_MSIX(qedi); id++) {
> + fp = &qedi->fp_array[id];
> + fp->sb_info = &qedi->sb_array[id];
> + fp->sb_id = id;
> + fp->qedi = qedi;
> + snprintf(fp->name, sizeof(fp->name), "%s-fp-%d",
> + "qedi", id);
> +
> + /* fp_array[i] ---- irq cookie
> + * So init data which is needed in int ctx
> + */
> + }
> +}
> +
> +static int qedi_prepare_fp(struct qedi_ctx *qedi)
> +{
> + struct qedi_fastpath *fp;
> + int id, ret = 0;
> +
> + ret = qedi_alloc_fp(qedi);
> + if (ret)
> + goto err;
> +
> + qedi_int_fp(qedi);
> +
> + for (id = 0; id < MIN_NUM_CPUS_MSIX(qedi); id++) {
> + fp = &qedi->fp_array[id];
> + ret = qedi_alloc_and_init_sb(qedi, fp->sb_info, fp->sb_id);
> + if (ret) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "SB allocation and initialization failed.\n");
> + ret = -EIO;
> + goto err_init;
> + }
> + }
> +
> + return 0;
> +
> +err_init:
> + qedi_free_sb(qedi);
> + qedi_free_fp(qedi);
> +err:
> + return ret;
> +}
> +
> +static enum qed_int_mode qedi_int_mode_to_enum(void)
> +{
> + switch (int_mode) {
> + case 0: return QED_INT_MODE_MSIX;
> + case 1: return QED_INT_MODE_INTA;
> + case 2: return QED_INT_MODE_MSI;
> + default:
> + QEDI_ERR(NULL, "Unknown qede_int_mode=%08x; "
> + "Defaulting to MSI-x\n", int_mode);
> + return QED_INT_MODE_MSIX;
> + }
> +}
Errm. A per-driver interrupt mode?
How very curious.
You surely want to make that per-HBA, right?
> +
> +static int qedi_setup_cid_que(struct qedi_ctx *qedi)
> +{
> + int i;
> +
> + qedi->cid_que.cid_que_base = kmalloc_array(qedi->max_active_conns,
> + sizeof(u32), GFP_KERNEL);
> + if (!qedi->cid_que.cid_que_base)
> + return -ENOMEM;
> +
> + qedi->cid_que.conn_cid_tbl = kmalloc_array(qedi->max_active_conns,
> + sizeof(struct qedi_conn *),
> + GFP_KERNEL);
> + if (!qedi->cid_que.conn_cid_tbl) {
> + kfree(qedi->cid_que.cid_que_base);
> + qedi->cid_que.cid_que_base = NULL;
> + return -ENOMEM;
> + }
> +
> + qedi->cid_que.cid_que = (u32 *)qedi->cid_que.cid_que_base;
> + qedi->cid_que.cid_q_prod_idx = 0;
> + qedi->cid_que.cid_q_cons_idx = 0;
> + qedi->cid_que.cid_q_max_idx = qedi->max_active_conns;
> + qedi->cid_que.cid_free_cnt = qedi->max_active_conns;
> +
> + for (i = 0; i < qedi->max_active_conns; i++) {
> + qedi->cid_que.cid_que[i] = i;
> + qedi->cid_que.conn_cid_tbl[i] = NULL;
> + }
> +
> + return 0;
> +}
> +
> +static void qedi_release_cid_que(struct qedi_ctx *qedi)
> +{
> + kfree(qedi->cid_que.cid_que_base);
> + qedi->cid_que.cid_que_base = NULL;
> +
> + kfree(qedi->cid_que.conn_cid_tbl);
> + qedi->cid_que.conn_cid_tbl = NULL;
> +}
> +
> +static int qedi_init_id_tbl(struct qedi_portid_tbl *id_tbl, u16 size,
> + u16 start_id, u16 next)
> +{
> + id_tbl->start = start_id;
> + id_tbl->max = size;
> + id_tbl->next = next;
> + spin_lock_init(&id_tbl->lock);
> + id_tbl->table = kzalloc(DIV_ROUND_UP(size, 32) * 4, GFP_KERNEL);
> + if (!id_tbl->table)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static void qedi_free_id_tbl(struct qedi_portid_tbl *id_tbl)
> +{
> + kfree(id_tbl->table);
> + id_tbl->table = NULL;
> +}
> +
> +int qedi_alloc_id(struct qedi_portid_tbl *id_tbl, u16 id)
> +{
> + int ret = -1;
> +
> + id -= id_tbl->start;
> + if (id >= id_tbl->max)
> + return ret;
> +
> + spin_lock(&id_tbl->lock);
> + if (!test_bit(id, id_tbl->table)) {
> + set_bit(id, id_tbl->table);
> + ret = 0;
> + }
> + spin_unlock(&id_tbl->lock);
> + return ret;
> +}
> +
> +u16 qedi_alloc_new_id(struct qedi_portid_tbl *id_tbl)
> +{
> + u16 id;
> +
> + spin_lock(&id_tbl->lock);
> + id = find_next_zero_bit(id_tbl->table, id_tbl->max, id_tbl->next);
> + if (id >= id_tbl->max) {
> + id = QEDI_LOCAL_PORT_INVALID;
> + if (id_tbl->next != 0) {
> + id = find_first_zero_bit(id_tbl->table, id_tbl->next);
> + if (id >= id_tbl->next)
> + id = QEDI_LOCAL_PORT_INVALID;
> + }
> + }
> +
> + if (id < id_tbl->max) {
> + set_bit(id, id_tbl->table);
> + id_tbl->next = (id + 1) & (id_tbl->max - 1);
> + id += id_tbl->start;
> + }
> +
> + spin_unlock(&id_tbl->lock);
> +
> + return id;
> +}
> +
> +void qedi_free_id(struct qedi_portid_tbl *id_tbl, u16 id)
> +{
> + if (id == QEDI_LOCAL_PORT_INVALID)
> + return;
> +
> + id -= id_tbl->start;
> + if (id >= id_tbl->max)
> + return;
> +
> + clear_bit(id, id_tbl->table);
> +}
> +
> +static void qedi_cm_free_mem(struct qedi_ctx *qedi)
> +{
> + kfree(qedi->ep_tbl);
> + qedi->ep_tbl = NULL;
> + qedi_free_id_tbl(&qedi->lcl_port_tbl);
> +}
> +
> +static int qedi_cm_alloc_mem(struct qedi_ctx *qedi)
> +{
> + u16 port_id;
> +
> + qedi->ep_tbl = kzalloc((qedi->max_active_conns *
> + sizeof(struct qedi_endpoint *)), GFP_KERNEL);
> + if (!qedi->ep_tbl)
> + return -ENOMEM;
> + port_id = prandom_u32() % QEDI_LOCAL_PORT_RANGE;
> + if (qedi_init_id_tbl(&qedi->lcl_port_tbl, QEDI_LOCAL_PORT_RANGE,
> + QEDI_LOCAL_PORT_MIN, port_id)) {
> + qedi_cm_free_mem(qedi);
> + return -ENOMEM;
> + }
> +
> + return 0;
> +}
> +
> +static struct qedi_ctx *qedi_host_alloc(struct pci_dev *pdev)
> +{
> + struct Scsi_Host *shost;
> + struct qedi_ctx *qedi = NULL;
> +
> + shost = iscsi_host_alloc(&qedi_host_template,
> + sizeof(struct qedi_ctx), 0);
> + if (!shost) {
> + QEDI_ERR(NULL, "Could not allocate shost\n");
> + goto exit_setup_shost;
> + }
> +
> + shost->max_id = QEDI_MAX_ISCSI_CONNS_PER_HBA;
> + shost->max_channel = 0;
> + shost->max_lun = ~0;
> + shost->max_cmd_len = 16;
> + shost->transportt = qedi_scsi_transport;
> +
> + qedi = iscsi_host_priv(shost);
> + memset(qedi, 0, sizeof(*qedi));
> + qedi->shost = shost;
> + qedi->dbg_ctx.host_no = shost->host_no;
> + qedi->pdev = pdev;
> + qedi->dbg_ctx.pdev = pdev;
> + qedi->max_active_conns = ISCSI_MAX_SESS_PER_HBA;
> + qedi->max_sqes = QEDI_SQ_SIZE;
> +
> + if (shost_use_blk_mq(shost))
> + shost->nr_hw_queues = MIN_NUM_CPUS_MSIX(qedi);
> +
> + pci_set_drvdata(pdev, qedi);
> +
> +exit_setup_shost:
> + return qedi;
> +}
> +
A round of silent applause for implementing scsi-mq support ...
> +static int qedi_set_iscsi_pf_param(struct qedi_ctx *qedi)
> +{
> + u8 num_sq_pages;
> + u32 log_page_size;
> + int rval = 0;
> +
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC, "Min number of MSIX %d\n",
> + MIN_NUM_CPUS_MSIX(qedi));
> +
> + num_sq_pages = (MAX_OUSTANDING_TASKS_PER_CON * 8) / PAGE_SIZE;
> +
> + qedi->num_queues = MIN_NUM_CPUS_MSIX(qedi);
> +
> + memset(&qedi->pf_params.iscsi_pf_params, 0,
> + sizeof(qedi->pf_params.iscsi_pf_params));
> +
> + qedi->p_cpuq = pci_alloc_consistent(qedi->pdev,
> + qedi->num_queues * sizeof(struct qedi_glbl_q_params),
> + &qedi->hw_p_cpuq);
> + if (!qedi->p_cpuq) {
> + QEDI_ERR(&qedi->dbg_ctx, "pci_alloc_consistent fail\n");
> + rval = -1;
> + goto err_alloc_mem;
> + }
> +
> + rval = qedi_alloc_global_queues(qedi);
> + if (rval) {
> + QEDI_ERR(&qedi->dbg_ctx, "Global queue allocation failed.\n");
> + rval = -1;
> + goto err_alloc_mem;
> + }
> +
> + qedi->pf_params.iscsi_pf_params.num_cons = QEDI_MAX_ISCSI_CONNS_PER_HBA;
> + qedi->pf_params.iscsi_pf_params.num_tasks = QEDI_MAX_ISCSI_TASK;
> + qedi->pf_params.iscsi_pf_params.half_way_close_timeout = 10;
> + qedi->pf_params.iscsi_pf_params.num_sq_pages_in_ring = num_sq_pages;
> + qedi->pf_params.iscsi_pf_params.num_r2tq_pages_in_ring = num_sq_pages;
> + qedi->pf_params.iscsi_pf_params.num_uhq_pages_in_ring = num_sq_pages;
> + qedi->pf_params.iscsi_pf_params.num_queues = qedi->num_queues;
> + qedi->pf_params.iscsi_pf_params.debug_mode = fw_debug;
> +
> + for (log_page_size = 0 ; log_page_size < 32 ; log_page_size++) {
> + if ((1 << log_page_size) == PAGE_SIZE)
> + break;
> + }
> + qedi->pf_params.iscsi_pf_params.log_page_size = log_page_size;
> +
> + qedi->pf_params.iscsi_pf_params.glbl_q_params_addr = qedi->hw_p_cpuq;
> +
> + /* RQ BDQ initializations.
> + * rq_num_entries: suggested value for Initiator is 16 (4KB RQ)
> + * rqe_log_size: 8 for 256B RQE
> + */
> + qedi->pf_params.iscsi_pf_params.rqe_log_size = 8;
> + /* BDQ address and size */
> + qedi->pf_params.iscsi_pf_params.bdq_pbl_base_addr[BDQ_ID_RQ] =
> + qedi->bdq_pbl_list_dma;
> + qedi->pf_params.iscsi_pf_params.bdq_pbl_num_entries[BDQ_ID_RQ] =
> + qedi->bdq_pbl_list_num_entries;
> + qedi->pf_params.iscsi_pf_params.rq_buffer_size = QEDI_BDQ_BUF_SIZE;
> +
> + /* cq_num_entries: num_tasks + rq_num_entries */
> + qedi->pf_params.iscsi_pf_params.cq_num_entries = 2048;
> +
> + qedi->pf_params.iscsi_pf_params.gl_rq_pi = QEDI_PROTO_CQ_PROD_IDX;
> + qedi->pf_params.iscsi_pf_params.gl_cmd_pi = 1;
> + qedi->pf_params.iscsi_pf_params.ooo_enable = 1;
> +
> +err_alloc_mem:
> + return rval;
> +}
> +
> +/* Free DMA coherent memory for array of queue pointers we pass to qed */
> +static void qedi_free_iscsi_pf_param(struct qedi_ctx *qedi)
> +{
> + size_t size = 0;
> +
> + if (qedi->p_cpuq) {
> + size = qedi->num_queues * sizeof(struct qedi_glbl_q_params);
> + pci_free_consistent(qedi->pdev, size, qedi->p_cpuq,
> + qedi->hw_p_cpuq);
> + }
> +
> + qedi_free_global_queues(qedi);
> +
> + kfree(qedi->global_queues);
> +}
> +
> +static void qedi_link_update(void *dev, struct qed_link_output *link)
> +{
> + struct qedi_ctx *qedi = (struct qedi_ctx *)dev;
> +
> + if (link->link_up) {
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_INFO, "Link Up event.\n");
> + atomic_set(&qedi->link_state, QEDI_LINK_UP);
> + } else {
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_INFO,
> + "Link Down event.\n");
> + atomic_set(&qedi->link_state, QEDI_LINK_DOWN);
> + }
> +}
> +
> +static struct qed_iscsi_cb_ops qedi_cb_ops = {
> + {
> + .link_update = qedi_link_update,
> + }
> +};
> +
> +static int qedi_queue_cqe(struct qedi_ctx *qedi, union iscsi_cqe *cqe,
> + u16 que_idx, struct qedi_percpu_s *p)
> +{
> + struct qedi_work *qedi_work;
> + struct qedi_conn *q_conn;
> + struct iscsi_conn *conn;
> + struct qedi_cmd *qedi_cmd;
> + u32 iscsi_cid;
> + int rc = 0;
> +
> + iscsi_cid = cqe->cqe_common.conn_id;
> + q_conn = qedi->cid_que.conn_cid_tbl[iscsi_cid];
> + if (!q_conn) {
> + QEDI_WARN(&qedi->dbg_ctx,
> + "Session no longer exists for cid=0x%x!!\n",
> + iscsi_cid);
> + return -1;
> + }
> + conn = q_conn->cls_conn->dd_data;
> +
> + switch (cqe->cqe_common.cqe_type) {
> + case ISCSI_CQE_TYPE_SOLICITED:
> + case ISCSI_CQE_TYPE_SOLICITED_WITH_SENSE:
> + qedi_cmd = qedi_get_cmd_from_tid(qedi, cqe->cqe_solicited.itid);
> + if (!qedi_cmd) {
> + rc = -1;
> + break;
> + }
> + INIT_LIST_HEAD(&qedi_cmd->cqe_work.list);
> + qedi_cmd->cqe_work.qedi = qedi;
> + memcpy(&qedi_cmd->cqe_work.cqe, cqe, sizeof(union iscsi_cqe));
> + qedi_cmd->cqe_work.que_idx = que_idx;
> + qedi_cmd->cqe_work.is_solicited = true;
> + list_add_tail(&qedi_cmd->cqe_work.list, &p->work_list);
> + break;
> + case ISCSI_CQE_TYPE_UNSOLICITED:
> + case ISCSI_CQE_TYPE_DUMMY:
> + case ISCSI_CQE_TYPE_TASK_CLEANUP:
> + qedi_work = kzalloc(sizeof(*qedi_work), GFP_ATOMIC);
> + if (!qedi_work) {
> + rc = -1;
> + break;
> + }
> + INIT_LIST_HEAD(&qedi_work->list);
> + qedi_work->qedi = qedi;
> + memcpy(&qedi_work->cqe, cqe, sizeof(union iscsi_cqe));
> + qedi_work->que_idx = que_idx;
> + qedi_work->is_solicited = false;
> + list_add_tail(&qedi_work->list, &p->work_list);
> + break;
> + default:
> + rc = -1;
> + QEDI_ERR(&qedi->dbg_ctx, "FW Error cqe.\n");
> + }
> + return rc;
> +}
> +
> +static bool qedi_process_completions(struct qedi_fastpath *fp)
> +{
> + struct qedi_ctx *qedi = fp->qedi;
> + struct qed_sb_info *sb_info = fp->sb_info;
> + struct status_block *sb = sb_info->sb_virt;
> + struct qedi_percpu_s *p = NULL;
> + struct global_queue *que;
> + u16 prod_idx;
> + unsigned long flags;
> + union iscsi_cqe *cqe;
> + int cpu;
> + int ret;
> +
> + /* Get the current firmware producer index */
> + prod_idx = sb->pi_array[QEDI_PROTO_CQ_PROD_IDX];
> +
> + if (prod_idx >= QEDI_CQ_SIZE)
> + prod_idx = prod_idx % QEDI_CQ_SIZE;
> +
> + que = qedi->global_queues[fp->sb_id];
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_IO,
> + "Before: global queue=%p prod_idx=%d cons_idx=%d, sb_id=%d\n",
> + que, prod_idx, que->cq_cons_idx, fp->sb_id);
> +
> + qedi->intr_cpu = fp->sb_id;
> + cpu = smp_processor_id();
> + p = &per_cpu(qedi_percpu, cpu);
> +
> + if (unlikely(!p->iothread))
> + WARN_ON(1);
> +
> + spin_lock_irqsave(&p->p_work_lock, flags);
> + while (que->cq_cons_idx != prod_idx) {
> + cqe = &que->cq[que->cq_cons_idx];
> +
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_IO,
> + "cqe=%p prod_idx=%d cons_idx=%d.\n",
> + cqe, prod_idx, que->cq_cons_idx);
> +
> + ret = qedi_queue_cqe(qedi, cqe, fp->sb_id, p);
> + if (ret)
> + continue;
> +
> + que->cq_cons_idx++;
> + if (que->cq_cons_idx == QEDI_CQ_SIZE)
> + que->cq_cons_idx = 0;
> + }
> + wake_up_process(p->iothread);
> + spin_unlock_irqrestore(&p->p_work_lock, flags);
> +
> + return true;
> +}
> +
> +static bool qedi_fp_has_work(struct qedi_fastpath *fp)
> +{
> + struct qedi_ctx *qedi = fp->qedi;
> + struct global_queue *que;
> + struct qed_sb_info *sb_info = fp->sb_info;
> + struct status_block *sb = sb_info->sb_virt;
> + u16 prod_idx;
> +
> + barrier();
> +
> + /* Get the current firmware producer index */
> + prod_idx = sb->pi_array[QEDI_PROTO_CQ_PROD_IDX];
> +
> + /* Get the pointer to the global CQ this completion is on */
> + que = qedi->global_queues[fp->sb_id];
> +
> + /* prod idx wrap around uint16 */
> + if (prod_idx >= QEDI_CQ_SIZE)
> + prod_idx = prod_idx % QEDI_CQ_SIZE;
> +
> + return (que->cq_cons_idx != prod_idx);
> +}
> +
> +/* MSI-X fastpath handler code */
> +static irqreturn_t qedi_msix_handler(int irq, void *dev_id)
> +{
> + struct qedi_fastpath *fp = dev_id;
> + struct qedi_ctx *qedi = fp->qedi;
> + bool wake_io_thread = true;
> +
> + qed_sb_ack(fp->sb_info, IGU_INT_DISABLE, 0);
> +
> +process_again:
> + wake_io_thread = qedi_process_completions(fp);
> + if (wake_io_thread) {
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "process already running\n");
> + }
> +
> + if (qedi_fp_has_work(fp) == 0)
> + qed_sb_update_sb_idx(fp->sb_info);
> +
> + /* Check for more work */
> + rmb();
> +
> + if (qedi_fp_has_work(fp) == 0)
> + qed_sb_ack(fp->sb_info, IGU_INT_ENABLE, 1);
> + else
> + goto process_again;
> +
> + return IRQ_HANDLED;
> +}
> +
> +/* simd handler for MSI/INTa */
> +static void qedi_simd_int_handler(void *cookie)
> +{
> + /* Cookie is qedi_ctx struct */
> + struct qedi_ctx *qedi = (struct qedi_ctx *)cookie;
> +
> + QEDI_WARN(&qedi->dbg_ctx, "qedi=%p.\n", qedi);
> +}
> +
> +#define QEDI_SIMD_HANDLER_NUM 0
> +static void qedi_sync_free_irqs(struct qedi_ctx *qedi)
> +{
> + int i;
> +
> + if (qedi->int_info.msix_cnt) {
> + for (i = 0; i < qedi->int_info.used_cnt; i++) {
> + synchronize_irq(qedi->int_info.msix[i].vector);
> + irq_set_affinity_hint(qedi->int_info.msix[i].vector,
> + NULL);
> + free_irq(qedi->int_info.msix[i].vector,
> + &qedi->fp_array[i]);
> + }
> + } else {
> + qedi_ops->common->simd_handler_clean(qedi->cdev,
> + QEDI_SIMD_HANDLER_NUM);
> + }
> +
> + qedi->int_info.used_cnt = 0;
> + qedi_ops->common->set_fp_int(qedi->cdev, 0);
> +}
> +
> +static int qedi_request_msix_irq(struct qedi_ctx *qedi)
> +{
> + int i, rc, cpu;
> +
> + cpu = cpumask_first(cpu_online_mask);
> + for (i = 0; i < MIN_NUM_CPUS_MSIX(qedi); i++) {
> + rc = request_irq(qedi->int_info.msix[i].vector,
> + qedi_msix_handler, 0, "qedi",
> + &qedi->fp_array[i]);
> +
> + if (rc) {
> + QEDI_WARN(&qedi->dbg_ctx, "request_irq failed.\n");
> + qedi_sync_free_irqs(qedi);
> + return rc;
> + }
> + qedi->int_info.used_cnt++;
> + rc = irq_set_affinity_hint(qedi->int_info.msix[i].vector,
> + get_cpu_mask(cpu));
> + cpu = cpumask_next(cpu, cpu_online_mask);
> + }
> +
> + return 0;
> +}
Please use the irq-affinity rework from Christoph here; that'll save you
the additional msix vectors allocation.
> +
> +static int qedi_setup_int(struct qedi_ctx *qedi)
> +{
> + int rc = 0;
> +
> + rc = qedi_ops->common->set_fp_int(qedi->cdev, num_online_cpus());
> + rc = qedi_ops->common->get_fp_int(qedi->cdev, &qedi->int_info);
> + if (rc)
> + goto exit_setup_int;
> +
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "Number of msix_cnt = 0x%x num of cpus = 0x%x\n",
> + qedi->int_info.msix_cnt, num_online_cpus());
> +
> + if (qedi->int_info.msix_cnt) {
> + rc = qedi_request_msix_irq(qedi);
> + goto exit_setup_int;
> + } else {
> + qedi_ops->common->simd_handler_config(qedi->cdev, &qedi,
> + QEDI_SIMD_HANDLER_NUM,
> + qedi_simd_int_handler);
> + qedi->int_info.used_cnt = 1;
> + }
> +
> +exit_setup_int:
> + return rc;
> +}
> +
> +static void qedi_free_bdq(struct qedi_ctx *qedi)
> +{
> + int i;
> +
> + if (qedi->bdq_pbl_list)
> + dma_free_coherent(&qedi->pdev->dev, PAGE_SIZE,
> + qedi->bdq_pbl_list, qedi->bdq_pbl_list_dma);
> +
> + if (qedi->bdq_pbl)
> + dma_free_coherent(&qedi->pdev->dev, qedi->bdq_pbl_mem_size,
> + qedi->bdq_pbl, qedi->bdq_pbl_dma);
> +
> + for (i = 0; i < QEDI_BDQ_NUM; i++) {
> + if (qedi->bdq[i].buf_addr) {
> + dma_free_coherent(&qedi->pdev->dev, QEDI_BDQ_BUF_SIZE,
> + qedi->bdq[i].buf_addr,
> + qedi->bdq[i].buf_dma);
> + }
> + }
> +}
> +
> +static void qedi_free_global_queues(struct qedi_ctx *qedi)
> +{
> + int i;
> + struct global_queue **gl = qedi->global_queues;
> +
> + for (i = 0; i < qedi->num_queues; i++) {
> + if (!gl[i])
> + continue;
> +
> + if (gl[i]->cq)
> + dma_free_coherent(&qedi->pdev->dev, gl[i]->cq_mem_size,
> + gl[i]->cq, gl[i]->cq_dma);
> + if (gl[i]->cq_pbl)
> + dma_free_coherent(&qedi->pdev->dev, gl[i]->cq_pbl_size,
> + gl[i]->cq_pbl, gl[i]->cq_pbl_dma);
> +
> + kfree(gl[i]);
> + }
> + qedi_free_bdq(qedi);
> +}
> +
> +static int qedi_alloc_bdq(struct qedi_ctx *qedi)
> +{
> + int i;
> + struct scsi_bd *pbl;
> + u64 *list;
> + dma_addr_t page;
> +
> + /* Alloc dma memory for BDQ buffers */
> + for (i = 0; i < QEDI_BDQ_NUM; i++) {
> + qedi->bdq[i].buf_addr =
> + dma_alloc_coherent(&qedi->pdev->dev,
> + QEDI_BDQ_BUF_SIZE,
> + &qedi->bdq[i].buf_dma,
> + GFP_KERNEL);
> + if (!qedi->bdq[i].buf_addr) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not allocate BDQ buffer %d.\n", i);
> + return -ENOMEM;
> + }
> + }
> +
> + /* Alloc dma memory for BDQ page buffer list */
> + qedi->bdq_pbl_mem_size = QEDI_BDQ_NUM * sizeof(struct scsi_bd);
> + qedi->bdq_pbl_mem_size = ALIGN(qedi->bdq_pbl_mem_size, PAGE_SIZE);
> + qedi->rq_num_entries = qedi->bdq_pbl_mem_size / sizeof(struct scsi_bd);
> +
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_CONN, "rq_num_entries = %d.\n",
> + qedi->rq_num_entries);
> +
> + qedi->bdq_pbl = dma_alloc_coherent(&qedi->pdev->dev,
> + qedi->bdq_pbl_mem_size,
> + &qedi->bdq_pbl_dma, GFP_KERNEL);
> + if (!qedi->bdq_pbl) {
> + QEDI_ERR(&qedi->dbg_ctx, "Could not allocate BDQ PBL.\n");
> + return -ENOMEM;
> + }
> +
> + /*
> + * Populate BDQ PBL with physical and virtual address of individual
> + * BDQ buffers
> + */
> + pbl = (struct scsi_bd *)qedi->bdq_pbl;
> + for (i = 0; i < QEDI_BDQ_NUM; i++) {
> + pbl->address.hi =
> + cpu_to_le32(QEDI_U64_HI(qedi->bdq[i].buf_dma));
> + pbl->address.lo =
> + cpu_to_le32(QEDI_U64_LO(qedi->bdq[i].buf_dma));
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_CONN,
> + "pbl [0x%p] pbl->address hi [0x%llx] lo [0x%llx], idx [%d]\n",
> + pbl, pbl->address.hi, pbl->address.lo, i);
> + pbl->opaque.hi = 0;
> + pbl->opaque.lo = cpu_to_le32(QEDI_U64_LO(i));
> + pbl++;
> + }
> +
> + /* Allocate list of PBL pages */
> + qedi->bdq_pbl_list = dma_alloc_coherent(&qedi->pdev->dev,
> + PAGE_SIZE,
> + &qedi->bdq_pbl_list_dma,
> + GFP_KERNEL);
> + if (!qedi->bdq_pbl_list) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not allocate list of PBL pages.\n");
> + return -ENOMEM;
> + }
> + memset(qedi->bdq_pbl_list, 0, PAGE_SIZE);
> +
> + /*
> + * Now populate PBL list with pages that contain pointers to the
> + * individual buffers.
> + */
> + qedi->bdq_pbl_list_num_entries = qedi->bdq_pbl_mem_size / PAGE_SIZE;
> + list = (u64 *)qedi->bdq_pbl_list;
> + page = qedi->bdq_pbl_list_dma;
> + for (i = 0; i < qedi->bdq_pbl_list_num_entries; i++) {
> + *list = qedi->bdq_pbl_dma;
> + list++;
> + page += PAGE_SIZE;
> + }
> +
> + return 0;
> +}
> +
> +static int qedi_alloc_global_queues(struct qedi_ctx *qedi)
> +{
> + u32 *list;
> + int i;
> + int status = 0, rc;
> + u32 *pbl;
> + dma_addr_t page;
> + int num_pages;
> +
> + /*
> + * Number of global queues (CQ / RQ). This should
> + * be <= number of available MSIX vectors for the PF
> + */
> + if (!qedi->num_queues) {
> + QEDI_ERR(&qedi->dbg_ctx, "No MSI-X vectors available!\n");
> + return 1;
> + }
> +
> + /* Make sure we allocated the PBL that will contain the physical
> + * addresses of our queues
> + */
> + if (!qedi->p_cpuq) {
> + status = 1;
> + goto mem_alloc_failure;
> + }
> +
> + qedi->global_queues = kzalloc((sizeof(struct global_queue *) *
> + qedi->num_queues), GFP_KERNEL);
> + if (!qedi->global_queues) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Unable to allocate global queues array ptr memory\n");
> + return -ENOMEM;
> + }
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "qedi->global_queues=%p.\n", qedi->global_queues);
> +
> + /* Allocate DMA coherent buffers for BDQ */
> + rc = qedi_alloc_bdq(qedi);
> + if (rc)
> + goto mem_alloc_failure;
> +
> + /* Allocate a CQ and an associated PBL for each MSI-X
> + * vector.
> + */
> + for (i = 0; i < qedi->num_queues; i++) {
> + qedi->global_queues[i] =
> + kzalloc(sizeof(*qedi->global_queues[0]),
> + GFP_KERNEL);
> + if (!qedi->global_queues[i]) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Unable to allocation global queue %d.\n", i);
> + goto mem_alloc_failure;
> + }
> +
> + qedi->global_queues[i]->cq_mem_size =
> + (QEDI_CQ_SIZE + 8) * sizeof(union iscsi_cqe);
> + qedi->global_queues[i]->cq_mem_size =
> + (qedi->global_queues[i]->cq_mem_size +
> + (QEDI_PAGE_SIZE - 1));
> +
> + qedi->global_queues[i]->cq_pbl_size =
> + (qedi->global_queues[i]->cq_mem_size /
> + QEDI_PAGE_SIZE) * sizeof(void *);
> + qedi->global_queues[i]->cq_pbl_size =
> + (qedi->global_queues[i]->cq_pbl_size +
> + (QEDI_PAGE_SIZE - 1));
> +
> + qedi->global_queues[i]->cq =
> + dma_alloc_coherent(&qedi->pdev->dev,
> + qedi->global_queues[i]->cq_mem_size,
> + &qedi->global_queues[i]->cq_dma,
> + GFP_KERNEL);
> +
> + if (!qedi->global_queues[i]->cq) {
> + QEDI_WARN(&qedi->dbg_ctx,
> + "Could not allocate cq.\n");
> + status = -ENOMEM;
> + goto mem_alloc_failure;
> + }
> + memset(qedi->global_queues[i]->cq, 0,
> + qedi->global_queues[i]->cq_mem_size);
> +
> + qedi->global_queues[i]->cq_pbl =
> + dma_alloc_coherent(&qedi->pdev->dev,
> + qedi->global_queues[i]->cq_pbl_size,
> + &qedi->global_queues[i]->cq_pbl_dma,
> + GFP_KERNEL);
> +
> + if (!qedi->global_queues[i]->cq_pbl) {
> + QEDI_WARN(&qedi->dbg_ctx,
> + "Could not allocate cq PBL.\n");
> + status = -ENOMEM;
> + goto mem_alloc_failure;
> + }
> + memset(qedi->global_queues[i]->cq_pbl, 0,
> + qedi->global_queues[i]->cq_pbl_size);
> +
> + /* Create PBL */
> + num_pages = qedi->global_queues[i]->cq_mem_size /
> + QEDI_PAGE_SIZE;
> + page = qedi->global_queues[i]->cq_dma;
> + pbl = (u32 *)qedi->global_queues[i]->cq_pbl;
> +
> + while (num_pages--) {
> + *pbl = (u32)page;
> + pbl++;
> + *pbl = (u32)((u64)page >> 32);
> + pbl++;
> + page += QEDI_PAGE_SIZE;
> + }
> + }
> +
> + list = (u32 *)qedi->p_cpuq;
> +
> + /*
> + * The list is built as follows: CQ#0 PBL pointer, RQ#0 PBL pointer,
> + * CQ#1 PBL pointer, RQ#1 PBL pointer, etc. Each PBL pointer points
> + * to the physical address which contains an array of pointers to the
> + * physical addresses of the specific queue pages.
> + */
> + for (i = 0; i < qedi->num_queues; i++) {
> + *list = (u32)qedi->global_queues[i]->cq_pbl_dma;
> + list++;
> + *list = (u32)((u64)qedi->global_queues[i]->cq_pbl_dma >> 32);
> + list++;
> +
> + *list = (u32)0;
> + list++;
> + *list = (u32)((u64)0 >> 32);
> + list++;
> + }
> +
> + return 0;
> +
> +mem_alloc_failure:
> + qedi_free_global_queues(qedi);
> + return status;
> +}
> +
> +struct qedi_cmd *qedi_get_cmd_from_tid(struct qedi_ctx *qedi, u32 tid)
> +{
> + struct qedi_cmd *cmd = NULL;
> +
> + if (tid > MAX_ISCSI_TASK_ENTRIES)
> + return NULL;
> +
> + cmd = qedi->itt_map[tid].p_cmd;
> + if (cmd->task_id != tid)
> + return NULL;
> +
> + qedi->itt_map[tid].p_cmd = NULL;
> +
> + return cmd;
> +}
> +
> +static int qedi_alloc_itt(struct qedi_ctx *qedi)
> +{
> + qedi->itt_map = kcalloc(MAX_ISCSI_TASK_ENTRIES,
> + sizeof(struct qedi_itt_map), GFP_KERNEL);
> + if (!qedi->itt_map) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Unable to allocate itt map array memory\n");
> + return -ENOMEM;
> + }
> + return 0;
> +}
> +
> +static void qedi_free_itt(struct qedi_ctx *qedi)
> +{
> + kfree(qedi->itt_map);
> +}
> +
> +static struct qed_ll2_cb_ops qedi_ll2_cb_ops = {
> + .rx_cb = qedi_ll2_rx,
> + .tx_cb = NULL,
> +};
> +
> +static int qedi_percpu_io_thread(void *arg)
> +{
> + struct qedi_percpu_s *p = arg;
> + struct qedi_work *work, *tmp;
> + unsigned long flags;
> + LIST_HEAD(work_list);
> +
> + set_user_nice(current, -20);
> +
> + while (!kthread_should_stop()) {
> + spin_lock_irqsave(&p->p_work_lock, flags);
> + while (!list_empty(&p->work_list)) {
> + list_splice_init(&p->work_list, &work_list);
> + spin_unlock_irqrestore(&p->p_work_lock, flags);
> +
> + list_for_each_entry_safe(work, tmp, &work_list, list) {
> + list_del_init(&work->list);
> + qedi_fp_process_cqes(work);
> + if (!work->is_solicited)
> + kfree(work);
> + }
> + cond_resched();
> + spin_lock_irqsave(&p->p_work_lock, flags);
> + }
> + set_current_state(TASK_INTERRUPTIBLE);
> + spin_unlock_irqrestore(&p->p_work_lock, flags);
> + schedule();
> + }
> + __set_current_state(TASK_RUNNING);
> +
> + return 0;
> +}
> +
> +static void qedi_percpu_thread_create(unsigned int cpu)
> +{
> + struct qedi_percpu_s *p;
> + struct task_struct *thread;
> +
> + p = &per_cpu(qedi_percpu, cpu);
> +
> + thread = kthread_create_on_node(qedi_percpu_io_thread, (void *)p,
> + cpu_to_node(cpu),
> + "qedi_thread/%d", cpu);
> + if (likely(!IS_ERR(thread))) {
> + kthread_bind(thread, cpu);
> + p->iothread = thread;
> + wake_up_process(thread);
> + }
> +}
> +
> +static void qedi_percpu_thread_destroy(unsigned int cpu)
> +{
> + struct qedi_percpu_s *p;
> + struct task_struct *thread;
> + struct qedi_work *work, *tmp;
> +
> + p = &per_cpu(qedi_percpu, cpu);
> + spin_lock_bh(&p->p_work_lock);
> + thread = p->iothread;
> + p->iothread = NULL;
> +
> + list_for_each_entry_safe(work, tmp, &p->work_list, list) {
> + list_del_init(&work->list);
> + qedi_fp_process_cqes(work);
> + if (!work->is_solicited)
> + kfree(work);
> + }
> +
> + spin_unlock_bh(&p->p_work_lock);
> + if (thread)
> + kthread_stop(thread);
> +}
> +
> +static int qedi_cpu_callback(struct notifier_block *nfb,
> + unsigned long action, void *hcpu)
> +{
> + unsigned int cpu = (unsigned long)hcpu;
> +
> + switch (action) {
> + case CPU_ONLINE:
> + case CPU_ONLINE_FROZEN:
> + QEDI_ERR(NULL, "CPU %d online.\n", cpu);
> + qedi_percpu_thread_create(cpu);
> + break;
> + case CPU_DEAD:
> + case CPU_DEAD_FROZEN:
> + QEDI_ERR(NULL, "CPU %d offline.\n", cpu);
> + qedi_percpu_thread_destroy(cpu);
> + break;
> + default:
> + break;
> + }
> +
> + return NOTIFY_OK;
> +}
> +
> +static struct notifier_block qedi_cpu_notifier = {
> + .notifier_call = qedi_cpu_callback,
> +};
> +
> +static void __qedi_remove(struct pci_dev *pdev, int mode)
> +{
> + struct qedi_ctx *qedi = pci_get_drvdata(pdev);
> +
> + if (qedi->tmf_thread) {
> + flush_workqueue(qedi->tmf_thread);
> + destroy_workqueue(qedi->tmf_thread);
> + qedi->tmf_thread = NULL;
> + }
> +
> + if (qedi->offload_thread) {
> + flush_workqueue(qedi->offload_thread);
> + destroy_workqueue(qedi->offload_thread);
> + qedi->offload_thread = NULL;
> + }
> +
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_host_exit(&qedi->dbg_ctx);
> +#endif
> + if (!test_bit(QEDI_IN_OFFLINE, &qedi->flags))
> + qedi_ops->common->set_power_state(qedi->cdev, PCI_D0);
> +
> + qedi_sync_free_irqs(qedi);
> +
> + if (!test_bit(QEDI_IN_OFFLINE, &qedi->flags)) {
> + qedi_ops->stop(qedi->cdev);
> + qedi_ops->ll2->stop(qedi->cdev);
> + }
> +
> + if (mode == QEDI_MODE_NORMAL)
> + qedi_free_iscsi_pf_param(qedi);
> +
> + if (!test_bit(QEDI_IN_OFFLINE, &qedi->flags)) {
> + qedi_ops->common->slowpath_stop(qedi->cdev);
> + qedi_ops->common->remove(qedi->cdev);
> + }
> +
> + qedi_destroy_fp(qedi);
> +
> + if (mode == QEDI_MODE_NORMAL) {
> + qedi_release_cid_que(qedi);
> + qedi_cm_free_mem(qedi);
> + qedi_free_uio(qedi->udev);
> + qedi_free_itt(qedi);
> +
> + iscsi_host_remove(qedi->shost);
> + iscsi_host_free(qedi->shost);
> +
> + if (qedi->ll2_recv_thread) {
> + kthread_stop(qedi->ll2_recv_thread);
> + qedi->ll2_recv_thread = NULL;
> + }
> + qedi_ll2_free_skbs(qedi);
> + }
> +}
> +
> +static int __qedi_probe(struct pci_dev *pdev, int mode)
> +{
> + struct qedi_ctx *qedi;
> + struct qed_ll2_params params;
> + u32 dp_module = 0;
> + u8 dp_level = 0;
> + bool is_vf = false;
> + char host_buf[16];
> + struct qed_link_params link_params;
> + struct qed_slowpath_params sp_params;
> + struct qed_probe_params qed_params;
> + void *task_start, *task_end;
> + int rc;
> + u16 tmp;
> +
> + if (mode != QEDI_MODE_RECOVERY) {
> + qedi = qedi_host_alloc(pdev);
> + if (!qedi) {
> + rc = -ENOMEM;
> + goto exit_probe;
> + }
> + } else {
> + qedi = pci_get_drvdata(pdev);
> + }
> +
> + memset(&qed_params, 0, sizeof(qed_params));
> + qed_params.protocol = QED_PROTOCOL_ISCSI;
> + qed_params.dp_module = dp_module;
> + qed_params.dp_level = dp_level;
> + qed_params.is_vf = is_vf;
> + qedi->cdev = qedi_ops->common->probe(pdev, &qed_params);
> + if (!qedi->cdev) {
> + rc = -ENODEV;
> + QEDI_ERR(&qedi->dbg_ctx, "Cannot initialize hardware\n");
> + goto free_host;
> + }
> +
> + qedi->msix_count = MAX_NUM_MSIX_PF;
> + atomic_set(&qedi->link_state, QEDI_LINK_DOWN);
> +
> + if (mode != QEDI_MODE_RECOVERY) {
> + rc = qedi_set_iscsi_pf_param(qedi);
> + if (rc) {
> + rc = -ENOMEM;
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Set iSCSI pf param fail\n");
> + goto free_host;
> + }
> + }
> +
> + qedi_ops->common->update_pf_params(qedi->cdev, &qedi->pf_params);
> +
> + rc = qedi_prepare_fp(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx, "Cannot start slowpath.\n");
> + goto free_pf_params;
> + }
> +
> + /* Start the Slowpath-process */
> + memset(&sp_params, 0, sizeof(struct qed_slowpath_params));
> + sp_params.int_mode = qedi_int_mode_to_enum();
> + sp_params.drv_major = QEDI_DRIVER_MAJOR_VER;
> + sp_params.drv_minor = QEDI_DRIVER_MINOR_VER;
> + sp_params.drv_rev = QEDI_DRIVER_REV_VER;
> + sp_params.drv_eng = QEDI_DRIVER_ENG_VER;
> + strlcpy(sp_params.name, "qedi iSCSI", QED_DRV_VER_STR_SIZE);
> + rc = qedi_ops->common->slowpath_start(qedi->cdev, &sp_params);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx, "Cannot start slowpath\n");
> + goto stop_hw;
> + }
> +
> + /* update_pf_params needs to be called before and after slowpath
> + * start
> + */
> + qedi_ops->common->update_pf_params(qedi->cdev, &qedi->pf_params);
> +
> + qedi_setup_int(qedi);
> + if (rc)
> + goto stop_iscsi_func;
> +
> + qedi_ops->common->set_power_state(qedi->cdev, PCI_D0);
> +
> + /* Learn information crucial for qedi to progress */
> + rc = qedi_ops->fill_dev_info(qedi->cdev, &qedi->dev_info);
> + if (rc)
> + goto stop_iscsi_func;
> +
> + /* Record BDQ producer doorbell addresses */
> + qedi->bdq_primary_prod = qedi->dev_info.primary_dbq_rq_addr;
> + qedi->bdq_secondary_prod = qedi->dev_info.secondary_bdq_rq_addr;
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "BDQ primary_prod=%p secondary_prod=%p.\n",
> + qedi->bdq_primary_prod,
> + qedi->bdq_secondary_prod);
> +
> + /*
> + * We need to write the number of BDs in the BDQ we've preallocated so
> + * the f/w will do a prefetch and we'll get an unsolicited CQE when a
> + * packet arrives.
> + */
> + qedi->bdq_prod_idx = QEDI_BDQ_NUM;
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "Writing %d to primary and secondary BDQ doorbell registers.\n",
> + qedi->bdq_prod_idx);
> + writew(qedi->bdq_prod_idx, qedi->bdq_primary_prod);
> + tmp = readw(qedi->bdq_primary_prod);
> + writew(qedi->bdq_prod_idx, qedi->bdq_secondary_prod);
> + tmp = readw(qedi->bdq_secondary_prod);
> +
> + ether_addr_copy(qedi->mac, qedi->dev_info.common.hw_mac);
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC, "MAC address is %pM.\n",
> + qedi->mac);
> +
> + sprintf(host_buf, "host_%d", qedi->shost->host_no);
> + qedi_ops->common->set_id(qedi->cdev, host_buf, QEDI_MODULE_VERSION);
> +
> + qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi);
> +
> + memset(¶ms, 0, sizeof(params));
> + params.mtu = DEF_PATH_MTU + IPV6_HDR_LEN + TCP_HDR_LEN;
> + qedi->ll2_mtu = DEF_PATH_MTU;
> + params.drop_ttl0_packets = 0;
> + params.rx_vlan_stripping = 1;
> + ether_addr_copy(params.ll2_mac_address, qedi->dev_info.common.hw_mac);
> +
> + if (mode != QEDI_MODE_RECOVERY) {
> + /* set up rx path */
> + INIT_LIST_HEAD(&qedi->ll2_skb_list);
> + spin_lock_init(&qedi->ll2_lock);
> + /* start qedi context */
> + spin_lock_init(&qedi->hba_lock);
> + spin_lock_init(&qedi->task_idx_lock);
> + }
> + qedi_ops->ll2->register_cb_ops(qedi->cdev, &qedi_ll2_cb_ops, qedi);
> + qedi_ops->ll2->start(qedi->cdev, ¶ms);
> +
> + if (mode != QEDI_MODE_RECOVERY) {
> + qedi->ll2_recv_thread = kthread_run(qedi_ll2_recv_thread,
> + (void *)qedi,
> + "qedi_ll2_thread");
> + }
> +
> + rc = qedi_ops->start(qedi->cdev, &qedi->tasks,
> + qedi, qedi_iscsi_event_cb);
> + if (rc) {
> + rc = -ENODEV;
> + QEDI_ERR(&qedi->dbg_ctx, "Cannot start iSCSI function\n");
> + goto stop_slowpath;
> + }
> +
> + task_start = qedi_get_task_mem(&qedi->tasks, 0);
> + task_end = qedi_get_task_mem(&qedi->tasks, MAX_TID_BLOCKS_ISCSI - 1);
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_DISC,
> + "Task context start=%p, end=%p block_size=%u.\n",
> + task_start, task_end, qedi->tasks.size);
> +
> + memset(&link_params, 0, sizeof(link_params));
> + link_params.link_up = true;
> + rc = qedi_ops->common->set_link(qedi->cdev, &link_params);
> + if (rc) {
> + QEDI_WARN(&qedi->dbg_ctx, "Link set up failed.\n");
> + atomic_set(&qedi->link_state, QEDI_LINK_DOWN);
> + }
> +
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_host_init(&qedi->dbg_ctx, &qedi_debugfs_ops,
> + &qedi_dbg_fops);
> +#endif
> + QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_INFO,
> + "QLogic FastLinQ iSCSI Module qedi %s, FW %d.%d.%d.%d\n",
> + QEDI_MODULE_VERSION, FW_MAJOR_VERSION, FW_MINOR_VERSION,
> + FW_REVISION_VERSION, FW_ENGINEERING_VERSION);
> +
> + if (mode == QEDI_MODE_NORMAL) {
> + if (iscsi_host_add(qedi->shost, &pdev->dev)) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not add iscsi host\n");
> + rc = -ENOMEM;
> + goto remove_host;
> + }
> +
> + /* Allocate uio buffers */
> + rc = qedi_alloc_uio_rings(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "UIO alloc ring failed err=%d\n", rc);
> + goto remove_host;
> + }
> +
> + rc = qedi_init_uio(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "UIO init failed, err=%d\n", rc);
> + goto free_uio;
> + }
> +
> + /* host the array on iscsi_conn */
> + rc = qedi_setup_cid_que(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not setup cid que\n");
> + goto free_uio;
> + }
> +
> + rc = qedi_cm_alloc_mem(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not alloc cm memory\n");
> + goto free_cid_que;
> + }
> +
> + rc = qedi_alloc_itt(qedi);
> + if (rc) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Could not alloc itt memory\n");
> + goto free_cid_que;
> + }
> +
> + sprintf(host_buf, "host_%d", qedi->shost->host_no);
> + qedi->tmf_thread = create_singlethread_workqueue(host_buf);
> + if (!qedi->tmf_thread) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Unable to start tmf thread!\n");
> + rc = -ENODEV;
> + goto free_cid_que;
> + }
> +
> + sprintf(host_buf, "qedi_ofld%d", qedi->shost->host_no);
> + qedi->offload_thread = create_workqueue(host_buf);
> + if (!qedi->offload_thread) {
> + QEDI_ERR(&qedi->dbg_ctx,
> + "Unable to start offload thread!\n");
> + rc = -ENODEV;
> + goto free_cid_que;
> + }
> +
> + /* F/w needs 1st task context memory entry for performance */
> + set_bit(QEDI_RESERVE_TASK_ID, qedi->task_idx_map);
> + atomic_set(&qedi->num_offloads, 0);
> + }
> +
> + return 0;
> +
> +free_cid_que:
> + qedi_release_cid_que(qedi);
> +free_uio:
> + qedi_free_uio(qedi->udev);
> +remove_host:
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_host_exit(&qedi->dbg_ctx);
> +#endif
> + iscsi_host_remove(qedi->shost);
> +stop_iscsi_func:
> + qedi_ops->stop(qedi->cdev);
> +stop_slowpath:
> + qedi_ops->common->slowpath_stop(qedi->cdev);
> +stop_hw:
> + qedi_ops->common->remove(qedi->cdev);
> +free_pf_params:
> + qedi_free_iscsi_pf_param(qedi);
> +free_host:
> + iscsi_host_free(qedi->shost);
> +exit_probe:
> + return rc;
> +}
> +
> +static int qedi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> + return __qedi_probe(pdev, QEDI_MODE_NORMAL);
> +}
> +
> +static void qedi_remove(struct pci_dev *pdev)
> +{
> + __qedi_remove(pdev, QEDI_MODE_NORMAL);
> +}
> +
> +static struct pci_device_id qedi_pci_tbl[] = {
> + { PCI_DEVICE(PCI_VENDOR_ID_QLOGIC, 0x165E) },
> + { 0 },
> +};
> +MODULE_DEVICE_TABLE(pci, qedi_pci_tbl);
> +
> +static struct pci_driver qedi_pci_driver = {
> + .name = QEDI_MODULE_NAME,
> + .id_table = qedi_pci_tbl,
> + .probe = qedi_probe,
> + .remove = qedi_remove,
> +};
> +
> +static int __init qedi_init(void)
> +{
> + int rc = 0;
> + int ret;
> + struct qedi_percpu_s *p;
> + unsigned int cpu = 0;
> +
> + qedi_ops = qed_get_iscsi_ops();
> + if (!qedi_ops) {
> + QEDI_ERR(NULL, "Failed to get qed iSCSI operations\n");
> + rc = -EINVAL;
> + goto exit_qedi_init_0;
> + }
> +
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_init("qedi");
> +#endif
> +
> + register_hotcpu_notifier(&qedi_cpu_notifier);
> +
> + ret = pci_register_driver(&qedi_pci_driver);
> + if (ret) {
> + QEDI_ERR(NULL, "Failed to register driver\n");
> + rc = -EINVAL;
> + goto exit_qedi_init_2;
> + }
> +
> + for_each_possible_cpu(cpu) {
> + p = &per_cpu(qedi_percpu, cpu);
> + INIT_LIST_HEAD(&p->work_list);
> + spin_lock_init(&p->p_work_lock);
> + p->iothread = NULL;
> + }
> +
> + for_each_online_cpu(cpu)
> + qedi_percpu_thread_create(cpu);
> +
> + return rc;
> +
> +exit_qedi_init_2:
> +exit_qedi_init_1:
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_exit();
> +#endif
> + qed_put_iscsi_ops();
> +exit_qedi_init_0:
> + return rc;
> +}
> +
> +static void __exit qedi_cleanup(void)
> +{
> + unsigned int cpu = 0;
> +
> + for_each_online_cpu(cpu)
> + qedi_percpu_thread_destroy(cpu);
> +
> + pci_unregister_driver(&qedi_pci_driver);
> + unregister_hotcpu_notifier(&qedi_cpu_notifier);
> +
> +#ifdef CONFIG_DEBUG_FS
> + qedi_dbg_exit();
> +#endif
> + qed_put_iscsi_ops();
> +}
> +
> +MODULE_DESCRIPTION("QLogic FastLinQ 4xxxx iSCSI Module");
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("QLogic Corporation");
> +MODULE_VERSION(QEDI_MODULE_VERSION);
> +module_init(qedi_init);
> +module_exit(qedi_cleanup);
> diff --git a/drivers/scsi/qedi/qedi_sysfs.c b/drivers/scsi/qedi/qedi_sysfs.c
> new file mode 100644
> index 0000000..a2cc3ed
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_sysfs.c
> @@ -0,0 +1,52 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#include "qedi.h"
> +#include "qedi_gbl.h"
> +#include "qedi_iscsi.h"
> +#include "qedi_dbg.h"
> +
> +static inline struct qedi_ctx *qedi_dev_to_hba(struct device *dev)
> +{
> + struct Scsi_Host *shost = class_to_shost(dev);
> +
> + return iscsi_host_priv(shost);
> +}
> +
> +static ssize_t qedi_show_port_state(struct device *dev,
> + struct device_attribute *attr,
> + char *buf)
> +{
> + struct qedi_ctx *qedi = qedi_dev_to_hba(dev);
> +
> + if (atomic_read(&qedi->link_state) == QEDI_LINK_UP)
> + return sprintf(buf, "Online\n");
> + else
> + return sprintf(buf, "Linkdown\n");
> +}
> +
> +static ssize_t qedi_show_speed(struct device *dev,
> + struct device_attribute *attr, char *buf)
> +{
> + struct qedi_ctx *qedi = qedi_dev_to_hba(dev);
> + struct qed_link_output if_link;
> +
> + qedi_ops->common->get_link(qedi->cdev, &if_link);
> +
> + return sprintf(buf, "%d Gbit\n", if_link.speed / 1000);
> +}
> +
> +static DEVICE_ATTR(port_state, S_IRUGO, qedi_show_port_state, NULL);
> +static DEVICE_ATTR(speed, S_IRUGO, qedi_show_speed, NULL);
> +
> +struct device_attribute *qedi_shost_attrs[] = {
> + &dev_attr_port_state,
> + &dev_attr_speed,
> + NULL
> +};
> diff --git a/drivers/scsi/qedi/qedi_version.h b/drivers/scsi/qedi/qedi_version.h
> new file mode 100644
> index 0000000..9543a1b
> --- /dev/null
> +++ b/drivers/scsi/qedi/qedi_version.h
> @@ -0,0 +1,14 @@
> +/*
> + * QLogic iSCSI Offload Driver
> + * Copyright (c) 2016 Cavium Inc.
> + *
> + * This software is available under the terms of the GNU General Public License
> + * (GPL) Version 2, available from the file COPYING in the main directory of
> + * this source tree.
> + */
> +
> +#define QEDI_MODULE_VERSION "8.10.3.0"
> +#define QEDI_DRIVER_MAJOR_VER 8
> +#define QEDI_DRIVER_MINOR_VER 10
> +#define QEDI_DRIVER_REV_VER 3
> +#define QEDI_DRIVER_ENG_VER 0
>
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox