* Re: Information about napi_struct and net_device
From: Stephen Hemminger @ 2012-10-29 1:09 UTC (permalink / raw)
To: Javier Domingo; +Cc: netdev
In-Reply-To: <CALZVap=ybdWdK22LHsfSRu+TVCJnvL95KeChtCdX8DE-C=1gAg@mail.gmail.com>
On Sat, 27 Oct 2012 18:18:39 +0200
Javier Domingo <javierdo1@gmail.com> wrote:
> Hello,
>
> I am updating a kernel patch from 2.6.23-rc7 to v3.6, using git
> facilities, and I have found that there have been lots of changes in
> networking. Now the drivers are all by vendor, and more important,
> net_device is not used always, as napi_struct took its place.
>
> I have asked in #kernel but no one answered me, so I subscribed here.
> Is there any place I can read about how is it now designed the use of
> napi_struct and net_device? I have found a presentation[1] about how
> networking was changing etc, and I wondered if there is anything more
> precise about how it is actually implemented.
>
> I understand a little why napi_struct, but don't really know much
> about the change,
>
> Hope someone can help me,
>
> Javier Domingo
>
>
> [1]: http://vger.kernel.org/~davem/davem_nyc09.pdf
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
First, off put your driver upstream. The kernel developers do not care
about drivers not in the tree. They can wither up and die.
Second, learn to look in the right place.
http://lwn.net/Articles/244640/
^ permalink raw reply
* Re: [PATCH v7 04/16] workqueue: use new hashtable implementation
From: Tejun Heo @ 2012-10-29 1:25 UTC (permalink / raw)
To: Sasha Levin
Cc: snitzer-H+wXaHxf7aLQT0dZR+AlfA, neilb-l3A5Bk7waGM,
fweisbec-Re5JQEeQqe8AvxtiuMwx3w,
Trond.Myklebust-HgOvQuBEEgTQT0dZR+AlfA,
bfields-uC3wQj2KruNg9hUCZPvPmw,
paul.gortmaker-CWA4WttNNZF54TAoqtyWWQ,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA,
aarcange-H+wXaHxf7aLQT0dZR+AlfA, rds-devel-N0ozoZBvEnrZJqsBc5GL+g,
eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w,
venkat.x.venkatsubra-QHcLZuEGTsvQT0dZR+AlfA,
ccaulfie-H+wXaHxf7aLQT0dZR+AlfA, mingo-X9Un+BFzKDI,
dev-yBygre7rU0TnMu66kgdUjQ, ericvh-Re5JQEeQqe8AvxtiuMwx3w,
josh-iaAMLnmF4UmaiuxdJuQwMA, rostedt-nx8X9YLhiw1AfugRpC6u6w,
mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w,
axboe-tSWWG44O7X1aa/9Udqfwiw, linux-nfs-u79uwXL29TY76Z2rM5mHXA,
edumazet-hpIqsD4AKlfQT0dZR+AlfA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, ejt-H+wXaHxf7aLQT0dZR+AlfA,
ebiederm-aS9lmoZGLiVWk0Htik3J/w, lw-BthXqXjhjHXQFUHtdCDX3A,
teigland-H+wXaHxf7aLQT0dZR+AlfA,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
davem-fT/PcQaiUtIeIZ0/mPfg9Q
In-Reply-To: <1351450948-15618-4-git-send-email-levinsasha928-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
On Sun, Oct 28, 2012 at 03:02:16PM -0400, Sasha Levin wrote:
> Switch workqueues to use the new hashtable implementation. This reduces the amount of
> generic unrelated code in the workqueues.
>
> Signed-off-by: Sasha Levin <levinsasha928-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Acked-by: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Thanks!
--
tejun
^ permalink raw reply
* Re: [PATCH v7 08/16] block,elevator: use new hashtable implementation
From: Tejun Heo @ 2012-10-29 1:29 UTC (permalink / raw)
To: Sasha Levin
Cc: torvalds, akpm, linux-kernel, linux-mm, paul.gortmaker, davem,
rostedt, mingo, ebiederm, aarcange, ericvh, netdev, josh,
eric.dumazet, mathieu.desnoyers, axboe, agk, dm-devel, neilb,
ccaulfie, teigland, Trond.Myklebust, bfields, fweisbec, jesse,
venkat.x.venkatsubra, ejt, snitzer, edumazet, linux-nfs, dev,
rds-devel, lw
In-Reply-To: <1351450948-15618-8-git-send-email-levinsasha928@gmail.com>
On Sun, Oct 28, 2012 at 03:02:20PM -0400, Sasha Levin wrote:
> Switch elevator to use the new hashtable implementation. This reduces the amount of
> generic unrelated code in the elevator.
>
> This also removes the dymanic allocation of the hash table. The size of the table is
> constant so there's no point in paying the price of an extra dereference when accessing
> it.
>
> Signed-off-by: Sasha Levin <levinsasha928@gmail.com>
Reviewed-by: Tejun Heo <tj@kernel.orG>
But please reformat commit message to fit inside 80col (preferably 74
or something like that).
Thanks.
--
tejun
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [net-next:master 86/92] net/ipv6/addrconf.c:557 dev_forward_change() error: we previously assumed 'dev' could be null (see line 541)
From: Fengguang Wu @ 2012-10-29 1:57 UTC (permalink / raw)
To: Nicolas Dichtel; +Cc: netdev
In-Reply-To: <508ddd10.Fq4xdOIJOPuT2J+A%yuanhan.liu@linux.intel.com>
Hi Nicolas,
FYI, there are new smatch warnings show up in
tree: git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git master
head: a932657f51eadb8280166e82dc7034dfbff3985a
commit: f3a1bfb11ccbc72d44f0b58c92115a40128979c3 [86/92] rtnl/ipv6: use netconf msg to advertise forwarding status
net/ipv6/addrconf.c:394 ipv6_add_dev() warn: 'dev' held on error path.
+ net/ipv6/addrconf.c:557 dev_forward_change() error: we previously assumed 'dev' could be null (see line 541)
net/ipv6/addrconf.c:885 ipv6_del_addr() error: potential NULL dereference 'rt'.
vim +557 +/dev net/ipv6/addrconf.c
^1da177e Linus Torvalds 2005-04-16 535
^1da177e Linus Torvalds 2005-04-16 536 if (!idev)
^1da177e Linus Torvalds 2005-04-16 537 return;
^1da177e Linus Torvalds 2005-04-16 538 dev = idev->dev;
0187bdfb Ben Hutchings 2008-06-19 539 if (idev->cnf.forwarding)
0187bdfb Ben Hutchings 2008-06-19 540 dev_disable_lro(dev);
^1da177e Linus Torvalds 2005-04-16 @541 if (dev && (dev->flags & IFF_MULTICAST)) {
^1da177e Linus Torvalds 2005-04-16 542 if (idev->cnf.forwarding)
f3ee4010 YOSHIFUJI Hideaki 2008-04-10 543 ipv6_dev_mc_inc(dev, &in6addr_linklocal_allrouters);
^1da177e Linus Torvalds 2005-04-16 544 else
f3ee4010 YOSHIFUJI Hideaki 2008-04-10 545 ipv6_dev_mc_dec(dev, &in6addr_linklocal_allrouters);
^1da177e Linus Torvalds 2005-04-16 546 }
502a2ffd stephen hemminger 2010-03-17 547
502a2ffd stephen hemminger 2010-03-17 548 list_for_each_entry(ifa, &idev->addr_list, if_list) {
2c12a74c Michal Wrobel 2007-02-26 549 if (ifa->flags&IFA_F_TENTATIVE)
2c12a74c Michal Wrobel 2007-02-26 550 continue;
^1da177e Linus Torvalds 2005-04-16 551 if (idev->cnf.forwarding)
^1da177e Linus Torvalds 2005-04-16 552 addrconf_join_anycast(ifa);
^1da177e Linus Torvalds 2005-04-16 553 else
^1da177e Linus Torvalds 2005-04-16 554 addrconf_leave_anycast(ifa);
^1da177e Linus Torvalds 2005-04-16 555 }
f3a1bfb1 Nicolas Dichtel 2012-10-25 556 inet6_netconf_notify_devconf(dev_net(dev), NETCONFA_FORWARDING,
f3a1bfb1 Nicolas Dichtel 2012-10-25 @557 dev->ifindex, &idev->cnf);
^1da177e Linus Torvalds 2005-04-16 558 }
^1da177e Linus Torvalds 2005-04-16 559
^1da177e Linus Torvalds 2005-04-16 560
---
0-DAY kernel build testing backend Open Source Technology Center
Fengguang Wu, Yuanhan Liu Intel Corporation
^ permalink raw reply
* Re: Information about napi_struct and net_device
From: Ben Hutchings @ 2012-10-29 2:41 UTC (permalink / raw)
To: Javier Domingo; +Cc: netdev
In-Reply-To: <CALZVap=ybdWdK22LHsfSRu+TVCJnvL95KeChtCdX8DE-C=1gAg@mail.gmail.com>
On Sat, 2012-10-27 at 18:18 +0200, Javier Domingo wrote:
> Hello,
>
> I am updating a kernel patch from 2.6.23-rc7 to v3.6, using git
> facilities, and I have found that there have been lots of changes in
> networking. Now the drivers are all by vendor, and more important,
> net_device is not used always, as napi_struct took its place.
>
> I have asked in #kernel but no one answered me, so I subscribed here.
> Is there any place I can read about how is it now designed the use of
> napi_struct and net_device? I have found a presentation[1] about how
> networking was changing etc, and I wondered if there is anything more
> precise about how it is actually implemented.
>
> I understand a little why napi_struct, but don't really know much
> about the change,
>
> Hope someone can help me,
The initial change to napi_struct is explained in
<http://lwn.net/Articles/244640/>.
Since then there have been further changes:
- netif_napi_del() has been added. You must call it to clean up NAPI
contexts before freeing the associated net device(s).
- Instead of netif_rx_schedule(), netif_rx_complete(), etc. you must use
napi_schedule(), napi_complete() etc. which just take a napi_struct
pointer.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* [Patch net-next] ipv6: remove a useless NULL check
From: Cong Wang @ 2012-10-29 3:43 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller, Cong Wang
In dev_forward_change(), it is useless to check if idev->dev
is NULL, it is always non-NULL here.
Reported-by: Fengguang Wu <fengguang.wu@intel.com>
Cc: David S. Miller <davem@davemloft.net>
Signed-off-by: Cong Wang <amwang@redhat.com>
---
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 8f0b12a..c9b1cf3 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -607,7 +607,7 @@ static void dev_forward_change(struct inet6_dev *idev)
dev = idev->dev;
if (idev->cnf.forwarding)
dev_disable_lro(dev);
- if (dev && (dev->flags & IFF_MULTICAST)) {
+ if (dev->flags & IFF_MULTICAST) {
if (idev->cnf.forwarding)
ipv6_dev_mc_inc(dev, &in6addr_linklocal_allrouters);
else
^ permalink raw reply related
* [Patch net-next] ipv6: fix a potential NULL deref
From: Cong Wang @ 2012-10-29 3:50 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller, Cong Wang
In ipv6_del_addr():
if (rt != net->ipv6.ip6_null_entry &&
addrconf_is_prefix_route(rt)) {
if (onlink == 0) {
ip6_del_rt(rt);
rt = NULL;
} else if (!(rt->rt6i_flags & RTF_EXPIRES)) {
rt6_set_expires(rt, expires);
}
}
dst_release(&rt->dst);
obviously rt could be NULL'd before dst_release(), so
we have to check if rt is NULL before calling it.
Reported-by: Fengguang Wu <fengguang.wu@intel.com>
Cc: David S. Miller <davem@davemloft.net>
Signed-off-by: Cong Wang <amwang@redhat.com>
---
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 8f0b12a..c467dbb 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -951,7 +951,8 @@ static void ipv6_del_addr(struct inet6_ifaddr *ifp)
rt6_set_expires(rt, expires);
}
}
- dst_release(&rt->dst);
+ if (rt)
+ dst_release(&rt->dst);
}
/* clean up prefsrc entries */
^ permalink raw reply related
* [PATCH net-next] net_cls: traffic counter based on classification control cgroup
From: Alexey Perevalov @ 2012-10-29 4:55 UTC (permalink / raw)
To: netdev
Hello.
First of all, It's request for comments.
I want to suggest a patch for counting ingress and engress traffic for
application placed to net_cls control group.
It's based on res_counters and holds counter per a network interfaces.
It's maybe to complex, and it should be separated:
I move menu entry for "Control group classifier" from network/QoS to
General Option/Control Group.
I don't like too many #ifdefs in tcp.c and udp.c which I introduced.
Also there is a problem as builtin module loaded before network device
module, due I initialize iface.usage_in_bytes in init function,
I have a plan to register cgroup files in callback invoked at
registration of network devices.
Here is entire patch:
From b13afb5b7f09e2a858a56ef5f9dfe7e12c4e8501 Mon Sep 17 00:00:00 2001
From: Alexey Perevalov <a.perevalov@samsung.com>
Date: Fri, 26 Oct 2012 17:45:44 +0400
Subject: Traffic statistics based on packet classification control group
---
include/net/cls_cgroup.h | 203
++++++++++++++++++++++++++++++++++----
include/net/cls_counter_holder.h | 26 +++++
init/Kconfig | 25 +++++
kernel/res_counter.c | 4 +
net/core/dev.c | 6 ++
net/ipv4/tcp.c | 27 ++++-
net/ipv4/udp.c | 6 ++
net/sched/Kconfig | 11 ---
net/sched/Makefile | 1 +
net/sched/cls_cgroup.c | 142 +++++++++++++++++++++++++-
net/sched/cls_counter_holder.c | 145 +++++++++++++++++++++++++++
11 files changed, 561 insertions(+), 35 deletions(-)
create mode 100644 include/net/cls_counter_holder.h
create mode 100644 net/sched/cls_counter_holder.c
diff --git a/include/net/cls_cgroup.h b/include/net/cls_cgroup.h
index 2581638..3a6954f 100644
--- a/include/net/cls_cgroup.h
+++ b/include/net/cls_cgroup.h
@@ -17,50 +17,198 @@
#include <linux/hardirq.h>
#include <linux/rcupdate.h>
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+#include <linux/nsproxy.h>
+#include <linux/res_counter.h>
+#include <linux/string.h>
+#include <linux/skbuff.h>
+#include <net/cls_counter_holder.h>
+#include <net/sock.h>
+
+/*TODO hide all it to separate file*/
+
+struct cls_iface_cntrs {
+ char *dev_name;
+ struct res_counter snd_counter;
+ struct res_counter rcv_counter;
+ struct list_head link;
+};
+
+#endif /*CONFIG_NET_CLS_COUNTER*/
+
+
#if IS_ENABLED(CONFIG_NET_CLS_CGROUP)
struct cgroup_cls_state
{
struct cgroup_subsys_state css;
u32 classid;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ struct cls_iface_cntrs iface_stats;
+#endif /*CONFIG_NET_CLS_COUNTER*/
};
extern void sock_update_classid(struct sock *sk, struct task_struct
*task);
-#if IS_BUILTIN(CONFIG_NET_CLS_CGROUP)
-static inline u32 task_cls_classid(struct task_struct *p)
+#if IS_MODULE(CONFIG_NET_CLS_CGROUP)
+static inline struct cgroup_cls_state *get_cls_cgroup(struct
task_struct *p)
{
- u32 classid;
+ struct cgroup_subsys_state *css = task_subsys_state(p,
+ net_cls_subsys_id);
+ if (css)
+ return container_of(css,
+ struct cgroup_cls_state, css);
+ return NULL;
+}
+#elif IS_BUILTIN(CONFIG_NET_CLS_CGROUP)
+static inline struct cgroup_cls_state *get_cls_cgroup(struct
task_struct *p)
+{
+ return container_of(task_subsys_state(p, net_cls_subsys_id),
+ struct cgroup_cls_state, css);
+}
+#endif
- if (in_interrupt())
- return 0;
- rcu_read_lock();
- classid = container_of(task_subsys_state(p, net_cls_subsys_id),
- struct cgroup_cls_state, css)->classid;
- rcu_read_unlock();
+#endif /*CONFIG_NET_CLS_CGROUP*/
- return classid;
+#if IS_ENABLED(CONFIG_NET_CLS_CGROUP)
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+static inline u32 skb_cls_classid(const struct sk_buff *skb)
+{
+ return (skb && skb->sk) ? skb->sk->sk_classid : 0;
+}
+
+static inline int get_ifindex_from_skb(const struct sk_buff *skb)
+{
+ int ifindex = 0;
+ if (skb)
+ ifindex = skb->skb_iif;
+ return ifindex;
+}
+
+static struct cls_iface_cntrs *find_cls_counter(
+ struct cgroup_cls_state *cls_cgroup,
+ const char *dev_name,
+ bool create)
+{
+ /*TODO Add lock*/
+ struct cls_iface_cntrs *entry = NULL;
+
+ if (!dev_name) {
+ pr_err("cls please provide valid dev name");
+ return NULL;
+ }
+
+ list_for_each_entry(entry, &cls_cgroup->iface_stats.link, link)
+ if (!strcmp(entry->dev_name, dev_name))
+ return entry;
+
+ if (!create)
+ return entry;
+
+ /*not found, insert*/
+ entry = kmalloc(sizeof(struct cls_iface_cntrs), GFP_ATOMIC);
+ entry->dev_name = kstrdup(dev_name, GFP_ATOMIC);
+ memset(&entry->rcv_counter, 0, sizeof(struct res_counter));
+ memset(&entry->snd_counter, 0, sizeof(struct res_counter));
+ res_counter_init(&entry->rcv_counter, NULL);
+ res_counter_init(&entry->snd_counter, NULL);
+ list_add_tail(&entry->link, &cls_cgroup->iface_stats.link);
+ return entry;
}
-#elif IS_MODULE(CONFIG_NET_CLS_CGROUP)
+
+static void charge_net_cls_snd(struct cgroup_cls_state *cls_cgroup,
+ const u32 copied, const char *dev_name)
+{
+ struct res_counter *fail_res;
+ int res;
+ struct cls_iface_cntrs *cnt = find_cls_counter(cls_cgroup,
+ dev_name, true);
+
+ if (!cnt)
+ return;
+
+ res = res_counter_charge(&cnt->snd_counter, copied, &fail_res);
+}
+
+static char *get_dev_name(const int ifindex)
+{
+ struct net *net = NULL;
+ struct nsproxy *nsproxy = NULL;
+ struct net_device *net_dev = NULL;
+
+ nsproxy = task_nsproxy(current);
+ if (!nsproxy) {
+ pr_debug("cls cant find task_nsproxy");
+ return NULL;
+ }
+
+ net = get_net(nsproxy->net_ns);
+ if (!net) {
+ pr_debug("cls cant find net");
+ return NULL;
+ }
+ net_dev = dev_get_by_index(net, ifindex);
+
+ return net_dev ? net_dev->name : NULL;
+}
+
+static void charge_net_cls_rcv(struct cgroup_cls_state *cls_cgroup,
+ const u32 copied, const int ifindex)
+{
+ char *dev_name = get_dev_name(ifindex);
+ struct res_counter *fail_res;
+ int res;
+ struct cls_iface_cntrs *cnt = find_cls_counter(cls_cgroup,
+ dev_name, true);
+
+ if (!cnt)
+ return;
+
+ res = res_counter_charge(&cnt->rcv_counter, copied, &fail_res);
+}
+
+static inline void count_cls_rcv(struct task_struct *p, const u32
copied, const int ifindex)
+{
+ struct cgroup_cls_state *cls_cgroup;
+
+ cls_cgroup = get_cls_cgroup(p);
+
+ if (cls_cgroup)
+ charge_net_cls_rcv(cls_cgroup, copied, ifindex);
+}
+
+static inline void count_cls_snd(u32 classid, const u32 copied,
+ const char *dev_name)
+{
+ struct cgroup_cls_state *cls_cgroup;
+
+ cls_cgroup = find_cls_cgroup_by_classid(classid);
+
+ if (cls_cgroup)
+ charge_net_cls_snd(cls_cgroup, copied, dev_name);
+}
+#endif /*CONFIG_NET_CLS_COUNTER*/
+
static inline u32 task_cls_classid(struct task_struct *p)
{
- struct cgroup_subsys_state *css;
- u32 classid = 0;
+ int classid = 0;
+ struct cgroup_cls_state *cls_cgroup = NULL;
if (in_interrupt())
return 0;
rcu_read_lock();
- css = task_subsys_state(p, net_cls_subsys_id);
- if (css)
- classid = container_of(css,
- struct cgroup_cls_state, css)->classid;
+
+ cls_cgroup = get_cls_cgroup(p);
+ if (cls_cgroup)
+ classid = cls_cgroup->classid;
+
rcu_read_unlock();
return classid;
}
-#endif
-#else /* !CGROUP_NET_CLS_CGROUP */
+
+#else /* !CONFIG_NET_CLS_CGROUP */
static inline void sock_update_classid(struct sock *sk, struct
task_struct *task)
{
}
@@ -69,5 +217,20 @@ static inline u32 task_cls_classid(struct
task_struct *p)
{
return 0;
}
-#endif /* CGROUP_NET_CLS_CGROUP */
+#endif /* CONFIG_NET_CLS_CGROUP */
+
+#if !IS_ENABLED(CONFIG_NET_CLS_CGROUP) ||
!IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+static inline void count_cls_rcv(struct task_struct *p, const u32
copied, const int ifindex)
+{
+}
+
+static inline void count_cls_snd(u32 classid, const u32 copied, const
char *dev_name)
+{
+}
+
+static inline u32 skb_cls_classid(const struct sk_buff *skb)
+{
+ return 0;
+}
+#endif
#endif /* _NET_CLS_CGROUP_H */
diff --git a/include/net/cls_counter_holder.h
b/include/net/cls_counter_holder.h
new file mode 100644
index 0000000..a129baa
--- /dev/null
+++ b/include/net/cls_counter_holder.h
@@ -0,0 +1,26 @@
+/*
+ * cls_counter_holder.c Interface for holding references of the
+ * net cls cgroup instances.
+ *
+ * Authors: Alexey Perevalov, <a.perevalov@samsung.com>
+ *
+ * Changes:
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _NET_CLS_COUNTER_HOLDER_H_
+#define _NET_CLS_COUNTER_HOLDER_H_
+
+#include <net/cls_cgroup.h>
+
+struct cgroup_cls_state;
+
+void insert_cls_cgroup_entry(struct cgroup_cls_state *obj);
+void delete_cls_cgroup_entry(const u32 classid);
+struct cgroup_cls_state *find_cls_cgroup_by_classid(const u32 classid);
+
+
+#endif /* _NET_CLS_COUNTER_HOLDER_H_ */
diff --git a/init/Kconfig b/init/Kconfig
index 6fdd6e3..2e6af85 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -841,6 +841,31 @@ config CGROUP_HUGETLB
control group is tracked in the third page lru pointer. This means
that we cannot use the controller with huge page less than 3 pages.
+menuconfig NET_CLS_CGROUP
+ tristate "Control Group Classifier"
+ select NET_CLS
+ depends on CGROUPS
+ ---help---
+ Say Y here if you want to classify packets based on the control
+ cgroup of their process.
+
+ To compile this code as a module, choose M here: the
+ module will be called cls_cgroup.
+
+if NET_CLS_CGROUP
+config NET_CLS_COUNTER
+ bool "Network traffic counter for network Control Group Classifier"
+ select NET_CLS
+ default n
+ depends on NET_CLS_CGROUP && RESOURCE_COUNTERS
+ ---help---
+ Say Y here if you want to count traffic associate with the control
+ cgroup.
+
+ To add functionality to cls_cgroup select y.
+
+endif #NET_CLS_CGROUP
+
config CGROUP_PERF
bool "Enable perf_event per-cpu per-container group (cgroup)
monitoring"
depends on PERF_EVENTS && CGROUPS
diff --git a/kernel/res_counter.c b/kernel/res_counter.c
index ad581aa..f5767af 100644
--- a/kernel/res_counter.c
+++ b/kernel/res_counter.c
@@ -13,6 +13,8 @@
#include <linux/res_counter.h>
#include <linux/uaccess.h>
#include <linux/mm.h>
+#include <linux/export.h>
+
void res_counter_init(struct res_counter *counter, struct res_counter
*parent)
{
@@ -21,6 +23,7 @@ void res_counter_init(struct res_counter *counter,
struct res_counter *parent)
counter->soft_limit = RESOURCE_MAX;
counter->parent = parent;
}
+EXPORT_SYMBOL(res_counter_init);
int res_counter_charge_locked(struct res_counter *counter, unsigned
long val,
bool force)
@@ -170,6 +173,7 @@ u64 res_counter_read_u64(struct res_counter
*counter, int member)
return *res_counter_member(counter, member);
}
#endif
+EXPORT_SYMBOL(res_counter_read_u64);
int res_counter_memparse_write_strategy(const char *buf,
unsigned long long *res)
diff --git a/net/core/dev.c b/net/core/dev.c
index b4978e2..61c9a61 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -135,6 +135,7 @@
#include <linux/net_tstamp.h>
#include <linux/static_key.h>
#include <net/flow_keys.h>
+#include <net/cls_cgroup.h>
#include "net-sysfs.h"
@@ -2570,6 +2571,11 @@ int dev_queue_xmit(struct sk_buff *skb)
*/
rcu_read_lock_bh();
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ if (dev)
+ count_cls_snd(skb_cls_classid(skb), skb->len, dev->name);
+#endif
+
skb_update_prio(skb);
txq = netdev_pick_tx(dev, skb);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index eace049..3013509 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -276,6 +276,7 @@
#include <net/ip.h>
#include <net/netdma.h>
#include <net/sock.h>
+#include <net/cls_cgroup.h>
#include <asm/uaccess.h>
#include <asm/ioctls.h>
@@ -1467,6 +1468,9 @@ int tcp_read_sock(struct sock *sk,
read_descriptor_t *desc,
u32 seq = tp->copied_seq;
u32 offset;
int copied = 0;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ int ifindex = 0;
+#endif
if (sk->sk_state == TCP_LISTEN)
return -ENOTCONN;
@@ -1509,6 +1513,9 @@ int tcp_read_sock(struct sock *sk,
read_descriptor_t *desc,
++seq;
break;
}
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ ifindex = get_ifindex_from_skb(skb);
+#endif
sk_eat_skb(sk, skb, false);
if (!desc->count)
break;
@@ -1519,8 +1526,12 @@ int tcp_read_sock(struct sock *sk,
read_descriptor_t *desc,
tcp_rcv_space_adjust(sk);
/* Clean up data we have read: This will do ACK frames. */
- if (copied > 0)
+ if (copied > 0) {
tcp_cleanup_rbuf(sk, copied);
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ count_cls_rcv(current, copied, ifindex);
+#endif
+ }
return copied;
}
EXPORT_SYMBOL(tcp_read_sock);
@@ -1548,6 +1559,9 @@ int tcp_recvmsg(struct kiocb *iocb, struct sock
*sk, struct msghdr *msg,
bool copied_early = false;
struct sk_buff *skb;
u32 urg_hole = 0;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ int ifindex = 0;
+#endif
lock_sock(sk);
@@ -1872,6 +1886,9 @@ skip_copy:
if (tcp_hdr(skb)->fin)
goto found_fin_ok;
if (!(flags & MSG_PEEK)) {
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ ifindex = get_ifindex_from_skb(skb);
+#endif
sk_eat_skb(sk, skb, copied_early);
copied_early = false;
}
@@ -1881,6 +1898,9 @@ skip_copy:
/* Process the FIN. */
++*seq;
if (!(flags & MSG_PEEK)) {
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ ifindex = get_ifindex_from_skb(skb);
+#endif
sk_eat_skb(sk, skb, copied_early);
copied_early = false;
}
@@ -1923,6 +1943,11 @@ skip_copy:
/* Clean up data we have read: This will do ACK frames. */
tcp_cleanup_rbuf(sk, copied);
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ if (copied > 0)
+ count_cls_rcv(current, copied, ifindex);
+#endif
+
release_sock(sk);
return copied;
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 79c8dbe..a143629 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -101,6 +101,7 @@
#include <linux/skbuff.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
+#include <net/cls_cgroup.h>
#include <net/net_namespace.h>
#include <net/icmp.h>
#include <net/route.h>
@@ -1254,6 +1255,11 @@ try_again:
if (flags & MSG_TRUNC)
err = ulen;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ if (ulen > 0)
+ count_cls_rcv(current, ulen, get_ifindex_from_skb(skb));
+#endif
+
out_free:
skb_free_datagram_locked(sk, skb);
out:
diff --git a/net/sched/Kconfig b/net/sched/Kconfig
index 62fb51f..926dedf 100644
--- a/net/sched/Kconfig
+++ b/net/sched/Kconfig
@@ -418,17 +418,6 @@ config NET_CLS_FLOW
To compile this code as a module, choose M here: the
module will be called cls_flow.
-config NET_CLS_CGROUP
- tristate "Control Group Classifier"
- select NET_CLS
- depends on CGROUPS
- ---help---
- Say Y here if you want to classify packets based on the control
- cgroup of their process.
-
- To compile this code as a module, choose M here: the
- module will be called cls_cgroup.
-
config NET_EMATCH
bool "Extended Matches"
select NET_CLS
diff --git a/net/sched/Makefile b/net/sched/Makefile
index 978cbf0..95dbb12 100644
--- a/net/sched/Makefile
+++ b/net/sched/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_NET_CLS_RSVP6) += cls_rsvp6.o
obj-$(CONFIG_NET_CLS_BASIC) += cls_basic.o
obj-$(CONFIG_NET_CLS_FLOW) += cls_flow.o
obj-$(CONFIG_NET_CLS_CGROUP) += cls_cgroup.o
+obj-$(CONFIG_NET_CLS_COUNTER) += cls_counter_holder.o
obj-$(CONFIG_NET_EMATCH) += ematch.o
obj-$(CONFIG_NET_EMATCH_CMP) += em_cmp.o
obj-$(CONFIG_NET_EMATCH_NBYTE) += em_nbyte.o
diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c
index 709b0fb..dcf7f60 100644
--- a/net/sched/cls_cgroup.c
+++ b/net/sched/cls_cgroup.c
@@ -22,6 +22,11 @@
#include <net/pkt_cls.h>
#include <net/sock.h>
#include <net/cls_cgroup.h>
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+#include <linux/rbtree.h>
+#include <linux/res_counter.h>
+#include <net/cls_counter_holder.h>
+#endif
static inline struct cgroup_cls_state *cgrp_cls_state(struct cgroup *cgrp)
{
@@ -46,11 +51,49 @@ static struct cgroup_subsys_state
*cgrp_create(struct cgroup *cgrp)
if (cgrp->parent)
cs->classid = cgrp_cls_state(cgrp->parent)->classid;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ res_counter_init(&cs->iface_stats.snd_counter, NULL);
+ res_counter_init(&cs->iface_stats.rcv_counter, NULL);
+ cs->iface_stats.dev_name = 0;
+ INIT_LIST_HEAD(&cs->iface_stats.link);
+#endif
+
return &cs->css;
}
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+static struct cftype *cft_first;
+
+static inline void cgrp_counter_destroy(struct cgroup_cls_state *cs)
+{
+ struct list_head *pos, *q;
+ delete_cls_cgroup_entry(cs->classid);
+
+ list_for_each_safe(pos, q, &cs->iface_stats.link) {
+ struct cls_iface_cntrs *tmp = list_entry(
+ pos, struct cls_iface_cntrs, link);
+ list_del(pos);
+ if (!tmp)
+ continue;
+
+ if (!tmp->dev_name)
+ kfree(tmp->dev_name);
+ kfree(tmp);
+ }
+
+}
+#endif
+
static void cgrp_destroy(struct cgroup *cgrp)
{
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+
+ struct cgroup_cls_state *cs = cgrp_cls_state(cgrp);
+
+ if (!cs)
+ return;
+ cgrp_counter_destroy(cs);
+#endif
kfree(cgrp_cls_state(cgrp));
}
@@ -81,9 +124,59 @@ static u64 read_classid(struct cgroup *cgrp, struct
cftype *cft)
return cgrp_cls_state(cgrp)->classid;
}
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+static const char *rcv_label = "rcv:";
+static const char *snd_label = "snd:";
+
+static const char *extract_dev_name(const char *cgroup_file_name)
+{
+ const char *dot = strchr(cgroup_file_name, '.');
+ const size_t len = dot ? dot - cgroup_file_name :
strlen(cgroup_file_name);
+
+ return kstrndup(cgroup_file_name, len, GFP_KERNEL);
+}
+
+static int read_stat(struct cgroup *cgrp, struct cftype *cft,
+ struct cgroup_map_cb *cb)
+{
+ struct cgroup_cls_state *cs = cgrp_cls_state(cgrp);
+ const char *dev_name = extract_dev_name(cft->name);
+ struct cls_iface_cntrs *res = find_cls_counter(cs, dev_name, false);
+
+ if (!res) {
+ pr_debug("cls cant read for cls");
+ return -EINVAL;
+ }
+
+ cb->fill(cb, rcv_label,
+ res_counter_read_u64(&res->rcv_counter, RES_USAGE));
+ cb->fill(cb, snd_label,
+ res_counter_read_u64(&res->snd_counter, RES_USAGE));
+
+ kfree(dev_name);
+ return 0;
+}
+#endif /*CONFIG_NET_CLS_COUNTER*/
+
static int write_classid(struct cgroup *cgrp, struct cftype *cft, u64
value)
{
- cgrp_cls_state(cgrp)->classid = (u32) value;
+ struct cgroup_cls_state *cgrp_cls = cgrp_cls_state(cgrp);
+ u32 *classid = &cgrp_cls->classid;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ u32 oldclassid = *classid;
+
+ if(find_cls_cgroup_by_classid(value)) {
+ pr_err("cls: classid %llu already exists\n", value);
+ return -EINVAL;
+ }
+
+ insert_cls_cgroup_entry(cgrp_cls);
+
+ if (oldclassid)
+ delete_cls_cgroup_entry(oldclassid);
+#endif /*CONFIG_NET_CLS_COUNTER*/
+ *classid = (u32) value;
+
return 0;
}
@@ -307,17 +400,57 @@ static struct tcf_proto_ops cls_cgroup_ops
__read_mostly = {
.owner = THIS_MODULE,
};
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+static inline int init_cgroup_counter(void)
+{
+ int dev_count = 0, ret = 0;
+ struct net_device *dev;
+ struct cftype *cft;
+
+ for_each_netdev(&init_net, dev)
+ ++dev_count;
+
+ cft = kmalloc(sizeof(struct cftype) * (dev_count + 1), GFP_KERNEL);
+ memset(cft, 0, sizeof(struct cftype) * (dev_count + 1));
+
+ cft_first = cft;
+
+ for_each_netdev(&init_net, dev) {
+ snprintf(cft->name, MAX_CFTYPE_NAME,
+ "%s.usage_in_bytes", dev->name);
+ cft->read_map = read_stat;
+ cft->private = RES_USAGE;
+ ++cft;
+ }
+/*terminate element*/
+ ret = cgroup_add_cftypes(&net_cls_subsys, cft_first);
+ if (ret)
+ pr_err("error adding cft for counting at cls_cgroup %d\n", ret);
+
+ return ret;
+}
+#endif
+
static int __init init_cgroup_cls(void)
{
int ret;
-
ret = cgroup_load_subsys(&net_cls_subsys);
if (ret)
goto out;
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ ret = init_cgroup_counter();
+ if (ret)
+ goto unload;
+#endif
+
ret = register_tcf_proto_ops(&cls_cgroup_ops);
if (ret)
- cgroup_unload_subsys(&net_cls_subsys);
+ goto unload;
+
+ return 0;
+unload:
+ cgroup_unload_subsys(&net_cls_subsys);
out:
return ret;
@@ -328,6 +461,9 @@ static void __exit exit_cgroup_cls(void)
unregister_tcf_proto_ops(&cls_cgroup_ops);
cgroup_unload_subsys(&net_cls_subsys);
+#if IS_ENABLED(CONFIG_NET_CLS_COUNTER)
+ kfree(cft_first);
+#endif
}
module_init(init_cgroup_cls);
diff --git a/net/sched/cls_counter_holder.c b/net/sched/cls_counter_holder.c
new file mode 100644
index 0000000..eb56298
--- /dev/null
+++ b/net/sched/cls_counter_holder.c
@@ -0,0 +1,145 @@
+/*
+ * net/sched/cls_counter_holder.c Interface for holding references of the
+ * net cls cgroup instances.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ *
+ * Authors: Alexey Perevalov <a.perevalov@samsung.com>
+ */
+
+
+#include <linux/export.h>
+#include <linux/module.h>
+#include <net/cls_cgroup.h>
+#include <net/cls_counter_holder.h>
+
+static struct rb_root classid_tree = RB_ROOT;
+static DEFINE_SPINLOCK(classid_tree_lock);
+
+struct entry {
+ struct cgroup_cls_state *data;
+ struct rb_node node;
+};
+
+static struct entry *find_entry(struct rb_root *root, const u32 classid)
+{
+ struct rb_node *node = root->rb_node;
+
+ while (node) {
+ struct entry *cls_entry = rb_entry(node, struct entry, node);
+ int result = 0;
+ if (!cls_entry || !cls_entry->data)
+ break;
+ result = cls_entry->data->classid - classid;
+
+ if (result < 0)
+ node = node->rb_left;
+ else if (result > 0)
+ node = node->rb_right;
+ else
+ return cls_entry;
+ }
+ return NULL;
+}
+
+void insert_cls_cgroup_entry(struct cgroup_cls_state *obj)
+{
+ struct rb_node **new;
+ struct rb_node *parent = NULL;
+ struct entry *new_entry;
+ unsigned long irq_flags = 0;
+
+ struct rb_root *root = &classid_tree;
+
+ spin_lock_irqsave(&classid_tree_lock, irq_flags);
+
+ new = &root->rb_node;
+
+ while (*new) {
+ struct entry *this = rb_entry(*new, struct entry, node);
+ /* Sort by classid, then by ifindex */
+ int result =
+ (this->data->classid - obj->classid);
+ parent = *new;
+ if (result < 0)
+ new = &((*new)->rb_left);
+ else if (result > 0)
+ new = &((*new)->rb_right);
+ else
+ goto unlock;
+ }
+
+ /* If we here, we need to insert new entry into tree */
+ new_entry = kmalloc(sizeof(struct entry), GFP_ATOMIC);
+ if (!new_entry)
+ goto unlock;
+
+ new_entry->data = obj;
+ /* Add new node and rebalance tree */
+ rb_link_node(&new_entry->node, parent, new);
+ rb_insert_color(&new_entry->node, root);
+
+unlock:
+ spin_unlock_irqrestore(&classid_tree_lock, irq_flags);
+}
+EXPORT_SYMBOL(insert_cls_cgroup_entry);
+
+void delete_cls_cgroup_entry(const u32 classid)
+{
+ unsigned long irq_flags = 0;
+ struct entry *data = NULL;
+ struct rb_root *root = &classid_tree;
+ spin_lock_irqsave(&classid_tree_lock, irq_flags);
+
+ data = find_entry(root, classid);
+
+ if (data) {
+ rb_erase(&data->node, root);
+ kfree(data);
+ }
+ spin_unlock_irqrestore(&classid_tree_lock, irq_flags);
+}
+EXPORT_SYMBOL(delete_cls_cgroup_entry);
+
+static void free_node(struct rb_node *root)
+{
+ struct entry *cur_entry = rb_entry(root, struct entry, node);
+ if (root->rb_left)
+ free_node(root->rb_left);
+ if (root->rb_right)
+ free_node(root->rb_right);
+ if (cur_entry)
+ kfree(cur_entry);
+}
+
+static void free_classid_tree(void)
+{
+ unsigned long irq_flags = 0;
+
+ spin_lock_irqsave(&classid_tree_lock, irq_flags);
+
+ free_node(classid_tree.rb_node);
+
+ spin_unlock_irqrestore(&classid_tree_lock, irq_flags);
+}
+
+struct cgroup_cls_state *find_cls_cgroup_by_classid(const u32 classid)
+{
+ struct entry *cls_entry = find_entry(&classid_tree, classid);
+ if (cls_entry)
+ return cls_entry->data;
+
+ return NULL;
+}
+EXPORT_SYMBOL(find_cls_cgroup_by_classid);
+
+static void __exit exit_cls_counter_holder(void)
+{
+ free_classid_tree();
+}
+
+module_exit(exit_cls_counter_holder);
+MODULE_LICENSE("GPL");
--
1.7.9.5
--
Best regards,
Alexey Perevalov
^ permalink raw reply related
* [net-next v4 0/7] Multiqueue support in tuntap
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
Hello All:
This is an update of multiqueue support in tuntap from V3. Please consider to
merge.
The main idea for this series is to let tun/tap device to be benefited from
multiqueue network cards and multi-core host. We used to have a single queue for
tuntap which could be a bottleneck in a multiqueue/core environment. So this
series let the device could be attched with multiple sockets and expose them
through fd to the userspace as multiqueues. The sereis were orignally designed
to serve as backend for multiqueue virtio-net in KVM, but the design is generic
for other application to be used.
Some quick overview of the design:
- Moving socket from tun_device to tun_file.
- Allowing multiple sockets to be attached to a tun/tap devices.
- Using RCU to synchronize the data path and system call.
- Two new ioctls were added for the usespace to attach and detach socket to the device.
- API compatibility were maintained without userspace notable changes, so legacy
userspace that only use one queue won't need any changes.
- A flow(rxhash) to queue table were maintained by tuntap which choose the txq
based on the last rxq where it comes.
Performance test:
Pktgen is used to generate the traffic and a simple program that only does the
receiving in userspace.
#q #thread kpps aggregate kpps
1q 1thread 818kpps +0%
2q 2threads 1926kpps +135%
3q 3threads 2642kpps +223%
4q 4threads 3536kpps +332%
Changes from V3:
- Rebase to net-next
- A separate RCUiying patch to simply the reviewing
- Add a simple "tx follows rx" policy when choosing txq
- Various bug fixes
Changes from V2:
- Rebase to the latest net-next
- Fix netdev leak when tun_attach fails
- Fix return value of TUNSETOWNER
- Purge the receive queue in socket destructor
- Enable multiqueue tun (V1 and V2 only allows mq to be eanbled for tap
- Add per-queue u64 statistics
- Fix wrong BUG_ON() check in tun_detach()
- Check numqueues instead of tfile[0] in tun_set_iff() to let tunctl -d works
correctly
- Set numqueues to MAX_TAP_QUEUES during tun_detach_all() to prevent the
attaching.
Changes from V1:
- Simplify the sockets array management by not leaving NULL in the slot.
- Optimization on the tx queue selecting.
- Fix the bug in tun_deatch_all()
Reference:
- V3 https://lkml.org/lkml/2012/6/25/191
- V2 http://lwn.net/Articles/459270/
- V1 http://www.mail-archive.com/kvm@vger.kernel.org/msg59479.html
Jason Wang (7):
tuntap: log the unsigned informaiton with %u
tuntap: move socket to tun_file
tuntap: RCUify dereferencing between tun_struct and tun_file
tuntap: introduce multiqueue flags
tuntap: multiqueue support
tuntap: add ioctl to attach or detach a file form tuntap device
tuntap: choose the txq based on rxq
drivers/net/tun.c | 845 ++++++++++++++++++++++++++++++++-----------
include/uapi/linux/if_tun.h | 5 +
2 files changed, 640 insertions(+), 210 deletions(-)
^ permalink raw reply
* [PATCH] tuntap: choose the txq based on rxq
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
This patch implements a simple multiqueue flow steering policy - tx follows rx
for tun/tap. The idea is simple, it just choose the txq based on which rxq it
comes. The flow were identified through the rxhash of a skb, and the hash to
queue mapping were recorded in a hlist with an ageing timer to retire the
mapping. The mapping were created when tun receives packet from userspace, and
was quired in .ndo_select_queue().
I run co-current TCP_CRR test and didn't see any mapping manipulation helpers in
perf top, so the overhead could be negelected.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
Changes from last version:
- Add the missing barrior to fix the race of module unloading.
- Add TODO in tun_flow_update() and comments of rcu read lock holding in
tun_select_queue()
- Use a macro to define the ageing time.
- Don't trigger the timer when there's no flow in use.
drivers/net/tun.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 236 insertions(+), 3 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 78dcda8..158ef1d 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -115,6 +115,8 @@ struct tap_filter {
* to match a queue per guest CPU. */
#define MAX_TAP_QUEUES 1024
+#define TUN_FLOW_EXPIRE 3 * HZ
+
/* A tun_file connects an open character device to a tuntap netdevice. It
* also contains all socket related strctures (except sock_fprog and tap_filter)
* to serve as one transmit queue for tuntap device. The sock_fprog and
@@ -138,6 +140,18 @@ struct tun_file {
u16 queue_index;
};
+struct tun_flow_entry {
+ struct hlist_node hash_link;
+ struct rcu_head rcu;
+ struct tun_struct *tun;
+
+ u32 rxhash;
+ int queue_index;
+ unsigned long updated;
+};
+
+#define TUN_NUM_FLOW_ENTRIES 1024
+
/* Since the socket were moved to tun_file, to preserve the behavior of persist
* device, socket fileter, sndbuf and vnet header size were restore when the
* file were attached to a persist device.
@@ -163,8 +177,176 @@ struct tun_struct {
#ifdef TUN_DEBUG
int debug;
#endif
+ spinlock_t lock;
+ struct kmem_cache *flow_cache;
+ struct hlist_head flows[TUN_NUM_FLOW_ENTRIES];
+ struct timer_list flow_gc_timer;
+ unsigned long ageing_time;
};
+static inline u32 tun_hashfn(u32 rxhash)
+{
+ return rxhash & 0x3ff;
+}
+
+static struct tun_flow_entry *tun_flow_find(struct hlist_head *head, u32 rxhash)
+{
+ struct tun_flow_entry *e;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_rcu(e, n, head, hash_link) {
+ if (e->rxhash == rxhash)
+ return e;
+ }
+ return NULL;
+}
+
+static struct tun_flow_entry *tun_flow_create(struct tun_struct *tun,
+ struct hlist_head *head,
+ u32 rxhash, u16 queue_index)
+{
+ struct tun_flow_entry *e = kmem_cache_alloc(tun->flow_cache,
+ GFP_ATOMIC);
+ if (e) {
+ tun_debug(KERN_INFO, tun, "create flow: hash %u index %u\n",
+ rxhash, queue_index);
+ e->updated = jiffies;
+ e->rxhash = rxhash;
+ e->queue_index = queue_index;
+ e->tun = tun;
+ hlist_add_head_rcu(&e->hash_link, head);
+ }
+ return e;
+}
+
+static void tun_flow_free(struct rcu_head *head)
+{
+ struct tun_flow_entry *e
+ = container_of(head, struct tun_flow_entry, rcu);
+ kmem_cache_free(e->tun->flow_cache, e);
+}
+
+static void tun_flow_delete(struct tun_struct *tun, struct tun_flow_entry *e)
+{
+ tun_debug(KERN_INFO, tun, "delete flow: hash %u index %u\n",
+ e->rxhash, e->queue_index);
+ hlist_del_rcu(&e->hash_link);
+ call_rcu(&e->rcu, tun_flow_free);
+}
+
+/* caller hold tun->lock */
+static int tun_delete_by_rxhash(struct tun_struct *tun, u32 rxhash)
+{
+ struct hlist_head *head = &tun->flows[tun_hashfn(rxhash)];
+ struct tun_flow_entry *e = tun_flow_find(head, rxhash);
+
+ if (!e)
+ return -ENOENT;
+
+ tun_flow_delete(tun, e);
+}
+
+static void tun_flow_flush(struct tun_struct *tun)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link)
+ tun_flow_delete(tun, e);
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_delete_by_queue(struct tun_struct *tun, u16 queue_index)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ if (e->queue_index == queue_index)
+ tun_flow_delete(tun, e);
+ }
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_cleanup(unsigned long data)
+{
+ struct tun_struct *tun = (struct tun_struct *)data;
+ unsigned long delay = tun->ageing_time;
+ unsigned long next_timer = jiffies + delay;
+ unsigned long count = 0;
+ int i;
+
+ tun_debug(KERN_INFO, tun, "tun_flow_cleanup\n");
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ unsigned long this_timer;
+ count ++;
+ this_timer = e->updated + delay;
+ if (time_before_eq(this_timer, jiffies))
+ tun_flow_delete(tun, e);
+ else if (time_before(this_timer, next_timer))
+ next_timer = this_timer;
+ }
+ }
+
+ if (count)
+ mod_timer(&tun->flow_gc_timer, round_jiffies_up(next_timer));
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_update(struct tun_struct *tun, struct sk_buff *skb,
+ u16 queue_index)
+{
+ struct hlist_head *head;
+ struct tun_flow_entry *e;
+ unsigned long delay = tun->ageing_time;
+ u32 rxhash = skb_get_rxhash(skb);
+
+ if (!rxhash)
+ return;
+ else
+ head = &tun->flows[tun_hashfn(rxhash)];
+
+ rcu_read_lock();
+
+ if (tun->numqueues == 1)
+ goto unlock;
+
+ e = tun_flow_find(head, rxhash);
+ if (likely(e)) {
+ /* TODO: keep queueing to old queue until it's empty? */
+ e->queue_index = queue_index;
+ e->updated = jiffies;
+ } else {
+ spin_lock_bh(&tun->lock);
+ if (!tun_flow_find(head, rxhash))
+ tun_flow_create(tun, head, rxhash, queue_index);
+
+ if (!timer_pending(&tun->flow_gc_timer))
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + delay));
+ spin_unlock_bh(&tun->lock);
+ }
+
+unlock:
+ rcu_read_unlock();
+}
+
/* We try to identify a flow through its rxhash first. The reason that
* we do not check rxq no. is becuase some cards(e.g 82599), chooses
* the rxq based on the txq where the last packet of the flow comes. As
@@ -175,6 +357,7 @@ struct tun_struct {
static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
{
struct tun_struct *tun = netdev_priv(dev);
+ struct tun_flow_entry *e;
u32 txq = 0;
u32 numqueues = 0;
@@ -183,8 +366,12 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
txq = skb_get_rxhash(skb);
if (txq) {
- /* use multiply and shift instead of expensive divide */
- txq = ((u64)txq * numqueues) >> 32;
+ e = tun_flow_find(&tun->flows[tun_hashfn(txq)], txq);
+ if (e)
+ txq = e->queue_index;
+ else
+ /* use multiply and shift instead of expensive divide */
+ txq = ((u64)txq * numqueues) >> 32;
} else if (likely(skb_rx_queue_recorded(skb))) {
txq = skb_get_rx_queue(skb);
while (unlikely(txq >= numqueues))
@@ -234,6 +421,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
sock_put(&tfile->sk);
synchronize_net();
+ tun_flow_delete_by_queue(tun, tun->numqueues + 1);
/* Drop read queue */
skb_queue_purge(&tfile->sk.sk_receive_queue);
tun_set_real_num_queues(tun);
@@ -629,6 +817,37 @@ static const struct net_device_ops tap_netdev_ops = {
#endif
};
+static int tun_flow_init(struct tun_struct *tun)
+{
+ int i;
+
+ tun->flow_cache = kmem_cache_create("tun_flow_cache",
+ sizeof(struct tun_flow_entry), 0, 0,
+ NULL);
+ if (!tun->flow_cache)
+ return -ENOMEM;
+
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++)
+ INIT_HLIST_HEAD(&tun->flows[i]);
+
+ tun->ageing_time = TUN_FLOW_EXPIRE;
+ setup_timer(&tun->flow_gc_timer, tun_flow_cleanup, (unsigned long)tun);
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + tun->ageing_time));
+
+ return 0;
+}
+
+static void tun_flow_uninit(struct tun_struct *tun)
+{
+ del_timer_sync(&tun->flow_gc_timer);
+ tun_flow_flush(tun);
+
+ /* Wait for completion of call_rcu()'s */
+ rcu_barrier();
+ kmem_cache_destroy(tun->flow_cache);
+}
+
/* Initialize net device. */
static void tun_net_init(struct net_device *dev)
{
@@ -973,6 +1192,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
tun->dev->stats.rx_packets++;
tun->dev->stats.rx_bytes += len;
+ tun_flow_update(tun, skb, tfile->queue_index);
return total_len;
}
@@ -1150,6 +1370,14 @@ out:
return ret;
}
+static void tun_free_netdev(struct net_device *dev)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+
+ tun_flow_uninit(tun);
+ free_netdev(dev);
+}
+
static void tun_setup(struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
@@ -1158,7 +1386,7 @@ static void tun_setup(struct net_device *dev)
tun->group = INVALID_GID;
dev->ethtool_ops = &tun_ethtool_ops;
- dev->destructor = free_netdev;
+ dev->destructor = tun_free_netdev;
}
/* Trivial set of netlink ops to allow deleting tun or tap
@@ -1381,10 +1609,15 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
tun->filter_attached = false;
tun->sndbuf = tfile->socket.sk->sk_sndbuf;
+ spin_lock_init(&tun->lock);
+
security_tun_dev_post_create(&tfile->sk);
tun_net_init(dev);
+ if (tun_flow_init(tun))
+ goto err_free_dev;
+
dev->hw_features = NETIF_F_SG | NETIF_F_FRAGLIST |
TUN_USER_FEATURES;
dev->features = dev->hw_features;
--
1.7.1
^ permalink raw reply related
* [net-next v4 1/7] tuntap: log the unsigned informaiton with %u
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 0873cdc..ef13cf0 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1422,7 +1422,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if (!tun)
goto unlock;
- tun_debug(KERN_INFO, tun, "tun_chr_ioctl cmd %d\n", cmd);
+ tun_debug(KERN_INFO, tun, "tun_chr_ioctl cmd %u\n", cmd);
ret = 0;
switch (cmd) {
@@ -1462,7 +1462,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
tun->owner = owner;
- tun_debug(KERN_INFO, tun, "owner set to %d\n",
+ tun_debug(KERN_INFO, tun, "owner set to %u\n",
from_kuid(&init_user_ns, tun->owner));
break;
@@ -1474,7 +1474,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
tun->group = group;
- tun_debug(KERN_INFO, tun, "group set to %d\n",
+ tun_debug(KERN_INFO, tun, "group set to %u\n",
from_kgid(&init_user_ns, tun->group));
break;
--
1.7.1
^ permalink raw reply related
* [net-next v4 2/7] tuntap: move socket to tun_file
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Current tuntap makes use of the socket receive queue as its tx queue. To
implement multiple tx queues for tuntap and enable the ability of adding and
removing queues during workload, the first step is to move the socket related
structures to tun_file. Then we could let multiple fds/sockets to be attached to
the tuntap.
This patch removes tun_sock and moves socket related structures from tun_sock or
tun_struct to tun_file. Two exceptions are tap_filter and sock_fprog, they are
still kept in tun_structure since they are used to filter packets for the net
device instead of per transmit queue (at least I see no requirements for
them). After those changes, socket were created and destroyed during file open
and close (instead of device creation and destroy), the socket structures could
be dereferenced from tun_file instead of the file of tun_struct structure
itself.
For persisent device, since we purge during datching and wouldn't queue any
packets when no interface were attached, there's no behaviod changes before and
after this patch, so the changes were transparent to the userspace. To keep the
attributes such as sndbuf, socket filter and vnet header, those would be
re-initialize after a new interface were attached to an persist device.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 266 +++++++++++++++++++++++++++++------------------------
1 files changed, 145 insertions(+), 121 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index ef13cf0..e8cedb0 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -110,14 +110,29 @@ struct tap_filter {
unsigned char addr[FLT_EXACT_COUNT][ETH_ALEN];
};
+/* A tun_file connects an open character device to a tuntap netdevice. It
+ * also contains all socket related strctures (except sock_fprog and tap_filter)
+ * to serve as one transmit queue for tuntap device. The sock_fprog and
+ * tap_filter were kept in tun_struct since they were used for filtering for the
+ * netdevice not for a specific queue (at least I didn't see the reqirement for
+ * this).
+ */
struct tun_file {
+ struct sock sk;
+ struct socket socket;
+ struct socket_wq wq;
atomic_t count;
struct tun_struct *tun;
struct net *net;
+ struct fasync_struct *fasync;
+ /* only used for fasnyc */
+ unsigned int flags;
};
-struct tun_sock;
-
+/* Since the socket were moved to tun_file, to preserve the behavior of persist
+ * device, socket fileter, sndbuf and vnet header size were restore when the
+ * file were attached to a persist device.
+ */
struct tun_struct {
struct tun_file *tfile;
unsigned int flags;
@@ -128,29 +143,18 @@ struct tun_struct {
netdev_features_t set_features;
#define TUN_USER_FEATURES (NETIF_F_HW_CSUM|NETIF_F_TSO_ECN|NETIF_F_TSO| \
NETIF_F_TSO6|NETIF_F_UFO)
- struct fasync_struct *fasync;
-
- struct tap_filter txflt;
- struct socket socket;
- struct socket_wq wq;
int vnet_hdr_sz;
-
+ int sndbuf;
+ struct tap_filter txflt;
+ struct sock_fprog fprog;
+ /* protected by rtnl lock */
+ bool filter_attached;
#ifdef TUN_DEBUG
int debug;
#endif
};
-struct tun_sock {
- struct sock sk;
- struct tun_struct *tun;
-};
-
-static inline struct tun_sock *tun_sk(struct sock *sk)
-{
- return container_of(sk, struct tun_sock, sk);
-}
-
static int tun_attach(struct tun_struct *tun, struct file *file)
{
struct tun_file *tfile = file->private_data;
@@ -169,12 +173,19 @@ static int tun_attach(struct tun_struct *tun, struct file *file)
goto out;
err = 0;
+
+ /* Re-attach filter when attaching to a persist device */
+ if (tun->filter_attached == true) {
+ err = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (!err)
+ goto out;
+ }
tfile->tun = tun;
+ tfile->socket.sk->sk_sndbuf = tun->sndbuf;
tun->tfile = tfile;
- tun->socket.file = file;
netif_carrier_on(tun->dev);
dev_hold(tun->dev);
- sock_hold(tun->socket.sk);
+ sock_hold(&tfile->sk);
atomic_inc(&tfile->count);
out:
@@ -184,14 +195,16 @@ out:
static void __tun_detach(struct tun_struct *tun)
{
+ struct tun_file *tfile = tun->tfile;
/* Detach from net device */
netif_tx_lock_bh(tun->dev);
netif_carrier_off(tun->dev);
tun->tfile = NULL;
+ tfile->tun = NULL;
netif_tx_unlock_bh(tun->dev);
/* Drop read queue */
- skb_queue_purge(&tun->socket.sk->sk_receive_queue);
+ skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
/* Drop the extra count on the net device */
dev_put(tun->dev);
@@ -350,21 +363,12 @@ static void tun_net_uninit(struct net_device *dev)
/* Inform the methods they need to stop using the dev.
*/
if (tfile) {
- wake_up_all(&tun->wq.wait);
+ wake_up_all(&tfile->wq.wait);
if (atomic_dec_and_test(&tfile->count))
__tun_detach(tun);
}
}
-static void tun_free_netdev(struct net_device *dev)
-{
- struct tun_struct *tun = netdev_priv(dev);
-
- BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED, &tun->socket.flags));
-
- sk_release_kernel(tun->socket.sk);
-}
-
/* Net device open. */
static int tun_net_open(struct net_device *dev)
{
@@ -383,11 +387,12 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
+ struct tun_file *tfile = tun->tfile;
tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
/* Drop packet if interface is not attached */
- if (!tun->tfile)
+ if (!tfile)
goto drop;
/* Drop if the filter does not like it.
@@ -396,11 +401,12 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
if (!check_filter(&tun->txflt, skb))
goto drop;
- if (tun->socket.sk->sk_filter &&
- sk_filter(tun->socket.sk, skb))
+ if (tfile->socket.sk->sk_filter &&
+ sk_filter(tfile->socket.sk, skb))
goto drop;
- if (skb_queue_len(&tun->socket.sk->sk_receive_queue) >= dev->tx_queue_len) {
+ if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
+ >= dev->tx_queue_len) {
if (!(tun->flags & TUN_ONE_QUEUE)) {
/* Normal queueing mode. */
/* Packet scheduler handles dropping of further packets. */
@@ -423,12 +429,12 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
skb_orphan(skb);
/* Enqueue packet */
- skb_queue_tail(&tun->socket.sk->sk_receive_queue, skb);
+ skb_queue_tail(&tfile->socket.sk->sk_receive_queue, skb);
/* Notify and wake up reader process */
- if (tun->flags & TUN_FASYNC)
- kill_fasync(&tun->fasync, SIGIO, POLL_IN);
- wake_up_interruptible_poll(&tun->wq.wait, POLLIN |
+ if (tfile->flags & TUN_FASYNC)
+ kill_fasync(&tfile->fasync, SIGIO, POLL_IN);
+ wake_up_interruptible_poll(&tfile->wq.wait, POLLIN |
POLLRDNORM | POLLRDBAND);
return NETDEV_TX_OK;
@@ -556,11 +562,11 @@ static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
if (!tun)
return POLLERR;
- sk = tun->socket.sk;
+ sk = tfile->socket.sk;
tun_debug(KERN_INFO, tun, "tun_chr_poll\n");
- poll_wait(file, &tun->wq.wait, wait);
+ poll_wait(file, &tfile->wq.wait, wait);
if (!skb_queue_empty(&sk->sk_receive_queue))
mask |= POLLIN | POLLRDNORM;
@@ -579,11 +585,11 @@ static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
/* prepad is the amount to reserve at front. len is length after that.
* linear is a hint as to how much to copy (usually headers). */
-static struct sk_buff *tun_alloc_skb(struct tun_struct *tun,
+static struct sk_buff *tun_alloc_skb(struct tun_file *tfile,
size_t prepad, size_t len,
size_t linear, int noblock)
{
- struct sock *sk = tun->socket.sk;
+ struct sock *sk = tfile->socket.sk;
struct sk_buff *skb;
int err;
@@ -685,9 +691,9 @@ static int zerocopy_sg_from_iovec(struct sk_buff *skb, const struct iovec *from,
}
/* Get packet from user space buffer */
-static ssize_t tun_get_user(struct tun_struct *tun, void *msg_control,
- const struct iovec *iv, size_t total_len,
- size_t count, int noblock)
+static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
+ void *msg_control, const struct iovec *iv,
+ size_t total_len, size_t count, int noblock)
{
struct tun_pi pi = { 0, cpu_to_be16(ETH_P_IP) };
struct sk_buff *skb;
@@ -757,7 +763,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, void *msg_control,
} else
copylen = len;
- skb = tun_alloc_skb(tun, align, copylen, gso.hdr_len, noblock);
+ skb = tun_alloc_skb(tfile, align, copylen, gso.hdr_len, noblock);
if (IS_ERR(skb)) {
if (PTR_ERR(skb) != -EAGAIN)
tun->dev->stats.rx_dropped++;
@@ -862,6 +868,7 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
{
struct file *file = iocb->ki_filp;
struct tun_struct *tun = tun_get(file);
+ struct tun_file *tfile = file->private_data;
ssize_t result;
if (!tun)
@@ -869,8 +876,8 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
tun_debug(KERN_INFO, tun, "tun_chr_write %ld\n", count);
- result = tun_get_user(tun, NULL, iv, iov_length(iv, count), count,
- file->f_flags & O_NONBLOCK);
+ result = tun_get_user(tun, tfile, NULL, iv, iov_length(iv, count),
+ count, file->f_flags & O_NONBLOCK);
tun_put(tun);
return result;
@@ -878,6 +885,7 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
/* Put packet to the user space buffer */
static ssize_t tun_put_user(struct tun_struct *tun,
+ struct tun_file *tfile,
struct sk_buff *skb,
const struct iovec *iv, int len)
{
@@ -957,7 +965,7 @@ static ssize_t tun_put_user(struct tun_struct *tun,
return total;
}
-static ssize_t tun_do_read(struct tun_struct *tun,
+static ssize_t tun_do_read(struct tun_struct *tun,struct tun_file *tfile,
struct kiocb *iocb, const struct iovec *iv,
ssize_t len, int noblock)
{
@@ -968,12 +976,12 @@ static ssize_t tun_do_read(struct tun_struct *tun,
tun_debug(KERN_INFO, tun, "tun_chr_read\n");
if (unlikely(!noblock))
- add_wait_queue(&tun->wq.wait, &wait);
+ add_wait_queue(&tfile->wq.wait, &wait);
while (len) {
current->state = TASK_INTERRUPTIBLE;
/* Read frames from the queue */
- if (!(skb=skb_dequeue(&tun->socket.sk->sk_receive_queue))) {
+ if (!(skb = skb_dequeue(&tfile->socket.sk->sk_receive_queue))) {
if (noblock) {
ret = -EAGAIN;
break;
@@ -993,14 +1001,14 @@ static ssize_t tun_do_read(struct tun_struct *tun,
}
netif_wake_queue(tun->dev);
- ret = tun_put_user(tun, skb, iv, len);
+ ret = tun_put_user(tun, tfile, skb, iv, len);
kfree_skb(skb);
break;
}
current->state = TASK_RUNNING;
if (unlikely(!noblock))
- remove_wait_queue(&tun->wq.wait, &wait);
+ remove_wait_queue(&tfile->wq.wait, &wait);
return ret;
}
@@ -1021,7 +1029,8 @@ static ssize_t tun_chr_aio_read(struct kiocb *iocb, const struct iovec *iv,
goto out;
}
- ret = tun_do_read(tun, iocb, iv, len, file->f_flags & O_NONBLOCK);
+ ret = tun_do_read(tun, tfile, iocb, iv, len,
+ file->f_flags & O_NONBLOCK);
ret = min_t(ssize_t, ret, len);
out:
tun_put(tun);
@@ -1036,7 +1045,7 @@ static void tun_setup(struct net_device *dev)
tun->group = INVALID_GID;
dev->ethtool_ops = &tun_ethtool_ops;
- dev->destructor = tun_free_netdev;
+ dev->destructor = free_netdev;
}
/* Trivial set of netlink ops to allow deleting tun or tap
@@ -1056,7 +1065,7 @@ static struct rtnl_link_ops tun_link_ops __read_mostly = {
static void tun_sock_write_space(struct sock *sk)
{
- struct tun_struct *tun;
+ struct tun_file *tfile;
wait_queue_head_t *wqueue;
if (!sock_writeable(sk))
@@ -1070,37 +1079,47 @@ static void tun_sock_write_space(struct sock *sk)
wake_up_interruptible_sync_poll(wqueue, POLLOUT |
POLLWRNORM | POLLWRBAND);
- tun = tun_sk(sk)->tun;
- kill_fasync(&tun->fasync, SIGIO, POLL_OUT);
-}
-
-static void tun_sock_destruct(struct sock *sk)
-{
- free_netdev(tun_sk(sk)->tun->dev);
+ tfile = container_of(sk, struct tun_file, sk);
+ kill_fasync(&tfile->fasync, SIGIO, POLL_OUT);
}
static int tun_sendmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *m, size_t total_len)
{
- struct tun_struct *tun = container_of(sock, struct tun_struct, socket);
- return tun_get_user(tun, m->msg_control, m->msg_iov, total_len,
- m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
+ int ret;
+ struct tun_file *tfile = container_of(sock, struct tun_file, socket);
+ struct tun_struct *tun = __tun_get(tfile);
+
+ if (!tun)
+ return -EBADFD;
+
+ ret = tun_get_user(tun, tfile, m->msg_control, m->msg_iov, total_len,
+ m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
+ tun_put(tun);
+ return ret;
}
+
static int tun_recvmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *m, size_t total_len,
int flags)
{
- struct tun_struct *tun = container_of(sock, struct tun_struct, socket);
+ struct tun_file *tfile = container_of(sock, struct tun_file, socket);
+ struct tun_struct *tun = __tun_get(tfile);
int ret;
+
+ if (!tun)
+ return -EBADFD;
+
if (flags & ~(MSG_DONTWAIT|MSG_TRUNC))
return -EINVAL;
- ret = tun_do_read(tun, iocb, m->msg_iov, total_len,
+ ret = tun_do_read(tun, tfile, iocb, m->msg_iov, total_len,
flags & MSG_DONTWAIT);
if (ret > total_len) {
m->msg_flags |= MSG_TRUNC;
ret = flags & MSG_TRUNC ? ret : total_len;
}
+ tun_put(tun);
return ret;
}
@@ -1121,7 +1140,7 @@ static const struct proto_ops tun_socket_ops = {
static struct proto tun_proto = {
.name = "tun",
.owner = THIS_MODULE,
- .obj_size = sizeof(struct tun_sock),
+ .obj_size = sizeof(struct tun_file),
};
static int tun_flags(struct tun_struct *tun)
@@ -1178,8 +1197,8 @@ static DEVICE_ATTR(group, 0444, tun_show_group, NULL);
static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
{
- struct sock *sk;
struct tun_struct *tun;
+ struct tun_file *tfile = file->private_data;
struct net_device *dev;
int err;
@@ -1200,7 +1219,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
(gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
!capable(CAP_NET_ADMIN))
return -EPERM;
- err = security_tun_dev_attach(tun->socket.sk);
+ err = security_tun_dev_attach(tfile->socket.sk);
if (err < 0)
return err;
@@ -1246,25 +1265,11 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
tun->flags = flags;
tun->txflt.count = 0;
tun->vnet_hdr_sz = sizeof(struct virtio_net_hdr);
- set_bit(SOCK_EXTERNALLY_ALLOCATED, &tun->socket.flags);
-
- err = -ENOMEM;
- sk = sk_alloc(&init_net, AF_UNSPEC, GFP_KERNEL, &tun_proto);
- if (!sk)
- goto err_free_dev;
- sk_change_net(sk, net);
- tun->socket.wq = &tun->wq;
- init_waitqueue_head(&tun->wq.wait);
- tun->socket.ops = &tun_socket_ops;
- sock_init_data(&tun->socket, sk);
- sk->sk_write_space = tun_sock_write_space;
- sk->sk_sndbuf = INT_MAX;
- sock_set_flag(sk, SOCK_ZEROCOPY);
+ tun->filter_attached = false;
+ tun->sndbuf = tfile->socket.sk->sk_sndbuf;
- tun_sk(sk)->tun = tun;
-
- security_tun_dev_post_create(sk);
+ security_tun_dev_post_create(&tfile->sk);
tun_net_init(dev);
@@ -1274,15 +1279,13 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
err = register_netdevice(tun->dev);
if (err < 0)
- goto err_free_sk;
+ goto err_free_dev;
if (device_create_file(&tun->dev->dev, &dev_attr_tun_flags) ||
device_create_file(&tun->dev->dev, &dev_attr_owner) ||
device_create_file(&tun->dev->dev, &dev_attr_group))
pr_err("Failed to create tun sysfs files\n");
- sk->sk_destruct = tun_sock_destruct;
-
err = tun_attach(tun, file);
if (err < 0)
goto failed;
@@ -1314,8 +1317,6 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
strcpy(ifr->ifr_name, tun->dev->name);
return 0;
- err_free_sk:
- tun_free_netdev(dev);
err_free_dev:
free_netdev(dev);
failed:
@@ -1379,7 +1380,6 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
struct tun_file *tfile = file->private_data;
struct tun_struct *tun;
void __user* argp = (void __user*)arg;
- struct sock_fprog fprog;
struct ifreq ifr;
kuid_t owner;
kgid_t group;
@@ -1444,11 +1444,16 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
case TUNSETPERSIST:
- /* Disable/Enable persist mode */
- if (arg)
+ /* Disable/Enable persist mode. Keep an extra reference to the
+ * module to prevent the module being unprobed.
+ */
+ if (arg) {
tun->flags |= TUN_PERSIST;
- else
+ __module_get(THIS_MODULE);
+ } else {
tun->flags &= ~TUN_PERSIST;
+ module_put(THIS_MODULE);
+ }
tun_debug(KERN_INFO, tun, "persist %s\n",
arg ? "enabled" : "disabled");
@@ -1526,7 +1531,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
case TUNGETSNDBUF:
- sndbuf = tun->socket.sk->sk_sndbuf;
+ sndbuf = tfile->socket.sk->sk_sndbuf;
if (copy_to_user(argp, &sndbuf, sizeof(sndbuf)))
ret = -EFAULT;
break;
@@ -1537,7 +1542,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
- tun->socket.sk->sk_sndbuf = sndbuf;
+ tun->sndbuf = tfile->socket.sk->sk_sndbuf = sndbuf;
break;
case TUNGETVNETHDRSZ:
@@ -1565,10 +1570,12 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
ret = -EFAULT;
- if (copy_from_user(&fprog, argp, sizeof(fprog)))
+ if (copy_from_user(&tun->fprog, argp, sizeof(tun->fprog)))
break;
- ret = sk_attach_filter(&fprog, tun->socket.sk);
+ ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (!ret)
+ tun->filter_attached = true;
break;
case TUNDETACHFILTER:
@@ -1576,7 +1583,9 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
ret = -EINVAL;
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
- ret = sk_detach_filter(tun->socket.sk);
+ ret = sk_detach_filter(tfile->socket.sk);
+ if (!ret)
+ tun->filter_attached = false;
break;
default:
@@ -1628,27 +1637,21 @@ static long tun_chr_compat_ioctl(struct file *file,
static int tun_chr_fasync(int fd, struct file *file, int on)
{
- struct tun_struct *tun = tun_get(file);
+ struct tun_file *tfile = file->private_data;
int ret;
- if (!tun)
- return -EBADFD;
-
- tun_debug(KERN_INFO, tun, "tun_chr_fasync %d\n", on);
-
- if ((ret = fasync_helper(fd, file, on, &tun->fasync)) < 0)
+ if ((ret = fasync_helper(fd, file, on, &tfile->fasync)) < 0)
goto out;
if (on) {
ret = __f_setown(file, task_pid(current), PIDTYPE_PID, 0);
if (ret)
goto out;
- tun->flags |= TUN_FASYNC;
+ tfile->flags |= TUN_FASYNC;
} else
- tun->flags &= ~TUN_FASYNC;
+ tfile->flags &= ~TUN_FASYNC;
ret = 0;
out:
- tun_put(tun);
return ret;
}
@@ -1658,13 +1661,30 @@ static int tun_chr_open(struct inode *inode, struct file * file)
DBG1(KERN_INFO, "tunX: tun_chr_open\n");
- tfile = kmalloc(sizeof(*tfile), GFP_KERNEL);
+ tfile = (struct tun_file *)sk_alloc(&init_net, AF_UNSPEC, GFP_KERNEL,
+ &tun_proto);
if (!tfile)
return -ENOMEM;
atomic_set(&tfile->count, 0);
tfile->tun = NULL;
tfile->net = get_net(current->nsproxy->net_ns);
+ tfile->flags = 0;
+
+ rcu_assign_pointer(tfile->socket.wq, &tfile->wq);
+ init_waitqueue_head(&tfile->wq.wait);
+
+ tfile->socket.file = file;
+ tfile->socket.ops = &tun_socket_ops;
+
+ sock_init_data(&tfile->socket, &tfile->sk);
+ sk_change_net(&tfile->sk, tfile->net);
+
+ tfile->sk.sk_write_space = tun_sock_write_space;
+ tfile->sk.sk_sndbuf = INT_MAX;
+
file->private_data = tfile;
+ set_bit(SOCK_EXTERNALLY_ALLOCATED, &tfile->socket.flags);
+
return 0;
}
@@ -1672,6 +1692,7 @@ static int tun_chr_close(struct inode *inode, struct file *file)
{
struct tun_file *tfile = file->private_data;
struct tun_struct *tun;
+ struct net *net = tfile->net;
tun = __tun_get(tfile);
if (tun) {
@@ -1688,14 +1709,16 @@ static int tun_chr_close(struct inode *inode, struct file *file)
unregister_netdevice(dev);
rtnl_unlock();
}
- }
- tun = tfile->tun;
- if (tun)
- sock_put(tun->socket.sk);
+ /* drop the reference that netdevice holds */
+ sock_put(&tfile->sk);
+ }
- put_net(tfile->net);
- kfree(tfile);
+ /* drop the reference that file holds */
+ BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
+ &tfile->socket.flags));
+ sk_release_kernel(&tfile->sk);
+ put_net(net);
return 0;
}
@@ -1823,13 +1846,14 @@ static void tun_cleanup(void)
struct socket *tun_get_socket(struct file *file)
{
struct tun_struct *tun;
+ struct tun_file *tfile = file->private_data;
if (file->f_op != &tun_fops)
return ERR_PTR(-EINVAL);
tun = tun_get(file);
if (!tun)
return ERR_PTR(-EBADFD);
tun_put(tun);
- return &tun->socket;
+ return &tfile->socket;
}
EXPORT_SYMBOL_GPL(tun_get_socket);
--
1.7.1
^ permalink raw reply related
* [net-next v4 3/7] tuntap: RCUify dereferencing between tun_struct and tun_file
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
RCU were introduced in this patch to synchronize the dereferences between
tun_struct and tun_file. All tun_{get|put} were replaced with RCU, the
dereference from one to other must be done under rtnl lock or rcu read critical
region.
This is needed for the following patches since the one of the goal of multiqueue
tuntap is to allow adding or removing queues during workload. Without RCU,
control path would hold tx locks when adding or removing queues (which may cause
sme delay) and it's hard to change the number of queues without stopping the net
device. With the help of rcu, there's also no need for tun_file hold an refcnt
to tun_struct.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 95 ++++++++++++++++++++++++++---------------------------
1 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index e8cedb0..d332cb8 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -116,13 +116,16 @@ struct tap_filter {
* tap_filter were kept in tun_struct since they were used for filtering for the
* netdevice not for a specific queue (at least I didn't see the reqirement for
* this).
+ *
+ * RCU usage:
+ * The tun_file and tun_struct are loosely coupled, the pointer from on to the
+ * other can only be read while rcu_read_lock or rtnl_lock is held.
*/
struct tun_file {
struct sock sk;
struct socket socket;
struct socket_wq wq;
- atomic_t count;
- struct tun_struct *tun;
+ struct tun_struct __rcu *tun;
struct net *net;
struct fasync_struct *fasync;
/* only used for fasnyc */
@@ -134,7 +137,7 @@ struct tun_file {
* file were attached to a persist device.
*/
struct tun_struct {
- struct tun_file *tfile;
+ struct tun_file __rcu *tfile;
unsigned int flags;
kuid_t owner;
kgid_t group;
@@ -180,13 +183,11 @@ static int tun_attach(struct tun_struct *tun, struct file *file)
if (!err)
goto out;
}
- tfile->tun = tun;
+ rcu_assign_pointer(tfile->tun, tun);
tfile->socket.sk->sk_sndbuf = tun->sndbuf;
- tun->tfile = tfile;
+ rcu_assign_pointer(tun->tfile, tfile);
netif_carrier_on(tun->dev);
- dev_hold(tun->dev);
sock_hold(&tfile->sk);
- atomic_inc(&tfile->count);
out:
netif_tx_unlock_bh(tun->dev);
@@ -195,34 +196,29 @@ out:
static void __tun_detach(struct tun_struct *tun)
{
- struct tun_file *tfile = tun->tfile;
+ struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
+ lockdep_rtnl_is_held());
/* Detach from net device */
- netif_tx_lock_bh(tun->dev);
netif_carrier_off(tun->dev);
- tun->tfile = NULL;
- tfile->tun = NULL;
- netif_tx_unlock_bh(tun->dev);
-
- /* Drop read queue */
- skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
-
- /* Drop the extra count on the net device */
- dev_put(tun->dev);
-}
+ rcu_assign_pointer(tun->tfile, NULL);
+ if (tfile) {
+ rcu_assign_pointer(tfile->tun, NULL);
-static void tun_detach(struct tun_struct *tun)
-{
- rtnl_lock();
- __tun_detach(tun);
- rtnl_unlock();
+ synchronize_net();
+ /* Drop read queue */
+ skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
+ }
}
static struct tun_struct *__tun_get(struct tun_file *tfile)
{
- struct tun_struct *tun = NULL;
+ struct tun_struct *tun;
- if (atomic_inc_not_zero(&tfile->count))
- tun = tfile->tun;
+ rcu_read_lock();
+ tun = rcu_dereference(tfile->tun);
+ if (tun)
+ dev_hold(tun->dev);
+ rcu_read_unlock();
return tun;
}
@@ -234,10 +230,7 @@ static struct tun_struct *tun_get(struct file *file)
static void tun_put(struct tun_struct *tun)
{
- struct tun_file *tfile = tun->tfile;
-
- if (atomic_dec_and_test(&tfile->count))
- tun_detach(tfile->tun);
+ dev_put(tun->dev);
}
/* TAP filtering */
@@ -358,14 +351,15 @@ static const struct ethtool_ops tun_ethtool_ops;
static void tun_net_uninit(struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = tun->tfile;
+ struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
+ lockdep_rtnl_is_held());
/* Inform the methods they need to stop using the dev.
*/
if (tfile) {
wake_up_all(&tfile->wq.wait);
- if (atomic_dec_and_test(&tfile->count))
- __tun_detach(tun);
+ __tun_detach(tun);
+ synchronize_net();
}
}
@@ -387,14 +381,16 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = tun->tfile;
-
- tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+ struct tun_file *tfile;
+ rcu_read_lock();
+ tfile = rcu_dereference(tun->tfile);
/* Drop packet if interface is not attached */
if (!tfile)
goto drop;
+ tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+
/* Drop if the filter does not like it.
* This is a noop if the filter is disabled.
* Filter can be enabled only for the TAP devices. */
@@ -436,11 +432,14 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
kill_fasync(&tfile->fasync, SIGIO, POLL_IN);
wake_up_interruptible_poll(&tfile->wq.wait, POLLIN |
POLLRDNORM | POLLRDBAND);
+
+ rcu_read_unlock();
return NETDEV_TX_OK;
drop:
dev->stats.tx_dropped++;
kfree_skb(skb);
+ rcu_read_unlock();
return NETDEV_TX_OK;
}
@@ -1092,7 +1091,6 @@ static int tun_sendmsg(struct kiocb *iocb, struct socket *sock,
if (!tun)
return -EBADFD;
-
ret = tun_get_user(tun, tfile, m->msg_control, m->msg_iov, total_len,
m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
tun_put(tun);
@@ -1665,8 +1663,7 @@ static int tun_chr_open(struct inode *inode, struct file * file)
&tun_proto);
if (!tfile)
return -ENOMEM;
- atomic_set(&tfile->count, 0);
- tfile->tun = NULL;
+ rcu_assign_pointer(tfile->tun, NULL);
tfile->net = get_net(current->nsproxy->net_ns);
tfile->flags = 0;
@@ -1694,7 +1691,9 @@ static int tun_chr_close(struct inode *inode, struct file *file)
struct tun_struct *tun;
struct net *net = tfile->net;
- tun = __tun_get(tfile);
+ rtnl_lock();
+
+ tun = rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held());
if (tun) {
struct net_device *dev = tun->dev;
@@ -1702,18 +1701,20 @@ static int tun_chr_close(struct inode *inode, struct file *file)
__tun_detach(tun);
+ synchronize_net();
+
/* If desirable, unregister the netdevice. */
if (!(tun->flags & TUN_PERSIST)) {
- rtnl_lock();
if (dev->reg_state == NETREG_REGISTERED)
unregister_netdevice(dev);
- rtnl_unlock();
}
/* drop the reference that netdevice holds */
sock_put(&tfile->sk);
}
+ rtnl_unlock();
+
/* drop the reference that file holds */
BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
&tfile->socket.flags));
@@ -1845,14 +1846,12 @@ static void tun_cleanup(void)
* holding a reference to the file for as long as the socket is in use. */
struct socket *tun_get_socket(struct file *file)
{
- struct tun_struct *tun;
- struct tun_file *tfile = file->private_data;
+ struct tun_file *tfile;
if (file->f_op != &tun_fops)
return ERR_PTR(-EINVAL);
- tun = tun_get(file);
- if (!tun)
+ tfile = file->private_data;
+ if (!tfile)
return ERR_PTR(-EBADFD);
- tun_put(tun);
return &tfile->socket;
}
EXPORT_SYMBOL_GPL(tun_get_socket);
--
1.7.1
^ permalink raw reply related
* [net-next v4 4/7] tuntap: introduce multiqueue flags
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Add flags to be used by creating multiqueue tuntap device.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
include/uapi/linux/if_tun.h | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h
index 25a585c..8ef3a87 100644
--- a/include/uapi/linux/if_tun.h
+++ b/include/uapi/linux/if_tun.h
@@ -34,6 +34,7 @@
#define TUN_ONE_QUEUE 0x0080
#define TUN_PERSIST 0x0100
#define TUN_VNET_HDR 0x0200
+#define TUN_TAP_MQ 0x0400
/* Ioctl defines */
#define TUNSETNOCSUM _IOW('T', 200, int)
@@ -61,6 +62,7 @@
#define IFF_ONE_QUEUE 0x2000
#define IFF_VNET_HDR 0x4000
#define IFF_TUN_EXCL 0x8000
+#define IFF_MULTI_QUEUE 0x0100
/* Features for GSO (TUNSETOFFLOAD). */
#define TUN_F_CSUM 0x01 /* You can hand me unchecksummed packets. */
--
1.7.1
^ permalink raw reply related
* [net-next v4 5/7] tuntap: multiqueue support
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
This patch converts tun/tap to a multiqueue devices and expose the multiqueue
queues as multiple file descriptors to userspace. Internally, each tun_file were
abstracted as a queue, and an array of pointers to tun_file structurs were
stored in tun_structure device, so multiple tun_files were allowed to be
attached to the device as multiple queues.
When choosing txq, we first try to identify a flow through its rxhash, if it
does not have such one, we could try recorded rxq and then use them to choose
the transmit queue. This policy may be changed in the future.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 305 +++++++++++++++++++++++++++++++++++++---------------
1 files changed, 217 insertions(+), 88 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index d332cb8..59235ba 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -110,6 +110,11 @@ struct tap_filter {
unsigned char addr[FLT_EXACT_COUNT][ETH_ALEN];
};
+/* 1024 is probably a high enough limit: modern hypervisors seem to support on
+ * the order of 100-200 CPUs so this leaves us some breathing space if we want
+ * to match a queue per guest CPU. */
+#define MAX_TAP_QUEUES 1024
+
/* A tun_file connects an open character device to a tuntap netdevice. It
* also contains all socket related strctures (except sock_fprog and tap_filter)
* to serve as one transmit queue for tuntap device. The sock_fprog and
@@ -130,6 +135,7 @@ struct tun_file {
struct fasync_struct *fasync;
/* only used for fasnyc */
unsigned int flags;
+ u16 queue_index;
};
/* Since the socket were moved to tun_file, to preserve the behavior of persist
@@ -137,7 +143,8 @@ struct tun_file {
* file were attached to a persist device.
*/
struct tun_struct {
- struct tun_file __rcu *tfile;
+ struct tun_file __rcu *tfiles[MAX_TAP_QUEUES];
+ unsigned int numqueues;
unsigned int flags;
kuid_t owner;
kgid_t group;
@@ -158,56 +165,156 @@ struct tun_struct {
#endif
};
+/* We try to identify a flow through its rxhash first. The reason that
+ * we do not check rxq no. is becuase some cards(e.g 82599), chooses
+ * the rxq based on the txq where the last packet of the flow comes. As
+ * the userspace application move between processors, we may get a
+ * different rxq no. here. If we could not get rxhash, then we would
+ * hope the rxq no. may help here.
+ */
+static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+ u32 txq = 0;
+ u32 numqueues = 0;
+
+ rcu_read_lock();
+ numqueues = tun->numqueues;
+
+ txq = skb_get_rxhash(skb);
+ if (txq) {
+ /* use multiply and shift instead of expensive divide */
+ txq = ((u64)txq * numqueues) >> 32;
+ } else if (likely(skb_rx_queue_recorded(skb))) {
+ txq = skb_get_rx_queue(skb);
+ while (unlikely(txq >= numqueues))
+ txq -= numqueues;
+ }
+
+ rcu_read_unlock();
+ return txq;
+}
+
+static void tun_set_real_num_queues(struct tun_struct *tun)
+{
+ netif_set_real_num_tx_queues(tun->dev, tun->numqueues);
+ netif_set_real_num_rx_queues(tun->dev, tun->numqueues);
+}
+
+static void __tun_detach(struct tun_file *tfile, bool clean)
+{
+ struct tun_file *ntfile;
+ struct tun_struct *tun;
+ struct net_device *dev;
+
+ tun = rcu_dereference_protected(tfile->tun,
+ lockdep_rtnl_is_held());
+ if (tun) {
+ u16 index = tfile->queue_index;
+ BUG_ON(index >= tun->numqueues);
+ dev = tun->dev;
+
+ rcu_assign_pointer(tun->tfiles[index],
+ tun->tfiles[tun->numqueues - 1]);
+ rcu_assign_pointer(tfile->tun, NULL);
+ ntfile = rcu_dereference_protected(tun->tfiles[index],
+ lockdep_rtnl_is_held());
+ ntfile->queue_index = index;
+
+ --tun->numqueues;
+ sock_put(&tfile->sk);
+
+ synchronize_net();
+ /* Drop read queue */
+ skb_queue_purge(&tfile->sk.sk_receive_queue);
+ tun_set_real_num_queues(tun);
+
+ if (tun->numqueues == 0 && !(tun->flags & TUN_PERSIST))
+ if (dev->reg_state == NETREG_REGISTERED)
+ unregister_netdevice(dev);
+ }
+
+ if (clean) {
+ BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
+ &tfile->socket.flags));
+ sk_release_kernel(&tfile->sk);
+ }
+}
+
+static void tun_detach(struct tun_file *tfile, bool clean)
+{
+ rtnl_lock();
+ __tun_detach(tfile, clean);
+ rtnl_unlock();
+}
+
+static void tun_detach_all(struct net_device *dev)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+ struct tun_file *tfile;
+ int i, n = tun->numqueues;
+
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ BUG_ON(!tfile);
+ wake_up_all(&tfile->wq.wait);
+ rcu_assign_pointer(tfile->tun, NULL);
+ --tun->numqueues;
+ }
+ BUG_ON(tun->numqueues != 0);
+
+ synchronize_net();
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ /* Drop read queue */
+ skb_queue_purge(&tfile->sk.sk_receive_queue);
+ sock_put(&tfile->sk);
+ }
+}
+
static int tun_attach(struct tun_struct *tun, struct file *file)
{
struct tun_file *tfile = file->private_data;
int err;
- ASSERT_RTNL();
-
- netif_tx_lock_bh(tun->dev);
-
err = -EINVAL;
- if (tfile->tun)
+ if (rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held()))
goto out;
err = -EBUSY;
- if (tun->tfile)
+ if (!(tun->flags & TUN_TAP_MQ) && tun->numqueues == 1)
+ goto out;
+
+ err = -E2BIG;
+ if (tun->numqueues == MAX_TAP_QUEUES)
goto out;
err = 0;
- /* Re-attach filter when attaching to a persist device */
+ /* Re-attach the filter to presist device */
if (tun->filter_attached == true) {
err = sk_attach_filter(&tun->fprog, tfile->socket.sk);
if (!err)
goto out;
}
+ tfile->queue_index = tun->numqueues;
rcu_assign_pointer(tfile->tun, tun);
- tfile->socket.sk->sk_sndbuf = tun->sndbuf;
- rcu_assign_pointer(tun->tfile, tfile);
- netif_carrier_on(tun->dev);
+ rcu_assign_pointer(tun->tfiles[tun->numqueues], tfile);
sock_hold(&tfile->sk);
+ tun->numqueues++;
-out:
- netif_tx_unlock_bh(tun->dev);
- return err;
-}
+ tun_set_real_num_queues(tun);
-static void __tun_detach(struct tun_struct *tun)
-{
- struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
- lockdep_rtnl_is_held());
- /* Detach from net device */
- netif_carrier_off(tun->dev);
- rcu_assign_pointer(tun->tfile, NULL);
- if (tfile) {
- rcu_assign_pointer(tfile->tun, NULL);
+ if (tun->numqueues == 1)
+ netif_carrier_on(tun->dev);
- synchronize_net();
- /* Drop read queue */
- skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
- }
+ /* device is allowed to go away first, so no need to hold extra
+ * refcnt. */
+
+out:
+ return err;
}
static struct tun_struct *__tun_get(struct tun_file *tfile)
@@ -350,30 +457,20 @@ static const struct ethtool_ops tun_ethtool_ops;
/* Net device detach from fd. */
static void tun_net_uninit(struct net_device *dev)
{
- struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
- lockdep_rtnl_is_held());
-
- /* Inform the methods they need to stop using the dev.
- */
- if (tfile) {
- wake_up_all(&tfile->wq.wait);
- __tun_detach(tun);
- synchronize_net();
- }
+ tun_detach_all(dev);
}
/* Net device open. */
static int tun_net_open(struct net_device *dev)
{
- netif_start_queue(dev);
+ netif_tx_start_all_queues(dev);
return 0;
}
/* Net device close. */
static int tun_net_close(struct net_device *dev)
{
- netif_stop_queue(dev);
+ netif_tx_stop_all_queues(dev);
return 0;
}
@@ -381,16 +478,20 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
+ int txq = skb->queue_mapping;
struct tun_file *tfile;
rcu_read_lock();
- tfile = rcu_dereference(tun->tfile);
+ tfile = rcu_dereference(tun->tfiles[txq]);
+
/* Drop packet if interface is not attached */
- if (!tfile)
+ if (txq >= tun->numqueues)
goto drop;
tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+ BUG_ON(!tfile);
+
/* Drop if the filter does not like it.
* This is a noop if the filter is disabled.
* Filter can be enabled only for the TAP devices. */
@@ -401,12 +502,14 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
sk_filter(tfile->socket.sk, skb))
goto drop;
+ /* Limit the number of packets queued by divining txq length with the
+ * number of queues. */
if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
- >= dev->tx_queue_len) {
+ >= dev->tx_queue_len / tun->numqueues){
if (!(tun->flags & TUN_ONE_QUEUE)) {
/* Normal queueing mode. */
/* Packet scheduler handles dropping of further packets. */
- netif_stop_queue(dev);
+ netif_stop_subqueue(dev, txq);
/* We won't see all dropped packets individually, so overrun
* error is more appropriate. */
@@ -495,6 +598,7 @@ static const struct net_device_ops tun_netdev_ops = {
.ndo_start_xmit = tun_net_xmit,
.ndo_change_mtu = tun_net_change_mtu,
.ndo_fix_features = tun_net_fix_features,
+ .ndo_select_queue = tun_select_queue,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = tun_poll_controller,
#endif
@@ -510,6 +614,7 @@ static const struct net_device_ops tap_netdev_ops = {
.ndo_set_rx_mode = tun_net_mclist,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
+ .ndo_select_queue = tun_select_queue,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = tun_poll_controller,
#endif
@@ -551,7 +656,7 @@ static void tun_net_init(struct net_device *dev)
/* Character device part */
/* Poll */
-static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
+static unsigned int tun_chr_poll(struct file *file, poll_table *wait)
{
struct tun_file *tfile = file->private_data;
struct tun_struct *tun = __tun_get(tfile);
@@ -998,7 +1103,7 @@ static ssize_t tun_do_read(struct tun_struct *tun,struct tun_file *tfile,
schedule();
continue;
}
- netif_wake_queue(tun->dev);
+ netif_wake_subqueue(tun->dev, tfile->queue_index);
ret = tun_put_user(tun, tfile, skb, iv, len);
kfree_skb(skb);
@@ -1159,6 +1264,9 @@ static int tun_flags(struct tun_struct *tun)
if (tun->flags & TUN_VNET_HDR)
flags |= IFF_VNET_HDR;
+ if (tun->flags & TUN_TAP_MQ)
+ flags |= IFF_MULTI_QUEUE;
+
return flags;
}
@@ -1250,8 +1358,9 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
if (*ifr->ifr_name)
name = ifr->ifr_name;
- dev = alloc_netdev(sizeof(struct tun_struct), name,
- tun_setup);
+ dev = alloc_netdev_mqs(sizeof(struct tun_struct), name,
+ tun_setup,
+ MAX_TAP_QUEUES, MAX_TAP_QUEUES);
if (!dev)
return -ENOMEM;
@@ -1286,7 +1395,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
err = tun_attach(tun, file);
if (err < 0)
- goto failed;
+ goto err_free_dev;
}
tun_debug(KERN_INFO, tun, "tun_set_iff\n");
@@ -1306,18 +1415,22 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
else
tun->flags &= ~TUN_VNET_HDR;
+ if (ifr->ifr_flags & IFF_MULTI_QUEUE)
+ tun->flags |= TUN_TAP_MQ;
+ else
+ tun->flags &= ~TUN_TAP_MQ;
+
/* Make sure persistent devices do not get stuck in
* xoff state.
*/
if (netif_running(tun->dev))
- netif_wake_queue(tun->dev);
+ netif_tx_wake_all_queues(tun->dev);
strcpy(ifr->ifr_name, tun->dev->name);
return 0;
err_free_dev:
free_netdev(dev);
- failed:
return err;
}
@@ -1372,6 +1485,51 @@ static int set_offload(struct tun_struct *tun, unsigned long arg)
return 0;
}
+static void tun_detach_filter(struct tun_struct *tun, int n)
+{
+ int i;
+ struct tun_file *tfile;
+
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ sk_detach_filter(tfile->socket.sk);
+ }
+
+ tun->filter_attached = false;
+}
+
+static int tun_attach_filter(struct tun_struct *tun)
+{
+ int i, ret = 0;
+ struct tun_file *tfile;
+
+ for (i = 0; i < tun->numqueues; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (ret) {
+ tun_detach_filter(tun, i);
+ return ret;
+ }
+ }
+
+ tun->filter_attached = true;
+ return ret;
+}
+
+static void tun_set_sndbuf(struct tun_struct *tun)
+{
+ struct tun_file *tfile;
+ int i;
+
+ for (i = 0; i < tun->numqueues; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ tfile->socket.sk->sk_sndbuf = tun->sndbuf;
+ }
+}
+
static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
unsigned long arg, int ifreq_len)
{
@@ -1400,6 +1558,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
(unsigned int __user*)argp);
}
+ ret = 0;
rtnl_lock();
tun = __tun_get(tfile);
@@ -1540,7 +1699,8 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
- tun->sndbuf = tfile->socket.sk->sk_sndbuf = sndbuf;
+ tun->sndbuf = sndbuf;
+ tun_set_sndbuf(tun);
break;
case TUNGETVNETHDRSZ:
@@ -1571,9 +1731,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if (copy_from_user(&tun->fprog, argp, sizeof(tun->fprog)))
break;
- ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
- if (!ret)
- tun->filter_attached = true;
+ ret = tun_attach_filter(tun);
break;
case TUNDETACHFILTER:
@@ -1581,9 +1739,8 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
ret = -EINVAL;
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
- ret = sk_detach_filter(tfile->socket.sk);
- if (!ret)
- tun->filter_attached = false;
+ ret = 0;
+ tun_detach_filter(tun, tun->numqueues);
break;
default:
@@ -1688,37 +1845,9 @@ static int tun_chr_open(struct inode *inode, struct file * file)
static int tun_chr_close(struct inode *inode, struct file *file)
{
struct tun_file *tfile = file->private_data;
- struct tun_struct *tun;
struct net *net = tfile->net;
- rtnl_lock();
-
- tun = rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held());
- if (tun) {
- struct net_device *dev = tun->dev;
-
- tun_debug(KERN_INFO, tun, "tun_chr_close\n");
-
- __tun_detach(tun);
-
- synchronize_net();
-
- /* If desirable, unregister the netdevice. */
- if (!(tun->flags & TUN_PERSIST)) {
- if (dev->reg_state == NETREG_REGISTERED)
- unregister_netdevice(dev);
- }
-
- /* drop the reference that netdevice holds */
- sock_put(&tfile->sk);
- }
-
- rtnl_unlock();
-
- /* drop the reference that file holds */
- BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
- &tfile->socket.flags));
- sk_release_kernel(&tfile->sk);
+ tun_detach(tfile, true);
put_net(net);
return 0;
--
1.7.1
^ permalink raw reply related
* [net-next v4 6/7] tuntap: add ioctl to attach or detach a file form tuntap device
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Sometimes usespace may need to active/deactive a queue, this could be done by
detaching and attaching a file from tuntap device.
This patch introduces a new ioctls - TUNSETQUEUE which could be used to do
this. Flag IFF_ATTACH_QUEUE were introduced to do attaching while
IFF_DETACH_QUEUE were introduced to do the detaching.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 56 ++++++++++++++++++++++++++++++++++++------
include/uapi/linux/if_tun.h | 3 ++
2 files changed, 51 insertions(+), 8 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 59235ba..78dcda8 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -195,6 +195,15 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
return txq;
}
+static inline bool tun_not_capable(struct tun_struct *tun)
+{
+ const struct cred *cred = current_cred();
+
+ return (((uid_valid(tun->owner) && !uid_eq(cred->euid, tun->owner)) ||
+ (gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
+ !capable(CAP_NET_ADMIN));
+}
+
static void tun_set_real_num_queues(struct tun_struct *tun)
{
netif_set_real_num_tx_queues(tun->dev, tun->numqueues);
@@ -1310,8 +1319,6 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
dev = __dev_get_by_name(net, ifr->ifr_name);
if (dev) {
- const struct cred *cred = current_cred();
-
if (ifr->ifr_flags & IFF_TUN_EXCL)
return -EBUSY;
if ((ifr->ifr_flags & IFF_TUN) && dev->netdev_ops == &tun_netdev_ops)
@@ -1321,9 +1328,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
else
return -EINVAL;
- if (((uid_valid(tun->owner) && !uid_eq(cred->euid, tun->owner)) ||
- (gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
- !capable(CAP_NET_ADMIN))
+ if (tun_not_capable(tun))
return -EPERM;
err = security_tun_dev_attach(tfile->socket.sk);
if (err < 0)
@@ -1530,6 +1535,40 @@ static void tun_set_sndbuf(struct tun_struct *tun)
}
}
+static int tun_set_queue(struct file *file, struct ifreq *ifr)
+{
+ struct tun_file *tfile = file->private_data;
+ struct tun_struct *tun;
+ struct net_device *dev;
+ int ret = 0;
+
+ rtnl_lock();
+
+ if (ifr->ifr_flags & IFF_ATTACH_QUEUE) {
+ dev = __dev_get_by_name(tfile->net, ifr->ifr_name);
+ if (!dev) {
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ tun = netdev_priv(dev);
+ if (dev->netdev_ops != &tap_netdev_ops &&
+ dev->netdev_ops != &tun_netdev_ops)
+ ret = -EINVAL;
+ else if (tun_not_capable(tun))
+ ret = -EPERM;
+ else
+ ret = tun_attach(tun, file);
+ } else if (ifr->ifr_flags & IFF_DETACH_QUEUE)
+ __tun_detach(tfile, false);
+ else
+ ret = -EINVAL;
+
+unlock:
+ rtnl_unlock();
+ return ret;
+}
+
static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
unsigned long arg, int ifreq_len)
{
@@ -1543,7 +1582,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
int vnet_hdr_sz;
int ret;
- if (cmd == TUNSETIFF || _IOC_TYPE(cmd) == 0x89) {
+ if (cmd == TUNSETIFF || cmd == TUNSETQUEUE || _IOC_TYPE(cmd) == 0x89) {
if (copy_from_user(&ifr, argp, ifreq_len))
return -EFAULT;
} else {
@@ -1554,9 +1593,10 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
* This is needed because we never checked for invalid flags on
* TUNSETIFF. */
return put_user(IFF_TUN | IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE |
- IFF_VNET_HDR,
+ IFF_VNET_HDR | IFF_MULTI_QUEUE,
(unsigned int __user*)argp);
- }
+ } else if (cmd == TUNSETQUEUE)
+ return tun_set_queue(file, &ifr);
ret = 0;
rtnl_lock();
diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h
index 8ef3a87..958497a 100644
--- a/include/uapi/linux/if_tun.h
+++ b/include/uapi/linux/if_tun.h
@@ -54,6 +54,7 @@
#define TUNDETACHFILTER _IOW('T', 214, struct sock_fprog)
#define TUNGETVNETHDRSZ _IOR('T', 215, int)
#define TUNSETVNETHDRSZ _IOW('T', 216, int)
+#define TUNSETQUEUE _IOW('T', 217, int)
/* TUNSETIFF ifr flags */
#define IFF_TUN 0x0001
@@ -63,6 +64,8 @@
#define IFF_VNET_HDR 0x4000
#define IFF_TUN_EXCL 0x8000
#define IFF_MULTI_QUEUE 0x0100
+#define IFF_ATTACH_QUEUE 0x0200
+#define IFF_DETACH_QUEUE 0x0400
/* Features for GSO (TUNSETOFFLOAD). */
#define TUN_F_CSUM 0x01 /* You can hand me unchecksummed packets. */
--
1.7.1
^ permalink raw reply related
* [net-next v4 7/7] tuntap: choose the txq based on rxq
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
This patch implements a simple multiqueue flow steering policy - tx follows rx
for tun/tap. The idea is simple, it just choose the txq based on which rxq it
comes. The flow were identified through the rxhash of a skb, and the hash to
queue mapping were recorded in a hlist with an ageing timer to retire the
mapping. The mapping were created when tun receives packet from userspace, and
was quired in .ndo_select_queue().
I run co-current TCP_CRR test and didn't see any mapping manipulation helpers in
perf top, so the overhead could be negelected.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 236 insertions(+), 3 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 78dcda8..158ef1d 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -115,6 +115,8 @@ struct tap_filter {
* to match a queue per guest CPU. */
#define MAX_TAP_QUEUES 1024
+#define TUN_FLOW_EXPIRE 3 * HZ
+
/* A tun_file connects an open character device to a tuntap netdevice. It
* also contains all socket related strctures (except sock_fprog and tap_filter)
* to serve as one transmit queue for tuntap device. The sock_fprog and
@@ -138,6 +140,18 @@ struct tun_file {
u16 queue_index;
};
+struct tun_flow_entry {
+ struct hlist_node hash_link;
+ struct rcu_head rcu;
+ struct tun_struct *tun;
+
+ u32 rxhash;
+ int queue_index;
+ unsigned long updated;
+};
+
+#define TUN_NUM_FLOW_ENTRIES 1024
+
/* Since the socket were moved to tun_file, to preserve the behavior of persist
* device, socket fileter, sndbuf and vnet header size were restore when the
* file were attached to a persist device.
@@ -163,8 +177,176 @@ struct tun_struct {
#ifdef TUN_DEBUG
int debug;
#endif
+ spinlock_t lock;
+ struct kmem_cache *flow_cache;
+ struct hlist_head flows[TUN_NUM_FLOW_ENTRIES];
+ struct timer_list flow_gc_timer;
+ unsigned long ageing_time;
};
+static inline u32 tun_hashfn(u32 rxhash)
+{
+ return rxhash & 0x3ff;
+}
+
+static struct tun_flow_entry *tun_flow_find(struct hlist_head *head, u32 rxhash)
+{
+ struct tun_flow_entry *e;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_rcu(e, n, head, hash_link) {
+ if (e->rxhash == rxhash)
+ return e;
+ }
+ return NULL;
+}
+
+static struct tun_flow_entry *tun_flow_create(struct tun_struct *tun,
+ struct hlist_head *head,
+ u32 rxhash, u16 queue_index)
+{
+ struct tun_flow_entry *e = kmem_cache_alloc(tun->flow_cache,
+ GFP_ATOMIC);
+ if (e) {
+ tun_debug(KERN_INFO, tun, "create flow: hash %u index %u\n",
+ rxhash, queue_index);
+ e->updated = jiffies;
+ e->rxhash = rxhash;
+ e->queue_index = queue_index;
+ e->tun = tun;
+ hlist_add_head_rcu(&e->hash_link, head);
+ }
+ return e;
+}
+
+static void tun_flow_free(struct rcu_head *head)
+{
+ struct tun_flow_entry *e
+ = container_of(head, struct tun_flow_entry, rcu);
+ kmem_cache_free(e->tun->flow_cache, e);
+}
+
+static void tun_flow_delete(struct tun_struct *tun, struct tun_flow_entry *e)
+{
+ tun_debug(KERN_INFO, tun, "delete flow: hash %u index %u\n",
+ e->rxhash, e->queue_index);
+ hlist_del_rcu(&e->hash_link);
+ call_rcu(&e->rcu, tun_flow_free);
+}
+
+/* caller hold tun->lock */
+static int tun_delete_by_rxhash(struct tun_struct *tun, u32 rxhash)
+{
+ struct hlist_head *head = &tun->flows[tun_hashfn(rxhash)];
+ struct tun_flow_entry *e = tun_flow_find(head, rxhash);
+
+ if (!e)
+ return -ENOENT;
+
+ tun_flow_delete(tun, e);
+}
+
+static void tun_flow_flush(struct tun_struct *tun)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link)
+ tun_flow_delete(tun, e);
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_delete_by_queue(struct tun_struct *tun, u16 queue_index)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ if (e->queue_index == queue_index)
+ tun_flow_delete(tun, e);
+ }
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_cleanup(unsigned long data)
+{
+ struct tun_struct *tun = (struct tun_struct *)data;
+ unsigned long delay = tun->ageing_time;
+ unsigned long next_timer = jiffies + delay;
+ unsigned long count = 0;
+ int i;
+
+ tun_debug(KERN_INFO, tun, "tun_flow_cleanup\n");
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ unsigned long this_timer;
+ count ++;
+ this_timer = e->updated + delay;
+ if (time_before_eq(this_timer, jiffies))
+ tun_flow_delete(tun, e);
+ else if (time_before(this_timer, next_timer))
+ next_timer = this_timer;
+ }
+ }
+
+ if (count)
+ mod_timer(&tun->flow_gc_timer, round_jiffies_up(next_timer));
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_update(struct tun_struct *tun, struct sk_buff *skb,
+ u16 queue_index)
+{
+ struct hlist_head *head;
+ struct tun_flow_entry *e;
+ unsigned long delay = tun->ageing_time;
+ u32 rxhash = skb_get_rxhash(skb);
+
+ if (!rxhash)
+ return;
+ else
+ head = &tun->flows[tun_hashfn(rxhash)];
+
+ rcu_read_lock();
+
+ if (tun->numqueues == 1)
+ goto unlock;
+
+ e = tun_flow_find(head, rxhash);
+ if (likely(e)) {
+ /* TODO: keep queueing to old queue until it's empty? */
+ e->queue_index = queue_index;
+ e->updated = jiffies;
+ } else {
+ spin_lock_bh(&tun->lock);
+ if (!tun_flow_find(head, rxhash))
+ tun_flow_create(tun, head, rxhash, queue_index);
+
+ if (!timer_pending(&tun->flow_gc_timer))
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + delay));
+ spin_unlock_bh(&tun->lock);
+ }
+
+unlock:
+ rcu_read_unlock();
+}
+
/* We try to identify a flow through its rxhash first. The reason that
* we do not check rxq no. is becuase some cards(e.g 82599), chooses
* the rxq based on the txq where the last packet of the flow comes. As
@@ -175,6 +357,7 @@ struct tun_struct {
static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
{
struct tun_struct *tun = netdev_priv(dev);
+ struct tun_flow_entry *e;
u32 txq = 0;
u32 numqueues = 0;
@@ -183,8 +366,12 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
txq = skb_get_rxhash(skb);
if (txq) {
- /* use multiply and shift instead of expensive divide */
- txq = ((u64)txq * numqueues) >> 32;
+ e = tun_flow_find(&tun->flows[tun_hashfn(txq)], txq);
+ if (e)
+ txq = e->queue_index;
+ else
+ /* use multiply and shift instead of expensive divide */
+ txq = ((u64)txq * numqueues) >> 32;
} else if (likely(skb_rx_queue_recorded(skb))) {
txq = skb_get_rx_queue(skb);
while (unlikely(txq >= numqueues))
@@ -234,6 +421,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
sock_put(&tfile->sk);
synchronize_net();
+ tun_flow_delete_by_queue(tun, tun->numqueues + 1);
/* Drop read queue */
skb_queue_purge(&tfile->sk.sk_receive_queue);
tun_set_real_num_queues(tun);
@@ -629,6 +817,37 @@ static const struct net_device_ops tap_netdev_ops = {
#endif
};
+static int tun_flow_init(struct tun_struct *tun)
+{
+ int i;
+
+ tun->flow_cache = kmem_cache_create("tun_flow_cache",
+ sizeof(struct tun_flow_entry), 0, 0,
+ NULL);
+ if (!tun->flow_cache)
+ return -ENOMEM;
+
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++)
+ INIT_HLIST_HEAD(&tun->flows[i]);
+
+ tun->ageing_time = TUN_FLOW_EXPIRE;
+ setup_timer(&tun->flow_gc_timer, tun_flow_cleanup, (unsigned long)tun);
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + tun->ageing_time));
+
+ return 0;
+}
+
+static void tun_flow_uninit(struct tun_struct *tun)
+{
+ del_timer_sync(&tun->flow_gc_timer);
+ tun_flow_flush(tun);
+
+ /* Wait for completion of call_rcu()'s */
+ rcu_barrier();
+ kmem_cache_destroy(tun->flow_cache);
+}
+
/* Initialize net device. */
static void tun_net_init(struct net_device *dev)
{
@@ -973,6 +1192,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
tun->dev->stats.rx_packets++;
tun->dev->stats.rx_bytes += len;
+ tun_flow_update(tun, skb, tfile->queue_index);
return total_len;
}
@@ -1150,6 +1370,14 @@ out:
return ret;
}
+static void tun_free_netdev(struct net_device *dev)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+
+ tun_flow_uninit(tun);
+ free_netdev(dev);
+}
+
static void tun_setup(struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
@@ -1158,7 +1386,7 @@ static void tun_setup(struct net_device *dev)
tun->group = INVALID_GID;
dev->ethtool_ops = &tun_ethtool_ops;
- dev->destructor = free_netdev;
+ dev->destructor = tun_free_netdev;
}
/* Trivial set of netlink ops to allow deleting tun or tap
@@ -1381,10 +1609,15 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
tun->filter_attached = false;
tun->sndbuf = tfile->socket.sk->sk_sndbuf;
+ spin_lock_init(&tun->lock);
+
security_tun_dev_post_create(&tfile->sk);
tun_net_init(dev);
+ if (tun_flow_init(tun))
+ goto err_free_dev;
+
dev->hw_features = NETIF_F_SG | NETIF_F_FRAGLIST |
TUN_USER_FEATURES;
dev->features = dev->hw_features;
--
1.7.1
^ permalink raw reply related
* [net-next v4 0/7] Multiqueue support in tuntap
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
Hello All:
This is an update of multiqueue support in tuntap from V3. Please consider to
merge.
The main idea for this series is to let tun/tap device to be benefited from
multiqueue network cards and multi-core host. We used to have a single queue for
tuntap which could be a bottleneck in a multiqueue/core environment. So this
series let the device could be attched with multiple sockets and expose them
through fd to the userspace as multiqueues. The sereis were orignally designed
to serve as backend for multiqueue virtio-net in KVM, but the design is generic
for other application to be used.
Some quick overview of the design:
- Moving socket from tun_device to tun_file.
- Allowing multiple sockets to be attached to a tun/tap devices.
- Using RCU to synchronize the data path and system call.
- Two new ioctls were added for the usespace to attach and detach socket to the device.
- API compatibility were maintained without userspace notable changes, so legacy
userspace that only use one queue won't need any changes.
- A flow(rxhash) to queue table were maintained by tuntap which choose the txq
based on the last rxq where it comes.
Performance test:
Pktgen is used to generate the traffic and a simple program that only does the
receiving in userspace.
#q #thread kpps aggregate kpps
1q 1thread 818kpps +0%
2q 2threads 1926kpps +135%
3q 3threads 2642kpps +223%
4q 4threads 3536kpps +332%
Changes from V3:
- Rebase to net-next
- A separate RCUiying patch to simply the reviewing
- Add a simple "tx follows rx" policy when choosing txq
- Various bug fixes
Changes from V2:
- Rebase to the latest net-next
- Fix netdev leak when tun_attach fails
- Fix return value of TUNSETOWNER
- Purge the receive queue in socket destructor
- Enable multiqueue tun (V1 and V2 only allows mq to be eanbled for tap
- Add per-queue u64 statistics
- Fix wrong BUG_ON() check in tun_detach()
- Check numqueues instead of tfile[0] in tun_set_iff() to let tunctl -d works
correctly
- Set numqueues to MAX_TAP_QUEUES during tun_detach_all() to prevent the
attaching.
Changes from V1:
- Simplify the sockets array management by not leaving NULL in the slot.
- Optimization on the tx queue selecting.
- Fix the bug in tun_deatch_all()
Reference:
- V3 https://lkml.org/lkml/2012/6/25/191
- V2 http://lwn.net/Articles/459270/
- V1 http://www.mail-archive.com/kvm@vger.kernel.org/msg59479.html
Jason Wang (7):
tuntap: log the unsigned informaiton with %u
tuntap: move socket to tun_file
tuntap: RCUify dereferencing between tun_struct and tun_file
tuntap: introduce multiqueue flags
tuntap: multiqueue support
tuntap: add ioctl to attach or detach a file form tuntap device
tuntap: choose the txq based on rxq
drivers/net/tun.c | 845 ++++++++++++++++++++++++++++++++-----------
include/uapi/linux/if_tun.h | 5 +
2 files changed, 640 insertions(+), 210 deletions(-)
^ permalink raw reply
* [net-next v4 1/7] tuntap: log the unsigned informaiton with %u
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 0873cdc..ef13cf0 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1422,7 +1422,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if (!tun)
goto unlock;
- tun_debug(KERN_INFO, tun, "tun_chr_ioctl cmd %d\n", cmd);
+ tun_debug(KERN_INFO, tun, "tun_chr_ioctl cmd %u\n", cmd);
ret = 0;
switch (cmd) {
@@ -1462,7 +1462,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
tun->owner = owner;
- tun_debug(KERN_INFO, tun, "owner set to %d\n",
+ tun_debug(KERN_INFO, tun, "owner set to %u\n",
from_kuid(&init_user_ns, tun->owner));
break;
@@ -1474,7 +1474,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
tun->group = group;
- tun_debug(KERN_INFO, tun, "group set to %d\n",
+ tun_debug(KERN_INFO, tun, "group set to %u\n",
from_kgid(&init_user_ns, tun->group));
break;
--
1.7.1
^ permalink raw reply related
* [net-next v4 2/7] tuntap: move socket to tun_file
From: Jason Wang @ 2012-10-29 5:35 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Current tuntap makes use of the socket receive queue as its tx queue. To
implement multiple tx queues for tuntap and enable the ability of adding and
removing queues during workload, the first step is to move the socket related
structures to tun_file. Then we could let multiple fds/sockets to be attached to
the tuntap.
This patch removes tun_sock and moves socket related structures from tun_sock or
tun_struct to tun_file. Two exceptions are tap_filter and sock_fprog, they are
still kept in tun_structure since they are used to filter packets for the net
device instead of per transmit queue (at least I see no requirements for
them). After those changes, socket were created and destroyed during file open
and close (instead of device creation and destroy), the socket structures could
be dereferenced from tun_file instead of the file of tun_struct structure
itself.
For persisent device, since we purge during datching and wouldn't queue any
packets when no interface were attached, there's no behaviod changes before and
after this patch, so the changes were transparent to the userspace. To keep the
attributes such as sndbuf, socket filter and vnet header, those would be
re-initialize after a new interface were attached to an persist device.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 266 +++++++++++++++++++++++++++++------------------------
1 files changed, 145 insertions(+), 121 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index ef13cf0..e8cedb0 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -110,14 +110,29 @@ struct tap_filter {
unsigned char addr[FLT_EXACT_COUNT][ETH_ALEN];
};
+/* A tun_file connects an open character device to a tuntap netdevice. It
+ * also contains all socket related strctures (except sock_fprog and tap_filter)
+ * to serve as one transmit queue for tuntap device. The sock_fprog and
+ * tap_filter were kept in tun_struct since they were used for filtering for the
+ * netdevice not for a specific queue (at least I didn't see the reqirement for
+ * this).
+ */
struct tun_file {
+ struct sock sk;
+ struct socket socket;
+ struct socket_wq wq;
atomic_t count;
struct tun_struct *tun;
struct net *net;
+ struct fasync_struct *fasync;
+ /* only used for fasnyc */
+ unsigned int flags;
};
-struct tun_sock;
-
+/* Since the socket were moved to tun_file, to preserve the behavior of persist
+ * device, socket fileter, sndbuf and vnet header size were restore when the
+ * file were attached to a persist device.
+ */
struct tun_struct {
struct tun_file *tfile;
unsigned int flags;
@@ -128,29 +143,18 @@ struct tun_struct {
netdev_features_t set_features;
#define TUN_USER_FEATURES (NETIF_F_HW_CSUM|NETIF_F_TSO_ECN|NETIF_F_TSO| \
NETIF_F_TSO6|NETIF_F_UFO)
- struct fasync_struct *fasync;
-
- struct tap_filter txflt;
- struct socket socket;
- struct socket_wq wq;
int vnet_hdr_sz;
-
+ int sndbuf;
+ struct tap_filter txflt;
+ struct sock_fprog fprog;
+ /* protected by rtnl lock */
+ bool filter_attached;
#ifdef TUN_DEBUG
int debug;
#endif
};
-struct tun_sock {
- struct sock sk;
- struct tun_struct *tun;
-};
-
-static inline struct tun_sock *tun_sk(struct sock *sk)
-{
- return container_of(sk, struct tun_sock, sk);
-}
-
static int tun_attach(struct tun_struct *tun, struct file *file)
{
struct tun_file *tfile = file->private_data;
@@ -169,12 +173,19 @@ static int tun_attach(struct tun_struct *tun, struct file *file)
goto out;
err = 0;
+
+ /* Re-attach filter when attaching to a persist device */
+ if (tun->filter_attached == true) {
+ err = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (!err)
+ goto out;
+ }
tfile->tun = tun;
+ tfile->socket.sk->sk_sndbuf = tun->sndbuf;
tun->tfile = tfile;
- tun->socket.file = file;
netif_carrier_on(tun->dev);
dev_hold(tun->dev);
- sock_hold(tun->socket.sk);
+ sock_hold(&tfile->sk);
atomic_inc(&tfile->count);
out:
@@ -184,14 +195,16 @@ out:
static void __tun_detach(struct tun_struct *tun)
{
+ struct tun_file *tfile = tun->tfile;
/* Detach from net device */
netif_tx_lock_bh(tun->dev);
netif_carrier_off(tun->dev);
tun->tfile = NULL;
+ tfile->tun = NULL;
netif_tx_unlock_bh(tun->dev);
/* Drop read queue */
- skb_queue_purge(&tun->socket.sk->sk_receive_queue);
+ skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
/* Drop the extra count on the net device */
dev_put(tun->dev);
@@ -350,21 +363,12 @@ static void tun_net_uninit(struct net_device *dev)
/* Inform the methods they need to stop using the dev.
*/
if (tfile) {
- wake_up_all(&tun->wq.wait);
+ wake_up_all(&tfile->wq.wait);
if (atomic_dec_and_test(&tfile->count))
__tun_detach(tun);
}
}
-static void tun_free_netdev(struct net_device *dev)
-{
- struct tun_struct *tun = netdev_priv(dev);
-
- BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED, &tun->socket.flags));
-
- sk_release_kernel(tun->socket.sk);
-}
-
/* Net device open. */
static int tun_net_open(struct net_device *dev)
{
@@ -383,11 +387,12 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
+ struct tun_file *tfile = tun->tfile;
tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
/* Drop packet if interface is not attached */
- if (!tun->tfile)
+ if (!tfile)
goto drop;
/* Drop if the filter does not like it.
@@ -396,11 +401,12 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
if (!check_filter(&tun->txflt, skb))
goto drop;
- if (tun->socket.sk->sk_filter &&
- sk_filter(tun->socket.sk, skb))
+ if (tfile->socket.sk->sk_filter &&
+ sk_filter(tfile->socket.sk, skb))
goto drop;
- if (skb_queue_len(&tun->socket.sk->sk_receive_queue) >= dev->tx_queue_len) {
+ if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
+ >= dev->tx_queue_len) {
if (!(tun->flags & TUN_ONE_QUEUE)) {
/* Normal queueing mode. */
/* Packet scheduler handles dropping of further packets. */
@@ -423,12 +429,12 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
skb_orphan(skb);
/* Enqueue packet */
- skb_queue_tail(&tun->socket.sk->sk_receive_queue, skb);
+ skb_queue_tail(&tfile->socket.sk->sk_receive_queue, skb);
/* Notify and wake up reader process */
- if (tun->flags & TUN_FASYNC)
- kill_fasync(&tun->fasync, SIGIO, POLL_IN);
- wake_up_interruptible_poll(&tun->wq.wait, POLLIN |
+ if (tfile->flags & TUN_FASYNC)
+ kill_fasync(&tfile->fasync, SIGIO, POLL_IN);
+ wake_up_interruptible_poll(&tfile->wq.wait, POLLIN |
POLLRDNORM | POLLRDBAND);
return NETDEV_TX_OK;
@@ -556,11 +562,11 @@ static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
if (!tun)
return POLLERR;
- sk = tun->socket.sk;
+ sk = tfile->socket.sk;
tun_debug(KERN_INFO, tun, "tun_chr_poll\n");
- poll_wait(file, &tun->wq.wait, wait);
+ poll_wait(file, &tfile->wq.wait, wait);
if (!skb_queue_empty(&sk->sk_receive_queue))
mask |= POLLIN | POLLRDNORM;
@@ -579,11 +585,11 @@ static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
/* prepad is the amount to reserve at front. len is length after that.
* linear is a hint as to how much to copy (usually headers). */
-static struct sk_buff *tun_alloc_skb(struct tun_struct *tun,
+static struct sk_buff *tun_alloc_skb(struct tun_file *tfile,
size_t prepad, size_t len,
size_t linear, int noblock)
{
- struct sock *sk = tun->socket.sk;
+ struct sock *sk = tfile->socket.sk;
struct sk_buff *skb;
int err;
@@ -685,9 +691,9 @@ static int zerocopy_sg_from_iovec(struct sk_buff *skb, const struct iovec *from,
}
/* Get packet from user space buffer */
-static ssize_t tun_get_user(struct tun_struct *tun, void *msg_control,
- const struct iovec *iv, size_t total_len,
- size_t count, int noblock)
+static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
+ void *msg_control, const struct iovec *iv,
+ size_t total_len, size_t count, int noblock)
{
struct tun_pi pi = { 0, cpu_to_be16(ETH_P_IP) };
struct sk_buff *skb;
@@ -757,7 +763,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, void *msg_control,
} else
copylen = len;
- skb = tun_alloc_skb(tun, align, copylen, gso.hdr_len, noblock);
+ skb = tun_alloc_skb(tfile, align, copylen, gso.hdr_len, noblock);
if (IS_ERR(skb)) {
if (PTR_ERR(skb) != -EAGAIN)
tun->dev->stats.rx_dropped++;
@@ -862,6 +868,7 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
{
struct file *file = iocb->ki_filp;
struct tun_struct *tun = tun_get(file);
+ struct tun_file *tfile = file->private_data;
ssize_t result;
if (!tun)
@@ -869,8 +876,8 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
tun_debug(KERN_INFO, tun, "tun_chr_write %ld\n", count);
- result = tun_get_user(tun, NULL, iv, iov_length(iv, count), count,
- file->f_flags & O_NONBLOCK);
+ result = tun_get_user(tun, tfile, NULL, iv, iov_length(iv, count),
+ count, file->f_flags & O_NONBLOCK);
tun_put(tun);
return result;
@@ -878,6 +885,7 @@ static ssize_t tun_chr_aio_write(struct kiocb *iocb, const struct iovec *iv,
/* Put packet to the user space buffer */
static ssize_t tun_put_user(struct tun_struct *tun,
+ struct tun_file *tfile,
struct sk_buff *skb,
const struct iovec *iv, int len)
{
@@ -957,7 +965,7 @@ static ssize_t tun_put_user(struct tun_struct *tun,
return total;
}
-static ssize_t tun_do_read(struct tun_struct *tun,
+static ssize_t tun_do_read(struct tun_struct *tun,struct tun_file *tfile,
struct kiocb *iocb, const struct iovec *iv,
ssize_t len, int noblock)
{
@@ -968,12 +976,12 @@ static ssize_t tun_do_read(struct tun_struct *tun,
tun_debug(KERN_INFO, tun, "tun_chr_read\n");
if (unlikely(!noblock))
- add_wait_queue(&tun->wq.wait, &wait);
+ add_wait_queue(&tfile->wq.wait, &wait);
while (len) {
current->state = TASK_INTERRUPTIBLE;
/* Read frames from the queue */
- if (!(skb=skb_dequeue(&tun->socket.sk->sk_receive_queue))) {
+ if (!(skb = skb_dequeue(&tfile->socket.sk->sk_receive_queue))) {
if (noblock) {
ret = -EAGAIN;
break;
@@ -993,14 +1001,14 @@ static ssize_t tun_do_read(struct tun_struct *tun,
}
netif_wake_queue(tun->dev);
- ret = tun_put_user(tun, skb, iv, len);
+ ret = tun_put_user(tun, tfile, skb, iv, len);
kfree_skb(skb);
break;
}
current->state = TASK_RUNNING;
if (unlikely(!noblock))
- remove_wait_queue(&tun->wq.wait, &wait);
+ remove_wait_queue(&tfile->wq.wait, &wait);
return ret;
}
@@ -1021,7 +1029,8 @@ static ssize_t tun_chr_aio_read(struct kiocb *iocb, const struct iovec *iv,
goto out;
}
- ret = tun_do_read(tun, iocb, iv, len, file->f_flags & O_NONBLOCK);
+ ret = tun_do_read(tun, tfile, iocb, iv, len,
+ file->f_flags & O_NONBLOCK);
ret = min_t(ssize_t, ret, len);
out:
tun_put(tun);
@@ -1036,7 +1045,7 @@ static void tun_setup(struct net_device *dev)
tun->group = INVALID_GID;
dev->ethtool_ops = &tun_ethtool_ops;
- dev->destructor = tun_free_netdev;
+ dev->destructor = free_netdev;
}
/* Trivial set of netlink ops to allow deleting tun or tap
@@ -1056,7 +1065,7 @@ static struct rtnl_link_ops tun_link_ops __read_mostly = {
static void tun_sock_write_space(struct sock *sk)
{
- struct tun_struct *tun;
+ struct tun_file *tfile;
wait_queue_head_t *wqueue;
if (!sock_writeable(sk))
@@ -1070,37 +1079,47 @@ static void tun_sock_write_space(struct sock *sk)
wake_up_interruptible_sync_poll(wqueue, POLLOUT |
POLLWRNORM | POLLWRBAND);
- tun = tun_sk(sk)->tun;
- kill_fasync(&tun->fasync, SIGIO, POLL_OUT);
-}
-
-static void tun_sock_destruct(struct sock *sk)
-{
- free_netdev(tun_sk(sk)->tun->dev);
+ tfile = container_of(sk, struct tun_file, sk);
+ kill_fasync(&tfile->fasync, SIGIO, POLL_OUT);
}
static int tun_sendmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *m, size_t total_len)
{
- struct tun_struct *tun = container_of(sock, struct tun_struct, socket);
- return tun_get_user(tun, m->msg_control, m->msg_iov, total_len,
- m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
+ int ret;
+ struct tun_file *tfile = container_of(sock, struct tun_file, socket);
+ struct tun_struct *tun = __tun_get(tfile);
+
+ if (!tun)
+ return -EBADFD;
+
+ ret = tun_get_user(tun, tfile, m->msg_control, m->msg_iov, total_len,
+ m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
+ tun_put(tun);
+ return ret;
}
+
static int tun_recvmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *m, size_t total_len,
int flags)
{
- struct tun_struct *tun = container_of(sock, struct tun_struct, socket);
+ struct tun_file *tfile = container_of(sock, struct tun_file, socket);
+ struct tun_struct *tun = __tun_get(tfile);
int ret;
+
+ if (!tun)
+ return -EBADFD;
+
if (flags & ~(MSG_DONTWAIT|MSG_TRUNC))
return -EINVAL;
- ret = tun_do_read(tun, iocb, m->msg_iov, total_len,
+ ret = tun_do_read(tun, tfile, iocb, m->msg_iov, total_len,
flags & MSG_DONTWAIT);
if (ret > total_len) {
m->msg_flags |= MSG_TRUNC;
ret = flags & MSG_TRUNC ? ret : total_len;
}
+ tun_put(tun);
return ret;
}
@@ -1121,7 +1140,7 @@ static const struct proto_ops tun_socket_ops = {
static struct proto tun_proto = {
.name = "tun",
.owner = THIS_MODULE,
- .obj_size = sizeof(struct tun_sock),
+ .obj_size = sizeof(struct tun_file),
};
static int tun_flags(struct tun_struct *tun)
@@ -1178,8 +1197,8 @@ static DEVICE_ATTR(group, 0444, tun_show_group, NULL);
static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
{
- struct sock *sk;
struct tun_struct *tun;
+ struct tun_file *tfile = file->private_data;
struct net_device *dev;
int err;
@@ -1200,7 +1219,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
(gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
!capable(CAP_NET_ADMIN))
return -EPERM;
- err = security_tun_dev_attach(tun->socket.sk);
+ err = security_tun_dev_attach(tfile->socket.sk);
if (err < 0)
return err;
@@ -1246,25 +1265,11 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
tun->flags = flags;
tun->txflt.count = 0;
tun->vnet_hdr_sz = sizeof(struct virtio_net_hdr);
- set_bit(SOCK_EXTERNALLY_ALLOCATED, &tun->socket.flags);
-
- err = -ENOMEM;
- sk = sk_alloc(&init_net, AF_UNSPEC, GFP_KERNEL, &tun_proto);
- if (!sk)
- goto err_free_dev;
- sk_change_net(sk, net);
- tun->socket.wq = &tun->wq;
- init_waitqueue_head(&tun->wq.wait);
- tun->socket.ops = &tun_socket_ops;
- sock_init_data(&tun->socket, sk);
- sk->sk_write_space = tun_sock_write_space;
- sk->sk_sndbuf = INT_MAX;
- sock_set_flag(sk, SOCK_ZEROCOPY);
+ tun->filter_attached = false;
+ tun->sndbuf = tfile->socket.sk->sk_sndbuf;
- tun_sk(sk)->tun = tun;
-
- security_tun_dev_post_create(sk);
+ security_tun_dev_post_create(&tfile->sk);
tun_net_init(dev);
@@ -1274,15 +1279,13 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
err = register_netdevice(tun->dev);
if (err < 0)
- goto err_free_sk;
+ goto err_free_dev;
if (device_create_file(&tun->dev->dev, &dev_attr_tun_flags) ||
device_create_file(&tun->dev->dev, &dev_attr_owner) ||
device_create_file(&tun->dev->dev, &dev_attr_group))
pr_err("Failed to create tun sysfs files\n");
- sk->sk_destruct = tun_sock_destruct;
-
err = tun_attach(tun, file);
if (err < 0)
goto failed;
@@ -1314,8 +1317,6 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
strcpy(ifr->ifr_name, tun->dev->name);
return 0;
- err_free_sk:
- tun_free_netdev(dev);
err_free_dev:
free_netdev(dev);
failed:
@@ -1379,7 +1380,6 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
struct tun_file *tfile = file->private_data;
struct tun_struct *tun;
void __user* argp = (void __user*)arg;
- struct sock_fprog fprog;
struct ifreq ifr;
kuid_t owner;
kgid_t group;
@@ -1444,11 +1444,16 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
case TUNSETPERSIST:
- /* Disable/Enable persist mode */
- if (arg)
+ /* Disable/Enable persist mode. Keep an extra reference to the
+ * module to prevent the module being unprobed.
+ */
+ if (arg) {
tun->flags |= TUN_PERSIST;
- else
+ __module_get(THIS_MODULE);
+ } else {
tun->flags &= ~TUN_PERSIST;
+ module_put(THIS_MODULE);
+ }
tun_debug(KERN_INFO, tun, "persist %s\n",
arg ? "enabled" : "disabled");
@@ -1526,7 +1531,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
case TUNGETSNDBUF:
- sndbuf = tun->socket.sk->sk_sndbuf;
+ sndbuf = tfile->socket.sk->sk_sndbuf;
if (copy_to_user(argp, &sndbuf, sizeof(sndbuf)))
ret = -EFAULT;
break;
@@ -1537,7 +1542,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
- tun->socket.sk->sk_sndbuf = sndbuf;
+ tun->sndbuf = tfile->socket.sk->sk_sndbuf = sndbuf;
break;
case TUNGETVNETHDRSZ:
@@ -1565,10 +1570,12 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
ret = -EFAULT;
- if (copy_from_user(&fprog, argp, sizeof(fprog)))
+ if (copy_from_user(&tun->fprog, argp, sizeof(tun->fprog)))
break;
- ret = sk_attach_filter(&fprog, tun->socket.sk);
+ ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (!ret)
+ tun->filter_attached = true;
break;
case TUNDETACHFILTER:
@@ -1576,7 +1583,9 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
ret = -EINVAL;
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
- ret = sk_detach_filter(tun->socket.sk);
+ ret = sk_detach_filter(tfile->socket.sk);
+ if (!ret)
+ tun->filter_attached = false;
break;
default:
@@ -1628,27 +1637,21 @@ static long tun_chr_compat_ioctl(struct file *file,
static int tun_chr_fasync(int fd, struct file *file, int on)
{
- struct tun_struct *tun = tun_get(file);
+ struct tun_file *tfile = file->private_data;
int ret;
- if (!tun)
- return -EBADFD;
-
- tun_debug(KERN_INFO, tun, "tun_chr_fasync %d\n", on);
-
- if ((ret = fasync_helper(fd, file, on, &tun->fasync)) < 0)
+ if ((ret = fasync_helper(fd, file, on, &tfile->fasync)) < 0)
goto out;
if (on) {
ret = __f_setown(file, task_pid(current), PIDTYPE_PID, 0);
if (ret)
goto out;
- tun->flags |= TUN_FASYNC;
+ tfile->flags |= TUN_FASYNC;
} else
- tun->flags &= ~TUN_FASYNC;
+ tfile->flags &= ~TUN_FASYNC;
ret = 0;
out:
- tun_put(tun);
return ret;
}
@@ -1658,13 +1661,30 @@ static int tun_chr_open(struct inode *inode, struct file * file)
DBG1(KERN_INFO, "tunX: tun_chr_open\n");
- tfile = kmalloc(sizeof(*tfile), GFP_KERNEL);
+ tfile = (struct tun_file *)sk_alloc(&init_net, AF_UNSPEC, GFP_KERNEL,
+ &tun_proto);
if (!tfile)
return -ENOMEM;
atomic_set(&tfile->count, 0);
tfile->tun = NULL;
tfile->net = get_net(current->nsproxy->net_ns);
+ tfile->flags = 0;
+
+ rcu_assign_pointer(tfile->socket.wq, &tfile->wq);
+ init_waitqueue_head(&tfile->wq.wait);
+
+ tfile->socket.file = file;
+ tfile->socket.ops = &tun_socket_ops;
+
+ sock_init_data(&tfile->socket, &tfile->sk);
+ sk_change_net(&tfile->sk, tfile->net);
+
+ tfile->sk.sk_write_space = tun_sock_write_space;
+ tfile->sk.sk_sndbuf = INT_MAX;
+
file->private_data = tfile;
+ set_bit(SOCK_EXTERNALLY_ALLOCATED, &tfile->socket.flags);
+
return 0;
}
@@ -1672,6 +1692,7 @@ static int tun_chr_close(struct inode *inode, struct file *file)
{
struct tun_file *tfile = file->private_data;
struct tun_struct *tun;
+ struct net *net = tfile->net;
tun = __tun_get(tfile);
if (tun) {
@@ -1688,14 +1709,16 @@ static int tun_chr_close(struct inode *inode, struct file *file)
unregister_netdevice(dev);
rtnl_unlock();
}
- }
- tun = tfile->tun;
- if (tun)
- sock_put(tun->socket.sk);
+ /* drop the reference that netdevice holds */
+ sock_put(&tfile->sk);
+ }
- put_net(tfile->net);
- kfree(tfile);
+ /* drop the reference that file holds */
+ BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
+ &tfile->socket.flags));
+ sk_release_kernel(&tfile->sk);
+ put_net(net);
return 0;
}
@@ -1823,13 +1846,14 @@ static void tun_cleanup(void)
struct socket *tun_get_socket(struct file *file)
{
struct tun_struct *tun;
+ struct tun_file *tfile = file->private_data;
if (file->f_op != &tun_fops)
return ERR_PTR(-EINVAL);
tun = tun_get(file);
if (!tun)
return ERR_PTR(-EBADFD);
tun_put(tun);
- return &tun->socket;
+ return &tfile->socket;
}
EXPORT_SYMBOL_GPL(tun_get_socket);
--
1.7.1
^ permalink raw reply related
* [net-next v4 3/7] tuntap: RCUify dereferencing between tun_struct and tun_file
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
RCU were introduced in this patch to synchronize the dereferences between
tun_struct and tun_file. All tun_{get|put} were replaced with RCU, the
dereference from one to other must be done under rtnl lock or rcu read critical
region.
This is needed for the following patches since the one of the goal of multiqueue
tuntap is to allow adding or removing queues during workload. Without RCU,
control path would hold tx locks when adding or removing queues (which may cause
sme delay) and it's hard to change the number of queues without stopping the net
device. With the help of rcu, there's also no need for tun_file hold an refcnt
to tun_struct.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 95 ++++++++++++++++++++++++++---------------------------
1 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index e8cedb0..d332cb8 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -116,13 +116,16 @@ struct tap_filter {
* tap_filter were kept in tun_struct since they were used for filtering for the
* netdevice not for a specific queue (at least I didn't see the reqirement for
* this).
+ *
+ * RCU usage:
+ * The tun_file and tun_struct are loosely coupled, the pointer from on to the
+ * other can only be read while rcu_read_lock or rtnl_lock is held.
*/
struct tun_file {
struct sock sk;
struct socket socket;
struct socket_wq wq;
- atomic_t count;
- struct tun_struct *tun;
+ struct tun_struct __rcu *tun;
struct net *net;
struct fasync_struct *fasync;
/* only used for fasnyc */
@@ -134,7 +137,7 @@ struct tun_file {
* file were attached to a persist device.
*/
struct tun_struct {
- struct tun_file *tfile;
+ struct tun_file __rcu *tfile;
unsigned int flags;
kuid_t owner;
kgid_t group;
@@ -180,13 +183,11 @@ static int tun_attach(struct tun_struct *tun, struct file *file)
if (!err)
goto out;
}
- tfile->tun = tun;
+ rcu_assign_pointer(tfile->tun, tun);
tfile->socket.sk->sk_sndbuf = tun->sndbuf;
- tun->tfile = tfile;
+ rcu_assign_pointer(tun->tfile, tfile);
netif_carrier_on(tun->dev);
- dev_hold(tun->dev);
sock_hold(&tfile->sk);
- atomic_inc(&tfile->count);
out:
netif_tx_unlock_bh(tun->dev);
@@ -195,34 +196,29 @@ out:
static void __tun_detach(struct tun_struct *tun)
{
- struct tun_file *tfile = tun->tfile;
+ struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
+ lockdep_rtnl_is_held());
/* Detach from net device */
- netif_tx_lock_bh(tun->dev);
netif_carrier_off(tun->dev);
- tun->tfile = NULL;
- tfile->tun = NULL;
- netif_tx_unlock_bh(tun->dev);
-
- /* Drop read queue */
- skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
-
- /* Drop the extra count on the net device */
- dev_put(tun->dev);
-}
+ rcu_assign_pointer(tun->tfile, NULL);
+ if (tfile) {
+ rcu_assign_pointer(tfile->tun, NULL);
-static void tun_detach(struct tun_struct *tun)
-{
- rtnl_lock();
- __tun_detach(tun);
- rtnl_unlock();
+ synchronize_net();
+ /* Drop read queue */
+ skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
+ }
}
static struct tun_struct *__tun_get(struct tun_file *tfile)
{
- struct tun_struct *tun = NULL;
+ struct tun_struct *tun;
- if (atomic_inc_not_zero(&tfile->count))
- tun = tfile->tun;
+ rcu_read_lock();
+ tun = rcu_dereference(tfile->tun);
+ if (tun)
+ dev_hold(tun->dev);
+ rcu_read_unlock();
return tun;
}
@@ -234,10 +230,7 @@ static struct tun_struct *tun_get(struct file *file)
static void tun_put(struct tun_struct *tun)
{
- struct tun_file *tfile = tun->tfile;
-
- if (atomic_dec_and_test(&tfile->count))
- tun_detach(tfile->tun);
+ dev_put(tun->dev);
}
/* TAP filtering */
@@ -358,14 +351,15 @@ static const struct ethtool_ops tun_ethtool_ops;
static void tun_net_uninit(struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = tun->tfile;
+ struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
+ lockdep_rtnl_is_held());
/* Inform the methods they need to stop using the dev.
*/
if (tfile) {
wake_up_all(&tfile->wq.wait);
- if (atomic_dec_and_test(&tfile->count))
- __tun_detach(tun);
+ __tun_detach(tun);
+ synchronize_net();
}
}
@@ -387,14 +381,16 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = tun->tfile;
-
- tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+ struct tun_file *tfile;
+ rcu_read_lock();
+ tfile = rcu_dereference(tun->tfile);
/* Drop packet if interface is not attached */
if (!tfile)
goto drop;
+ tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+
/* Drop if the filter does not like it.
* This is a noop if the filter is disabled.
* Filter can be enabled only for the TAP devices. */
@@ -436,11 +432,14 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
kill_fasync(&tfile->fasync, SIGIO, POLL_IN);
wake_up_interruptible_poll(&tfile->wq.wait, POLLIN |
POLLRDNORM | POLLRDBAND);
+
+ rcu_read_unlock();
return NETDEV_TX_OK;
drop:
dev->stats.tx_dropped++;
kfree_skb(skb);
+ rcu_read_unlock();
return NETDEV_TX_OK;
}
@@ -1092,7 +1091,6 @@ static int tun_sendmsg(struct kiocb *iocb, struct socket *sock,
if (!tun)
return -EBADFD;
-
ret = tun_get_user(tun, tfile, m->msg_control, m->msg_iov, total_len,
m->msg_iovlen, m->msg_flags & MSG_DONTWAIT);
tun_put(tun);
@@ -1665,8 +1663,7 @@ static int tun_chr_open(struct inode *inode, struct file * file)
&tun_proto);
if (!tfile)
return -ENOMEM;
- atomic_set(&tfile->count, 0);
- tfile->tun = NULL;
+ rcu_assign_pointer(tfile->tun, NULL);
tfile->net = get_net(current->nsproxy->net_ns);
tfile->flags = 0;
@@ -1694,7 +1691,9 @@ static int tun_chr_close(struct inode *inode, struct file *file)
struct tun_struct *tun;
struct net *net = tfile->net;
- tun = __tun_get(tfile);
+ rtnl_lock();
+
+ tun = rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held());
if (tun) {
struct net_device *dev = tun->dev;
@@ -1702,18 +1701,20 @@ static int tun_chr_close(struct inode *inode, struct file *file)
__tun_detach(tun);
+ synchronize_net();
+
/* If desirable, unregister the netdevice. */
if (!(tun->flags & TUN_PERSIST)) {
- rtnl_lock();
if (dev->reg_state == NETREG_REGISTERED)
unregister_netdevice(dev);
- rtnl_unlock();
}
/* drop the reference that netdevice holds */
sock_put(&tfile->sk);
}
+ rtnl_unlock();
+
/* drop the reference that file holds */
BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
&tfile->socket.flags));
@@ -1845,14 +1846,12 @@ static void tun_cleanup(void)
* holding a reference to the file for as long as the socket is in use. */
struct socket *tun_get_socket(struct file *file)
{
- struct tun_struct *tun;
- struct tun_file *tfile = file->private_data;
+ struct tun_file *tfile;
if (file->f_op != &tun_fops)
return ERR_PTR(-EINVAL);
- tun = tun_get(file);
- if (!tun)
+ tfile = file->private_data;
+ if (!tfile)
return ERR_PTR(-EBADFD);
- tun_put(tun);
return &tfile->socket;
}
EXPORT_SYMBOL_GPL(tun_get_socket);
--
1.7.1
^ permalink raw reply related
* [net-next v4 4/7] tuntap: introduce multiqueue flags
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Add flags to be used by creating multiqueue tuntap device.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
include/uapi/linux/if_tun.h | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h
index 25a585c..8ef3a87 100644
--- a/include/uapi/linux/if_tun.h
+++ b/include/uapi/linux/if_tun.h
@@ -34,6 +34,7 @@
#define TUN_ONE_QUEUE 0x0080
#define TUN_PERSIST 0x0100
#define TUN_VNET_HDR 0x0200
+#define TUN_TAP_MQ 0x0400
/* Ioctl defines */
#define TUNSETNOCSUM _IOW('T', 200, int)
@@ -61,6 +62,7 @@
#define IFF_ONE_QUEUE 0x2000
#define IFF_VNET_HDR 0x4000
#define IFF_TUN_EXCL 0x8000
+#define IFF_MULTI_QUEUE 0x0100
/* Features for GSO (TUNSETOFFLOAD). */
#define TUN_F_CSUM 0x01 /* You can hand me unchecksummed packets. */
--
1.7.1
^ permalink raw reply related
* [net-next v4 5/7] tuntap: multiqueue support
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
This patch converts tun/tap to a multiqueue devices and expose the multiqueue
queues as multiple file descriptors to userspace. Internally, each tun_file were
abstracted as a queue, and an array of pointers to tun_file structurs were
stored in tun_structure device, so multiple tun_files were allowed to be
attached to the device as multiple queues.
When choosing txq, we first try to identify a flow through its rxhash, if it
does not have such one, we could try recorded rxq and then use them to choose
the transmit queue. This policy may be changed in the future.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 305 +++++++++++++++++++++++++++++++++++++---------------
1 files changed, 217 insertions(+), 88 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index d332cb8..59235ba 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -110,6 +110,11 @@ struct tap_filter {
unsigned char addr[FLT_EXACT_COUNT][ETH_ALEN];
};
+/* 1024 is probably a high enough limit: modern hypervisors seem to support on
+ * the order of 100-200 CPUs so this leaves us some breathing space if we want
+ * to match a queue per guest CPU. */
+#define MAX_TAP_QUEUES 1024
+
/* A tun_file connects an open character device to a tuntap netdevice. It
* also contains all socket related strctures (except sock_fprog and tap_filter)
* to serve as one transmit queue for tuntap device. The sock_fprog and
@@ -130,6 +135,7 @@ struct tun_file {
struct fasync_struct *fasync;
/* only used for fasnyc */
unsigned int flags;
+ u16 queue_index;
};
/* Since the socket were moved to tun_file, to preserve the behavior of persist
@@ -137,7 +143,8 @@ struct tun_file {
* file were attached to a persist device.
*/
struct tun_struct {
- struct tun_file __rcu *tfile;
+ struct tun_file __rcu *tfiles[MAX_TAP_QUEUES];
+ unsigned int numqueues;
unsigned int flags;
kuid_t owner;
kgid_t group;
@@ -158,56 +165,156 @@ struct tun_struct {
#endif
};
+/* We try to identify a flow through its rxhash first. The reason that
+ * we do not check rxq no. is becuase some cards(e.g 82599), chooses
+ * the rxq based on the txq where the last packet of the flow comes. As
+ * the userspace application move between processors, we may get a
+ * different rxq no. here. If we could not get rxhash, then we would
+ * hope the rxq no. may help here.
+ */
+static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+ u32 txq = 0;
+ u32 numqueues = 0;
+
+ rcu_read_lock();
+ numqueues = tun->numqueues;
+
+ txq = skb_get_rxhash(skb);
+ if (txq) {
+ /* use multiply and shift instead of expensive divide */
+ txq = ((u64)txq * numqueues) >> 32;
+ } else if (likely(skb_rx_queue_recorded(skb))) {
+ txq = skb_get_rx_queue(skb);
+ while (unlikely(txq >= numqueues))
+ txq -= numqueues;
+ }
+
+ rcu_read_unlock();
+ return txq;
+}
+
+static void tun_set_real_num_queues(struct tun_struct *tun)
+{
+ netif_set_real_num_tx_queues(tun->dev, tun->numqueues);
+ netif_set_real_num_rx_queues(tun->dev, tun->numqueues);
+}
+
+static void __tun_detach(struct tun_file *tfile, bool clean)
+{
+ struct tun_file *ntfile;
+ struct tun_struct *tun;
+ struct net_device *dev;
+
+ tun = rcu_dereference_protected(tfile->tun,
+ lockdep_rtnl_is_held());
+ if (tun) {
+ u16 index = tfile->queue_index;
+ BUG_ON(index >= tun->numqueues);
+ dev = tun->dev;
+
+ rcu_assign_pointer(tun->tfiles[index],
+ tun->tfiles[tun->numqueues - 1]);
+ rcu_assign_pointer(tfile->tun, NULL);
+ ntfile = rcu_dereference_protected(tun->tfiles[index],
+ lockdep_rtnl_is_held());
+ ntfile->queue_index = index;
+
+ --tun->numqueues;
+ sock_put(&tfile->sk);
+
+ synchronize_net();
+ /* Drop read queue */
+ skb_queue_purge(&tfile->sk.sk_receive_queue);
+ tun_set_real_num_queues(tun);
+
+ if (tun->numqueues == 0 && !(tun->flags & TUN_PERSIST))
+ if (dev->reg_state == NETREG_REGISTERED)
+ unregister_netdevice(dev);
+ }
+
+ if (clean) {
+ BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
+ &tfile->socket.flags));
+ sk_release_kernel(&tfile->sk);
+ }
+}
+
+static void tun_detach(struct tun_file *tfile, bool clean)
+{
+ rtnl_lock();
+ __tun_detach(tfile, clean);
+ rtnl_unlock();
+}
+
+static void tun_detach_all(struct net_device *dev)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+ struct tun_file *tfile;
+ int i, n = tun->numqueues;
+
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ BUG_ON(!tfile);
+ wake_up_all(&tfile->wq.wait);
+ rcu_assign_pointer(tfile->tun, NULL);
+ --tun->numqueues;
+ }
+ BUG_ON(tun->numqueues != 0);
+
+ synchronize_net();
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ /* Drop read queue */
+ skb_queue_purge(&tfile->sk.sk_receive_queue);
+ sock_put(&tfile->sk);
+ }
+}
+
static int tun_attach(struct tun_struct *tun, struct file *file)
{
struct tun_file *tfile = file->private_data;
int err;
- ASSERT_RTNL();
-
- netif_tx_lock_bh(tun->dev);
-
err = -EINVAL;
- if (tfile->tun)
+ if (rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held()))
goto out;
err = -EBUSY;
- if (tun->tfile)
+ if (!(tun->flags & TUN_TAP_MQ) && tun->numqueues == 1)
+ goto out;
+
+ err = -E2BIG;
+ if (tun->numqueues == MAX_TAP_QUEUES)
goto out;
err = 0;
- /* Re-attach filter when attaching to a persist device */
+ /* Re-attach the filter to presist device */
if (tun->filter_attached == true) {
err = sk_attach_filter(&tun->fprog, tfile->socket.sk);
if (!err)
goto out;
}
+ tfile->queue_index = tun->numqueues;
rcu_assign_pointer(tfile->tun, tun);
- tfile->socket.sk->sk_sndbuf = tun->sndbuf;
- rcu_assign_pointer(tun->tfile, tfile);
- netif_carrier_on(tun->dev);
+ rcu_assign_pointer(tun->tfiles[tun->numqueues], tfile);
sock_hold(&tfile->sk);
+ tun->numqueues++;
-out:
- netif_tx_unlock_bh(tun->dev);
- return err;
-}
+ tun_set_real_num_queues(tun);
-static void __tun_detach(struct tun_struct *tun)
-{
- struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
- lockdep_rtnl_is_held());
- /* Detach from net device */
- netif_carrier_off(tun->dev);
- rcu_assign_pointer(tun->tfile, NULL);
- if (tfile) {
- rcu_assign_pointer(tfile->tun, NULL);
+ if (tun->numqueues == 1)
+ netif_carrier_on(tun->dev);
- synchronize_net();
- /* Drop read queue */
- skb_queue_purge(&tfile->socket.sk->sk_receive_queue);
- }
+ /* device is allowed to go away first, so no need to hold extra
+ * refcnt. */
+
+out:
+ return err;
}
static struct tun_struct *__tun_get(struct tun_file *tfile)
@@ -350,30 +457,20 @@ static const struct ethtool_ops tun_ethtool_ops;
/* Net device detach from fd. */
static void tun_net_uninit(struct net_device *dev)
{
- struct tun_struct *tun = netdev_priv(dev);
- struct tun_file *tfile = rcu_dereference_protected(tun->tfile,
- lockdep_rtnl_is_held());
-
- /* Inform the methods they need to stop using the dev.
- */
- if (tfile) {
- wake_up_all(&tfile->wq.wait);
- __tun_detach(tun);
- synchronize_net();
- }
+ tun_detach_all(dev);
}
/* Net device open. */
static int tun_net_open(struct net_device *dev)
{
- netif_start_queue(dev);
+ netif_tx_start_all_queues(dev);
return 0;
}
/* Net device close. */
static int tun_net_close(struct net_device *dev)
{
- netif_stop_queue(dev);
+ netif_tx_stop_all_queues(dev);
return 0;
}
@@ -381,16 +478,20 @@ static int tun_net_close(struct net_device *dev)
static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
+ int txq = skb->queue_mapping;
struct tun_file *tfile;
rcu_read_lock();
- tfile = rcu_dereference(tun->tfile);
+ tfile = rcu_dereference(tun->tfiles[txq]);
+
/* Drop packet if interface is not attached */
- if (!tfile)
+ if (txq >= tun->numqueues)
goto drop;
tun_debug(KERN_INFO, tun, "tun_net_xmit %d\n", skb->len);
+ BUG_ON(!tfile);
+
/* Drop if the filter does not like it.
* This is a noop if the filter is disabled.
* Filter can be enabled only for the TAP devices. */
@@ -401,12 +502,14 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
sk_filter(tfile->socket.sk, skb))
goto drop;
+ /* Limit the number of packets queued by divining txq length with the
+ * number of queues. */
if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
- >= dev->tx_queue_len) {
+ >= dev->tx_queue_len / tun->numqueues){
if (!(tun->flags & TUN_ONE_QUEUE)) {
/* Normal queueing mode. */
/* Packet scheduler handles dropping of further packets. */
- netif_stop_queue(dev);
+ netif_stop_subqueue(dev, txq);
/* We won't see all dropped packets individually, so overrun
* error is more appropriate. */
@@ -495,6 +598,7 @@ static const struct net_device_ops tun_netdev_ops = {
.ndo_start_xmit = tun_net_xmit,
.ndo_change_mtu = tun_net_change_mtu,
.ndo_fix_features = tun_net_fix_features,
+ .ndo_select_queue = tun_select_queue,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = tun_poll_controller,
#endif
@@ -510,6 +614,7 @@ static const struct net_device_ops tap_netdev_ops = {
.ndo_set_rx_mode = tun_net_mclist,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
+ .ndo_select_queue = tun_select_queue,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = tun_poll_controller,
#endif
@@ -551,7 +656,7 @@ static void tun_net_init(struct net_device *dev)
/* Character device part */
/* Poll */
-static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
+static unsigned int tun_chr_poll(struct file *file, poll_table *wait)
{
struct tun_file *tfile = file->private_data;
struct tun_struct *tun = __tun_get(tfile);
@@ -998,7 +1103,7 @@ static ssize_t tun_do_read(struct tun_struct *tun,struct tun_file *tfile,
schedule();
continue;
}
- netif_wake_queue(tun->dev);
+ netif_wake_subqueue(tun->dev, tfile->queue_index);
ret = tun_put_user(tun, tfile, skb, iv, len);
kfree_skb(skb);
@@ -1159,6 +1264,9 @@ static int tun_flags(struct tun_struct *tun)
if (tun->flags & TUN_VNET_HDR)
flags |= IFF_VNET_HDR;
+ if (tun->flags & TUN_TAP_MQ)
+ flags |= IFF_MULTI_QUEUE;
+
return flags;
}
@@ -1250,8 +1358,9 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
if (*ifr->ifr_name)
name = ifr->ifr_name;
- dev = alloc_netdev(sizeof(struct tun_struct), name,
- tun_setup);
+ dev = alloc_netdev_mqs(sizeof(struct tun_struct), name,
+ tun_setup,
+ MAX_TAP_QUEUES, MAX_TAP_QUEUES);
if (!dev)
return -ENOMEM;
@@ -1286,7 +1395,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
err = tun_attach(tun, file);
if (err < 0)
- goto failed;
+ goto err_free_dev;
}
tun_debug(KERN_INFO, tun, "tun_set_iff\n");
@@ -1306,18 +1415,22 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
else
tun->flags &= ~TUN_VNET_HDR;
+ if (ifr->ifr_flags & IFF_MULTI_QUEUE)
+ tun->flags |= TUN_TAP_MQ;
+ else
+ tun->flags &= ~TUN_TAP_MQ;
+
/* Make sure persistent devices do not get stuck in
* xoff state.
*/
if (netif_running(tun->dev))
- netif_wake_queue(tun->dev);
+ netif_tx_wake_all_queues(tun->dev);
strcpy(ifr->ifr_name, tun->dev->name);
return 0;
err_free_dev:
free_netdev(dev);
- failed:
return err;
}
@@ -1372,6 +1485,51 @@ static int set_offload(struct tun_struct *tun, unsigned long arg)
return 0;
}
+static void tun_detach_filter(struct tun_struct *tun, int n)
+{
+ int i;
+ struct tun_file *tfile;
+
+ for (i = 0; i < n; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ sk_detach_filter(tfile->socket.sk);
+ }
+
+ tun->filter_attached = false;
+}
+
+static int tun_attach_filter(struct tun_struct *tun)
+{
+ int i, ret = 0;
+ struct tun_file *tfile;
+
+ for (i = 0; i < tun->numqueues; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
+ if (ret) {
+ tun_detach_filter(tun, i);
+ return ret;
+ }
+ }
+
+ tun->filter_attached = true;
+ return ret;
+}
+
+static void tun_set_sndbuf(struct tun_struct *tun)
+{
+ struct tun_file *tfile;
+ int i;
+
+ for (i = 0; i < tun->numqueues; i++) {
+ tfile = rcu_dereference_protected(tun->tfiles[i],
+ lockdep_rtnl_is_held());
+ tfile->socket.sk->sk_sndbuf = tun->sndbuf;
+ }
+}
+
static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
unsigned long arg, int ifreq_len)
{
@@ -1400,6 +1558,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
(unsigned int __user*)argp);
}
+ ret = 0;
rtnl_lock();
tun = __tun_get(tfile);
@@ -1540,7 +1699,8 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
break;
}
- tun->sndbuf = tfile->socket.sk->sk_sndbuf = sndbuf;
+ tun->sndbuf = sndbuf;
+ tun_set_sndbuf(tun);
break;
case TUNGETVNETHDRSZ:
@@ -1571,9 +1731,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
if (copy_from_user(&tun->fprog, argp, sizeof(tun->fprog)))
break;
- ret = sk_attach_filter(&tun->fprog, tfile->socket.sk);
- if (!ret)
- tun->filter_attached = true;
+ ret = tun_attach_filter(tun);
break;
case TUNDETACHFILTER:
@@ -1581,9 +1739,8 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
ret = -EINVAL;
if ((tun->flags & TUN_TYPE_MASK) != TUN_TAP_DEV)
break;
- ret = sk_detach_filter(tfile->socket.sk);
- if (!ret)
- tun->filter_attached = false;
+ ret = 0;
+ tun_detach_filter(tun, tun->numqueues);
break;
default:
@@ -1688,37 +1845,9 @@ static int tun_chr_open(struct inode *inode, struct file * file)
static int tun_chr_close(struct inode *inode, struct file *file)
{
struct tun_file *tfile = file->private_data;
- struct tun_struct *tun;
struct net *net = tfile->net;
- rtnl_lock();
-
- tun = rcu_dereference_protected(tfile->tun, lockdep_rtnl_is_held());
- if (tun) {
- struct net_device *dev = tun->dev;
-
- tun_debug(KERN_INFO, tun, "tun_chr_close\n");
-
- __tun_detach(tun);
-
- synchronize_net();
-
- /* If desirable, unregister the netdevice. */
- if (!(tun->flags & TUN_PERSIST)) {
- if (dev->reg_state == NETREG_REGISTERED)
- unregister_netdevice(dev);
- }
-
- /* drop the reference that netdevice holds */
- sock_put(&tfile->sk);
- }
-
- rtnl_unlock();
-
- /* drop the reference that file holds */
- BUG_ON(!test_bit(SOCK_EXTERNALLY_ALLOCATED,
- &tfile->socket.flags));
- sk_release_kernel(&tfile->sk);
+ tun_detach(tfile, true);
put_net(net);
return 0;
--
1.7.1
^ permalink raw reply related
* [net-next v4 6/7] tuntap: add ioctl to attach or detach a file form tuntap device
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
Sometimes usespace may need to active/deactive a queue, this could be done by
detaching and attaching a file from tuntap device.
This patch introduces a new ioctls - TUNSETQUEUE which could be used to do
this. Flag IFF_ATTACH_QUEUE were introduced to do attaching while
IFF_DETACH_QUEUE were introduced to do the detaching.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 56 ++++++++++++++++++++++++++++++++++++------
include/uapi/linux/if_tun.h | 3 ++
2 files changed, 51 insertions(+), 8 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 59235ba..78dcda8 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -195,6 +195,15 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
return txq;
}
+static inline bool tun_not_capable(struct tun_struct *tun)
+{
+ const struct cred *cred = current_cred();
+
+ return (((uid_valid(tun->owner) && !uid_eq(cred->euid, tun->owner)) ||
+ (gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
+ !capable(CAP_NET_ADMIN));
+}
+
static void tun_set_real_num_queues(struct tun_struct *tun)
{
netif_set_real_num_tx_queues(tun->dev, tun->numqueues);
@@ -1310,8 +1319,6 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
dev = __dev_get_by_name(net, ifr->ifr_name);
if (dev) {
- const struct cred *cred = current_cred();
-
if (ifr->ifr_flags & IFF_TUN_EXCL)
return -EBUSY;
if ((ifr->ifr_flags & IFF_TUN) && dev->netdev_ops == &tun_netdev_ops)
@@ -1321,9 +1328,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
else
return -EINVAL;
- if (((uid_valid(tun->owner) && !uid_eq(cred->euid, tun->owner)) ||
- (gid_valid(tun->group) && !in_egroup_p(tun->group))) &&
- !capable(CAP_NET_ADMIN))
+ if (tun_not_capable(tun))
return -EPERM;
err = security_tun_dev_attach(tfile->socket.sk);
if (err < 0)
@@ -1530,6 +1535,40 @@ static void tun_set_sndbuf(struct tun_struct *tun)
}
}
+static int tun_set_queue(struct file *file, struct ifreq *ifr)
+{
+ struct tun_file *tfile = file->private_data;
+ struct tun_struct *tun;
+ struct net_device *dev;
+ int ret = 0;
+
+ rtnl_lock();
+
+ if (ifr->ifr_flags & IFF_ATTACH_QUEUE) {
+ dev = __dev_get_by_name(tfile->net, ifr->ifr_name);
+ if (!dev) {
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ tun = netdev_priv(dev);
+ if (dev->netdev_ops != &tap_netdev_ops &&
+ dev->netdev_ops != &tun_netdev_ops)
+ ret = -EINVAL;
+ else if (tun_not_capable(tun))
+ ret = -EPERM;
+ else
+ ret = tun_attach(tun, file);
+ } else if (ifr->ifr_flags & IFF_DETACH_QUEUE)
+ __tun_detach(tfile, false);
+ else
+ ret = -EINVAL;
+
+unlock:
+ rtnl_unlock();
+ return ret;
+}
+
static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
unsigned long arg, int ifreq_len)
{
@@ -1543,7 +1582,7 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
int vnet_hdr_sz;
int ret;
- if (cmd == TUNSETIFF || _IOC_TYPE(cmd) == 0x89) {
+ if (cmd == TUNSETIFF || cmd == TUNSETQUEUE || _IOC_TYPE(cmd) == 0x89) {
if (copy_from_user(&ifr, argp, ifreq_len))
return -EFAULT;
} else {
@@ -1554,9 +1593,10 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd,
* This is needed because we never checked for invalid flags on
* TUNSETIFF. */
return put_user(IFF_TUN | IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE |
- IFF_VNET_HDR,
+ IFF_VNET_HDR | IFF_MULTI_QUEUE,
(unsigned int __user*)argp);
- }
+ } else if (cmd == TUNSETQUEUE)
+ return tun_set_queue(file, &ifr);
ret = 0;
rtnl_lock();
diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h
index 8ef3a87..958497a 100644
--- a/include/uapi/linux/if_tun.h
+++ b/include/uapi/linux/if_tun.h
@@ -54,6 +54,7 @@
#define TUNDETACHFILTER _IOW('T', 214, struct sock_fprog)
#define TUNGETVNETHDRSZ _IOR('T', 215, int)
#define TUNSETVNETHDRSZ _IOW('T', 216, int)
+#define TUNSETQUEUE _IOW('T', 217, int)
/* TUNSETIFF ifr flags */
#define IFF_TUN 0x0001
@@ -63,6 +64,8 @@
#define IFF_VNET_HDR 0x4000
#define IFF_TUN_EXCL 0x8000
#define IFF_MULTI_QUEUE 0x0100
+#define IFF_ATTACH_QUEUE 0x0200
+#define IFF_DETACH_QUEUE 0x0400
/* Features for GSO (TUNSETOFFLOAD). */
#define TUN_F_CSUM 0x01 /* You can hand me unchecksummed packets. */
--
1.7.1
^ permalink raw reply related
* [net-next v4 7/7] tuntap: choose the txq based on rxq
From: Jason Wang @ 2012-10-29 5:36 UTC (permalink / raw)
To: davem, netdev, linux-kernel, mst, davem, netdev, linux-kernel,
mst
Cc: maxk, haixiao, ernesto.martin, krkumar2, edumazet, Jason Wang
In-Reply-To: <1351488964-11109-1-git-send-email-jasowang@redhat.com>
This patch implements a simple multiqueue flow steering policy - tx follows rx
for tun/tap. The idea is simple, it just choose the txq based on which rxq it
comes. The flow were identified through the rxhash of a skb, and the hash to
queue mapping were recorded in a hlist with an ageing timer to retire the
mapping. The mapping were created when tun receives packet from userspace, and
was quired in .ndo_select_queue().
I run co-current TCP_CRR test and didn't see any mapping manipulation helpers in
perf top, so the overhead could be negelected.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/tun.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 236 insertions(+), 3 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 78dcda8..158ef1d 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -115,6 +115,8 @@ struct tap_filter {
* to match a queue per guest CPU. */
#define MAX_TAP_QUEUES 1024
+#define TUN_FLOW_EXPIRE 3 * HZ
+
/* A tun_file connects an open character device to a tuntap netdevice. It
* also contains all socket related strctures (except sock_fprog and tap_filter)
* to serve as one transmit queue for tuntap device. The sock_fprog and
@@ -138,6 +140,18 @@ struct tun_file {
u16 queue_index;
};
+struct tun_flow_entry {
+ struct hlist_node hash_link;
+ struct rcu_head rcu;
+ struct tun_struct *tun;
+
+ u32 rxhash;
+ int queue_index;
+ unsigned long updated;
+};
+
+#define TUN_NUM_FLOW_ENTRIES 1024
+
/* Since the socket were moved to tun_file, to preserve the behavior of persist
* device, socket fileter, sndbuf and vnet header size were restore when the
* file were attached to a persist device.
@@ -163,8 +177,176 @@ struct tun_struct {
#ifdef TUN_DEBUG
int debug;
#endif
+ spinlock_t lock;
+ struct kmem_cache *flow_cache;
+ struct hlist_head flows[TUN_NUM_FLOW_ENTRIES];
+ struct timer_list flow_gc_timer;
+ unsigned long ageing_time;
};
+static inline u32 tun_hashfn(u32 rxhash)
+{
+ return rxhash & 0x3ff;
+}
+
+static struct tun_flow_entry *tun_flow_find(struct hlist_head *head, u32 rxhash)
+{
+ struct tun_flow_entry *e;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_rcu(e, n, head, hash_link) {
+ if (e->rxhash == rxhash)
+ return e;
+ }
+ return NULL;
+}
+
+static struct tun_flow_entry *tun_flow_create(struct tun_struct *tun,
+ struct hlist_head *head,
+ u32 rxhash, u16 queue_index)
+{
+ struct tun_flow_entry *e = kmem_cache_alloc(tun->flow_cache,
+ GFP_ATOMIC);
+ if (e) {
+ tun_debug(KERN_INFO, tun, "create flow: hash %u index %u\n",
+ rxhash, queue_index);
+ e->updated = jiffies;
+ e->rxhash = rxhash;
+ e->queue_index = queue_index;
+ e->tun = tun;
+ hlist_add_head_rcu(&e->hash_link, head);
+ }
+ return e;
+}
+
+static void tun_flow_free(struct rcu_head *head)
+{
+ struct tun_flow_entry *e
+ = container_of(head, struct tun_flow_entry, rcu);
+ kmem_cache_free(e->tun->flow_cache, e);
+}
+
+static void tun_flow_delete(struct tun_struct *tun, struct tun_flow_entry *e)
+{
+ tun_debug(KERN_INFO, tun, "delete flow: hash %u index %u\n",
+ e->rxhash, e->queue_index);
+ hlist_del_rcu(&e->hash_link);
+ call_rcu(&e->rcu, tun_flow_free);
+}
+
+/* caller hold tun->lock */
+static int tun_delete_by_rxhash(struct tun_struct *tun, u32 rxhash)
+{
+ struct hlist_head *head = &tun->flows[tun_hashfn(rxhash)];
+ struct tun_flow_entry *e = tun_flow_find(head, rxhash);
+
+ if (!e)
+ return -ENOENT;
+
+ tun_flow_delete(tun, e);
+}
+
+static void tun_flow_flush(struct tun_struct *tun)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link)
+ tun_flow_delete(tun, e);
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_delete_by_queue(struct tun_struct *tun, u16 queue_index)
+{
+ int i;
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ if (e->queue_index == queue_index)
+ tun_flow_delete(tun, e);
+ }
+ }
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_cleanup(unsigned long data)
+{
+ struct tun_struct *tun = (struct tun_struct *)data;
+ unsigned long delay = tun->ageing_time;
+ unsigned long next_timer = jiffies + delay;
+ unsigned long count = 0;
+ int i;
+
+ tun_debug(KERN_INFO, tun, "tun_flow_cleanup\n");
+
+ spin_lock_bh(&tun->lock);
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++) {
+ struct tun_flow_entry *e;
+ struct hlist_node *h, *n;
+
+ hlist_for_each_entry_safe(e, h, n, &tun->flows[i], hash_link) {
+ unsigned long this_timer;
+ count ++;
+ this_timer = e->updated + delay;
+ if (time_before_eq(this_timer, jiffies))
+ tun_flow_delete(tun, e);
+ else if (time_before(this_timer, next_timer))
+ next_timer = this_timer;
+ }
+ }
+
+ if (count)
+ mod_timer(&tun->flow_gc_timer, round_jiffies_up(next_timer));
+ spin_unlock_bh(&tun->lock);
+}
+
+static void tun_flow_update(struct tun_struct *tun, struct sk_buff *skb,
+ u16 queue_index)
+{
+ struct hlist_head *head;
+ struct tun_flow_entry *e;
+ unsigned long delay = tun->ageing_time;
+ u32 rxhash = skb_get_rxhash(skb);
+
+ if (!rxhash)
+ return;
+ else
+ head = &tun->flows[tun_hashfn(rxhash)];
+
+ rcu_read_lock();
+
+ if (tun->numqueues == 1)
+ goto unlock;
+
+ e = tun_flow_find(head, rxhash);
+ if (likely(e)) {
+ /* TODO: keep queueing to old queue until it's empty? */
+ e->queue_index = queue_index;
+ e->updated = jiffies;
+ } else {
+ spin_lock_bh(&tun->lock);
+ if (!tun_flow_find(head, rxhash))
+ tun_flow_create(tun, head, rxhash, queue_index);
+
+ if (!timer_pending(&tun->flow_gc_timer))
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + delay));
+ spin_unlock_bh(&tun->lock);
+ }
+
+unlock:
+ rcu_read_unlock();
+}
+
/* We try to identify a flow through its rxhash first. The reason that
* we do not check rxq no. is becuase some cards(e.g 82599), chooses
* the rxq based on the txq where the last packet of the flow comes. As
@@ -175,6 +357,7 @@ struct tun_struct {
static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
{
struct tun_struct *tun = netdev_priv(dev);
+ struct tun_flow_entry *e;
u32 txq = 0;
u32 numqueues = 0;
@@ -183,8 +366,12 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb)
txq = skb_get_rxhash(skb);
if (txq) {
- /* use multiply and shift instead of expensive divide */
- txq = ((u64)txq * numqueues) >> 32;
+ e = tun_flow_find(&tun->flows[tun_hashfn(txq)], txq);
+ if (e)
+ txq = e->queue_index;
+ else
+ /* use multiply and shift instead of expensive divide */
+ txq = ((u64)txq * numqueues) >> 32;
} else if (likely(skb_rx_queue_recorded(skb))) {
txq = skb_get_rx_queue(skb);
while (unlikely(txq >= numqueues))
@@ -234,6 +421,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
sock_put(&tfile->sk);
synchronize_net();
+ tun_flow_delete_by_queue(tun, tun->numqueues + 1);
/* Drop read queue */
skb_queue_purge(&tfile->sk.sk_receive_queue);
tun_set_real_num_queues(tun);
@@ -629,6 +817,37 @@ static const struct net_device_ops tap_netdev_ops = {
#endif
};
+static int tun_flow_init(struct tun_struct *tun)
+{
+ int i;
+
+ tun->flow_cache = kmem_cache_create("tun_flow_cache",
+ sizeof(struct tun_flow_entry), 0, 0,
+ NULL);
+ if (!tun->flow_cache)
+ return -ENOMEM;
+
+ for (i = 0; i < TUN_NUM_FLOW_ENTRIES; i++)
+ INIT_HLIST_HEAD(&tun->flows[i]);
+
+ tun->ageing_time = TUN_FLOW_EXPIRE;
+ setup_timer(&tun->flow_gc_timer, tun_flow_cleanup, (unsigned long)tun);
+ mod_timer(&tun->flow_gc_timer,
+ round_jiffies_up(jiffies + tun->ageing_time));
+
+ return 0;
+}
+
+static void tun_flow_uninit(struct tun_struct *tun)
+{
+ del_timer_sync(&tun->flow_gc_timer);
+ tun_flow_flush(tun);
+
+ /* Wait for completion of call_rcu()'s */
+ rcu_barrier();
+ kmem_cache_destroy(tun->flow_cache);
+}
+
/* Initialize net device. */
static void tun_net_init(struct net_device *dev)
{
@@ -973,6 +1192,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
tun->dev->stats.rx_packets++;
tun->dev->stats.rx_bytes += len;
+ tun_flow_update(tun, skb, tfile->queue_index);
return total_len;
}
@@ -1150,6 +1370,14 @@ out:
return ret;
}
+static void tun_free_netdev(struct net_device *dev)
+{
+ struct tun_struct *tun = netdev_priv(dev);
+
+ tun_flow_uninit(tun);
+ free_netdev(dev);
+}
+
static void tun_setup(struct net_device *dev)
{
struct tun_struct *tun = netdev_priv(dev);
@@ -1158,7 +1386,7 @@ static void tun_setup(struct net_device *dev)
tun->group = INVALID_GID;
dev->ethtool_ops = &tun_ethtool_ops;
- dev->destructor = free_netdev;
+ dev->destructor = tun_free_netdev;
}
/* Trivial set of netlink ops to allow deleting tun or tap
@@ -1381,10 +1609,15 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
tun->filter_attached = false;
tun->sndbuf = tfile->socket.sk->sk_sndbuf;
+ spin_lock_init(&tun->lock);
+
security_tun_dev_post_create(&tfile->sk);
tun_net_init(dev);
+ if (tun_flow_init(tun))
+ goto err_free_dev;
+
dev->hw_features = NETIF_F_SG | NETIF_F_FRAGLIST |
TUN_USER_FEATURES;
dev->features = dev->hw_features;
--
1.7.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox