* Re: [PATCH net-next v16 4/8] netfilter: Add nf_ct_get_tuple_skb callback
From: Pablo Neira Ayuso @ 2018-05-28 19:49 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: netdev, cake, netfilter-devel
In-Reply-To: <152751766686.30935.14644567905547700823.stgit@alrua-kau>
On Mon, May 28, 2018 at 04:27:46PM +0200, Toke Høiland-Jørgensen wrote:
[...]
> diff --git a/net/netfilter/core.c b/net/netfilter/core.c
> index 0f6b8172fb9a..520565198f0e 100644
> --- a/net/netfilter/core.c
> +++ b/net/netfilter/core.c
> @@ -572,6 +572,27 @@ void nf_conntrack_destroy(struct nf_conntrack *nfct)
> }
> EXPORT_SYMBOL(nf_conntrack_destroy);
>
> +bool (*skb_ct_get_tuple)(struct nf_conntrack_tuple *,
> + const struct sk_buff *) __rcu __read_mostly;
> +EXPORT_SYMBOL(skb_ct_get_tuple);
Now we have struct nf_ct_hook in net-next, please add ->get_tuple to
that new object.
Thanks.
^ permalink raw reply
* Re: [PATCH net-next v16 5/8] sch_cake: Add NAT awareness to packet classifier
From: Pablo Neira Ayuso @ 2018-05-28 19:51 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: netdev, cake, netfilter-devel
In-Reply-To: <152751766690.30935.18178441475189968162.stgit@alrua-kau>
On Mon, May 28, 2018 at 04:27:46PM +0200, Toke Høiland-Jørgensen wrote:
> When CAKE is deployed on a gateway that also performs NAT (which is a
> common deployment mode), the host fairness mechanism cannot distinguish
> internal hosts from each other, and so fails to work correctly.
>
> To fix this, we add an optional NAT awareness mode, which will query the
> kernel conntrack mechanism to obtain the pre-NAT addresses for each packet
> and use that in the flow and host hashing.
>
> When the shaper is enabled and the host is already performing NAT, the cost
> of this lookup is negligible. However, in unlimited mode with no NAT being
> performed, there is a significant CPU cost at higher bandwidths. For this
> reason, the feature is turned off by default.
>
> Cc: netfilter-devel@vger.kernel.org
> Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
> ---
> net/sched/sch_cake.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 46 insertions(+)
>
> diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
> index 68ac908470f1..fecd9caac0cc 100644
> --- a/net/sched/sch_cake.c
> +++ b/net/sched/sch_cake.c
> @@ -71,6 +71,10 @@
> #include <net/tcp.h>
> #include <net/flow_dissector.h>
>
> +#if IS_ENABLED(CONFIG_NF_CONNTRACK)
> +#include <net/netfilter/nf_conntrack_core.h>
> +#endif
> +
> #define CAKE_SET_WAYS (8)
> #define CAKE_MAX_TINS (8)
> #define CAKE_QUEUES (1024)
> @@ -516,6 +520,29 @@ static bool cobalt_should_drop(struct cobalt_vars *vars,
> return drop;
> }
>
> +static void cake_update_flowkeys(struct flow_keys *keys,
> + const struct sk_buff *skb)
> +{
> +#if IS_ENABLED(CONFIG_NF_CONNTRACK)
I would remove the ifdef, not really needed, it will simplify things.
But I leave it to you to decide, this is not I deal breaker.
^ permalink raw reply
* Re: [RFC net-next 0/4] net: sched: support replay of filter offload when binding to block
From: Jakub Kicinski @ 2018-05-28 20:02 UTC (permalink / raw)
To: Or Gerlitz
Cc: John Hurley, Linux Netdev List, Jiri Pirko, Samudrala, Sridhar,
oss-drivers, Rabie Loulou
In-Reply-To: <CAJ3xEMjfSc=yg+o+Pv4AQw0P-1C4pDjKa=ctCbeS2hWWMR+RPw@mail.gmail.com>
On Mon, 28 May 2018 13:48:28 +0300, Or Gerlitz wrote:
> On Fri, May 25, 2018 at 5:25 AM, Jakub Kicinski wrote:
> > This series from John adds the ability to replay filter offload requests
> > when new offload callback is being registered on a TC block. This is most
> > likely to take place for shared blocks today, when a block which already
> > has rules is bound to another interface. Prior to this patch set if any
> > of the rules were offloaded the block bind would fail.
>
> Can you elaborate a little further here? is this something that you are planning
> to use for the uplink LAG use-case? AFAIU if we apply share-block to nfp as
> things are prior to this patch, it would work, so there's a case where
> it doesn't and this is now handled with the series?
Just looking at things as they stand today, no bond/forward looking
plans - nfp "supports" shared blocks by registering multiple callbacks
to the block. There are two problems:
(a) one can't install a second callback if some rules are already
offloaded because of:
/* At this point, playback of previous block cb calls is not supported,
* so forbid to register to block which already has some offloaded
* filters present.
*/
if (tcf_block_offload_in_use(block))
return ERR_PTR(-EOPNOTSUPP);
in __tcf_block_cb_register(), so block sharing has to be set up
before any rules are added.
(b) when block is unshared filters are not removed today and driver
would have to sweep its rule table, as John notes. It's not a big
deal but this series fixes it nicely in the core, too.
Looking forward there are two things we can use shared blocks for: we
can try to teach user space to share ingress blocks on all legs of bonds
instead of trying to propagate the rules from the bond down in the
kernel, which is more tricky to get right. We will need reliable
replay for that, because we want new links to be able to join and leave
the bond when rules are already present.
Second use case, which is more far fetched, is trying to discover and
register callbacks for blocks of tunnel devices directly, and avoid the
egdev infrastructure...
We should discuss the above further, but regardless, I think this
patchset is quite a nice addition on it's own. Would you agree?
^ permalink raw reply
* Proposal
From: Miss Zeliha Omer Faruk @ 2018-05-28 20:46 UTC (permalink / raw)
--
Hello
I have been trying to contact you. Did you get my business proposal?
Best Regards,
Miss.Zeliha ömer faruk
Esentepe Mahallesi Büyükdere
Caddesi Kristal Kule Binasi
No:215
Sisli - Istanbul, Turke
^ permalink raw reply
* Re: Is it possible to get device information via CMSG?
From: Eric S. Raymond @ 2018-05-28 21:02 UTC (permalink / raw)
To: Michal Kubecek; +Cc: netdev
In-Reply-To: <20180528065701.dqdjbrqb34zfsfvo@unicorn.suse.cz>
Michal Kubecek <mkubecek@suse.cz>:
> > 1. Is there a cmsg_level/cmsg_type combination that will return the
> > name of the device the packet arrived through?
>
> Not name directly, AFAIK, but you can set SOL_IP / IP_PKTINFO (or
> SOL_IPV6 / IPV6_RECVPKTINFO) socket option and get IP_PKTINFO
> (IPV6_PKTINFO) message with recvmsg(). This will tell you incoming
> interface index so that you can look the name up. See ip(7) or ipv6(7)
> for format of the message (struct ip_pktinfo, struct in6_pktinfo).
Thanks, that confirms something I found on Stack Overflow after I
queried your list.
> However, I suspect that userspace application is not really interested
> in incoming interface name but rather in destination address of the
> incoming packet which is also provided in IP_PKTINFO / IPV6_PKTINFO
> message.
NTP is weird that way. My group, NTPsec, inherited the reference
Mills codebase (what we now call "NTP Classic") which really does have
a filter-by-interface-name feature *in addition to* local address
filtering.
We want to simplify the way it works without discarding that feature,
because we've made promises about backward compatibility that we mean
to keep. We don't throw away features unless either they're security
holes or we are *dead certain* they are obsolete.
In case it's of interest to you, NTPsec is a drop-in replacement for
NTP Classic that solves its chronic security problems by stripping out
unused features and legacy code. We've actually shrunk the codebase
size by a factor of 4x. We have better monitoring and admin tools, too.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.
^ permalink raw reply
* Re: [PATCH net-next v16 4/8] netfilter: Add nf_ct_get_tuple_skb callback
From: Toke Høiland-Jørgensen @ 2018-05-28 21:28 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netdev, cake, netfilter-devel
In-Reply-To: <20180528194902.kooqoao3agt32nls@salvia>
Pablo Neira Ayuso <pablo@netfilter.org> writes:
> On Mon, May 28, 2018 at 04:27:46PM +0200, Toke Høiland-Jørgensen wrote:
> [...]
>> diff --git a/net/netfilter/core.c b/net/netfilter/core.c
>> index 0f6b8172fb9a..520565198f0e 100644
>> --- a/net/netfilter/core.c
>> +++ b/net/netfilter/core.c
>> @@ -572,6 +572,27 @@ void nf_conntrack_destroy(struct nf_conntrack *nfct)
>> }
>> EXPORT_SYMBOL(nf_conntrack_destroy);
>>
>> +bool (*skb_ct_get_tuple)(struct nf_conntrack_tuple *,
>> + const struct sk_buff *) __rcu __read_mostly;
>> +EXPORT_SYMBOL(skb_ct_get_tuple);
>
> Now we have struct nf_ct_hook in net-next, please add ->get_tuple to
> that new object.
Ah, right, will do :)
-Toke
^ permalink raw reply
* Re: [PATCH v3 11/11] net: sched: change action API to use array of pointers to actions
From: Marcelo Ricardo Leitner @ 2018-05-28 21:31 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-12-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:29AM +0300, Vlad Buslov wrote:
...
> -int tcf_action_destroy(struct list_head *actions, int bind)
> +int tcf_action_destroy(struct tc_action *actions[], int bind)
> {
> const struct tc_action_ops *ops;
> - struct tc_action *a, *tmp;
> - int ret = 0;
> + struct tc_action *a;
> + int ret = 0, i;
>
> - list_for_each_entry_safe(a, tmp, actions, list) {
> + for (i = 0; i < TCA_ACT_MAX_PRIO && actions[i]; i++) {
...
> @@ -878,10 +881,9 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> if (TC_ACT_EXT_CMP(a->tcfa_action, TC_ACT_GOTO_CHAIN)) {
> err = tcf_action_goto_chain_init(a, tp);
> if (err) {
> - LIST_HEAD(actions);
> + struct tc_action *actions[TCA_ACT_MAX_PRIO] = { a };
Somewhat nit.. Considering tcf_action_destroy will stop at the first
NULL, you need only 2 slots here.
>
> - list_add_tail(&a->list, &actions);
> - tcf_action_destroy(&actions, bind);
> + tcf_action_destroy(actions, bind);
> NL_SET_ERR_MSG(extack, "Failed to init TC action chain");
> return ERR_PTR(err);
> }
^ permalink raw reply
* [PATCH net-next] net: remove bypassed check in sch_direct_xmit()
From: Song Liu @ 2018-05-28 21:36 UTC (permalink / raw)
To: netdev
Cc: Song Liu, kernel-team, John Fastabend, Alexei Starovoitov,
David S . Miller
Check sch_direct_xmit() at the end of sch_direct_xmit() will be bypassed.
This is because "ret" from sch_direct_xmit() will be either NETDEV_TX_OK
or NETDEV_TX_BUSY, and only ret == NETDEV_TX_OK == 0 will reach the
condition:
if (ret && netif_xmit_frozen_or_stopped(txq))
return false;
This patch cleans up the code by removing the whole condition.
For more discussion about this, please refer to
https://marc.info/?t=152727195700008
Signed-off-by: Song Liu <songliubraving@fb.com>
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
---
net/sched/sch_generic.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 760ab1b..69078c8 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -346,9 +346,6 @@ bool sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
return false;
}
- if (ret && netif_xmit_frozen_or_stopped(txq))
- return false;
-
return true;
}
--
2.9.5
^ permalink raw reply related
* Re: [PATCH v3 01/11] net: sched: use rcu for action cookie update
From: Marcelo Ricardo Leitner @ 2018-05-28 21:37 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn,
Jiri Pirko
In-Reply-To: <1527455849-22327-2-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:19AM +0300, Vlad Buslov wrote:
> Implement functions to atomically update and free action cookie
> using rcu mechanism.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
> Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> include/net/act_api.h | 2 +-
> include/net/pkt_cls.h | 1 +
> net/sched/act_api.c | 44 ++++++++++++++++++++++++++++++--------------
> 3 files changed, 32 insertions(+), 15 deletions(-)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index 9e59ebfded62..f7b59ef7303d 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -37,7 +37,7 @@ struct tc_action {
> spinlock_t tcfa_lock;
> struct gnet_stats_basic_cpu __percpu *cpu_bstats;
> struct gnet_stats_queue __percpu *cpu_qstats;
> - struct tc_cookie *act_cookie;
> + struct tc_cookie __rcu *act_cookie;
> struct tcf_chain *goto_chain;
> };
> #define tcf_index common.tcfa_index
> diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h
> index e828d31be5da..3068cc8aa0f1 100644
> --- a/include/net/pkt_cls.h
> +++ b/include/net/pkt_cls.h
> @@ -769,6 +769,7 @@ struct tc_mqprio_qopt_offload {
> struct tc_cookie {
> u8 *data;
> u32 len;
> + struct rcu_head rcu;
> };
>
> struct tc_qopt_offload_stats {
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 3f4cf930f809..02670c7489e3 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -55,6 +55,24 @@ static void tcf_action_goto_chain_exec(const struct tc_action *a,
> res->goto_tp = rcu_dereference_bh(chain->filter_chain);
> }
>
> +static void tcf_free_cookie_rcu(struct rcu_head *p)
> +{
> + struct tc_cookie *cookie = container_of(p, struct tc_cookie, rcu);
> +
> + kfree(cookie->data);
> + kfree(cookie);
> +}
> +
> +static void tcf_set_action_cookie(struct tc_cookie __rcu **old_cookie,
> + struct tc_cookie *new_cookie)
> +{
> + struct tc_cookie *old;
> +
> + old = xchg(old_cookie, new_cookie);
> + if (old)
> + call_rcu(&old->rcu, tcf_free_cookie_rcu);
> +}
> +
> /* XXX: For standalone actions, we don't need a RCU grace period either, because
> * actions are always connected to filters and filters are already destroyed in
> * RCU callbacks, so after a RCU grace period actions are already disconnected
> @@ -65,10 +83,7 @@ static void free_tcf(struct tc_action *p)
> free_percpu(p->cpu_bstats);
> free_percpu(p->cpu_qstats);
>
> - if (p->act_cookie) {
> - kfree(p->act_cookie->data);
> - kfree(p->act_cookie);
> - }
> + tcf_set_action_cookie(&p->act_cookie, NULL);
> if (p->goto_chain)
> tcf_action_goto_chain_fini(p);
>
> @@ -567,16 +582,22 @@ tcf_action_dump_1(struct sk_buff *skb, struct tc_action *a, int bind, int ref)
> int err = -EINVAL;
> unsigned char *b = skb_tail_pointer(skb);
> struct nlattr *nest;
> + struct tc_cookie *cookie;
>
> if (nla_put_string(skb, TCA_KIND, a->ops->kind))
> goto nla_put_failure;
> if (tcf_action_copy_stats(skb, a, 0))
> goto nla_put_failure;
> - if (a->act_cookie) {
> - if (nla_put(skb, TCA_ACT_COOKIE, a->act_cookie->len,
> - a->act_cookie->data))
> +
> + rcu_read_lock();
> + cookie = rcu_dereference(a->act_cookie);
> + if (cookie) {
> + if (nla_put(skb, TCA_ACT_COOKIE, cookie->len, cookie->data)) {
> + rcu_read_unlock();
> goto nla_put_failure;
> + }
> }
> + rcu_read_unlock();
>
> nest = nla_nest_start(skb, TCA_OPTIONS);
> if (nest == NULL)
> @@ -719,13 +740,8 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> if (err < 0)
> goto err_mod;
>
> - if (name == NULL && tb[TCA_ACT_COOKIE]) {
> - if (a->act_cookie) {
> - kfree(a->act_cookie->data);
> - kfree(a->act_cookie);
> - }
> - a->act_cookie = cookie;
> - }
> + if (!name && tb[TCA_ACT_COOKIE])
> + tcf_set_action_cookie(&a->act_cookie, cookie);
>
> /* module count goes up only when brand new policy is created
> * if it exists and is only bound to in a_o->init() then
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 02/11] net: sched: change type of reference and bind counters
From: Marcelo Ricardo Leitner @ 2018-05-28 21:37 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn,
Jiri Pirko
In-Reply-To: <1527455849-22327-3-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:20AM +0300, Vlad Buslov wrote:
> Change type of action reference counter to refcount_t.
>
> Change type of action bind counter to atomic_t.
> This type is used to allow decrementing bind counter without testing
> for 0 result.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
> Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> include/net/act_api.h | 5 +++--
> net/sched/act_api.c | 32 ++++++++++++++++++++++----------
> net/sched/act_bpf.c | 4 ++--
> net/sched/act_connmark.c | 4 ++--
> net/sched/act_csum.c | 4 ++--
> net/sched/act_gact.c | 4 ++--
> net/sched/act_ife.c | 4 ++--
> net/sched/act_ipt.c | 4 ++--
> net/sched/act_mirred.c | 4 ++--
> net/sched/act_nat.c | 4 ++--
> net/sched/act_pedit.c | 4 ++--
> net/sched/act_police.c | 4 ++--
> net/sched/act_sample.c | 4 ++--
> net/sched/act_simple.c | 4 ++--
> net/sched/act_skbedit.c | 4 ++--
> net/sched/act_skbmod.c | 4 ++--
> net/sched/act_tunnel_key.c | 4 ++--
> net/sched/act_vlan.c | 4 ++--
> 18 files changed, 57 insertions(+), 44 deletions(-)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index f7b59ef7303d..e634014605cb 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -6,6 +6,7 @@
> * Public action API for classifiers/qdiscs
> */
>
> +#include <linux/refcount.h>
> #include <net/sch_generic.h>
> #include <net/pkt_sched.h>
> #include <net/net_namespace.h>
> @@ -26,8 +27,8 @@ struct tc_action {
> struct tcf_idrinfo *idrinfo;
>
> u32 tcfa_index;
> - int tcfa_refcnt;
> - int tcfa_bindcnt;
> + refcount_t tcfa_refcnt;
> + atomic_t tcfa_bindcnt;
> u32 tcfa_capab;
> int tcfa_action;
> struct tcf_t tcfa_tm;
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 02670c7489e3..4f064ecab882 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -105,14 +105,26 @@ int __tcf_idr_release(struct tc_action *p, bool bind, bool strict)
>
> ASSERT_RTNL();
>
> + /* Release with strict==1 and bind==0 is only called through act API
> + * interface (classifiers always bind). Only case when action with
> + * positive reference count and zero bind count can exist is when it was
> + * also created with act API (unbinding last classifier will destroy the
> + * action if it was created by classifier). So only case when bind count
> + * can be changed after initial check is when unbound action is
> + * destroyed by act API while classifier binds to action with same id
> + * concurrently. This result either creation of new action(same behavior
> + * as before), or reusing existing action if concurrent process
> + * increments reference count before action is deleted. Both scenarios
> + * are acceptable.
> + */
> if (p) {
> if (bind)
> - p->tcfa_bindcnt--;
> - else if (strict && p->tcfa_bindcnt > 0)
> + atomic_dec(&p->tcfa_bindcnt);
> + else if (strict && atomic_read(&p->tcfa_bindcnt) > 0)
> return -EPERM;
>
> - p->tcfa_refcnt--;
> - if (p->tcfa_bindcnt <= 0 && p->tcfa_refcnt <= 0) {
> + if (atomic_read(&p->tcfa_bindcnt) <= 0 &&
> + refcount_dec_and_test(&p->tcfa_refcnt)) {
> if (p->ops->cleanup)
> p->ops->cleanup(p);
> tcf_idr_remove(p->idrinfo, p);
> @@ -304,8 +316,8 @@ bool tcf_idr_check(struct tc_action_net *tn, u32 index, struct tc_action **a,
>
> if (index && p) {
> if (bind)
> - p->tcfa_bindcnt++;
> - p->tcfa_refcnt++;
> + atomic_inc(&p->tcfa_bindcnt);
> + refcount_inc(&p->tcfa_refcnt);
> *a = p;
> return true;
> }
> @@ -324,9 +336,9 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
>
> if (unlikely(!p))
> return -ENOMEM;
> - p->tcfa_refcnt = 1;
> + refcount_set(&p->tcfa_refcnt, 1);
> if (bind)
> - p->tcfa_bindcnt = 1;
> + atomic_set(&p->tcfa_bindcnt, 1);
>
> if (cpustats) {
> p->cpu_bstats = netdev_alloc_pcpu_stats(struct gnet_stats_basic_cpu);
> @@ -782,7 +794,7 @@ static void cleanup_a(struct list_head *actions, int ovr)
> return;
>
> list_for_each_entry(a, actions, list)
> - a->tcfa_refcnt--;
> + refcount_dec(&a->tcfa_refcnt);
> }
>
> int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> @@ -810,7 +822,7 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> act->order = i;
> sz += tcf_action_fill_size(act);
> if (ovr)
> - act->tcfa_refcnt++;
> + refcount_inc(&act->tcfa_refcnt);
> list_add_tail(&act->list, actions);
> }
>
> diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
> index 18089c02e557..15a2a53cbde1 100644
> --- a/net/sched/act_bpf.c
> +++ b/net/sched/act_bpf.c
> @@ -141,8 +141,8 @@ static int tcf_bpf_dump(struct sk_buff *skb, struct tc_action *act,
> struct tcf_bpf *prog = to_bpf(act);
> struct tc_act_bpf opt = {
> .index = prog->tcf_index,
> - .refcnt = prog->tcf_refcnt - ref,
> - .bindcnt = prog->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&prog->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&prog->tcf_bindcnt) - bind,
> .action = prog->tcf_action,
> };
> struct tcf_t tm;
> diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
> index e4b880fa51fe..188865034f9a 100644
> --- a/net/sched/act_connmark.c
> +++ b/net/sched/act_connmark.c
> @@ -154,8 +154,8 @@ static inline int tcf_connmark_dump(struct sk_buff *skb, struct tc_action *a,
>
> struct tc_connmark opt = {
> .index = ci->tcf_index,
> - .refcnt = ci->tcf_refcnt - ref,
> - .bindcnt = ci->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&ci->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&ci->tcf_bindcnt) - bind,
> .action = ci->tcf_action,
> .zone = ci->zone,
> };
> diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
> index 526a8e491626..da865f7b390a 100644
> --- a/net/sched/act_csum.c
> +++ b/net/sched/act_csum.c
> @@ -597,8 +597,8 @@ static int tcf_csum_dump(struct sk_buff *skb, struct tc_action *a, int bind,
> struct tcf_csum_params *params;
> struct tc_csum opt = {
> .index = p->tcf_index,
> - .refcnt = p->tcf_refcnt - ref,
> - .bindcnt = p->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&p->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&p->tcf_bindcnt) - bind,
> };
> struct tcf_t t;
>
> diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
> index 4dc4f153cad8..ca83debd5a70 100644
> --- a/net/sched/act_gact.c
> +++ b/net/sched/act_gact.c
> @@ -169,8 +169,8 @@ static int tcf_gact_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_gact *gact = to_gact(a);
> struct tc_gact opt = {
> .index = gact->tcf_index,
> - .refcnt = gact->tcf_refcnt - ref,
> - .bindcnt = gact->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&gact->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&gact->tcf_bindcnt) - bind,
> .action = gact->tcf_action,
> };
> struct tcf_t t;
> diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
> index 8527cfdc446d..706e84d6f912 100644
> --- a/net/sched/act_ife.c
> +++ b/net/sched/act_ife.c
> @@ -598,8 +598,8 @@ static int tcf_ife_dump(struct sk_buff *skb, struct tc_action *a, int bind,
> struct tcf_ife_params *p = rtnl_dereference(ife->params);
> struct tc_ife opt = {
> .index = ife->tcf_index,
> - .refcnt = ife->tcf_refcnt - ref,
> - .bindcnt = ife->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&ife->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&ife->tcf_bindcnt) - bind,
> .action = ife->tcf_action,
> .flags = p->flags,
> };
> diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
> index 14c312d7908f..7bce88dc11c9 100644
> --- a/net/sched/act_ipt.c
> +++ b/net/sched/act_ipt.c
> @@ -280,8 +280,8 @@ static int tcf_ipt_dump(struct sk_buff *skb, struct tc_action *a, int bind,
> if (unlikely(!t))
> goto nla_put_failure;
>
> - c.bindcnt = ipt->tcf_bindcnt - bind;
> - c.refcnt = ipt->tcf_refcnt - ref;
> + c.bindcnt = atomic_read(&ipt->tcf_bindcnt) - bind;
> + c.refcnt = refcount_read(&ipt->tcf_refcnt) - ref;
> strcpy(t->u.user.name, ipt->tcfi_t->u.kernel.target->name);
>
> if (nla_put(skb, TCA_IPT_TARG, ipt->tcfi_t->u.user.target_size, t) ||
> diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> index fd34015331ab..82a8bdd67c47 100644
> --- a/net/sched/act_mirred.c
> +++ b/net/sched/act_mirred.c
> @@ -250,8 +250,8 @@ static int tcf_mirred_dump(struct sk_buff *skb, struct tc_action *a, int bind,
> struct tc_mirred opt = {
> .index = m->tcf_index,
> .action = m->tcf_action,
> - .refcnt = m->tcf_refcnt - ref,
> - .bindcnt = m->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&m->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&m->tcf_bindcnt) - bind,
> .eaction = m->tcfm_eaction,
> .ifindex = dev ? dev->ifindex : 0,
> };
> diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
> index 4b5848b6c252..457c2ae3de46 100644
> --- a/net/sched/act_nat.c
> +++ b/net/sched/act_nat.c
> @@ -257,8 +257,8 @@ static int tcf_nat_dump(struct sk_buff *skb, struct tc_action *a,
>
> .index = p->tcf_index,
> .action = p->tcf_action,
> - .refcnt = p->tcf_refcnt - ref,
> - .bindcnt = p->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&p->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&p->tcf_bindcnt) - bind,
> };
> struct tcf_t t;
>
> diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
> index 8a925c72db5f..0102b2935fdb 100644
> --- a/net/sched/act_pedit.c
> +++ b/net/sched/act_pedit.c
> @@ -391,8 +391,8 @@ static int tcf_pedit_dump(struct sk_buff *skb, struct tc_action *a,
> opt->nkeys = p->tcfp_nkeys;
> opt->flags = p->tcfp_flags;
> opt->action = p->tcf_action;
> - opt->refcnt = p->tcf_refcnt - ref;
> - opt->bindcnt = p->tcf_bindcnt - bind;
> + opt->refcnt = refcount_read(&p->tcf_refcnt) - ref;
> + opt->bindcnt = atomic_read(&p->tcf_bindcnt) - bind;
>
> if (p->tcfp_keys_ex) {
> tcf_pedit_key_ex_dump(skb, p->tcfp_keys_ex, p->tcfp_nkeys);
> diff --git a/net/sched/act_police.c b/net/sched/act_police.c
> index 4e72bc2a0dfb..a789b8060968 100644
> --- a/net/sched/act_police.c
> +++ b/net/sched/act_police.c
> @@ -274,8 +274,8 @@ static int tcf_act_police_dump(struct sk_buff *skb, struct tc_action *a,
> .action = police->tcf_action,
> .mtu = police->tcfp_mtu,
> .burst = PSCHED_NS2TICKS(police->tcfp_burst),
> - .refcnt = police->tcf_refcnt - ref,
> - .bindcnt = police->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&police->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&police->tcf_bindcnt) - bind,
> };
> struct tcf_t t;
>
> diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
> index 5db358497c9e..4a46978db092 100644
> --- a/net/sched/act_sample.c
> +++ b/net/sched/act_sample.c
> @@ -173,8 +173,8 @@ static int tcf_sample_dump(struct sk_buff *skb, struct tc_action *a,
> struct tc_sample opt = {
> .index = s->tcf_index,
> .action = s->tcf_action,
> - .refcnt = s->tcf_refcnt - ref,
> - .bindcnt = s->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&s->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&s->tcf_bindcnt) - bind,
> };
> struct tcf_t t;
>
> diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
> index 9618b4a83cee..95d5985b8d67 100644
> --- a/net/sched/act_simple.c
> +++ b/net/sched/act_simple.c
> @@ -148,8 +148,8 @@ static int tcf_simp_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_defact *d = to_defact(a);
> struct tc_defact opt = {
> .index = d->tcf_index,
> - .refcnt = d->tcf_refcnt - ref,
> - .bindcnt = d->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&d->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&d->tcf_bindcnt) - bind,
> .action = d->tcf_action,
> };
> struct tcf_t t;
> diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
> index 6138d1d71900..d418ec3b0ab9 100644
> --- a/net/sched/act_skbedit.c
> +++ b/net/sched/act_skbedit.c
> @@ -173,8 +173,8 @@ static int tcf_skbedit_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_skbedit *d = to_skbedit(a);
> struct tc_skbedit opt = {
> .index = d->tcf_index,
> - .refcnt = d->tcf_refcnt - ref,
> - .bindcnt = d->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&d->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&d->tcf_bindcnt) - bind,
> .action = d->tcf_action,
> };
> struct tcf_t t;
> diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
> index ad050d7d4b46..ff90d720eda3 100644
> --- a/net/sched/act_skbmod.c
> +++ b/net/sched/act_skbmod.c
> @@ -205,8 +205,8 @@ static int tcf_skbmod_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_skbmod_params *p = rtnl_dereference(d->skbmod_p);
> struct tc_skbmod opt = {
> .index = d->tcf_index,
> - .refcnt = d->tcf_refcnt - ref,
> - .bindcnt = d->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&d->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&d->tcf_bindcnt) - bind,
> .action = d->tcf_action,
> };
> struct tcf_t t;
> diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
> index 626dac81a48a..c6e50695414b 100644
> --- a/net/sched/act_tunnel_key.c
> +++ b/net/sched/act_tunnel_key.c
> @@ -252,8 +252,8 @@ static int tunnel_key_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_tunnel_key_params *params;
> struct tc_tunnel_key opt = {
> .index = t->tcf_index,
> - .refcnt = t->tcf_refcnt - ref,
> - .bindcnt = t->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&t->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&t->tcf_bindcnt) - bind,
> };
> struct tcf_t tm;
>
> diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
> index 853604685965..8dda78473004 100644
> --- a/net/sched/act_vlan.c
> +++ b/net/sched/act_vlan.c
> @@ -237,8 +237,8 @@ static int tcf_vlan_dump(struct sk_buff *skb, struct tc_action *a,
> struct tcf_vlan_params *p = rtnl_dereference(v->vlan_p);
> struct tc_vlan opt = {
> .index = v->tcf_index,
> - .refcnt = v->tcf_refcnt - ref,
> - .bindcnt = v->tcf_bindcnt - bind,
> + .refcnt = refcount_read(&v->tcf_refcnt) - ref,
> + .bindcnt = atomic_read(&v->tcf_bindcnt) - bind,
> .action = v->tcf_action,
> .v_action = p->tcfv_action,
> };
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 03/11] net: sched: implement unlocked action init API
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-4-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:21AM +0300, Vlad Buslov wrote:
> Add additional 'rtnl_held' argument to act API init functions. It is
> required to implement actions that need to release rtnl lock before loading
> kernel module and reacquire if afterwards.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Rename "unlocked" to "rtnl_held" for clarity.
>
> include/net/act_api.h | 6 ++++--
> net/sched/act_api.c | 18 +++++++++++-------
> net/sched/act_bpf.c | 3 ++-
> net/sched/act_connmark.c | 2 +-
> net/sched/act_csum.c | 3 ++-
> net/sched/act_gact.c | 3 ++-
> net/sched/act_ife.c | 3 ++-
> net/sched/act_ipt.c | 6 ++++--
> net/sched/act_mirred.c | 5 +++--
> net/sched/act_nat.c | 2 +-
> net/sched/act_pedit.c | 3 ++-
> net/sched/act_police.c | 2 +-
> net/sched/act_sample.c | 3 ++-
> net/sched/act_simple.c | 3 ++-
> net/sched/act_skbedit.c | 3 ++-
> net/sched/act_skbmod.c | 3 ++-
> net/sched/act_tunnel_key.c | 3 ++-
> net/sched/act_vlan.c | 3 ++-
> net/sched/cls_api.c | 5 +++--
> 19 files changed, 50 insertions(+), 29 deletions(-)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index e634014605cb..888ff471bbf6 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -92,7 +92,8 @@ struct tc_action_ops {
> struct netlink_ext_ack *extack);
> int (*init)(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **act, int ovr,
> - int bind, struct netlink_ext_ack *extack);
> + int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack);
> int (*walk)(struct net *, struct sk_buff *,
> struct netlink_callback *, int,
> const struct tc_action_ops *,
> @@ -168,10 +169,11 @@ int tcf_action_exec(struct sk_buff *skb, struct tc_action **actions,
> int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> struct nlattr *est, char *name, int ovr, int bind,
> struct list_head *actions, size_t *attr_size,
> - struct netlink_ext_ack *extack);
> + bool rtnl_held, struct netlink_ext_ack *extack);
> struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> struct nlattr *nla, struct nlattr *est,
> char *name, int ovr, int bind,
> + bool rtnl_held,
> struct netlink_ext_ack *extack);
> int tcf_action_dump(struct sk_buff *skb, struct list_head *, int, int);
> int tcf_action_dump_old(struct sk_buff *skb, struct tc_action *a, int, int);
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 4f064ecab882..256b0c93916c 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -671,6 +671,7 @@ static struct tc_cookie *nla_memdup_cookie(struct nlattr **tb)
> struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> struct nlattr *nla, struct nlattr *est,
> char *name, int ovr, int bind,
> + bool rtnl_held,
> struct netlink_ext_ack *extack)
> {
> struct tc_action *a;
> @@ -721,9 +722,11 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> a_o = tc_lookup_action_n(act_name);
> if (a_o == NULL) {
> #ifdef CONFIG_MODULES
> - rtnl_unlock();
> + if (rtnl_held)
> + rtnl_unlock();
> request_module("act_%s", act_name);
> - rtnl_lock();
> + if (rtnl_held)
> + rtnl_lock();
>
> a_o = tc_lookup_action_n(act_name);
>
> @@ -746,9 +749,10 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> /* backward compatibility for policer */
> if (name == NULL)
> err = a_o->init(net, tb[TCA_ACT_OPTIONS], est, &a, ovr, bind,
> - extack);
> + rtnl_held, extack);
> else
> - err = a_o->init(net, nla, est, &a, ovr, bind, extack);
> + err = a_o->init(net, nla, est, &a, ovr, bind, rtnl_held,
> + extack);
> if (err < 0)
> goto err_mod;
>
> @@ -800,7 +804,7 @@ static void cleanup_a(struct list_head *actions, int ovr)
> int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> struct nlattr *est, char *name, int ovr, int bind,
> struct list_head *actions, size_t *attr_size,
> - struct netlink_ext_ack *extack)
> + bool rtnl_held, struct netlink_ext_ack *extack)
> {
> struct nlattr *tb[TCA_ACT_MAX_PRIO + 1];
> struct tc_action *act;
> @@ -814,7 +818,7 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
>
> for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) {
> act = tcf_action_init_1(net, tp, tb[i], est, name, ovr, bind,
> - extack);
> + rtnl_held, extack);
> if (IS_ERR(act)) {
> err = PTR_ERR(act);
> goto err;
> @@ -1173,7 +1177,7 @@ static int tcf_action_add(struct net *net, struct nlattr *nla,
> LIST_HEAD(actions);
>
> ret = tcf_action_init(net, NULL, nla, NULL, NULL, ovr, 0, &actions,
> - &attr_size, extack);
> + &attr_size, true, extack);
> if (ret)
> return ret;
>
> diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
> index 15a2a53cbde1..8ebf40a3506c 100644
> --- a/net/sched/act_bpf.c
> +++ b/net/sched/act_bpf.c
> @@ -276,7 +276,8 @@ static void tcf_bpf_prog_fill_cfg(const struct tcf_bpf *prog,
>
> static int tcf_bpf_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **act,
> - int replace, int bind, struct netlink_ext_ack *extack)
> + int replace, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, bpf_net_id);
> struct nlattr *tb[TCA_ACT_BPF_MAX + 1];
> diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
> index 188865034f9a..e3787aa0025a 100644
> --- a/net/sched/act_connmark.c
> +++ b/net/sched/act_connmark.c
> @@ -96,7 +96,7 @@ static const struct nla_policy connmark_policy[TCA_CONNMARK_MAX + 1] = {
>
> static int tcf_connmark_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind,
> + int ovr, int bind, bool rtnl_held,
> struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, connmark_net_id);
> diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
> index da865f7b390a..334261943f9f 100644
> --- a/net/sched/act_csum.c
> +++ b/net/sched/act_csum.c
> @@ -46,7 +46,8 @@ static struct tc_action_ops act_csum_ops;
>
> static int tcf_csum_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a, int ovr,
> - int bind, struct netlink_ext_ack *extack)
> + int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, csum_net_id);
> struct tcf_csum_params *params_old, *params_new;
> diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
> index ca83debd5a70..b4dfb2b4addc 100644
> --- a/net/sched/act_gact.c
> +++ b/net/sched/act_gact.c
> @@ -56,7 +56,8 @@ static const struct nla_policy gact_policy[TCA_GACT_MAX + 1] = {
>
> static int tcf_gact_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, gact_net_id);
> struct nlattr *tb[TCA_GACT_MAX + 1];
> diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
> index 706e84d6f912..3dccc4e1d378 100644
> --- a/net/sched/act_ife.c
> +++ b/net/sched/act_ife.c
> @@ -447,7 +447,8 @@ static int populate_metalist(struct tcf_ife_info *ife, struct nlattr **tb,
>
> static int tcf_ife_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, ife_net_id);
> struct nlattr *tb[TCA_IFE_MAX + 1];
> diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
> index 7bce88dc11c9..9c21663a86a6 100644
> --- a/net/sched/act_ipt.c
> +++ b/net/sched/act_ipt.c
> @@ -196,7 +196,8 @@ static int __tcf_ipt_init(struct net *net, unsigned int id, struct nlattr *nla,
>
> static int tcf_ipt_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a, int ovr,
> - int bind, struct netlink_ext_ack *extack)
> + int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> return __tcf_ipt_init(net, ipt_net_id, nla, est, a, &act_ipt_ops, ovr,
> bind);
> @@ -204,7 +205,8 @@ static int tcf_ipt_init(struct net *net, struct nlattr *nla,
>
> static int tcf_xt_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a, int ovr,
> - int bind, struct netlink_ext_ack *extack)
> + int bind, bool unlocked,
> + struct netlink_ext_ack *extack)
> {
> return __tcf_ipt_init(net, xt_net_id, nla, est, a, &act_xt_ops, ovr,
> bind);
> diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> index 82a8bdd67c47..5434f08f2eb7 100644
> --- a/net/sched/act_mirred.c
> +++ b/net/sched/act_mirred.c
> @@ -68,8 +68,9 @@ static unsigned int mirred_net_id;
> static struct tc_action_ops act_mirred_ops;
>
> static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> - struct nlattr *est, struct tc_action **a, int ovr,
> - int bind, struct netlink_ext_ack *extack)
> + struct nlattr *est, struct tc_action **a,
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, mirred_net_id);
> struct nlattr *tb[TCA_MIRRED_MAX + 1];
> diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
> index 457c2ae3de46..e6487ad1e4a8 100644
> --- a/net/sched/act_nat.c
> +++ b/net/sched/act_nat.c
> @@ -38,7 +38,7 @@ static const struct nla_policy nat_policy[TCA_NAT_MAX + 1] = {
>
> static int tcf_nat_init(struct net *net, struct nlattr *nla, struct nlattr *est,
> struct tc_action **a, int ovr, int bind,
> - struct netlink_ext_ack *extack)
> + bool rtnl_held, struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, nat_net_id);
> struct nlattr *tb[TCA_NAT_MAX + 1];
> diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
> index 0102b2935fdb..7c9a3f24edba 100644
> --- a/net/sched/act_pedit.c
> +++ b/net/sched/act_pedit.c
> @@ -132,7 +132,8 @@ static int tcf_pedit_key_ex_dump(struct sk_buff *skb,
>
> static int tcf_pedit_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, pedit_net_id);
> struct nlattr *tb[TCA_PEDIT_MAX + 1];
> diff --git a/net/sched/act_police.c b/net/sched/act_police.c
> index a789b8060968..0e1c2fb0ebea 100644
> --- a/net/sched/act_police.c
> +++ b/net/sched/act_police.c
> @@ -75,7 +75,7 @@ static const struct nla_policy police_policy[TCA_POLICE_MAX + 1] = {
>
> static int tcf_act_police_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind,
> + int ovr, int bind, bool rtnl_held,
> struct netlink_ext_ack *extack)
> {
> int ret = 0, err;
> diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
> index 4a46978db092..316fc645595d 100644
> --- a/net/sched/act_sample.c
> +++ b/net/sched/act_sample.c
> @@ -37,7 +37,8 @@ static const struct nla_policy sample_policy[TCA_SAMPLE_MAX + 1] = {
>
> static int tcf_sample_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a, int ovr,
> - int bind, struct netlink_ext_ack *extack)
> + int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, sample_net_id);
> struct nlattr *tb[TCA_SAMPLE_MAX + 1];
> diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
> index 95d5985b8d67..23fa893ea092 100644
> --- a/net/sched/act_simple.c
> +++ b/net/sched/act_simple.c
> @@ -79,7 +79,8 @@ static const struct nla_policy simple_policy[TCA_DEF_MAX + 1] = {
>
> static int tcf_simp_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, simp_net_id);
> struct nlattr *tb[TCA_DEF_MAX + 1];
> diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
> index d418ec3b0ab9..85ed9d603dc1 100644
> --- a/net/sched/act_skbedit.c
> +++ b/net/sched/act_skbedit.c
> @@ -66,7 +66,8 @@ static const struct nla_policy skbedit_policy[TCA_SKBEDIT_MAX + 1] = {
>
> static int tcf_skbedit_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, skbedit_net_id);
> struct nlattr *tb[TCA_SKBEDIT_MAX + 1];
> diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
> index ff90d720eda3..026d6f58eda1 100644
> --- a/net/sched/act_skbmod.c
> +++ b/net/sched/act_skbmod.c
> @@ -84,7 +84,8 @@ static const struct nla_policy skbmod_policy[TCA_SKBMOD_MAX + 1] = {
>
> static int tcf_skbmod_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, skbmod_net_id);
> struct nlattr *tb[TCA_SKBMOD_MAX + 1];
> diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
> index c6e50695414b..ed698fcb0e5a 100644
> --- a/net/sched/act_tunnel_key.c
> +++ b/net/sched/act_tunnel_key.c
> @@ -70,7 +70,8 @@ static const struct nla_policy tunnel_key_policy[TCA_TUNNEL_KEY_MAX + 1] = {
>
> static int tunnel_key_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, tunnel_key_net_id);
> struct nlattr *tb[TCA_TUNNEL_KEY_MAX + 1];
> diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
> index 8dda78473004..72d2d78fb942 100644
> --- a/net/sched/act_vlan.c
> +++ b/net/sched/act_vlan.c
> @@ -109,7 +109,8 @@ static const struct nla_policy vlan_policy[TCA_VLAN_MAX + 1] = {
>
> static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> struct nlattr *est, struct tc_action **a,
> - int ovr, int bind, struct netlink_ext_ack *extack)
> + int ovr, int bind, bool rtnl_held,
> + struct netlink_ext_ack *extack)
> {
> struct tc_action_net *tn = net_generic(net, vlan_net_id);
> struct nlattr *tb[TCA_VLAN_MAX + 1];
> diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
> index 963e4bf0aab8..efbf01ce14c2 100644
> --- a/net/sched/cls_api.c
> +++ b/net/sched/cls_api.c
> @@ -1438,7 +1438,7 @@ int tcf_exts_validate(struct net *net, struct tcf_proto *tp, struct nlattr **tb,
> if (exts->police && tb[exts->police]) {
> act = tcf_action_init_1(net, tp, tb[exts->police],
> rate_tlv, "police", ovr,
> - TCA_ACT_BIND, extack);
> + TCA_ACT_BIND, true, extack);
> if (IS_ERR(act))
> return PTR_ERR(act);
>
> @@ -1451,7 +1451,8 @@ int tcf_exts_validate(struct net *net, struct tcf_proto *tp, struct nlattr **tb,
>
> err = tcf_action_init(net, tp, tb[exts->action],
> rate_tlv, NULL, ovr, TCA_ACT_BIND,
> - &actions, &attr_size, extack);
> + &actions, &attr_size, true,
> + extack);
> if (err)
> return err;
> list_for_each_entry(act, &actions, list)
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 04/11] net: sched: always take reference to action
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-5-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:22AM +0300, Vlad Buslov wrote:
> Without rtnl lock protection it is no longer safe to use pointer to tc
> action without holding reference to it. (it can be destroyed concurrently)
>
> Remove unsafe action idr lookup function. Instead of it, implement safe tcf
> idr check function that atomically looks up action in idr and increments
> its reference and bind counters. Implement both action search and check
> using new safe function
>
> Reference taken by idr check is temporal and should not be accounted by
> userspace clients (both logically and to preserver current API behavior).
> Subtract temporal reference when dumping action to userspace using existing
> tca_get_fill function arguments.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Make __tcf_idr_check function static
> - Merge changes that take reference to action when performing lookup and
> changes that account for this additional reference when dumping action
> to user space into single patch.
>
> net/sched/act_api.c | 46 ++++++++++++++++++++--------------------------
> 1 file changed, 20 insertions(+), 26 deletions(-)
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 256b0c93916c..aa304d36fee0 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -284,44 +284,38 @@ int tcf_generic_walker(struct tc_action_net *tn, struct sk_buff *skb,
> }
> EXPORT_SYMBOL(tcf_generic_walker);
>
> -static struct tc_action *tcf_idr_lookup(u32 index, struct tcf_idrinfo *idrinfo)
> +static bool __tcf_idr_check(struct tc_action_net *tn, u32 index,
> + struct tc_action **a, int bind)
> {
> - struct tc_action *p = NULL;
> + struct tcf_idrinfo *idrinfo = tn->idrinfo;
> + struct tc_action *p;
>
> spin_lock(&idrinfo->lock);
> p = idr_find(&idrinfo->action_idr, index);
> + if (p) {
> + refcount_inc(&p->tcfa_refcnt);
> + if (bind)
> + atomic_inc(&p->tcfa_bindcnt);
> + }
> spin_unlock(&idrinfo->lock);
>
> - return p;
> + if (p) {
> + *a = p;
> + return true;
> + }
> + return false;
> }
>
> int tcf_idr_search(struct tc_action_net *tn, struct tc_action **a, u32 index)
> {
> - struct tcf_idrinfo *idrinfo = tn->idrinfo;
> - struct tc_action *p = tcf_idr_lookup(index, idrinfo);
> -
> - if (p) {
> - *a = p;
> - return 1;
> - }
> - return 0;
> + return __tcf_idr_check(tn, index, a, 0);
> }
> EXPORT_SYMBOL(tcf_idr_search);
>
> bool tcf_idr_check(struct tc_action_net *tn, u32 index, struct tc_action **a,
> int bind)
> {
> - struct tcf_idrinfo *idrinfo = tn->idrinfo;
> - struct tc_action *p = tcf_idr_lookup(index, idrinfo);
> -
> - if (index && p) {
> - if (bind)
> - atomic_inc(&p->tcfa_bindcnt);
> - refcount_inc(&p->tcfa_refcnt);
> - *a = p;
> - return true;
> - }
> - return false;
> + return __tcf_idr_check(tn, index, a, bind);
> }
> EXPORT_SYMBOL(tcf_idr_check);
>
> @@ -932,7 +926,7 @@ tcf_get_notify(struct net *net, u32 portid, struct nlmsghdr *n,
> if (!skb)
> return -ENOBUFS;
> if (tca_get_fill(skb, actions, portid, n->nlmsg_seq, 0, event,
> - 0, 0) <= 0) {
> + 0, 1) <= 0) {
> NL_SET_ERR_MSG(extack, "Failed to fill netlink attributes while adding TC action");
> kfree_skb(skb);
> return -EINVAL;
> @@ -1072,7 +1066,7 @@ tcf_del_notify(struct net *net, struct nlmsghdr *n, struct list_head *actions,
> return -ENOBUFS;
>
> if (tca_get_fill(skb, actions, portid, n->nlmsg_seq, 0, RTM_DELACTION,
> - 0, 1) <= 0) {
> + 0, 2) <= 0) {
> NL_SET_ERR_MSG(extack, "Failed to fill netlink TC action attributes");
> kfree_skb(skb);
> return -EINVAL;
> @@ -1131,14 +1125,14 @@ tca_action_gd(struct net *net, struct nlattr *nla, struct nlmsghdr *n,
> if (event == RTM_GETACTION)
> ret = tcf_get_notify(net, portid, n, &actions, event, extack);
> else { /* delete */
> + cleanup_a(&actions, 1); /* lookup took reference */
> ret = tcf_del_notify(net, n, &actions, portid, attr_size, extack);
> if (ret)
> goto err;
> return ret;
> }
> err:
> - if (event != RTM_GETACTION)
> - tcf_action_destroy(&actions, 0);
> + tcf_action_destroy(&actions, 0);
> return ret;
> }
>
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 05/11] net: sched: implement action API that deletes action by index
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-6-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:23AM +0300, Vlad Buslov wrote:
> Implement new action API function that atomically finds and deletes action
> from idr by index. Intended to be used by lockless actions that do not rely
> on rtnl lock.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Rename tcf_idr_find_delete to tcf_idr_delete_index.
>
> include/net/act_api.h | 1 +
> net/sched/act_api.c | 39 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 40 insertions(+)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index 888ff471bbf6..d94ec6400673 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -153,6 +153,7 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> int bind, bool cpustats);
> void tcf_idr_insert(struct tc_action_net *tn, struct tc_action *a);
>
> +int tcf_idr_delete_index(struct tc_action_net *tn, u32 index);
> int __tcf_idr_release(struct tc_action *a, bool bind, bool strict);
>
> static inline int tcf_idr_release(struct tc_action *a, bool bind)
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index aa304d36fee0..0f31f09946ab 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -319,6 +319,45 @@ bool tcf_idr_check(struct tc_action_net *tn, u32 index, struct tc_action **a,
> }
> EXPORT_SYMBOL(tcf_idr_check);
>
> +int tcf_idr_delete_index(struct tc_action_net *tn, u32 index)
> +{
> + struct tcf_idrinfo *idrinfo = tn->idrinfo;
> + struct tc_action *p;
> + int ret = 0;
> +
> + spin_lock(&idrinfo->lock);
> + p = idr_find(&idrinfo->action_idr, index);
> + if (!p) {
> + spin_unlock(&idrinfo->lock);
> + return -ENOENT;
> + }
> +
> + if (!atomic_read(&p->tcfa_bindcnt)) {
> + if (refcount_dec_and_test(&p->tcfa_refcnt)) {
> + struct module *owner = p->ops->owner;
> +
> + WARN_ON(p != idr_remove(&idrinfo->action_idr,
> + p->tcfa_index));
> + spin_unlock(&idrinfo->lock);
> +
> + if (p->ops->cleanup)
> + p->ops->cleanup(p);
> +
> + gen_kill_estimator(&p->tcfa_rate_est);
> + free_tcf(p);
> + module_put(owner);
> + return 0;
> + }
> + ret = 0;
> + } else {
> + ret = -EPERM;
> + }
> +
> + spin_unlock(&idrinfo->lock);
> + return ret;
> +}
> +EXPORT_SYMBOL(tcf_idr_delete_index);
> +
> int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> struct tc_action **a, const struct tc_action_ops *ops,
> int bind, bool cpustats)
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 06/11] net: sched: add 'delete' function to action ops
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-7-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:24AM +0300, Vlad Buslov wrote:
> Extend action ops with 'delete' function. Each action type to implements
> its own delete function that doesn't depend on rtnl lock.
>
> Implement delete function that is required to delete actions without
> holding rtnl lock. Use action API function that atomically deletes action
> only if it is still in action idr. This implementation prevents concurrent
> threads from deleting same action twice.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Merge action ops delete definition and implementation.
>
> include/net/act_api.h | 1 +
> net/sched/act_bpf.c | 8 ++++++++
> net/sched/act_connmark.c | 8 ++++++++
> net/sched/act_csum.c | 8 ++++++++
> net/sched/act_gact.c | 8 ++++++++
> net/sched/act_ife.c | 8 ++++++++
> net/sched/act_ipt.c | 16 ++++++++++++++++
> net/sched/act_mirred.c | 8 ++++++++
> net/sched/act_nat.c | 8 ++++++++
> net/sched/act_pedit.c | 8 ++++++++
> net/sched/act_police.c | 8 ++++++++
> net/sched/act_sample.c | 8 ++++++++
> net/sched/act_simple.c | 8 ++++++++
> net/sched/act_skbedit.c | 8 ++++++++
> net/sched/act_skbmod.c | 8 ++++++++
> net/sched/act_tunnel_key.c | 8 ++++++++
> net/sched/act_vlan.c | 8 ++++++++
> 17 files changed, 137 insertions(+)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index d94ec6400673..d256e20507b9 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -101,6 +101,7 @@ struct tc_action_ops {
> void (*stats_update)(struct tc_action *, u64, u32, u64);
> size_t (*get_fill_size)(const struct tc_action *act);
> struct net_device *(*get_dev)(const struct tc_action *a);
> + int (*delete)(struct net *net, u32 index);
> };
>
> struct tc_action_net {
> diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
> index 8ebf40a3506c..7941dd66ff83 100644
> --- a/net/sched/act_bpf.c
> +++ b/net/sched/act_bpf.c
> @@ -388,6 +388,13 @@ static int tcf_bpf_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_bpf_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, bpf_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_bpf_ops __read_mostly = {
> .kind = "bpf",
> .type = TCA_ACT_BPF,
> @@ -398,6 +405,7 @@ static struct tc_action_ops act_bpf_ops __read_mostly = {
> .init = tcf_bpf_init,
> .walk = tcf_bpf_walker,
> .lookup = tcf_bpf_search,
> + .delete = tcf_bpf_delete,
> .size = sizeof(struct tcf_bpf),
> };
>
> diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
> index e3787aa0025a..143c2d3de723 100644
> --- a/net/sched/act_connmark.c
> +++ b/net/sched/act_connmark.c
> @@ -193,6 +193,13 @@ static int tcf_connmark_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_connmark_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, connmark_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_connmark_ops = {
> .kind = "connmark",
> .type = TCA_ACT_CONNMARK,
> @@ -202,6 +209,7 @@ static struct tc_action_ops act_connmark_ops = {
> .init = tcf_connmark_init,
> .walk = tcf_connmark_walker,
> .lookup = tcf_connmark_search,
> + .delete = tcf_connmark_delete,
> .size = sizeof(struct tcf_connmark_info),
> };
>
> diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
> index 334261943f9f..3768539340e0 100644
> --- a/net/sched/act_csum.c
> +++ b/net/sched/act_csum.c
> @@ -654,6 +654,13 @@ static size_t tcf_csum_get_fill_size(const struct tc_action *act)
> return nla_total_size(sizeof(struct tc_csum));
> }
>
> +static int tcf_csum_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, csum_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_csum_ops = {
> .kind = "csum",
> .type = TCA_ACT_CSUM,
> @@ -665,6 +672,7 @@ static struct tc_action_ops act_csum_ops = {
> .walk = tcf_csum_walker,
> .lookup = tcf_csum_search,
> .get_fill_size = tcf_csum_get_fill_size,
> + .delete = tcf_csum_delete,
> .size = sizeof(struct tcf_csum),
> };
>
> diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
> index b4dfb2b4addc..a431a711f0dd 100644
> --- a/net/sched/act_gact.c
> +++ b/net/sched/act_gact.c
> @@ -231,6 +231,13 @@ static size_t tcf_gact_get_fill_size(const struct tc_action *act)
> return sz;
> }
>
> +static int tcf_gact_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, gact_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_gact_ops = {
> .kind = "gact",
> .type = TCA_ACT_GACT,
> @@ -242,6 +249,7 @@ static struct tc_action_ops act_gact_ops = {
> .walk = tcf_gact_walker,
> .lookup = tcf_gact_search,
> .get_fill_size = tcf_gact_get_fill_size,
> + .delete = tcf_gact_delete,
> .size = sizeof(struct tcf_gact),
> };
>
> diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
> index 3dccc4e1d378..027c305dcb37 100644
> --- a/net/sched/act_ife.c
> +++ b/net/sched/act_ife.c
> @@ -846,6 +846,13 @@ static int tcf_ife_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_ife_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, ife_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_ife_ops = {
> .kind = "ife",
> .type = TCA_ACT_IFE,
> @@ -856,6 +863,7 @@ static struct tc_action_ops act_ife_ops = {
> .init = tcf_ife_init,
> .walk = tcf_ife_walker,
> .lookup = tcf_ife_search,
> + .delete = tcf_ife_delete,
> .size = sizeof(struct tcf_ife_info),
> };
>
> diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
> index 9c21663a86a6..6c234411c771 100644
> --- a/net/sched/act_ipt.c
> +++ b/net/sched/act_ipt.c
> @@ -324,6 +324,13 @@ static int tcf_ipt_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_ipt_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, ipt_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_ipt_ops = {
> .kind = "ipt",
> .type = TCA_ACT_IPT,
> @@ -334,6 +341,7 @@ static struct tc_action_ops act_ipt_ops = {
> .init = tcf_ipt_init,
> .walk = tcf_ipt_walker,
> .lookup = tcf_ipt_search,
> + .delete = tcf_ipt_delete,
> .size = sizeof(struct tcf_ipt),
> };
>
> @@ -374,6 +382,13 @@ static int tcf_xt_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_xt_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, xt_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_xt_ops = {
> .kind = "xt",
> .type = TCA_ACT_XT,
> @@ -384,6 +399,7 @@ static struct tc_action_ops act_xt_ops = {
> .init = tcf_xt_init,
> .walk = tcf_xt_walker,
> .lookup = tcf_xt_search,
> + .delete = tcf_xt_delete,
> .size = sizeof(struct tcf_ipt),
> };
>
> diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> index 5434f08f2eb7..3d8300bce7e4 100644
> --- a/net/sched/act_mirred.c
> +++ b/net/sched/act_mirred.c
> @@ -322,6 +322,13 @@ static struct net_device *tcf_mirred_get_dev(const struct tc_action *a)
> return rtnl_dereference(m->tcfm_dev);
> }
>
> +static int tcf_mirred_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, mirred_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_mirred_ops = {
> .kind = "mirred",
> .type = TCA_ACT_MIRRED,
> @@ -335,6 +342,7 @@ static struct tc_action_ops act_mirred_ops = {
> .lookup = tcf_mirred_search,
> .size = sizeof(struct tcf_mirred),
> .get_dev = tcf_mirred_get_dev,
> + .delete = tcf_mirred_delete,
> };
>
> static __net_init int mirred_init_net(struct net *net)
> diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
> index e6487ad1e4a8..9eb27c89dc46 100644
> --- a/net/sched/act_nat.c
> +++ b/net/sched/act_nat.c
> @@ -294,6 +294,13 @@ static int tcf_nat_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_nat_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, nat_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_nat_ops = {
> .kind = "nat",
> .type = TCA_ACT_NAT,
> @@ -303,6 +310,7 @@ static struct tc_action_ops act_nat_ops = {
> .init = tcf_nat_init,
> .walk = tcf_nat_walker,
> .lookup = tcf_nat_search,
> + .delete = tcf_nat_delete,
> .size = sizeof(struct tcf_nat),
> };
>
> diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
> index 7c9a3f24edba..b8857035e3f8 100644
> --- a/net/sched/act_pedit.c
> +++ b/net/sched/act_pedit.c
> @@ -436,6 +436,13 @@ static int tcf_pedit_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_pedit_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, pedit_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_pedit_ops = {
> .kind = "pedit",
> .type = TCA_ACT_PEDIT,
> @@ -446,6 +453,7 @@ static struct tc_action_ops act_pedit_ops = {
> .init = tcf_pedit_init,
> .walk = tcf_pedit_walker,
> .lookup = tcf_pedit_search,
> + .delete = tcf_pedit_delete,
> .size = sizeof(struct tcf_pedit),
> };
>
> diff --git a/net/sched/act_police.c b/net/sched/act_police.c
> index 0e1c2fb0ebea..c955fb0d4f3f 100644
> --- a/net/sched/act_police.c
> +++ b/net/sched/act_police.c
> @@ -314,6 +314,13 @@ static int tcf_police_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_police_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, police_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> MODULE_AUTHOR("Alexey Kuznetsov");
> MODULE_DESCRIPTION("Policing actions");
> MODULE_LICENSE("GPL");
> @@ -327,6 +334,7 @@ static struct tc_action_ops act_police_ops = {
> .init = tcf_act_police_init,
> .walk = tcf_act_police_walker,
> .lookup = tcf_police_search,
> + .delete = tcf_police_delete,
> .size = sizeof(struct tcf_police),
> };
>
> diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
> index 316fc645595d..6f79d2afcba2 100644
> --- a/net/sched/act_sample.c
> +++ b/net/sched/act_sample.c
> @@ -220,6 +220,13 @@ static int tcf_sample_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_sample_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, sample_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_sample_ops = {
> .kind = "sample",
> .type = TCA_ACT_SAMPLE,
> @@ -230,6 +237,7 @@ static struct tc_action_ops act_sample_ops = {
> .cleanup = tcf_sample_cleanup,
> .walk = tcf_sample_walker,
> .lookup = tcf_sample_search,
> + .delete = tcf_sample_delete,
> .size = sizeof(struct tcf_sample),
> };
>
> diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
> index 23fa893ea092..b570e7ca7e33 100644
> --- a/net/sched/act_simple.c
> +++ b/net/sched/act_simple.c
> @@ -187,6 +187,13 @@ static int tcf_simp_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_simp_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, simp_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_simp_ops = {
> .kind = "simple",
> .type = TCA_ACT_SIMP,
> @@ -197,6 +204,7 @@ static struct tc_action_ops act_simp_ops = {
> .init = tcf_simp_init,
> .walk = tcf_simp_walker,
> .lookup = tcf_simp_search,
> + .delete = tcf_simp_delete,
> .size = sizeof(struct tcf_defact),
> };
>
> diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
> index 85ed9d603dc1..dc0cb350aa45 100644
> --- a/net/sched/act_skbedit.c
> +++ b/net/sched/act_skbedit.c
> @@ -226,6 +226,13 @@ static int tcf_skbedit_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_skbedit_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, skbedit_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_skbedit_ops = {
> .kind = "skbedit",
> .type = TCA_ACT_SKBEDIT,
> @@ -235,6 +242,7 @@ static struct tc_action_ops act_skbedit_ops = {
> .init = tcf_skbedit_init,
> .walk = tcf_skbedit_walker,
> .lookup = tcf_skbedit_search,
> + .delete = tcf_skbedit_delete,
> .size = sizeof(struct tcf_skbedit),
> };
>
> diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
> index 026d6f58eda1..30be3f767495 100644
> --- a/net/sched/act_skbmod.c
> +++ b/net/sched/act_skbmod.c
> @@ -253,6 +253,13 @@ static int tcf_skbmod_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_skbmod_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, skbmod_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_skbmod_ops = {
> .kind = "skbmod",
> .type = TCA_ACT_SKBMOD,
> @@ -263,6 +270,7 @@ static struct tc_action_ops act_skbmod_ops = {
> .cleanup = tcf_skbmod_cleanup,
> .walk = tcf_skbmod_walker,
> .lookup = tcf_skbmod_search,
> + .delete = tcf_skbmod_delete,
> .size = sizeof(struct tcf_skbmod),
> };
>
> diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
> index ed698fcb0e5a..4b7f9a3b47d7 100644
> --- a/net/sched/act_tunnel_key.c
> +++ b/net/sched/act_tunnel_key.c
> @@ -310,6 +310,13 @@ static int tunnel_key_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tunnel_key_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, tunnel_key_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_tunnel_key_ops = {
> .kind = "tunnel_key",
> .type = TCA_ACT_TUNNEL_KEY,
> @@ -320,6 +327,7 @@ static struct tc_action_ops act_tunnel_key_ops = {
> .cleanup = tunnel_key_release,
> .walk = tunnel_key_walker,
> .lookup = tunnel_key_search,
> + .delete = tunnel_key_delete,
> .size = sizeof(struct tcf_tunnel_key),
> };
>
> diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
> index 72d2d78fb942..b44377c951b6 100644
> --- a/net/sched/act_vlan.c
> +++ b/net/sched/act_vlan.c
> @@ -285,6 +285,13 @@ static int tcf_vlan_search(struct net *net, struct tc_action **a, u32 index,
> return tcf_idr_search(tn, a, index);
> }
>
> +static int tcf_vlan_delete(struct net *net, u32 index)
> +{
> + struct tc_action_net *tn = net_generic(net, vlan_net_id);
> +
> + return tcf_idr_delete_index(tn, index);
> +}
> +
> static struct tc_action_ops act_vlan_ops = {
> .kind = "vlan",
> .type = TCA_ACT_VLAN,
> @@ -295,6 +302,7 @@ static struct tc_action_ops act_vlan_ops = {
> .cleanup = tcf_vlan_cleanup,
> .walk = tcf_vlan_walker,
> .lookup = tcf_vlan_search,
> + .delete = tcf_vlan_delete,
> .size = sizeof(struct tcf_vlan),
> };
>
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 07/11] net: sched: implement reference counted action release
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-8-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:25AM +0300, Vlad Buslov wrote:
> Implement helper delete function that uses new action ops 'delete', instead
> of destroying action directly. This is required so act API could delete
> actions by index, without holding any references to action that is being
> deleted.
>
> Implement function __tcf_action_put() that releases reference to action and
> frees it, if necessary. Refactor action deletion code to use new put
> function and not to rely on rtnl lock. Remove rtnl lock assertions that are
> no longer needed.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Removed redundant actions ops lookup during delete.
> - Assume all actions have delete implemented and don't check for it
> explicitly.
> - Rearrange variable definitions in tcf_action_delete.
>
> net/sched/act_api.c | 84 +++++++++++++++++++++++++++++++++++++++--------------
> net/sched/cls_api.c | 1 -
> 2 files changed, 62 insertions(+), 23 deletions(-)
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 0f31f09946ab..a023873db713 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -90,21 +90,39 @@ static void free_tcf(struct tc_action *p)
> kfree(p);
> }
>
> -static void tcf_idr_remove(struct tcf_idrinfo *idrinfo, struct tc_action *p)
> +static void tcf_action_cleanup(struct tc_action *p)
> {
> - spin_lock(&idrinfo->lock);
> - idr_remove(&idrinfo->action_idr, p->tcfa_index);
> - spin_unlock(&idrinfo->lock);
> + if (p->ops->cleanup)
> + p->ops->cleanup(p);
> +
> gen_kill_estimator(&p->tcfa_rate_est);
> free_tcf(p);
> }
>
> +static int __tcf_action_put(struct tc_action *p, bool bind)
> +{
> + struct tcf_idrinfo *idrinfo = p->idrinfo;
> +
> + if (refcount_dec_and_lock(&p->tcfa_refcnt, &idrinfo->lock)) {
> + if (bind)
> + atomic_dec(&p->tcfa_bindcnt);
> + idr_remove(&idrinfo->action_idr, p->tcfa_index);
> + spin_unlock(&idrinfo->lock);
> +
> + tcf_action_cleanup(p);
> + return 1;
> + }
> +
> + if (bind)
> + atomic_dec(&p->tcfa_bindcnt);
> +
> + return 0;
> +}
> +
> int __tcf_idr_release(struct tc_action *p, bool bind, bool strict)
> {
> int ret = 0;
>
> - ASSERT_RTNL();
> -
> /* Release with strict==1 and bind==0 is only called through act API
> * interface (classifiers always bind). Only case when action with
> * positive reference count and zero bind count can exist is when it was
> @@ -118,18 +136,11 @@ int __tcf_idr_release(struct tc_action *p, bool bind, bool strict)
> * are acceptable.
> */
> if (p) {
> - if (bind)
> - atomic_dec(&p->tcfa_bindcnt);
> - else if (strict && atomic_read(&p->tcfa_bindcnt) > 0)
> + if (!bind && strict && atomic_read(&p->tcfa_bindcnt) > 0)
> return -EPERM;
>
> - if (atomic_read(&p->tcfa_bindcnt) <= 0 &&
> - refcount_dec_and_test(&p->tcfa_refcnt)) {
> - if (p->ops->cleanup)
> - p->ops->cleanup(p);
> - tcf_idr_remove(p->idrinfo, p);
> + if (__tcf_action_put(p, bind))
> ret = ACT_P_DELETED;
> - }
> }
>
> return ret;
> @@ -340,11 +351,7 @@ int tcf_idr_delete_index(struct tc_action_net *tn, u32 index)
> p->tcfa_index));
> spin_unlock(&idrinfo->lock);
>
> - if (p->ops->cleanup)
> - p->ops->cleanup(p);
> -
> - gen_kill_estimator(&p->tcfa_rate_est);
> - free_tcf(p);
> + tcf_action_cleanup(p);
> module_put(owner);
> return 0;
> }
> @@ -615,6 +622,11 @@ int tcf_action_destroy(struct list_head *actions, int bind)
> return ret;
> }
>
> +static int tcf_action_put(struct tc_action *p)
> +{
> + return __tcf_action_put(p, false);
> +}
> +
> int
> tcf_action_dump_old(struct sk_buff *skb, struct tc_action *a, int bind, int ref)
> {
> @@ -1092,6 +1104,35 @@ static int tca_action_flush(struct net *net, struct nlattr *nla,
> return err;
> }
>
> +static int tcf_action_delete(struct net *net, struct list_head *actions,
> + struct netlink_ext_ack *extack)
> +{
> + struct tc_action *a, *tmp;
> + u32 act_index;
> + int ret;
> +
> + list_for_each_entry_safe(a, tmp, actions, list) {
> + const struct tc_action_ops *ops = a->ops;
> +
> + /* Actions can be deleted concurrently so we must save their
> + * type and id to search again after reference is released.
> + */
> + act_index = a->tcfa_index;
> +
> + list_del(&a->list);
> + if (tcf_action_put(a)) {
> + /* last reference, action was deleted concurrently */
> + module_put(ops->owner);
> + } else {
> + /* now do the delete */
> + ret = ops->delete(net, act_index);
> + if (ret < 0)
> + return ret;
> + }
> + }
> + return 0;
> +}
> +
> static int
> tcf_del_notify(struct net *net, struct nlmsghdr *n, struct list_head *actions,
> u32 portid, size_t attr_size, struct netlink_ext_ack *extack)
> @@ -1112,7 +1153,7 @@ tcf_del_notify(struct net *net, struct nlmsghdr *n, struct list_head *actions,
> }
>
> /* now do the delete */
> - ret = tcf_action_destroy(actions, 0);
> + ret = tcf_action_delete(net, actions, extack);
> if (ret < 0) {
> NL_SET_ERR_MSG(extack, "Failed to delete TC action");
> kfree_skb(skb);
> @@ -1164,7 +1205,6 @@ tca_action_gd(struct net *net, struct nlattr *nla, struct nlmsghdr *n,
> if (event == RTM_GETACTION)
> ret = tcf_get_notify(net, portid, n, &actions, event, extack);
> else { /* delete */
> - cleanup_a(&actions, 1); /* lookup took reference */
> ret = tcf_del_notify(net, n, &actions, portid, attr_size, extack);
> if (ret)
> goto err;
> diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
> index efbf01ce14c2..dd37d0ae3fce 100644
> --- a/net/sched/cls_api.c
> +++ b/net/sched/cls_api.c
> @@ -1417,7 +1417,6 @@ void tcf_exts_destroy(struct tcf_exts *exts)
> #ifdef CONFIG_NET_CLS_ACT
> LIST_HEAD(actions);
>
> - ASSERT_RTNL();
> tcf_exts_to_list(exts, &actions);
> tcf_action_destroy(&actions, TCA_ACT_UNBIND);
> kfree(exts->actions);
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 08/11] net: sched: don't release reference on action overwrite
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-9-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:26AM +0300, Vlad Buslov wrote:
> Return from action init function with reference to action taken,
> even when overwriting existing action.
>
> Action init API initializes its fourth argument (pointer to pointer to tc
> action) to either existing action with same index or newly created action.
> In case of existing index(and bind argument is zero), init function returns
> without incrementing action reference counter. Caller of action init then
> proceeds working with action, without actually holding reference to it.
> This means that action could be deleted concurrently.
>
> Change action init behavior to always take reference to action before
> returning successfully, in order to protect from concurrent deletion.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Resplit action lookup/release code to prevent memory leaks in
> individual patches.
> - Change convoluted commit message.
>
> net/sched/act_api.c | 2 --
> net/sched/act_bpf.c | 8 ++++----
> net/sched/act_connmark.c | 5 +++--
> net/sched/act_csum.c | 8 ++++----
> net/sched/act_gact.c | 5 +++--
> net/sched/act_ife.c | 12 +++++-------
> net/sched/act_ipt.c | 5 +++--
> net/sched/act_mirred.c | 5 ++---
> net/sched/act_nat.c | 5 +++--
> net/sched/act_pedit.c | 5 +++--
> net/sched/act_police.c | 8 +++-----
> net/sched/act_sample.c | 8 +++-----
> net/sched/act_simple.c | 5 +++--
> net/sched/act_skbedit.c | 5 +++--
> net/sched/act_skbmod.c | 8 +++-----
> net/sched/act_tunnel_key.c | 8 +++-----
> net/sched/act_vlan.c | 8 +++-----
> 17 files changed, 51 insertions(+), 59 deletions(-)
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index a023873db713..f019f0464cec 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -870,8 +870,6 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> }
> act->order = i;
> sz += tcf_action_fill_size(act);
> - if (ovr)
> - refcount_inc(&act->tcfa_refcnt);
> list_add_tail(&act->list, actions);
> }
>
> diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
> index 7941dd66ff83..d3f4ac6f2c4b 100644
> --- a/net/sched/act_bpf.c
> +++ b/net/sched/act_bpf.c
> @@ -311,9 +311,10 @@ static int tcf_bpf_init(struct net *net, struct nlattr *nla,
> if (bind)
> return 0;
>
> - tcf_idr_release(*act, bind);
> - if (!replace)
> + if (!replace) {
> + tcf_idr_release(*act, bind);
> return -EEXIST;
> + }
> }
>
> is_bpf = tb[TCA_ACT_BPF_OPS_LEN] && tb[TCA_ACT_BPF_OPS];
> @@ -356,8 +357,7 @@ static int tcf_bpf_init(struct net *net, struct nlattr *nla,
>
> return res;
> out:
> - if (res == ACT_P_CREATED)
> - tcf_idr_release(*act, bind);
> + tcf_idr_release(*act, bind);
>
> return ret;
> }
> diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
> index 143c2d3de723..701e90244eff 100644
> --- a/net/sched/act_connmark.c
> +++ b/net/sched/act_connmark.c
> @@ -135,9 +135,10 @@ static int tcf_connmark_init(struct net *net, struct nlattr *nla,
> ci = to_connmark(*a);
> if (bind)
> return 0;
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> /* replacing action and zone */
> ci->tcf_action = parm->action;
> ci->zone = parm->zone;
> diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
> index 3768539340e0..5dbee136b0a1 100644
> --- a/net/sched/act_csum.c
> +++ b/net/sched/act_csum.c
> @@ -76,9 +76,10 @@ static int tcf_csum_init(struct net *net, struct nlattr *nla,
> } else {
> if (bind)/* dont override defaults */
> return 0;
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> }
>
> p = to_tcf_csum(*a);
> @@ -86,8 +87,7 @@ static int tcf_csum_init(struct net *net, struct nlattr *nla,
>
> params_new = kzalloc(sizeof(*params_new), GFP_KERNEL);
> if (unlikely(!params_new)) {
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return -ENOMEM;
> }
> params_old = rtnl_dereference(p->params);
> diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
> index a431a711f0dd..11c4de3f344e 100644
> --- a/net/sched/act_gact.c
> +++ b/net/sched/act_gact.c
> @@ -100,9 +100,10 @@ static int tcf_gact_init(struct net *net, struct nlattr *nla,
> } else {
> if (bind)/* dont override defaults */
> return 0;
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> }
>
> gact = to_gact(*a);
> diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
> index 027c305dcb37..3dd3d79c5a4b 100644
> --- a/net/sched/act_ife.c
> +++ b/net/sched/act_ife.c
> @@ -497,12 +497,10 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
> return ret;
> }
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr) {
> - kfree(p);
> - return -EEXIST;
> - }
> + kfree(p);
> + return -EEXIST;
> }
>
> ife = to_ife(*a);
> @@ -544,13 +542,13 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
> NULL, NULL);
> if (err) {
> metadata_parse_err:
> - if (exists)
> - tcf_idr_release(*a, bind);
> if (ret == ACT_P_CREATED)
> _tcf_ife_cleanup(*a);
>
> if (exists)
> spin_unlock_bh(&ife->tcf_lock);
> + tcf_idr_release(*a, bind);
> +
> kfree(p);
> return err;
> }
> diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
> index 6c234411c771..85e85dfba401 100644
> --- a/net/sched/act_ipt.c
> +++ b/net/sched/act_ipt.c
> @@ -145,10 +145,11 @@ static int __tcf_ipt_init(struct net *net, unsigned int id, struct nlattr *nla,
> } else {
> if (bind)/* dont override defaults */
> return 0;
> - tcf_idr_release(*a, bind);
>
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> }
> hook = nla_get_u32(tb[TCA_IPT_HOOK]);
>
> diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> index 3d8300bce7e4..e08aed06d7f8 100644
> --- a/net/sched/act_mirred.c
> +++ b/net/sched/act_mirred.c
> @@ -132,10 +132,9 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> if (ret)
> return ret;
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
> m = to_mirred(*a);
>
> diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
> index 9eb27c89dc46..1f91e8e66c0f 100644
> --- a/net/sched/act_nat.c
> +++ b/net/sched/act_nat.c
> @@ -66,9 +66,10 @@ static int tcf_nat_init(struct net *net, struct nlattr *nla, struct nlattr *est,
> } else {
> if (bind)
> return 0;
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> }
> p = to_tcf_nat(*a);
>
> diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
> index b8857035e3f8..fbf283f2ac34 100644
> --- a/net/sched/act_pedit.c
> +++ b/net/sched/act_pedit.c
> @@ -185,9 +185,10 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla,
> } else {
> if (bind)
> return 0;
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> p = to_pedit(*a);
> if (p->tcfp_nkeys && p->tcfp_nkeys != parm->nkeys) {
> keys = kmalloc(ksize, GFP_KERNEL);
> diff --git a/net/sched/act_police.c b/net/sched/act_police.c
> index c955fb0d4f3f..99335cca739e 100644
> --- a/net/sched/act_police.c
> +++ b/net/sched/act_police.c
> @@ -111,10 +111,9 @@ static int tcf_act_police_init(struct net *net, struct nlattr *nla,
> if (ret)
> return ret;
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
>
> police = to_police(*a);
> @@ -195,8 +194,7 @@ static int tcf_act_police_init(struct net *net, struct nlattr *nla,
> failure:
> qdisc_put_rtab(P_tab);
> qdisc_put_rtab(R_tab);
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return err;
> }
>
> diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
> index 6f79d2afcba2..a8582e1347db 100644
> --- a/net/sched/act_sample.c
> +++ b/net/sched/act_sample.c
> @@ -69,10 +69,9 @@ static int tcf_sample_init(struct net *net, struct nlattr *nla,
> if (ret)
> return ret;
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
> s = to_sample(*a);
>
> @@ -81,8 +80,7 @@ static int tcf_sample_init(struct net *net, struct nlattr *nla,
> s->psample_group_num = nla_get_u32(tb[TCA_SAMPLE_PSAMPLE_GROUP]);
> psample_group = psample_group_get(net, s->psample_group_num);
> if (!psample_group) {
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return -ENOMEM;
> }
> RCU_INIT_POINTER(s->psample_group, psample_group);
> diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
> index b570e7ca7e33..78fffd329ed9 100644
> --- a/net/sched/act_simple.c
> +++ b/net/sched/act_simple.c
> @@ -130,9 +130,10 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
> } else {
> d = to_defact(*a);
>
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
>
> reset_policy(d, defdata, parm);
> }
> diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
> index dc0cb350aa45..c0607d1319eb 100644
> --- a/net/sched/act_skbedit.c
> +++ b/net/sched/act_skbedit.c
> @@ -137,9 +137,10 @@ static int tcf_skbedit_init(struct net *net, struct nlattr *nla,
> ret = ACT_P_CREATED;
> } else {
> d = to_skbedit(*a);
> - tcf_idr_release(*a, bind);
> - if (!ovr)
> + if (!ovr) {
> + tcf_idr_release(*a, bind);
> return -EEXIST;
> + }
> }
>
> spin_lock_bh(&d->tcf_lock);
> diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
> index 30be3f767495..e844381af066 100644
> --- a/net/sched/act_skbmod.c
> +++ b/net/sched/act_skbmod.c
> @@ -145,10 +145,9 @@ static int tcf_skbmod_init(struct net *net, struct nlattr *nla,
> return ret;
>
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
>
> d = to_skbmod(*a);
> @@ -156,8 +155,7 @@ static int tcf_skbmod_init(struct net *net, struct nlattr *nla,
> ASSERT_RTNL();
> p = kzalloc(sizeof(struct tcf_skbmod_params), GFP_KERNEL);
> if (unlikely(!p)) {
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return -ENOMEM;
> }
>
> diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
> index 4b7f9a3b47d7..bd53f39a345b 100644
> --- a/net/sched/act_tunnel_key.c
> +++ b/net/sched/act_tunnel_key.c
> @@ -165,10 +165,9 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
> return ret;
>
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
>
> t = to_tunnel_key(*a);
> @@ -176,8 +175,7 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
> ASSERT_RTNL();
> params_new = kzalloc(sizeof(*params_new), GFP_KERNEL);
> if (unlikely(!params_new)) {
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return -ENOMEM;
> }
>
> diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
> index b44377c951b6..4ac0d565e437 100644
> --- a/net/sched/act_vlan.c
> +++ b/net/sched/act_vlan.c
> @@ -185,10 +185,9 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> return ret;
>
> ret = ACT_P_CREATED;
> - } else {
> + } else if (!ovr) {
> tcf_idr_release(*a, bind);
> - if (!ovr)
> - return -EEXIST;
> + return -EEXIST;
> }
>
> v = to_vlan(*a);
> @@ -196,8 +195,7 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> ASSERT_RTNL();
> p = kzalloc(sizeof(*p), GFP_KERNEL);
> if (!p) {
> - if (ret == ACT_P_CREATED)
> - tcf_idr_release(*a, bind);
> + tcf_idr_release(*a, bind);
> return -ENOMEM;
> }
>
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 09/11] net: sched: use reference counting action init
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-10-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:27AM +0300, Vlad Buslov wrote:
> Change action API to assume that action init function always takes
> reference to action, even when overwriting existing action. This is
> necessary because action API continues to use action pointer after init
> function is done. At this point action becomes accessible for concurrent
> modifications, so user must always hold reference to it.
>
> Implement helper put list function to atomically release list of actions
> after action API init code is done using them.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Resplit action lookup/release code to prevent memory leaks in
> individual patches.
>
> net/sched/act_api.c | 35 +++++++++++++++++------------------
> 1 file changed, 17 insertions(+), 18 deletions(-)
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index f019f0464cec..eefe8c2fe667 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -627,6 +627,18 @@ static int tcf_action_put(struct tc_action *p)
> return __tcf_action_put(p, false);
> }
>
> +static void tcf_action_put_lst(struct list_head *actions)
> +{
> + struct tc_action *a, *tmp;
> +
> + list_for_each_entry_safe(a, tmp, actions, list) {
> + const struct tc_action_ops *ops = a->ops;
> +
> + if (tcf_action_put(a))
> + module_put(ops->owner);
> + }
> +}
> +
> int
> tcf_action_dump_old(struct sk_buff *skb, struct tc_action *a, int bind, int ref)
> {
> @@ -835,17 +847,6 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp,
> return ERR_PTR(err);
> }
>
> -static void cleanup_a(struct list_head *actions, int ovr)
> -{
> - struct tc_action *a;
> -
> - if (!ovr)
> - return;
> -
> - list_for_each_entry(a, actions, list)
> - refcount_dec(&a->tcfa_refcnt);
> -}
> -
> int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> struct nlattr *est, char *name, int ovr, int bind,
> struct list_head *actions, size_t *attr_size,
> @@ -874,11 +875,6 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,
> }
>
> *attr_size = tcf_action_full_attrs_size(sz);
> -
> - /* Remove the temp refcnt which was necessary to protect against
> - * destroying an existing action which was being replaced
> - */
> - cleanup_a(actions, ovr);
> return 0;
>
> err:
> @@ -1209,7 +1205,7 @@ tca_action_gd(struct net *net, struct nlattr *nla, struct nlmsghdr *n,
> return ret;
> }
> err:
> - tcf_action_destroy(&actions, 0);
> + tcf_action_put_lst(&actions);
> return ret;
> }
>
> @@ -1251,8 +1247,11 @@ static int tcf_action_add(struct net *net, struct nlattr *nla,
> &attr_size, true, extack);
> if (ret)
> return ret;
> + ret = tcf_add_notify(net, n, &actions, portid, attr_size, extack);
> + if (ovr)
> + tcf_action_put_lst(&actions);
>
> - return tcf_add_notify(net, n, &actions, portid, attr_size, extack);
> + return ret;
> }
>
> static u32 tcaa_root_flags_allowed = TCA_FLAG_LARGE_DUMP_ON;
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH v3 10/11] net: sched: atomically check-allocate action
From: Marcelo Ricardo Leitner @ 2018-05-28 21:38 UTC (permalink / raw)
To: Vlad Buslov
Cc: jiri, netdev, jhs, xiyou.wangcong, davem, ast, daniel, kliteyn
In-Reply-To: <1527455849-22327-11-git-send-email-vladbu@mellanox.com>
On Mon, May 28, 2018 at 12:17:28AM +0300, Vlad Buslov wrote:
> Implement function that atomically checks if action exists and either takes
> reference to it, or allocates idr slot for action index to prevent
> concurrent allocations of actions with same index. Use EBUSY error pointer
> to indicate that idr slot is reserved.
>
> Implement cleanup helper function that removes temporary error pointer from
> idr. (in case of error between idr allocation and insertion of newly
> created action to specified index)
>
> Refactor all action init functions to insert new action to idr using this
> API.
>
> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Reviewed-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
> ---
> Changes from V1 to V2:
> - Remove unique idr insertion function. Change original idr insert to do
> the same thing.
> - Refactor action check-alloc code into standalone function.
>
> include/net/act_api.h | 3 ++
> net/sched/act_api.c | 92 ++++++++++++++++++++++++++++++++++++----------
> net/sched/act_bpf.c | 11 ++++--
> net/sched/act_connmark.c | 10 +++--
> net/sched/act_csum.c | 11 ++++--
> net/sched/act_gact.c | 11 ++++--
> net/sched/act_ife.c | 6 ++-
> net/sched/act_ipt.c | 13 ++++++-
> net/sched/act_mirred.c | 16 ++++++--
> net/sched/act_nat.c | 11 ++++--
> net/sched/act_pedit.c | 15 ++++++--
> net/sched/act_police.c | 9 ++++-
> net/sched/act_sample.c | 11 ++++--
> net/sched/act_simple.c | 11 +++++-
> net/sched/act_skbedit.c | 11 +++++-
> net/sched/act_skbmod.c | 11 +++++-
> net/sched/act_tunnel_key.c | 9 ++++-
> net/sched/act_vlan.c | 17 ++++++++-
> 18 files changed, 218 insertions(+), 60 deletions(-)
>
> diff --git a/include/net/act_api.h b/include/net/act_api.h
> index d256e20507b9..cd4547476074 100644
> --- a/include/net/act_api.h
> +++ b/include/net/act_api.h
> @@ -154,6 +154,9 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> int bind, bool cpustats);
> void tcf_idr_insert(struct tc_action_net *tn, struct tc_action *a);
>
> +void tcf_idr_cleanup(struct tc_action_net *tn, u32 index);
> +int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,
> + struct tc_action **a, int bind);
> int tcf_idr_delete_index(struct tc_action_net *tn, u32 index);
> int __tcf_idr_release(struct tc_action *a, bool bind, bool strict);
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index eefe8c2fe667..9511502e1cbb 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -303,7 +303,9 @@ static bool __tcf_idr_check(struct tc_action_net *tn, u32 index,
>
> spin_lock(&idrinfo->lock);
> p = idr_find(&idrinfo->action_idr, index);
> - if (p) {
> + if (IS_ERR(p)) {
> + p = NULL;
> + } else if (p) {
> refcount_inc(&p->tcfa_refcnt);
> if (bind)
> atomic_inc(&p->tcfa_bindcnt);
> @@ -371,7 +373,6 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> {
> struct tc_action *p = kzalloc(ops->size, GFP_KERNEL);
> struct tcf_idrinfo *idrinfo = tn->idrinfo;
> - struct idr *idr = &idrinfo->action_idr;
> int err = -ENOMEM;
>
> if (unlikely(!p))
> @@ -389,20 +390,6 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> goto err2;
> }
> spin_lock_init(&p->tcfa_lock);
> - idr_preload(GFP_KERNEL);
> - spin_lock(&idrinfo->lock);
> - /* user doesn't specify an index */
> - if (!index) {
> - index = 1;
> - err = idr_alloc_u32(idr, NULL, &index, UINT_MAX, GFP_ATOMIC);
> - } else {
> - err = idr_alloc_u32(idr, NULL, &index, index, GFP_ATOMIC);
> - }
> - spin_unlock(&idrinfo->lock);
> - idr_preload_end();
> - if (err)
> - goto err3;
> -
> p->tcfa_index = index;
> p->tcfa_tm.install = jiffies;
> p->tcfa_tm.lastuse = jiffies;
> @@ -412,7 +399,7 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> &p->tcfa_rate_est,
> &p->tcfa_lock, NULL, est);
> if (err)
> - goto err4;
> + goto err3;
> }
>
> p->idrinfo = idrinfo;
> @@ -420,8 +407,6 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
> INIT_LIST_HEAD(&p->list);
> *a = p;
> return 0;
> -err4:
> - idr_remove(idr, index);
> err3:
> free_percpu(p->cpu_qstats);
> err2:
> @@ -437,11 +422,78 @@ void tcf_idr_insert(struct tc_action_net *tn, struct tc_action *a)
> struct tcf_idrinfo *idrinfo = tn->idrinfo;
>
> spin_lock(&idrinfo->lock);
> - idr_replace(&idrinfo->action_idr, a, a->tcfa_index);
> + /* Replace ERR_PTR(-EBUSY) allocated by tcf_idr_check_alloc */
> + WARN_ON(!IS_ERR(idr_replace(&idrinfo->action_idr, a, a->tcfa_index)));
> spin_unlock(&idrinfo->lock);
> }
> EXPORT_SYMBOL(tcf_idr_insert);
>
> +/* Cleanup idr index that was allocated but not initialized. */
> +
> +void tcf_idr_cleanup(struct tc_action_net *tn, u32 index)
> +{
> + struct tcf_idrinfo *idrinfo = tn->idrinfo;
> +
> + spin_lock(&idrinfo->lock);
> + /* Remove ERR_PTR(-EBUSY) allocated by tcf_idr_check_alloc */
> + WARN_ON(!IS_ERR(idr_remove(&idrinfo->action_idr, index)));
> + spin_unlock(&idrinfo->lock);
> +}
> +EXPORT_SYMBOL(tcf_idr_cleanup);
> +
> +/* Check if action with specified index exists. If actions is found, increments
> + * its reference and bind counters, and return 1. Otherwise insert temporary
> + * error pointer (to prevent concurrent users from inserting actions with same
> + * index) and return 0.
> + */
> +
> +int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,
> + struct tc_action **a, int bind)
> +{
> + struct tcf_idrinfo *idrinfo = tn->idrinfo;
> + struct tc_action *p;
> + int ret;
> +
> +again:
> + spin_lock(&idrinfo->lock);
> + if (*index) {
> + p = idr_find(&idrinfo->action_idr, *index);
> + if (IS_ERR(p)) {
> + /* This means that another process allocated
> + * index but did not assign the pointer yet.
> + */
> + spin_unlock(&idrinfo->lock);
> + goto again;
> + }
> +
> + if (p) {
> + refcount_inc(&p->tcfa_refcnt);
> + if (bind)
> + atomic_inc(&p->tcfa_bindcnt);
> + *a = p;
> + ret = 1;
> + } else {
> + *a = NULL;
> + ret = idr_alloc_u32(&idrinfo->action_idr, NULL, index,
> + *index, GFP_ATOMIC);
> + if (!ret)
> + idr_replace(&idrinfo->action_idr,
> + ERR_PTR(-EBUSY), *index);
> + }
> + } else {
> + *index = 1;
> + *a = NULL;
> + ret = idr_alloc_u32(&idrinfo->action_idr, NULL, index,
> + UINT_MAX, GFP_ATOMIC);
> + if (!ret)
> + idr_replace(&idrinfo->action_idr, ERR_PTR(-EBUSY),
> + *index);
> + }
> + spin_unlock(&idrinfo->lock);
> + return ret;
> +}
> +EXPORT_SYMBOL(tcf_idr_check_alloc);
> +
> void tcf_idrinfo_destroy(const struct tc_action_ops *ops,
> struct tcf_idrinfo *idrinfo)
> {
> diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
> index d3f4ac6f2c4b..06f743d8ed41 100644
> --- a/net/sched/act_bpf.c
> +++ b/net/sched/act_bpf.c
> @@ -299,14 +299,17 @@ static int tcf_bpf_init(struct net *net, struct nlattr *nla,
>
> parm = nla_data(tb[TCA_ACT_BPF_PARMS]);
>
> - if (!tcf_idr_check(tn, parm->index, act, bind)) {
> + ret = tcf_idr_check_alloc(tn, &parm->index, act, bind);
> + if (!ret) {
> ret = tcf_idr_create(tn, parm->index, est, act,
> &act_bpf_ops, bind, true);
> - if (ret < 0)
> + if (ret < 0) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> res = ACT_P_CREATED;
> - } else {
> + } else if (ret > 0) {
> /* Don't override defaults. */
> if (bind)
> return 0;
> @@ -315,6 +318,8 @@ static int tcf_bpf_init(struct net *net, struct nlattr *nla,
> tcf_idr_release(*act, bind);
> return -EEXIST;
> }
> + } else {
> + return ret;
> }
>
> is_bpf = tb[TCA_ACT_BPF_OPS_LEN] && tb[TCA_ACT_BPF_OPS];
> diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
> index 701e90244eff..1e31f0e448e2 100644
> --- a/net/sched/act_connmark.c
> +++ b/net/sched/act_connmark.c
> @@ -118,11 +118,14 @@ static int tcf_connmark_init(struct net *net, struct nlattr *nla,
>
> parm = nla_data(tb[TCA_CONNMARK_PARMS]);
>
> - if (!tcf_idr_check(tn, parm->index, a, bind)) {
> + ret = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (!ret) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_connmark_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> ci = to_connmark(*a);
> ci->tcf_action = parm->action;
> @@ -131,7 +134,7 @@ static int tcf_connmark_init(struct net *net, struct nlattr *nla,
>
> tcf_idr_insert(tn, *a);
> ret = ACT_P_CREATED;
> - } else {
> + } else if (ret > 0) {
> ci = to_connmark(*a);
> if (bind)
> return 0;
> @@ -142,6 +145,7 @@ static int tcf_connmark_init(struct net *net, struct nlattr *nla,
> /* replacing action and zone */
> ci->tcf_action = parm->action;
> ci->zone = parm->zone;
> + ret = 0;
> }
>
> return ret;
> diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
> index 5dbee136b0a1..bd232d3bd022 100644
> --- a/net/sched/act_csum.c
> +++ b/net/sched/act_csum.c
> @@ -67,19 +67,24 @@ static int tcf_csum_init(struct net *net, struct nlattr *nla,
> return -EINVAL;
> parm = nla_data(tb[TCA_CSUM_PARMS]);
>
> - if (!tcf_idr_check(tn, parm->index, a, bind)) {
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (!err) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_csum_ops, bind, true);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> - } else {
> + } else if (err > 0) {
> if (bind)/* dont override defaults */
> return 0;
> if (!ovr) {
> tcf_idr_release(*a, bind);
> return -EEXIST;
> }
> + } else {
> + return err;
> }
>
> p = to_tcf_csum(*a);
> diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
> index 11c4de3f344e..661b72b9147d 100644
> --- a/net/sched/act_gact.c
> +++ b/net/sched/act_gact.c
> @@ -91,19 +91,24 @@ static int tcf_gact_init(struct net *net, struct nlattr *nla,
> }
> #endif
>
> - if (!tcf_idr_check(tn, parm->index, a, bind)) {
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (!err) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_gact_ops, bind, true);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> - } else {
> + } else if (err > 0) {
> if (bind)/* dont override defaults */
> return 0;
> if (!ovr) {
> tcf_idr_release(*a, bind);
> return -EEXIST;
> }
> + } else {
> + return err;
> }
>
> gact = to_gact(*a);
> diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
> index 3dd3d79c5a4b..5bf0e79796c0 100644
> --- a/net/sched/act_ife.c
> +++ b/net/sched/act_ife.c
> @@ -483,7 +483,10 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
> if (!p)
> return -ENOMEM;
>
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind) {
> kfree(p);
> return 0;
> @@ -493,6 +496,7 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
> ret = tcf_idr_create(tn, parm->index, est, a, &act_ife_ops,
> bind, true);
> if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> kfree(p);
> return ret;
> }
> diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
> index 85e85dfba401..0dc787a57798 100644
> --- a/net/sched/act_ipt.c
> +++ b/net/sched/act_ipt.c
> @@ -119,13 +119,18 @@ static int __tcf_ipt_init(struct net *net, unsigned int id, struct nlattr *nla,
> if (tb[TCA_IPT_INDEX] != NULL)
> index = nla_get_u32(tb[TCA_IPT_INDEX]);
>
> - exists = tcf_idr_check(tn, index, a, bind);
> + err = tcf_idr_check_alloc(tn, &index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (tb[TCA_IPT_HOOK] == NULL || tb[TCA_IPT_TARG] == NULL) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, index);
> return -EINVAL;
> }
>
> @@ -133,14 +138,18 @@ static int __tcf_ipt_init(struct net *net, unsigned int id, struct nlattr *nla,
> if (nla_len(tb[TCA_IPT_TARG]) < td->u.target_size) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, index);
> return -EINVAL;
> }
>
> if (!exists) {
> ret = tcf_idr_create(tn, index, est, a, ops, bind,
> false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> } else {
> if (bind)/* dont override defaults */
> diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
> index e08aed06d7f8..6afd89a36c69 100644
> --- a/net/sched/act_mirred.c
> +++ b/net/sched/act_mirred.c
> @@ -79,7 +79,7 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> struct tcf_mirred *m;
> struct net_device *dev;
> bool exists = false;
> - int ret;
> + int ret, err;
>
> if (!nla) {
> NL_SET_ERR_MSG_MOD(extack, "Mirred requires attributes to be passed");
> @@ -94,7 +94,10 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> }
> parm = nla_data(tb[TCA_MIRRED_PARMS]);
>
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> @@ -107,6 +110,8 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> default:
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> NL_SET_ERR_MSG_MOD(extack, "Unknown mirred option");
> return -EINVAL;
> }
> @@ -115,6 +120,8 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
> if (dev == NULL) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -ENODEV;
> }
> mac_header_xmit = dev_is_mac_header_xmit(dev);
> @@ -124,13 +131,16 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
>
> if (!exists) {
> if (!dev) {
> + tcf_idr_cleanup(tn, parm->index);
> NL_SET_ERR_MSG_MOD(extack, "Specified device does not exist");
> return -EINVAL;
> }
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_mirred_ops, bind, true);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> tcf_idr_release(*a, bind);
> diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
> index 1f91e8e66c0f..4dd9188a72fd 100644
> --- a/net/sched/act_nat.c
> +++ b/net/sched/act_nat.c
> @@ -57,19 +57,24 @@ static int tcf_nat_init(struct net *net, struct nlattr *nla, struct nlattr *est,
> return -EINVAL;
> parm = nla_data(tb[TCA_NAT_PARMS]);
>
> - if (!tcf_idr_check(tn, parm->index, a, bind)) {
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (!err) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_nat_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> - } else {
> + } else if (err > 0) {
> if (bind)
> return 0;
> if (!ovr) {
> tcf_idr_release(*a, bind);
> return -EEXIST;
> }
> + } else {
> + return err;
> }
> p = to_tcf_nat(*a);
>
> diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
> index fbf283f2ac34..2bd1d3f61488 100644
> --- a/net/sched/act_pedit.c
> +++ b/net/sched/act_pedit.c
> @@ -167,13 +167,18 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla,
> if (IS_ERR(keys_ex))
> return PTR_ERR(keys_ex);
>
> - if (!tcf_idr_check(tn, parm->index, a, bind)) {
> - if (!parm->nkeys)
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (!err) {
> + if (!parm->nkeys) {
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> + }
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_pedit_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> p = to_pedit(*a);
> keys = kmalloc(ksize, GFP_KERNEL);
> if (keys == NULL) {
> @@ -182,7 +187,7 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla,
> return -ENOMEM;
> }
> ret = ACT_P_CREATED;
> - } else {
> + } else if (err > 0) {
> if (bind)
> return 0;
> if (!ovr) {
> @@ -197,6 +202,8 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla,
> return -ENOMEM;
> }
> }
> + } else {
> + return err;
> }
>
> spin_lock_bh(&p->tcf_lock);
> diff --git a/net/sched/act_police.c b/net/sched/act_police.c
> index 99335cca739e..1f3192ea8df7 100644
> --- a/net/sched/act_police.c
> +++ b/net/sched/act_police.c
> @@ -101,15 +101,20 @@ static int tcf_act_police_init(struct net *net, struct nlattr *nla,
> return -EINVAL;
>
> parm = nla_data(tb[TCA_POLICE_TBF]);
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, NULL, a,
> &act_police_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> tcf_idr_release(*a, bind);
> diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
> index a8582e1347db..3079e7be5bde 100644
> --- a/net/sched/act_sample.c
> +++ b/net/sched/act_sample.c
> @@ -46,7 +46,7 @@ static int tcf_sample_init(struct net *net, struct nlattr *nla,
> struct tc_sample *parm;
> struct tcf_sample *s;
> bool exists = false;
> - int ret;
> + int ret, err;
>
> if (!nla)
> return -EINVAL;
> @@ -59,15 +59,20 @@ static int tcf_sample_init(struct net *net, struct nlattr *nla,
>
> parm = nla_data(tb[TCA_SAMPLE_PARMS]);
>
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_sample_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> tcf_idr_release(*a, bind);
> diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
> index 78fffd329ed9..2cc874f791df 100644
> --- a/net/sched/act_simple.c
> +++ b/net/sched/act_simple.c
> @@ -101,13 +101,18 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
> return -EINVAL;
>
> parm = nla_data(tb[TCA_DEF_PARMS]);
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (tb[TCA_DEF_DATA] == NULL) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> }
>
> @@ -116,8 +121,10 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_simp_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> d = to_defact(*a);
> ret = alloc_defdata(d, defdata);
> diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
> index c0607d1319eb..29a15172a99d 100644
> --- a/net/sched/act_skbedit.c
> +++ b/net/sched/act_skbedit.c
> @@ -117,21 +117,28 @@ static int tcf_skbedit_init(struct net *net, struct nlattr *nla,
>
> parm = nla_data(tb[TCA_SKBEDIT_PARMS]);
>
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (!flags) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> }
>
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_skbedit_ops, bind, false);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> d = to_skbedit(*a);
> ret = ACT_P_CREATED;
> diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
> index e844381af066..cdc6bacfb190 100644
> --- a/net/sched/act_skbmod.c
> +++ b/net/sched/act_skbmod.c
> @@ -128,21 +128,28 @@ static int tcf_skbmod_init(struct net *net, struct nlattr *nla,
> if (parm->flags & SKBMOD_F_SWAPMAC)
> lflags = SKBMOD_F_SWAPMAC;
>
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> if (!lflags) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> }
>
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_skbmod_ops, bind, true);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
> index bd53f39a345b..4679b620af12 100644
> --- a/net/sched/act_tunnel_key.c
> +++ b/net/sched/act_tunnel_key.c
> @@ -99,7 +99,10 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
> return -EINVAL;
>
> parm = nla_data(tb[TCA_TUNNEL_KEY_PARMS]);
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> @@ -162,7 +165,7 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_tunnel_key_ops, bind, true);
> if (ret)
> - return ret;
> + goto err_out;
>
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> @@ -198,6 +201,8 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
> err_out:
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> }
>
> diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
> index 4ac0d565e437..bae9822837d7 100644
> --- a/net/sched/act_vlan.c
> +++ b/net/sched/act_vlan.c
> @@ -134,7 +134,10 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> if (!tb[TCA_VLAN_PARMS])
> return -EINVAL;
> parm = nla_data(tb[TCA_VLAN_PARMS]);
> - exists = tcf_idr_check(tn, parm->index, a, bind);
> + err = tcf_idr_check_alloc(tn, &parm->index, a, bind);
> + if (err < 0)
> + return err;
> + exists = err;
> if (exists && bind)
> return 0;
>
> @@ -146,12 +149,16 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> if (!tb[TCA_VLAN_PUSH_VLAN_ID]) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> }
> push_vid = nla_get_u16(tb[TCA_VLAN_PUSH_VLAN_ID]);
> if (push_vid >= VLAN_VID_MASK) {
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -ERANGE;
> }
>
> @@ -162,6 +169,8 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> case htons(ETH_P_8021AD):
> break;
> default:
> + if (!exists)
> + tcf_idr_cleanup(tn, parm->index);
> return -EPROTONOSUPPORT;
> }
> } else {
> @@ -174,6 +183,8 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> default:
> if (exists)
> tcf_idr_release(*a, bind);
> + else
> + tcf_idr_cleanup(tn, parm->index);
> return -EINVAL;
> }
> action = parm->v_action;
> @@ -181,8 +192,10 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla,
> if (!exists) {
> ret = tcf_idr_create(tn, parm->index, est, a,
> &act_vlan_ops, bind, true);
> - if (ret)
> + if (ret) {
> + tcf_idr_cleanup(tn, parm->index);
> return ret;
> + }
>
> ret = ACT_P_CREATED;
> } else if (!ovr) {
> --
> 2.7.5
>
^ permalink raw reply
* Re: [PATCH] iptables-compat: homogenize error message
From: Pablo Neira Ayuso @ 2018-05-28 22:04 UTC (permalink / raw)
To: Arushi Singhal; +Cc: netfilter-devel, coreteam, netdev, linux-kernel
In-Reply-To: <20180527130901.GA18320@seema-Inspiron-15-3567>
On Sun, May 27, 2018 at 06:39:01PM +0530, Arushi Singhal wrote:
> There is a difference between error messages in iptables and
> iptables-compat:
>
> #sudo iptables-compat -D INPUT 4
> iptables: No chain/target/match by that name.
>
> #sudo iptables -D INPUT 4
> iptables: Index of deletion too big.
>
> Now, will show same error message.
Applied, thanks Arushi.
P.S: Just Cc: netfilter-devel@vger.kernel.org next time. Thanks.
^ permalink raw reply
* Re: [PATCH net-next v16 5/8] sch_cake: Add NAT awareness to packet classifier
From: Toke Høiland-Jørgensen @ 2018-05-28 22:19 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netdev, cake, netfilter-devel
In-Reply-To: <20180528195101.7mceghxngz5jhawa@salvia>
Pablo Neira Ayuso <pablo@netfilter.org> writes:
> On Mon, May 28, 2018 at 04:27:46PM +0200, Toke Høiland-Jørgensen wrote:
>> When CAKE is deployed on a gateway that also performs NAT (which is a
>> common deployment mode), the host fairness mechanism cannot distinguish
>> internal hosts from each other, and so fails to work correctly.
>>
>> To fix this, we add an optional NAT awareness mode, which will query the
>> kernel conntrack mechanism to obtain the pre-NAT addresses for each packet
>> and use that in the flow and host hashing.
>>
>> When the shaper is enabled and the host is already performing NAT, the cost
>> of this lookup is negligible. However, in unlimited mode with no NAT being
>> performed, there is a significant CPU cost at higher bandwidths. For this
>> reason, the feature is turned off by default.
>>
>> Cc: netfilter-devel@vger.kernel.org
>> Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
>> ---
>> net/sched/sch_cake.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 46 insertions(+)
>>
>> diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
>> index 68ac908470f1..fecd9caac0cc 100644
>> --- a/net/sched/sch_cake.c
>> +++ b/net/sched/sch_cake.c
>> @@ -71,6 +71,10 @@
>> #include <net/tcp.h>
>> #include <net/flow_dissector.h>
>>
>> +#if IS_ENABLED(CONFIG_NF_CONNTRACK)
>> +#include <net/netfilter/nf_conntrack_core.h>
>> +#endif
>> +
>> #define CAKE_SET_WAYS (8)
>> #define CAKE_MAX_TINS (8)
>> #define CAKE_QUEUES (1024)
>> @@ -516,6 +520,29 @@ static bool cobalt_should_drop(struct cobalt_vars *vars,
>> return drop;
>> }
>>
>> +static void cake_update_flowkeys(struct flow_keys *keys,
>> + const struct sk_buff *skb)
>> +{
>> +#if IS_ENABLED(CONFIG_NF_CONNTRACK)
>
> I would remove the ifdef, not really needed, it will simplify things.
>
> But I leave it to you to decide, this is not I deal breaker.
If I remove it I get a bunch of 'incomplete type' errors when compiling.
Besides, we use it to report an error to userspace when conntrack is
disabled anyway, so might as well keep the whole thing ifdef'ed.
-Toke
^ permalink raw reply
* [PATCH net-next v17 7/8] sch_cake: Add overhead compensation support to the rate shaper
From: Toke Høiland-Jørgensen @ 2018-05-28 22:28 UTC (permalink / raw)
To: netdev; +Cc: cake
In-Reply-To: <152754646847.18169.14016621463603092505.stgit@alrua-kau>
This commit adds configurable overhead compensation support to the rate
shaper. With this feature, userspace can configure the actual bottleneck
link overhead and encapsulation mode used, which will be used by the shaper
to calculate the precise duration of each packet on the wire.
This feature is needed because CAKE is often deployed one or two hops
upstream of the actual bottleneck (which can be, e.g., inside a DSL or
cable modem). In this case, the link layer characteristics and overhead
reported by the kernel does not match the actual bottleneck. Being able to
set the actual values in use makes it possible to configure the shaper rate
much closer to the actual bottleneck rate (our experience shows it is
possible to get with 0.1% of the actual physical bottleneck rate), thus
keeping latency low without sacrificing bandwidth.
The overhead compensation has three tunables: A fixed per-packet overhead
size (which, if set, will be accounted from the IP packet header), a
minimum packet size (MPU) and a framing mode supporting either ATM or PTM
framing. We include a set of common keywords in TC to help users configure
the right parameters. If no overhead value is set, the value reported by
the kernel is used.
Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
---
net/sched/sch_cake.c | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 123 insertions(+), 1 deletion(-)
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index b2760bc1a34f..10a968d43d44 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -270,6 +270,7 @@ enum {
struct cobalt_skb_cb {
ktime_t enqueue_time;
+ u32 adjusted_len;
};
static u64 us_to_ns(u64 us)
@@ -1282,6 +1283,88 @@ static u64 cake_ewma(u64 avg, u64 sample, u32 shift)
return avg;
}
+static u32 cake_calc_overhead(struct cake_sched_data *q, u32 len, u32 off)
+{
+ if (q->rate_flags & CAKE_FLAG_OVERHEAD)
+ len -= off;
+
+ if (q->max_netlen < len)
+ q->max_netlen = len;
+ if (q->min_netlen > len)
+ q->min_netlen = len;
+
+ len += q->rate_overhead;
+
+ if (len < q->rate_mpu)
+ len = q->rate_mpu;
+
+ if (q->atm_mode == CAKE_ATM_ATM) {
+ len += 47;
+ len /= 48;
+ len *= 53;
+ } else if (q->atm_mode == CAKE_ATM_PTM) {
+ /* Add one byte per 64 bytes or part thereof.
+ * This is conservative and easier to calculate than the
+ * precise value.
+ */
+ len += (len + 63) / 64;
+ }
+
+ if (q->max_adjlen < len)
+ q->max_adjlen = len;
+ if (q->min_adjlen > len)
+ q->min_adjlen = len;
+
+ return len;
+}
+
+static u32 cake_overhead(struct cake_sched_data *q, const struct sk_buff *skb)
+{
+ const struct skb_shared_info *shinfo = skb_shinfo(skb);
+ unsigned int hdr_len, last_len = 0;
+ u32 off = skb_network_offset(skb);
+ u32 len = qdisc_pkt_len(skb);
+ u16 segs = 1;
+
+ q->avg_netoff = cake_ewma(q->avg_netoff, off << 16, 8);
+
+ if (!shinfo->gso_size)
+ return cake_calc_overhead(q, len, off);
+
+ /* borrowed from qdisc_pkt_len_init() */
+ hdr_len = skb_transport_header(skb) - skb_mac_header(skb);
+
+ /* + transport layer */
+ if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 |
+ SKB_GSO_TCPV6))) {
+ const struct tcphdr *th;
+ struct tcphdr _tcphdr;
+
+ th = skb_header_pointer(skb, skb_transport_offset(skb),
+ sizeof(_tcphdr), &_tcphdr);
+ if (likely(th))
+ hdr_len += __tcp_hdrlen(th);
+ } else {
+ struct udphdr _udphdr;
+
+ if (skb_header_pointer(skb, skb_transport_offset(skb),
+ sizeof(_udphdr), &_udphdr))
+ hdr_len += sizeof(struct udphdr);
+ }
+
+ if (unlikely(shinfo->gso_type & SKB_GSO_DODGY))
+ segs = DIV_ROUND_UP(skb->len - hdr_len,
+ shinfo->gso_size);
+ else
+ segs = shinfo->gso_segs;
+
+ len = shinfo->gso_size + hdr_len;
+ last_len = skb->len - shinfo->gso_size * (segs - 1);
+
+ return (cake_calc_overhead(q, len, off) * (segs - 1) +
+ cake_calc_overhead(q, last_len, off));
+}
+
static void cake_heap_swap(struct cake_sched_data *q, u16 i, u16 j)
{
struct cake_heap_entry ii = q->overflow_heap[i];
@@ -1359,7 +1442,7 @@ static int cake_advance_shaper(struct cake_sched_data *q,
struct sk_buff *skb,
ktime_t now, bool drop)
{
- u32 len = qdisc_pkt_len(skb);
+ u32 len = get_cobalt_cb(skb)->adjusted_len;
/* charge packet bandwidth to this tin
* and to the global shaper.
@@ -1556,6 +1639,7 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
b->max_skblen = len;
cobalt_set_enqueue_time(skb, now);
+ get_cobalt_cb(skb)->adjusted_len = cake_overhead(q, skb);
flow_queue_add(flow, skb);
if (q->ack_filter)
@@ -2356,6 +2440,31 @@ static int cake_change(struct Qdisc *sch, struct nlattr *opt,
q->flow_mode = (nla_get_u32(tb[TCA_CAKE_FLOW_MODE]) &
CAKE_FLOW_MASK);
+ if (tb[TCA_CAKE_ATM])
+ q->atm_mode = nla_get_u32(tb[TCA_CAKE_ATM]);
+
+ if (tb[TCA_CAKE_OVERHEAD]) {
+ q->rate_overhead = nla_get_s32(tb[TCA_CAKE_OVERHEAD]);
+ q->rate_flags |= CAKE_FLAG_OVERHEAD;
+
+ q->max_netlen = 0;
+ q->max_adjlen = 0;
+ q->min_netlen = ~0;
+ q->min_adjlen = ~0;
+ }
+
+ if (tb[TCA_CAKE_RAW]) {
+ q->rate_flags &= ~CAKE_FLAG_OVERHEAD;
+
+ q->max_netlen = 0;
+ q->max_adjlen = 0;
+ q->min_netlen = ~0;
+ q->min_adjlen = ~0;
+ }
+
+ if (tb[TCA_CAKE_MPU])
+ q->rate_mpu = nla_get_u32(tb[TCA_CAKE_MPU]);
+
if (tb[TCA_CAKE_RTT]) {
q->interval = nla_get_u32(tb[TCA_CAKE_RTT]);
@@ -2532,6 +2641,19 @@ static int cake_dump(struct Qdisc *sch, struct sk_buff *skb)
!!(q->rate_flags & CAKE_FLAG_WASH)))
goto nla_put_failure;
+ if (nla_put_u32(skb, TCA_CAKE_OVERHEAD, q->rate_overhead))
+ goto nla_put_failure;
+
+ if (!(q->rate_flags & CAKE_FLAG_OVERHEAD))
+ if (nla_put_u32(skb, TCA_CAKE_RAW, 0))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_ATM, q->atm_mode))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_MPU, q->rate_mpu))
+ goto nla_put_failure;
+
return nla_nest_end(skb, opts);
nla_put_failure:
^ permalink raw reply related
* [PATCH net-next v17 2/8] sch_cake: Add ingress mode
From: Toke Høiland-Jørgensen @ 2018-05-28 22:28 UTC (permalink / raw)
To: netdev; +Cc: cake
In-Reply-To: <152754646847.18169.14016621463603092505.stgit@alrua-kau>
The ingress mode is meant to be enabled when CAKE runs downlink of the
actual bottleneck (such as on an IFB device). The mode changes the shaper
to also account dropped packets to the shaped rate, as these have already
traversed the bottleneck.
Enabling ingress mode will also tune the AQM to always keep at least two
packets queued *for each flow*. This is done by scaling the minimum queue
occupancy level that will disable the AQM by the number of active bulk
flows. The rationale for this is that retransmits are more expensive in
ingress mode, since dropped packets have to traverse the bottleneck again
when they are retransmitted; thus, being more lenient and keeping a minimum
number of packets queued will improve throughput in cases where the number
of active flows are so large that they saturate the bottleneck even at
their minimum window size.
This commit also adds a separate switch to enable ingress mode rate
autoscaling. If enabled, the autoscaling code will observe the actual
traffic rate and adjust the shaper rate to match it. This can help avoid
latency increases in the case where the actual bottleneck rate decreases
below the shaped rate. The scaling filters out spikes by an EWMA filter.
Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
---
net/sched/sch_cake.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 81 insertions(+), 4 deletions(-)
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 7ea4aa261cec..10e208e4255d 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -435,7 +435,8 @@ static bool cobalt_queue_empty(struct cobalt_vars *vars,
static bool cobalt_should_drop(struct cobalt_vars *vars,
struct cobalt_params *p,
ktime_t now,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ u32 bulk_flows)
{
bool next_due, over_target, drop = false;
ktime_t schedule;
@@ -459,6 +460,7 @@ static bool cobalt_should_drop(struct cobalt_vars *vars,
sojourn = ktime_to_ns(ktime_sub(now, cobalt_get_enqueue_time(skb)));
schedule = ktime_sub(now, vars->drop_next);
over_target = sojourn > p->target &&
+ sojourn > p->mtu_time * bulk_flows * 2 &&
sojourn > p->mtu_time * 4;
next_due = vars->count && ktime_to_ns(schedule) >= 0;
@@ -913,6 +915,9 @@ static unsigned int cake_drop(struct Qdisc *sch, struct sk_buff **to_free)
b->tin_dropped++;
sch->qstats.drops++;
+ if (q->rate_flags & CAKE_FLAG_INGRESS)
+ cake_advance_shaper(q, b, skb, now, true);
+
__qdisc_drop(skb, to_free);
sch->q.qlen--;
@@ -990,8 +995,46 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
cake_heapify_up(q, b->overflow_idx[idx]);
/* incoming bandwidth capacity estimate */
- q->avg_window_bytes = 0;
- q->last_packet_time = now;
+ if (q->rate_flags & CAKE_FLAG_AUTORATE_INGRESS) {
+ u64 packet_interval = \
+ ktime_to_ns(ktime_sub(now, q->last_packet_time));
+
+ if (packet_interval > NSEC_PER_SEC)
+ packet_interval = NSEC_PER_SEC;
+
+ /* filter out short-term bursts, eg. wifi aggregation */
+ q->avg_packet_interval = \
+ cake_ewma(q->avg_packet_interval,
+ packet_interval,
+ (packet_interval > q->avg_packet_interval ?
+ 2 : 8));
+
+ q->last_packet_time = now;
+
+ if (packet_interval > q->avg_packet_interval) {
+ u64 window_interval = \
+ ktime_to_ns(ktime_sub(now,
+ q->avg_window_begin));
+ u64 b = q->avg_window_bytes * (u64)NSEC_PER_SEC;
+
+ do_div(b, window_interval);
+ q->avg_peak_bandwidth =
+ cake_ewma(q->avg_peak_bandwidth, b,
+ b > q->avg_peak_bandwidth ? 2 : 8);
+ q->avg_window_bytes = 0;
+ q->avg_window_begin = now;
+
+ if (ktime_after(now,
+ ktime_add_ms(q->last_reconfig_time,
+ 250))) {
+ q->rate_bps = (q->avg_peak_bandwidth * 15) >> 4;
+ cake_reconfigure(sch);
+ }
+ }
+ } else {
+ q->avg_window_bytes = 0;
+ q->last_packet_time = now;
+ }
/* flowchain */
if (!flow->set || flow->set == CAKE_SET_DECAYING) {
@@ -1251,15 +1294,27 @@ static struct sk_buff *cake_dequeue(struct Qdisc *sch)
}
/* Last packet in queue may be marked, shouldn't be dropped */
- if (!cobalt_should_drop(&flow->cvars, &b->cparams, now, skb) ||
+ if (!cobalt_should_drop(&flow->cvars, &b->cparams, now, skb,
+ (b->bulk_flow_count *
+ !!(q->rate_flags &
+ CAKE_FLAG_INGRESS))) ||
!flow->head)
break;
+ /* drop this packet, get another one */
+ if (q->rate_flags & CAKE_FLAG_INGRESS) {
+ len = cake_advance_shaper(q, b, skb,
+ now, true);
+ flow->deficit -= len;
+ b->tin_deficit -= len;
+ }
flow->dropped++;
b->tin_dropped++;
qdisc_tree_reduce_backlog(sch, 1, qdisc_pkt_len(skb));
qdisc_qstats_drop(sch);
kfree_skb(skb);
+ if (q->rate_flags & CAKE_FLAG_INGRESS)
+ goto retry;
}
b->tin_ecn_mark += !!flow->cvars.ecn_marked;
@@ -1442,6 +1497,20 @@ static int cake_change(struct Qdisc *sch, struct nlattr *opt,
q->target = 1;
}
+ if (tb[TCA_CAKE_AUTORATE]) {
+ if (!!nla_get_u32(tb[TCA_CAKE_AUTORATE]))
+ q->rate_flags |= CAKE_FLAG_AUTORATE_INGRESS;
+ else
+ q->rate_flags &= ~CAKE_FLAG_AUTORATE_INGRESS;
+ }
+
+ if (tb[TCA_CAKE_INGRESS]) {
+ if (!!nla_get_u32(tb[TCA_CAKE_INGRESS]))
+ q->rate_flags |= CAKE_FLAG_INGRESS;
+ else
+ q->rate_flags &= ~CAKE_FLAG_INGRESS;
+ }
+
if (tb[TCA_CAKE_MEMORY])
q->buffer_config_limit = nla_get_u32(tb[TCA_CAKE_MEMORY]);
@@ -1565,6 +1634,14 @@ static int cake_dump(struct Qdisc *sch, struct sk_buff *skb)
if (nla_put_u32(skb, TCA_CAKE_MEMORY, q->buffer_config_limit))
goto nla_put_failure;
+ if (nla_put_u32(skb, TCA_CAKE_AUTORATE,
+ !!(q->rate_flags & CAKE_FLAG_AUTORATE_INGRESS)))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_INGRESS,
+ !!(q->rate_flags & CAKE_FLAG_INGRESS)))
+ goto nla_put_failure;
+
return nla_nest_end(skb, opts);
nla_put_failure:
^ permalink raw reply related
* [PATCH net-next v17 0/8] sched: Add Common Applications Kept Enhanced (cake) qdisc
From: Toke Høiland-Jørgensen @ 2018-05-28 22:28 UTC (permalink / raw)
To: netdev
Cc: Georgios Amanakis, Pete Heist, Yuchung Cheng, Neal Cardwell,
Dave Taht, netfilter-devel, cake
This patch series adds the CAKE qdisc, and has been split up to ease
review.
I have attempted to split out each configurable feature into its own patch.
The first commit adds the base shaper and packet scheduler, while
subsequent commits add the optional features. The full userspace API and
most data structures are included in this commit, but options not
understood in the base version will be ignored.
The result of applying the entire series is identical to the out of tree
version that have seen extensive testing in previous deployments, most
notably as an out of tree patch to OpenWrt. However, note that I have only
compile tested the individual patches; so the whole series should be
considered as a unit.
---
Changelog
v17:
- Rebase to newest net-next and move the conntrack callback to
nf_ct_hook
- Fix a compile error when NF_CONNTRACK is unset.
v16:
- Move conntrack lookup function into conntrack core and read it via
RCU so it is only active when the nf_conntrack module is loaded.
This avoids the module dependency on conntrack for NAT mode. Thanks
to Pablo for the idea.
v15:
- Handle ECN flags in ACK filter
v14:
- Handle seqno wraps and DSACKs in ACK filter
v13:
- Avoid ktime_t to scalar compares
- Add class dumping and basic stats
- Fail with ENOTSUPP when requesting NAT mode and conntrack is not
available.
- Parse all TCP options in ACK filter and make sure to only drop safe
ones. Also handle SACK ranges properly.
v12:
- Get rid of custom time typedefs. Use ktime_t for time and u64 for
duration instead.
v11:
- Fix overhead compensation calculation for GSO packets
- Change configured rate to be u64 (I ran out of bits before I ran out
of CPU when testing the effects of the above)
v10:
- Christmas tree gardening (fix variable declarations to be in reverse
line length order)
v9:
- Remove duplicated checks around kvfree() and just call it
unconditionally.
- Don't pass __GFP_NOWARN when allocating memory
- Move options in cake_dump() that are related to optional features to
later patches implementing the features.
- Support attaching filters to the qdisc and use the classification
result to select flow queue.
- Support overriding diffserv priority tin from skb->priority
v8:
- Remove inline keyword from function definitions
- Simplify ACK filter; remove the complex state handling to make the
logic easier to follow. This will potentially be a bit less efficient,
but I have not been able to measure a difference.
v7:
- Split up patch into a series to ease review.
- Constify the ACK filter.
v6:
- Fix 6in4 encapsulation checks in ACK filter code
- Checkpatch fixes
v5:
- Refactor ACK filter code and hopefully fix the safety issues
properly this time.
v4:
- Only split GSO packets if shaping at speeds <= 1Gbps
- Fix overhead calculation code to also work for GSO packets
- Don't re-implement kvzalloc()
- Remove local header include from out-of-tree build (fixes kbuild-bot
complaint).
- Several fixes to the ACK filter:
- Check pskb_may_pull() before deref of transport headers.
- Don't run ACK filter logic on split GSO packets
- Fix TCP sequence number compare to deal with wraparounds
v3:
- Use IS_REACHABLE() macro to fix compilation when sch_cake is
built-in and conntrack is a module.
- Switch the stats output to use nested netlink attributes instead
of a versioned struct.
- Remove GPL boilerplate.
- Fix array initialisation style.
v2:
- Fix kbuild test bot complaint
- Clean up the netlink ABI
- Fix checkpatch complaints
- A few tweaks to the behaviour of cake based on testing carried out
while writing the paper.
---
Toke Høiland-Jørgensen (8):
sched: Add Common Applications Kept Enhanced (cake) qdisc
sch_cake: Add ingress mode
sch_cake: Add optional ACK filter
netfilter: Add nf_ct_get_tuple_skb global lookup function
sch_cake: Add NAT awareness to packet classifier
sch_cake: Add DiffServ handling
sch_cake: Add overhead compensation support to the rate shaper
sch_cake: Conditionally split GSO segments
include/linux/netfilter.h | 11
include/uapi/linux/pkt_sched.h | 113 +
net/netfilter/core.c | 15
net/netfilter/nf_conntrack_core.c | 36
net/sched/Kconfig | 11
net/sched/Makefile | 1
net/sched/sch_cake.c | 2987 +++++++++++++++++++++++++++++++++++++
7 files changed, 3174 insertions(+)
create mode 100644 net/sched/sch_cake.c
^ permalink raw reply
* [PATCH net-next v17 1/8] sched: Add Common Applications Kept Enhanced (cake) qdisc
From: Toke Høiland-Jørgensen @ 2018-05-28 22:28 UTC (permalink / raw)
To: netdev; +Cc: Pete Heist, Georgios Amanakis, Dave Taht, cake
In-Reply-To: <152754646847.18169.14016621463603092505.stgit@alrua-kau>
sch_cake targets the home router use case and is intended to squeeze the
most bandwidth and latency out of even the slowest ISP links and routers,
while presenting an API simple enough that even an ISP can configure it.
Example of use on a cable ISP uplink:
tc qdisc add dev eth0 cake bandwidth 20Mbit nat docsis ack-filter
To shape a cable download link (ifb and tc-mirred setup elided)
tc qdisc add dev ifb0 cake bandwidth 200mbit nat docsis ingress wash
CAKE is filled with:
* A hybrid Codel/Blue AQM algorithm, "Cobalt", tied to an FQ_Codel
derived Flow Queuing system, which autoconfigures based on the bandwidth.
* A novel "triple-isolate" mode (the default) which balances per-host
and per-flow FQ even through NAT.
* An deficit based shaper, that can also be used in an unlimited mode.
* 8 way set associative hashing to reduce flow collisions to a minimum.
* A reasonable interpretation of various diffserv latency/loss tradeoffs.
* Support for zeroing diffserv markings for entering and exiting traffic.
* Support for interacting well with Docsis 3.0 shaper framing.
* Extensive support for DSL framing types.
* Support for ack filtering.
* Extensive statistics for measuring, loss, ecn markings, latency
variation.
A paper describing the design of CAKE is available at
https://arxiv.org/abs/1804.07617, and will be published at the 2018 IEEE
International Symposium on Local and Metropolitan Area Networks (LANMAN).
This patch adds the base shaper and packet scheduler, while subsequent
commits add the optional (configurable) features. The full userspace API
and most data structures are included in this commit, but options not
understood in the base version will be ignored.
Various versions baking have been available as an out of tree build for
kernel versions going back to 3.10, as the embedded router world has been
running a few years behind mainline Linux. A stable version has been
generally available on lede-17.01 and later.
sch_cake replaces a combination of iptables, tc filter, htb and fq_codel
in the sqm-scripts, with sane defaults and vastly simpler configuration.
CAKE's principal author is Jonathan Morton, with contributions from
Kevin Darbyshire-Bryant, Toke Høiland-Jørgensen, Sebastian Moeller,
Ryan Mounce, Guido Sarducci, Dean Scarff, Nils Andreas Svee, Dave Täht,
and Loganaden Velvindron.
Testing from Pete Heist, Georgios Amanakis, and the many other members of
the cake@lists.bufferbloat.net mailing list.
tc -s qdisc show dev eth2
qdisc cake 1: root refcnt 2 bandwidth 100Mbit diffserv3 triple-isolate rtt 100.0ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 5000000b
capacity estimate: 100Mbit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0
Bulk Best Effort Voice
thresh 6250Kbit 100Mbit 25Mbit
target 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 100.0ms
pk_delay 0us 0us 0us
av_delay 0us 0us 0us
sp_delay 0us 0us 0us
pkts 0 0 0
bytes 0 0 0
way_inds 0 0 0
way_miss 0 0 0
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 0 0
bk_flows 0 0 0
un_flows 0 0 0
max_len 0 0 0
quantum 300 1514 762
Tested-by: Pete Heist <peteheist@gmail.com>
Tested-by: Georgios Amanakis <gamanakis@gmail.com>
Signed-off-by: Dave Taht <dave.taht@gmail.com>
Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
---
include/uapi/linux/pkt_sched.h | 113 ++
net/sched/Kconfig | 11
net/sched/Makefile | 1
net/sched/sch_cake.c | 1850 ++++++++++++++++++++++++++++++++++++++++
4 files changed, 1975 insertions(+)
create mode 100644 net/sched/sch_cake.c
diff --git a/include/uapi/linux/pkt_sched.h b/include/uapi/linux/pkt_sched.h
index 37b5096ae97b..07648e6ea569 100644
--- a/include/uapi/linux/pkt_sched.h
+++ b/include/uapi/linux/pkt_sched.h
@@ -934,4 +934,117 @@ enum {
#define TCA_CBS_MAX (__TCA_CBS_MAX - 1)
+/* CAKE */
+enum {
+ TCA_CAKE_UNSPEC,
+ TCA_CAKE_PAD,
+ TCA_CAKE_BASE_RATE64,
+ TCA_CAKE_DIFFSERV_MODE,
+ TCA_CAKE_ATM,
+ TCA_CAKE_FLOW_MODE,
+ TCA_CAKE_OVERHEAD,
+ TCA_CAKE_RTT,
+ TCA_CAKE_TARGET,
+ TCA_CAKE_AUTORATE,
+ TCA_CAKE_MEMORY,
+ TCA_CAKE_NAT,
+ TCA_CAKE_RAW,
+ TCA_CAKE_WASH,
+ TCA_CAKE_MPU,
+ TCA_CAKE_INGRESS,
+ TCA_CAKE_ACK_FILTER,
+ TCA_CAKE_SPLIT_GSO,
+ __TCA_CAKE_MAX
+};
+#define TCA_CAKE_MAX (__TCA_CAKE_MAX - 1)
+
+enum {
+ __TCA_CAKE_STATS_INVALID,
+ TCA_CAKE_STATS_PAD,
+ TCA_CAKE_STATS_CAPACITY_ESTIMATE64,
+ TCA_CAKE_STATS_MEMORY_LIMIT,
+ TCA_CAKE_STATS_MEMORY_USED,
+ TCA_CAKE_STATS_AVG_NETOFF,
+ TCA_CAKE_STATS_MIN_NETLEN,
+ TCA_CAKE_STATS_MAX_NETLEN,
+ TCA_CAKE_STATS_MIN_ADJLEN,
+ TCA_CAKE_STATS_MAX_ADJLEN,
+ TCA_CAKE_STATS_TIN_STATS,
+ TCA_CAKE_STATS_DEFICIT,
+ TCA_CAKE_STATS_COBALT_COUNT,
+ TCA_CAKE_STATS_DROPPING,
+ TCA_CAKE_STATS_DROP_NEXT_US,
+ TCA_CAKE_STATS_P_DROP,
+ TCA_CAKE_STATS_BLUE_TIMER_US,
+ __TCA_CAKE_STATS_MAX
+};
+#define TCA_CAKE_STATS_MAX (__TCA_CAKE_STATS_MAX - 1)
+
+enum {
+ __TCA_CAKE_TIN_STATS_INVALID,
+ TCA_CAKE_TIN_STATS_PAD,
+ TCA_CAKE_TIN_STATS_SENT_PACKETS,
+ TCA_CAKE_TIN_STATS_SENT_BYTES64,
+ TCA_CAKE_TIN_STATS_DROPPED_PACKETS,
+ TCA_CAKE_TIN_STATS_DROPPED_BYTES64,
+ TCA_CAKE_TIN_STATS_ACKS_DROPPED_PACKETS,
+ TCA_CAKE_TIN_STATS_ACKS_DROPPED_BYTES64,
+ TCA_CAKE_TIN_STATS_ECN_MARKED_PACKETS,
+ TCA_CAKE_TIN_STATS_ECN_MARKED_BYTES64,
+ TCA_CAKE_TIN_STATS_BACKLOG_PACKETS,
+ TCA_CAKE_TIN_STATS_BACKLOG_BYTES,
+ TCA_CAKE_TIN_STATS_THRESHOLD_RATE64,
+ TCA_CAKE_TIN_STATS_TARGET_US,
+ TCA_CAKE_TIN_STATS_INTERVAL_US,
+ TCA_CAKE_TIN_STATS_WAY_INDIRECT_HITS,
+ TCA_CAKE_TIN_STATS_WAY_MISSES,
+ TCA_CAKE_TIN_STATS_WAY_COLLISIONS,
+ TCA_CAKE_TIN_STATS_PEAK_DELAY_US,
+ TCA_CAKE_TIN_STATS_AVG_DELAY_US,
+ TCA_CAKE_TIN_STATS_BASE_DELAY_US,
+ TCA_CAKE_TIN_STATS_SPARSE_FLOWS,
+ TCA_CAKE_TIN_STATS_BULK_FLOWS,
+ TCA_CAKE_TIN_STATS_UNRESPONSIVE_FLOWS,
+ TCA_CAKE_TIN_STATS_MAX_SKBLEN,
+ TCA_CAKE_TIN_STATS_FLOW_QUANTUM,
+ __TCA_CAKE_TIN_STATS_MAX
+};
+#define TCA_CAKE_TIN_STATS_MAX (__TCA_CAKE_TIN_STATS_MAX - 1)
+#define TC_CAKE_MAX_TINS (8)
+
+enum {
+ CAKE_FLOW_NONE = 0,
+ CAKE_FLOW_SRC_IP,
+ CAKE_FLOW_DST_IP,
+ CAKE_FLOW_HOSTS, /* = CAKE_FLOW_SRC_IP | CAKE_FLOW_DST_IP */
+ CAKE_FLOW_FLOWS,
+ CAKE_FLOW_DUAL_SRC, /* = CAKE_FLOW_SRC_IP | CAKE_FLOW_FLOWS */
+ CAKE_FLOW_DUAL_DST, /* = CAKE_FLOW_DST_IP | CAKE_FLOW_FLOWS */
+ CAKE_FLOW_TRIPLE, /* = CAKE_FLOW_HOSTS | CAKE_FLOW_FLOWS */
+ CAKE_FLOW_MAX,
+};
+
+enum {
+ CAKE_DIFFSERV_DIFFSERV3 = 0,
+ CAKE_DIFFSERV_DIFFSERV4,
+ CAKE_DIFFSERV_DIFFSERV8,
+ CAKE_DIFFSERV_BESTEFFORT,
+ CAKE_DIFFSERV_PRECEDENCE,
+ CAKE_DIFFSERV_MAX
+};
+
+enum {
+ CAKE_ACK_NONE = 0,
+ CAKE_ACK_FILTER,
+ CAKE_ACK_AGGRESSIVE,
+ CAKE_ACK_MAX
+};
+
+enum {
+ CAKE_ATM_NONE = 0,
+ CAKE_ATM_ATM,
+ CAKE_ATM_PTM,
+ CAKE_ATM_MAX
+};
+
#endif
diff --git a/net/sched/Kconfig b/net/sched/Kconfig
index a01169fb5325..6e7d614b5757 100644
--- a/net/sched/Kconfig
+++ b/net/sched/Kconfig
@@ -284,6 +284,17 @@ config NET_SCH_FQ_CODEL
If unsure, say N.
+config NET_SCH_CAKE
+ tristate "Common Applications Kept Enhanced (CAKE)"
+ help
+ Say Y here if you want to use the Common Applications Kept Enhanced
+ (CAKE) queue management algorithm.
+
+ To compile this driver as a module, choose M here: the module
+ will be called sch_cake.
+
+ If unsure, say N.
+
config NET_SCH_FQ
tristate "Fair Queue"
help
diff --git a/net/sched/Makefile b/net/sched/Makefile
index 8811d3804878..435054cee32c 100644
--- a/net/sched/Makefile
+++ b/net/sched/Makefile
@@ -50,6 +50,7 @@ obj-$(CONFIG_NET_SCH_CHOKE) += sch_choke.o
obj-$(CONFIG_NET_SCH_QFQ) += sch_qfq.o
obj-$(CONFIG_NET_SCH_CODEL) += sch_codel.o
obj-$(CONFIG_NET_SCH_FQ_CODEL) += sch_fq_codel.o
+obj-$(CONFIG_NET_SCH_CAKE) += sch_cake.o
obj-$(CONFIG_NET_SCH_FQ) += sch_fq.o
obj-$(CONFIG_NET_SCH_HHF) += sch_hhf.o
obj-$(CONFIG_NET_SCH_PIE) += sch_pie.o
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
new file mode 100644
index 000000000000..7ea4aa261cec
--- /dev/null
+++ b/net/sched/sch_cake.c
@@ -0,0 +1,1850 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+/* COMMON Applications Kept Enhanced (CAKE) discipline
+ *
+ * Copyright (C) 2014-2018 Jonathan Morton <chromatix99@gmail.com>
+ * Copyright (C) 2015-2018 Toke Høiland-Jørgensen <toke@toke.dk>
+ * Copyright (C) 2014-2018 Dave Täht <dave.taht@gmail.com>
+ * Copyright (C) 2015-2018 Sebastian Moeller <moeller0@gmx.de>
+ * (C) 2015-2018 Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
+ * Copyright (C) 2017-2018 Ryan Mounce <ryan@mounce.com.au>
+ *
+ * The CAKE Principles:
+ * (or, how to have your cake and eat it too)
+ *
+ * This is a combination of several shaping, AQM and FQ techniques into one
+ * easy-to-use package:
+ *
+ * - An overall bandwidth shaper, to move the bottleneck away from dumb CPE
+ * equipment and bloated MACs. This operates in deficit mode (as in sch_fq),
+ * eliminating the need for any sort of burst parameter (eg. token bucket
+ * depth). Burst support is limited to that necessary to overcome scheduling
+ * latency.
+ *
+ * - A Diffserv-aware priority queue, giving more priority to certain classes,
+ * up to a specified fraction of bandwidth. Above that bandwidth threshold,
+ * the priority is reduced to avoid starving other tins.
+ *
+ * - Each priority tin has a separate Flow Queue system, to isolate traffic
+ * flows from each other. This prevents a burst on one flow from increasing
+ * the delay to another. Flows are distributed to queues using a
+ * set-associative hash function.
+ *
+ * - Each queue is actively managed by Cobalt, which is a combination of the
+ * Codel and Blue AQM algorithms. This serves flows fairly, and signals
+ * congestion early via ECN (if available) and/or packet drops, to keep
+ * latency low. The codel parameters are auto-tuned based on the bandwidth
+ * setting, as is necessary at low bandwidths.
+ *
+ * The configuration parameters are kept deliberately simple for ease of use.
+ * Everything has sane defaults. Complete generality of configuration is *not*
+ * a goal.
+ *
+ * The priority queue operates according to a weighted DRR scheme, combined with
+ * a bandwidth tracker which reuses the shaper logic to detect which side of the
+ * bandwidth sharing threshold the tin is operating. This determines whether a
+ * priority-based weight (high) or a bandwidth-based weight (low) is used for
+ * that tin in the current pass.
+ *
+ * This qdisc was inspired by Eric Dumazet's fq_codel code, which he kindly
+ * granted us permission to leverage.
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/jiffies.h>
+#include <linux/string.h>
+#include <linux/in.h>
+#include <linux/errno.h>
+#include <linux/init.h>
+#include <linux/skbuff.h>
+#include <linux/jhash.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/reciprocal_div.h>
+#include <net/netlink.h>
+#include <linux/version.h>
+#include <linux/if_vlan.h>
+#include <net/pkt_sched.h>
+#include <net/pkt_cls.h>
+#include <net/tcp.h>
+#include <net/flow_dissector.h>
+
+#define CAKE_SET_WAYS (8)
+#define CAKE_MAX_TINS (8)
+#define CAKE_QUEUES (1024)
+#define CAKE_FLOW_MASK 63
+#define CAKE_FLOW_NAT_FLAG 64
+
+/* struct cobalt_params - contains codel and blue parameters
+ * @interval: codel initial drop rate
+ * @target: maximum persistent sojourn time & blue update rate
+ * @mtu_time: serialisation delay of maximum-size packet
+ * @p_inc: increment of blue drop probability (0.32 fxp)
+ * @p_dec: decrement of blue drop probability (0.32 fxp)
+ */
+struct cobalt_params {
+ u64 interval;
+ u64 target;
+ u64 mtu_time;
+ u32 p_inc;
+ u32 p_dec;
+};
+
+/* struct cobalt_vars - contains codel and blue variables
+ * @count: codel dropping frequency
+ * @rec_inv_sqrt: reciprocal value of sqrt(count) >> 1
+ * @drop_next: time to drop next packet, or when we dropped last
+ * @blue_timer: Blue time to next drop
+ * @p_drop: BLUE drop probability (0.32 fxp)
+ * @dropping: set if in dropping state
+ * @ecn_marked: set if marked
+ */
+struct cobalt_vars {
+ u32 count;
+ u32 rec_inv_sqrt;
+ ktime_t drop_next;
+ ktime_t blue_timer;
+ u32 p_drop;
+ bool dropping;
+ bool ecn_marked;
+};
+
+enum {
+ CAKE_SET_NONE = 0,
+ CAKE_SET_SPARSE,
+ CAKE_SET_SPARSE_WAIT, /* counted in SPARSE, actually in BULK */
+ CAKE_SET_BULK,
+ CAKE_SET_DECAYING
+};
+
+struct cake_flow {
+ /* this stuff is all needed per-flow at dequeue time */
+ struct sk_buff *head;
+ struct sk_buff *tail;
+ struct list_head flowchain;
+ s32 deficit;
+ u32 dropped;
+ struct cobalt_vars cvars;
+ u16 srchost; /* index into cake_host table */
+ u16 dsthost;
+ u8 set;
+}; /* please try to keep this structure <= 64 bytes */
+
+struct cake_host {
+ u32 srchost_tag;
+ u32 dsthost_tag;
+ u16 srchost_refcnt;
+ u16 dsthost_refcnt;
+};
+
+struct cake_heap_entry {
+ u16 t:3, b:10;
+};
+
+struct cake_tin_data {
+ struct cake_flow flows[CAKE_QUEUES];
+ u32 backlogs[CAKE_QUEUES];
+ u32 tags[CAKE_QUEUES]; /* for set association */
+ u16 overflow_idx[CAKE_QUEUES];
+ struct cake_host hosts[CAKE_QUEUES]; /* for triple isolation */
+ u16 flow_quantum;
+
+ struct cobalt_params cparams;
+ u32 drop_overlimit;
+ u16 bulk_flow_count;
+ u16 sparse_flow_count;
+ u16 decaying_flow_count;
+ u16 unresponsive_flow_count;
+
+ u32 max_skblen;
+
+ struct list_head new_flows;
+ struct list_head old_flows;
+ struct list_head decaying_flows;
+
+ /* time_next = time_this + ((len * rate_ns) >> rate_shft) */
+ ktime_t time_next_packet;
+ u64 tin_rate_ns;
+ u64 tin_rate_bps;
+ u16 tin_rate_shft;
+
+ u16 tin_quantum_prio;
+ u16 tin_quantum_band;
+ s32 tin_deficit;
+ u32 tin_backlog;
+ u32 tin_dropped;
+ u32 tin_ecn_mark;
+
+ u32 packets;
+ u64 bytes;
+
+ u32 ack_drops;
+
+ /* moving averages */
+ u64 avge_delay;
+ u64 peak_delay;
+ u64 base_delay;
+
+ /* hash function stats */
+ u32 way_directs;
+ u32 way_hits;
+ u32 way_misses;
+ u32 way_collisions;
+}; /* number of tins is small, so size of this struct doesn't matter much */
+
+struct cake_sched_data {
+ struct tcf_proto __rcu *filter_list; /* optional external classifier */
+ struct tcf_block *block;
+ struct cake_tin_data *tins;
+
+ struct cake_heap_entry overflow_heap[CAKE_QUEUES * CAKE_MAX_TINS];
+ u16 overflow_timeout;
+
+ u16 tin_cnt;
+ u8 tin_mode;
+ u8 flow_mode;
+ u8 ack_filter;
+ u8 atm_mode;
+
+ /* time_next = time_this + ((len * rate_ns) >> rate_shft) */
+ u16 rate_shft;
+ ktime_t time_next_packet;
+ ktime_t failsafe_next_packet;
+ u64 rate_ns;
+ u64 rate_bps;
+ u16 rate_flags;
+ s16 rate_overhead;
+ u16 rate_mpu;
+ u64 interval;
+ u64 target;
+
+ /* resource tracking */
+ u32 buffer_used;
+ u32 buffer_max_used;
+ u32 buffer_limit;
+ u32 buffer_config_limit;
+
+ /* indices for dequeue */
+ u16 cur_tin;
+ u16 cur_flow;
+
+ struct qdisc_watchdog watchdog;
+ const u8 *tin_index;
+ const u8 *tin_order;
+
+ /* bandwidth capacity estimate */
+ ktime_t last_packet_time;
+ ktime_t avg_window_begin;
+ u64 avg_packet_interval;
+ u64 avg_window_bytes;
+ u64 avg_peak_bandwidth;
+ ktime_t last_reconfig_time;
+
+ /* packet length stats */
+ u32 avg_netoff;
+ u16 max_netlen;
+ u16 max_adjlen;
+ u16 min_netlen;
+ u16 min_adjlen;
+};
+
+enum {
+ CAKE_FLAG_OVERHEAD = BIT(0),
+ CAKE_FLAG_AUTORATE_INGRESS = BIT(1),
+ CAKE_FLAG_INGRESS = BIT(2),
+ CAKE_FLAG_WASH = BIT(3),
+ CAKE_FLAG_SPLIT_GSO = BIT(4)
+};
+
+/* COBALT operates the Codel and BLUE algorithms in parallel, in order to
+ * obtain the best features of each. Codel is excellent on flows which
+ * respond to congestion signals in a TCP-like way. BLUE is more effective on
+ * unresponsive flows.
+ */
+
+struct cobalt_skb_cb {
+ ktime_t enqueue_time;
+};
+
+static u64 us_to_ns(u64 us)
+{
+ return us * NSEC_PER_USEC;
+}
+
+static struct cobalt_skb_cb *get_cobalt_cb(const struct sk_buff *skb)
+{
+ qdisc_cb_private_validate(skb, sizeof(struct cobalt_skb_cb));
+ return (struct cobalt_skb_cb *)qdisc_skb_cb(skb)->data;
+}
+
+static ktime_t cobalt_get_enqueue_time(const struct sk_buff *skb)
+{
+ return get_cobalt_cb(skb)->enqueue_time;
+}
+
+static void cobalt_set_enqueue_time(struct sk_buff *skb,
+ ktime_t now)
+{
+ get_cobalt_cb(skb)->enqueue_time = now;
+}
+
+static u16 quantum_div[CAKE_QUEUES + 1] = {0};
+
+#define REC_INV_SQRT_CACHE (16)
+static u32 cobalt_rec_inv_sqrt_cache[REC_INV_SQRT_CACHE] = {0};
+
+/* http://en.wikipedia.org/wiki/Methods_of_computing_square_roots
+ * new_invsqrt = (invsqrt / 2) * (3 - count * invsqrt^2)
+ *
+ * Here, invsqrt is a fixed point number (< 1.0), 32bit mantissa, aka Q0.32
+ */
+
+static void cobalt_newton_step(struct cobalt_vars *vars)
+{
+ u32 invsqrt, invsqrt2;
+ u64 val;
+
+ invsqrt = vars->rec_inv_sqrt;
+ invsqrt2 = ((u64)invsqrt * invsqrt) >> 32;
+ val = (3LL << 32) - ((u64)vars->count * invsqrt2);
+
+ val >>= 2; /* avoid overflow in following multiply */
+ val = (val * invsqrt) >> (32 - 2 + 1);
+
+ vars->rec_inv_sqrt = val;
+}
+
+static void cobalt_invsqrt(struct cobalt_vars *vars)
+{
+ if (vars->count < REC_INV_SQRT_CACHE)
+ vars->rec_inv_sqrt = cobalt_rec_inv_sqrt_cache[vars->count];
+ else
+ cobalt_newton_step(vars);
+}
+
+/* There is a big difference in timing between the accurate values placed in
+ * the cache and the approximations given by a single Newton step for small
+ * count values, particularly when stepping from count 1 to 2 or vice versa.
+ * Above 16, a single Newton step gives sufficient accuracy in either
+ * direction, given the precision stored.
+ *
+ * The magnitude of the error when stepping up to count 2 is such as to give
+ * the value that *should* have been produced at count 4.
+ */
+
+static void cobalt_cache_init(void)
+{
+ struct cobalt_vars v;
+
+ memset(&v, 0, sizeof(v));
+ v.rec_inv_sqrt = ~0U;
+ cobalt_rec_inv_sqrt_cache[0] = v.rec_inv_sqrt;
+
+ for (v.count = 1; v.count < REC_INV_SQRT_CACHE; v.count++) {
+ cobalt_newton_step(&v);
+ cobalt_newton_step(&v);
+ cobalt_newton_step(&v);
+ cobalt_newton_step(&v);
+
+ cobalt_rec_inv_sqrt_cache[v.count] = v.rec_inv_sqrt;
+ }
+}
+
+static void cobalt_vars_init(struct cobalt_vars *vars)
+{
+ memset(vars, 0, sizeof(*vars));
+
+ if (!cobalt_rec_inv_sqrt_cache[0]) {
+ cobalt_cache_init();
+ cobalt_rec_inv_sqrt_cache[0] = ~0;
+ }
+}
+
+/* CoDel control_law is t + interval/sqrt(count)
+ * We maintain in rec_inv_sqrt the reciprocal value of sqrt(count) to avoid
+ * both sqrt() and divide operation.
+ */
+static ktime_t cobalt_control(ktime_t t,
+ u64 interval,
+ u32 rec_inv_sqrt)
+{
+ return ktime_add_ns(t, reciprocal_scale(interval,
+ rec_inv_sqrt));
+}
+
+/* Call this when a packet had to be dropped due to queue overflow. Returns
+ * true if the BLUE state was quiescent before but active after this call.
+ */
+static bool cobalt_queue_full(struct cobalt_vars *vars,
+ struct cobalt_params *p,
+ ktime_t now)
+{
+ bool up = false;
+
+ if (ktime_to_ns(ktime_sub(now, vars->blue_timer)) > p->target) {
+ up = !vars->p_drop;
+ vars->p_drop += p->p_inc;
+ if (vars->p_drop < p->p_inc)
+ vars->p_drop = ~0;
+ vars->blue_timer = now;
+ }
+ vars->dropping = true;
+ vars->drop_next = now;
+ if (!vars->count)
+ vars->count = 1;
+
+ return up;
+}
+
+/* Call this when the queue was serviced but turned out to be empty. Returns
+ * true if the BLUE state was active before but quiescent after this call.
+ */
+static bool cobalt_queue_empty(struct cobalt_vars *vars,
+ struct cobalt_params *p,
+ ktime_t now)
+{
+ bool down = false;
+
+ if (vars->p_drop &&
+ ktime_to_ns(ktime_sub(now, vars->blue_timer)) > p->target) {
+ if (vars->p_drop < p->p_dec)
+ vars->p_drop = 0;
+ else
+ vars->p_drop -= p->p_dec;
+ vars->blue_timer = now;
+ down = !vars->p_drop;
+ }
+ vars->dropping = false;
+
+ if (vars->count && ktime_to_ns(ktime_sub(now, vars->drop_next)) >= 0) {
+ vars->count--;
+ cobalt_invsqrt(vars);
+ vars->drop_next = cobalt_control(vars->drop_next,
+ p->interval,
+ vars->rec_inv_sqrt);
+ }
+
+ return down;
+}
+
+/* Call this with a freshly dequeued packet for possible congestion marking.
+ * Returns true as an instruction to drop the packet, false for delivery.
+ */
+static bool cobalt_should_drop(struct cobalt_vars *vars,
+ struct cobalt_params *p,
+ ktime_t now,
+ struct sk_buff *skb)
+{
+ bool next_due, over_target, drop = false;
+ ktime_t schedule;
+ u64 sojourn;
+
+/* The 'schedule' variable records, in its sign, whether 'now' is before or
+ * after 'drop_next'. This allows 'drop_next' to be updated before the next
+ * scheduling decision is actually branched, without destroying that
+ * information. Similarly, the first 'schedule' value calculated is preserved
+ * in the boolean 'next_due'.
+ *
+ * As for 'drop_next', we take advantage of the fact that 'interval' is both
+ * the delay between first exceeding 'target' and the first signalling event,
+ * *and* the scaling factor for the signalling frequency. It's therefore very
+ * natural to use a single mechanism for both purposes, and eliminates a
+ * significant amount of reference Codel's spaghetti code. To help with this,
+ * both the '0' and '1' entries in the invsqrt cache are 0xFFFFFFFF, as close
+ * as possible to 1.0 in fixed-point.
+ */
+
+ sojourn = ktime_to_ns(ktime_sub(now, cobalt_get_enqueue_time(skb)));
+ schedule = ktime_sub(now, vars->drop_next);
+ over_target = sojourn > p->target &&
+ sojourn > p->mtu_time * 4;
+ next_due = vars->count && ktime_to_ns(schedule) >= 0;
+
+ vars->ecn_marked = false;
+
+ if (over_target) {
+ if (!vars->dropping) {
+ vars->dropping = true;
+ vars->drop_next = cobalt_control(now,
+ p->interval,
+ vars->rec_inv_sqrt);
+ }
+ if (!vars->count)
+ vars->count = 1;
+ } else if (vars->dropping) {
+ vars->dropping = false;
+ }
+
+ if (next_due && vars->dropping) {
+ /* Use ECN mark if possible, otherwise drop */
+ drop = !(vars->ecn_marked = INET_ECN_set_ce(skb));
+
+ vars->count++;
+ if (!vars->count)
+ vars->count--;
+ cobalt_invsqrt(vars);
+ vars->drop_next = cobalt_control(vars->drop_next,
+ p->interval,
+ vars->rec_inv_sqrt);
+ schedule = ktime_sub(now, vars->drop_next);
+ } else {
+ while (next_due) {
+ vars->count--;
+ cobalt_invsqrt(vars);
+ vars->drop_next = cobalt_control(vars->drop_next,
+ p->interval,
+ vars->rec_inv_sqrt);
+ schedule = ktime_sub(now, vars->drop_next);
+ next_due = vars->count && ktime_to_ns(schedule) >= 0;
+ }
+ }
+
+ /* Simple BLUE implementation. Lack of ECN is deliberate. */
+ if (vars->p_drop)
+ drop |= (prandom_u32() < vars->p_drop);
+
+ /* Overload the drop_next field as an activity timeout */
+ if (!vars->count)
+ vars->drop_next = ktime_add_ns(now, p->interval);
+ else if (ktime_to_ns(schedule) > 0 && !drop)
+ vars->drop_next = now;
+
+ return drop;
+}
+
+/* Cake has several subtle multiple bit settings. In these cases you
+ * would be matching triple isolate mode as well.
+ */
+
+static bool cake_dsrc(int flow_mode)
+{
+ return (flow_mode & CAKE_FLOW_DUAL_SRC) == CAKE_FLOW_DUAL_SRC;
+}
+
+static bool cake_ddst(int flow_mode)
+{
+ return (flow_mode & CAKE_FLOW_DUAL_DST) == CAKE_FLOW_DUAL_DST;
+}
+
+static u32 cake_hash(struct cake_tin_data *q, const struct sk_buff *skb,
+ int flow_mode)
+{
+ u32 flow_hash = 0, srchost_hash, dsthost_hash;
+ u16 reduced_hash, srchost_idx, dsthost_idx;
+ struct flow_keys keys, host_keys;
+
+ if (unlikely(flow_mode == CAKE_FLOW_NONE))
+ return 0;
+
+ skb_flow_dissect_flow_keys(skb, &keys,
+ FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL);
+
+ /* flow_hash_from_keys() sorts the addresses by value, so we have
+ * to preserve their order in a separate data structure to treat
+ * src and dst host addresses as independently selectable.
+ */
+ host_keys = keys;
+ host_keys.ports.ports = 0;
+ host_keys.basic.ip_proto = 0;
+ host_keys.keyid.keyid = 0;
+ host_keys.tags.flow_label = 0;
+
+ switch (host_keys.control.addr_type) {
+ case FLOW_DISSECTOR_KEY_IPV4_ADDRS:
+ host_keys.addrs.v4addrs.src = 0;
+ dsthost_hash = flow_hash_from_keys(&host_keys);
+ host_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
+ host_keys.addrs.v4addrs.dst = 0;
+ srchost_hash = flow_hash_from_keys(&host_keys);
+ break;
+
+ case FLOW_DISSECTOR_KEY_IPV6_ADDRS:
+ memset(&host_keys.addrs.v6addrs.src, 0,
+ sizeof(host_keys.addrs.v6addrs.src));
+ dsthost_hash = flow_hash_from_keys(&host_keys);
+ host_keys.addrs.v6addrs.src = keys.addrs.v6addrs.src;
+ memset(&host_keys.addrs.v6addrs.dst, 0,
+ sizeof(host_keys.addrs.v6addrs.dst));
+ srchost_hash = flow_hash_from_keys(&host_keys);
+ break;
+
+ default:
+ dsthost_hash = 0;
+ srchost_hash = 0;
+ }
+
+ /* This *must* be after the above switch, since as a
+ * side-effect it sorts the src and dst addresses.
+ */
+ if (flow_mode & CAKE_FLOW_FLOWS)
+ flow_hash = flow_hash_from_keys(&keys);
+
+ if (!(flow_mode & CAKE_FLOW_FLOWS)) {
+ if (flow_mode & CAKE_FLOW_SRC_IP)
+ flow_hash ^= srchost_hash;
+
+ if (flow_mode & CAKE_FLOW_DST_IP)
+ flow_hash ^= dsthost_hash;
+ }
+
+ reduced_hash = flow_hash % CAKE_QUEUES;
+
+ /* set-associative hashing */
+ /* fast path if no hash collision (direct lookup succeeds) */
+ if (likely(q->tags[reduced_hash] == flow_hash &&
+ q->flows[reduced_hash].set)) {
+ q->way_directs++;
+ } else {
+ u32 inner_hash = reduced_hash % CAKE_SET_WAYS;
+ u32 outer_hash = reduced_hash - inner_hash;
+ bool allocate_src = false;
+ bool allocate_dst = false;
+ u32 i, k;
+
+ /* check if any active queue in the set is reserved for
+ * this flow.
+ */
+ for (i = 0, k = inner_hash; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (q->tags[outer_hash + k] == flow_hash) {
+ if (i)
+ q->way_hits++;
+
+ if (!q->flows[outer_hash + k].set) {
+ /* need to increment host refcnts */
+ allocate_src = cake_dsrc(flow_mode);
+ allocate_dst = cake_ddst(flow_mode);
+ }
+
+ goto found;
+ }
+ }
+
+ /* no queue is reserved for this flow, look for an
+ * empty one.
+ */
+ for (i = 0; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (!q->flows[outer_hash + k].set) {
+ q->way_misses++;
+ allocate_src = cake_dsrc(flow_mode);
+ allocate_dst = cake_ddst(flow_mode);
+ goto found;
+ }
+ }
+
+ /* With no empty queues, default to the original
+ * queue, accept the collision, update the host tags.
+ */
+ q->way_collisions++;
+ q->hosts[q->flows[reduced_hash].srchost].srchost_refcnt--;
+ q->hosts[q->flows[reduced_hash].dsthost].dsthost_refcnt--;
+ allocate_src = cake_dsrc(flow_mode);
+ allocate_dst = cake_ddst(flow_mode);
+found:
+ /* reserve queue for future packets in same flow */
+ reduced_hash = outer_hash + k;
+ q->tags[reduced_hash] = flow_hash;
+
+ if (allocate_src) {
+ srchost_idx = srchost_hash % CAKE_QUEUES;
+ inner_hash = srchost_idx % CAKE_SET_WAYS;
+ outer_hash = srchost_idx - inner_hash;
+ for (i = 0, k = inner_hash; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (q->hosts[outer_hash + k].srchost_tag ==
+ srchost_hash)
+ goto found_src;
+ }
+ for (i = 0; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (!q->hosts[outer_hash + k].srchost_refcnt)
+ break;
+ }
+ q->hosts[outer_hash + k].srchost_tag = srchost_hash;
+found_src:
+ srchost_idx = outer_hash + k;
+ q->hosts[srchost_idx].srchost_refcnt++;
+ q->flows[reduced_hash].srchost = srchost_idx;
+ }
+
+ if (allocate_dst) {
+ dsthost_idx = dsthost_hash % CAKE_QUEUES;
+ inner_hash = dsthost_idx % CAKE_SET_WAYS;
+ outer_hash = dsthost_idx - inner_hash;
+ for (i = 0, k = inner_hash; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (q->hosts[outer_hash + k].dsthost_tag ==
+ dsthost_hash)
+ goto found_dst;
+ }
+ for (i = 0; i < CAKE_SET_WAYS;
+ i++, k = (k + 1) % CAKE_SET_WAYS) {
+ if (!q->hosts[outer_hash + k].dsthost_refcnt)
+ break;
+ }
+ q->hosts[outer_hash + k].dsthost_tag = dsthost_hash;
+found_dst:
+ dsthost_idx = outer_hash + k;
+ q->hosts[dsthost_idx].dsthost_refcnt++;
+ q->flows[reduced_hash].dsthost = dsthost_idx;
+ }
+ }
+
+ return reduced_hash;
+}
+
+static u32 cake_classify(struct Qdisc *sch, struct cake_tin_data *t,
+ struct sk_buff *skb, int flow_mode, int *qerr)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct tcf_proto *filter;
+ struct tcf_result res;
+ int result;
+
+ filter = rcu_dereference_bh(q->filter_list);
+ if (!filter)
+ return cake_hash(t, skb, flow_mode) + 1;
+
+ *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
+ result = tcf_classify(skb, filter, &res, false);
+ if (result >= 0) {
+#ifdef CONFIG_NET_CLS_ACT
+ switch (result) {
+ case TC_ACT_STOLEN:
+ case TC_ACT_QUEUED:
+ case TC_ACT_TRAP:
+ *qerr = NET_XMIT_SUCCESS | __NET_XMIT_STOLEN;
+ /* fall through */
+ case TC_ACT_SHOT:
+ return 0;
+ }
+#endif
+ if (TC_H_MIN(res.classid) <= CAKE_QUEUES)
+ return TC_H_MIN(res.classid);
+ }
+ return 0;
+}
+
+/* helper functions : might be changed when/if skb use a standard list_head */
+/* remove one skb from head of slot queue */
+
+static struct sk_buff *dequeue_head(struct cake_flow *flow)
+{
+ struct sk_buff *skb = flow->head;
+
+ if (skb) {
+ flow->head = skb->next;
+ skb->next = NULL;
+ }
+
+ return skb;
+}
+
+/* add skb to flow queue (tail add) */
+
+static void flow_queue_add(struct cake_flow *flow, struct sk_buff *skb)
+{
+ if (!flow->head)
+ flow->head = skb;
+ else
+ flow->tail->next = skb;
+ flow->tail = skb;
+ skb->next = NULL;
+}
+
+static u64 cake_ewma(u64 avg, u64 sample, u32 shift)
+{
+ avg -= avg >> shift;
+ avg += sample >> shift;
+ return avg;
+}
+
+static void cake_heap_swap(struct cake_sched_data *q, u16 i, u16 j)
+{
+ struct cake_heap_entry ii = q->overflow_heap[i];
+ struct cake_heap_entry jj = q->overflow_heap[j];
+
+ q->overflow_heap[i] = jj;
+ q->overflow_heap[j] = ii;
+
+ q->tins[ii.t].overflow_idx[ii.b] = j;
+ q->tins[jj.t].overflow_idx[jj.b] = i;
+}
+
+static u32 cake_heap_get_backlog(const struct cake_sched_data *q, u16 i)
+{
+ struct cake_heap_entry ii = q->overflow_heap[i];
+
+ return q->tins[ii.t].backlogs[ii.b];
+}
+
+static void cake_heapify(struct cake_sched_data *q, u16 i)
+{
+ static const u32 a = CAKE_MAX_TINS * CAKE_QUEUES;
+ u32 mb = cake_heap_get_backlog(q, i);
+ u32 m = i;
+
+ while (m < a) {
+ u32 l = m + m + 1;
+ u32 r = l + 1;
+
+ if (l < a) {
+ u32 lb = cake_heap_get_backlog(q, l);
+
+ if (lb > mb) {
+ m = l;
+ mb = lb;
+ }
+ }
+
+ if (r < a) {
+ u32 rb = cake_heap_get_backlog(q, r);
+
+ if (rb > mb) {
+ m = r;
+ mb = rb;
+ }
+ }
+
+ if (m != i) {
+ cake_heap_swap(q, i, m);
+ i = m;
+ } else {
+ break;
+ }
+ }
+}
+
+static void cake_heapify_up(struct cake_sched_data *q, u16 i)
+{
+ while (i > 0 && i < CAKE_MAX_TINS * CAKE_QUEUES) {
+ u16 p = (i - 1) >> 1;
+ u32 ib = cake_heap_get_backlog(q, i);
+ u32 pb = cake_heap_get_backlog(q, p);
+
+ if (ib > pb) {
+ cake_heap_swap(q, i, p);
+ i = p;
+ } else {
+ break;
+ }
+ }
+}
+
+static int cake_advance_shaper(struct cake_sched_data *q,
+ struct cake_tin_data *b,
+ struct sk_buff *skb,
+ ktime_t now, bool drop)
+{
+ u32 len = qdisc_pkt_len(skb);
+
+ /* charge packet bandwidth to this tin
+ * and to the global shaper.
+ */
+ if (q->rate_ns) {
+ u64 tin_dur = (len * b->tin_rate_ns) >> b->tin_rate_shft;
+ u64 global_dur = (len * q->rate_ns) >> q->rate_shft;
+ u64 failsafe_dur = global_dur + (global_dur >> 1);
+
+ if (ktime_before(b->time_next_packet, now))
+ b->time_next_packet = ktime_add_ns(b->time_next_packet,
+ tin_dur);
+
+ else if (ktime_before(b->time_next_packet,
+ ktime_add_ns(now, tin_dur)))
+ b->time_next_packet = ktime_add_ns(now, tin_dur);
+
+ q->time_next_packet = ktime_add_ns(q->time_next_packet,
+ global_dur);
+ if (!drop)
+ q->failsafe_next_packet = \
+ ktime_add_ns(q->failsafe_next_packet,
+ failsafe_dur);
+ }
+ return len;
+}
+
+static unsigned int cake_drop(struct Qdisc *sch, struct sk_buff **to_free)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ ktime_t now = ktime_get();
+ u32 idx = 0, tin = 0, len;
+ struct cake_heap_entry qq;
+ struct cake_tin_data *b;
+ struct cake_flow *flow;
+ struct sk_buff *skb;
+
+ if (!q->overflow_timeout) {
+ int i;
+ /* Build fresh max-heap */
+ for (i = CAKE_MAX_TINS * CAKE_QUEUES / 2; i >= 0; i--)
+ cake_heapify(q, i);
+ }
+ q->overflow_timeout = 65535;
+
+ /* select longest queue for pruning */
+ qq = q->overflow_heap[0];
+ tin = qq.t;
+ idx = qq.b;
+
+ b = &q->tins[tin];
+ flow = &b->flows[idx];
+ skb = dequeue_head(flow);
+ if (unlikely(!skb)) {
+ /* heap has gone wrong, rebuild it next time */
+ q->overflow_timeout = 0;
+ return idx + (tin << 16);
+ }
+
+ if (cobalt_queue_full(&flow->cvars, &b->cparams, now))
+ b->unresponsive_flow_count++;
+
+ len = qdisc_pkt_len(skb);
+ q->buffer_used -= skb->truesize;
+ b->backlogs[idx] -= len;
+ b->tin_backlog -= len;
+ sch->qstats.backlog -= len;
+ qdisc_tree_reduce_backlog(sch, 1, len);
+
+ flow->dropped++;
+ b->tin_dropped++;
+ sch->qstats.drops++;
+
+ __qdisc_drop(skb, to_free);
+ sch->q.qlen--;
+
+ cake_heapify(q, 0);
+
+ return idx + (tin << 16);
+}
+
+static void cake_reconfigure(struct Qdisc *sch);
+
+static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
+ struct sk_buff **to_free)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ int len = qdisc_pkt_len(skb);
+ int uninitialized_var(ret);
+ ktime_t now = ktime_get();
+ struct cake_tin_data *b;
+ struct cake_flow *flow;
+ u32 idx, tin;
+
+ tin = 0;
+ b = &q->tins[tin];
+
+ /* choose flow to insert into */
+ idx = cake_classify(sch, b, skb, q->flow_mode, &ret);
+ if (idx == 0) {
+ if (ret & __NET_XMIT_BYPASS)
+ qdisc_qstats_drop(sch);
+ __qdisc_drop(skb, to_free);
+ return ret;
+ }
+ idx--;
+ flow = &b->flows[idx];
+
+ /* ensure shaper state isn't stale */
+ if (!b->tin_backlog) {
+ if (ktime_before(b->time_next_packet, now))
+ b->time_next_packet = now;
+
+ if (!sch->q.qlen) {
+ if (ktime_before(q->time_next_packet, now)) {
+ q->failsafe_next_packet = now;
+ q->time_next_packet = now;
+ } else if (ktime_after(q->time_next_packet, now) &&
+ ktime_after(q->failsafe_next_packet, now)) {
+ u64 next = \
+ min(ktime_to_ns(q->time_next_packet),
+ ktime_to_ns(
+ q->failsafe_next_packet));
+ sch->qstats.overlimits++;
+ qdisc_watchdog_schedule_ns(&q->watchdog, next);
+ }
+ }
+ }
+
+ if (unlikely(len > b->max_skblen))
+ b->max_skblen = len;
+
+ cobalt_set_enqueue_time(skb, now);
+ flow_queue_add(flow, skb);
+
+ sch->q.qlen++;
+ q->buffer_used += skb->truesize;
+
+ /* stats */
+ b->packets++;
+ b->bytes += len;
+ b->backlogs[idx] += len;
+ b->tin_backlog += len;
+ sch->qstats.backlog += len;
+ q->avg_window_bytes += len;
+
+ if (q->overflow_timeout)
+ cake_heapify_up(q, b->overflow_idx[idx]);
+
+ /* incoming bandwidth capacity estimate */
+ q->avg_window_bytes = 0;
+ q->last_packet_time = now;
+
+ /* flowchain */
+ if (!flow->set || flow->set == CAKE_SET_DECAYING) {
+ struct cake_host *srchost = &b->hosts[flow->srchost];
+ struct cake_host *dsthost = &b->hosts[flow->dsthost];
+ u16 host_load = 1;
+
+ if (!flow->set) {
+ list_add_tail(&flow->flowchain, &b->new_flows);
+ } else {
+ b->decaying_flow_count--;
+ list_move_tail(&flow->flowchain, &b->new_flows);
+ }
+ flow->set = CAKE_SET_SPARSE;
+ b->sparse_flow_count++;
+
+ if (cake_dsrc(q->flow_mode))
+ host_load = max(host_load, srchost->srchost_refcnt);
+
+ if (cake_ddst(q->flow_mode))
+ host_load = max(host_load, dsthost->dsthost_refcnt);
+
+ flow->deficit = (b->flow_quantum *
+ quantum_div[host_load]) >> 16;
+ } else if (flow->set == CAKE_SET_SPARSE_WAIT) {
+ /* this flow was empty, accounted as a sparse flow, but actually
+ * in the bulk rotation.
+ */
+ flow->set = CAKE_SET_BULK;
+ b->sparse_flow_count--;
+ b->bulk_flow_count++;
+ }
+
+ if (q->buffer_used > q->buffer_max_used)
+ q->buffer_max_used = q->buffer_used;
+
+ if (q->buffer_used > q->buffer_limit) {
+ u32 dropped = 0;
+
+ while (q->buffer_used > q->buffer_limit) {
+ dropped++;
+ cake_drop(sch, to_free);
+ }
+ b->drop_overlimit += dropped;
+ }
+ return NET_XMIT_SUCCESS;
+}
+
+static struct sk_buff *cake_dequeue_one(struct Qdisc *sch)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct cake_tin_data *b = &q->tins[q->cur_tin];
+ struct cake_flow *flow = &b->flows[q->cur_flow];
+ struct sk_buff *skb = NULL;
+ u32 len;
+
+ if (flow->head) {
+ skb = dequeue_head(flow);
+ len = qdisc_pkt_len(skb);
+ b->backlogs[q->cur_flow] -= len;
+ b->tin_backlog -= len;
+ sch->qstats.backlog -= len;
+ q->buffer_used -= skb->truesize;
+ sch->q.qlen--;
+
+ if (q->overflow_timeout)
+ cake_heapify(q, b->overflow_idx[q->cur_flow]);
+ }
+ return skb;
+}
+
+/* Discard leftover packets from a tin no longer in use. */
+static void cake_clear_tin(struct Qdisc *sch, u16 tin)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct sk_buff *skb;
+
+ q->cur_tin = tin;
+ for (q->cur_flow = 0; q->cur_flow < CAKE_QUEUES; q->cur_flow++)
+ while (!!(skb = cake_dequeue_one(sch)))
+ kfree_skb(skb);
+}
+
+static struct sk_buff *cake_dequeue(struct Qdisc *sch)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct cake_tin_data *b = &q->tins[q->cur_tin];
+ struct cake_host *srchost, *dsthost;
+ ktime_t now = ktime_get();
+ struct cake_flow *flow;
+ struct list_head *head;
+ bool first_flow = true;
+ struct sk_buff *skb;
+ u16 host_load;
+ u64 delay;
+ u32 len;
+
+begin:
+ if (!sch->q.qlen)
+ return NULL;
+
+ /* global hard shaper */
+ if (ktime_after(q->time_next_packet, now) &&
+ ktime_after(q->failsafe_next_packet, now)) {
+ u64 next = min(ktime_to_ns(q->time_next_packet),
+ ktime_to_ns(q->failsafe_next_packet));
+
+ sch->qstats.overlimits++;
+ qdisc_watchdog_schedule_ns(&q->watchdog, next);
+ return NULL;
+ }
+
+ /* Choose a class to work on. */
+ if (!q->rate_ns) {
+ /* In unlimited mode, can't rely on shaper timings, just balance
+ * with DRR
+ */
+ while (b->tin_deficit < 0 ||
+ !(b->sparse_flow_count + b->bulk_flow_count)) {
+ if (b->tin_deficit <= 0)
+ b->tin_deficit += b->tin_quantum_band;
+
+ q->cur_tin++;
+ b++;
+ if (q->cur_tin >= q->tin_cnt) {
+ q->cur_tin = 0;
+ b = q->tins;
+ }
+ }
+ } else {
+ /* In shaped mode, choose:
+ * - Highest-priority tin with queue and meeting schedule, or
+ * - The earliest-scheduled tin with queue.
+ */
+ ktime_t best_time = KTIME_MAX;
+ int tin, best_tin = 0;
+
+ for (tin = 0; tin < q->tin_cnt; tin++) {
+ b = q->tins + tin;
+ if ((b->sparse_flow_count + b->bulk_flow_count) > 0) {
+ ktime_t time_to_pkt = \
+ ktime_sub(b->time_next_packet, now);
+
+ if (ktime_to_ns(time_to_pkt) <= 0 ||
+ ktime_compare(time_to_pkt,
+ best_time) <= 0) {
+ best_time = time_to_pkt;
+ best_tin = tin;
+ }
+ }
+ }
+
+ q->cur_tin = best_tin;
+ b = q->tins + best_tin;
+ }
+
+retry:
+ /* service this class */
+ head = &b->decaying_flows;
+ if (!first_flow || list_empty(head)) {
+ head = &b->new_flows;
+ if (list_empty(head)) {
+ head = &b->old_flows;
+ if (unlikely(list_empty(head))) {
+ head = &b->decaying_flows;
+ if (unlikely(list_empty(head)))
+ goto begin;
+ }
+ }
+ }
+ flow = list_first_entry(head, struct cake_flow, flowchain);
+ q->cur_flow = flow - b->flows;
+ first_flow = false;
+
+ /* triple isolation (modified DRR++) */
+ srchost = &b->hosts[flow->srchost];
+ dsthost = &b->hosts[flow->dsthost];
+ host_load = 1;
+
+ if (cake_dsrc(q->flow_mode))
+ host_load = max(host_load, srchost->srchost_refcnt);
+
+ if (cake_ddst(q->flow_mode))
+ host_load = max(host_load, dsthost->dsthost_refcnt);
+
+ WARN_ON(host_load > CAKE_QUEUES);
+
+ /* flow isolation (DRR++) */
+ if (flow->deficit <= 0) {
+ /* The shifted prandom_u32() is a way to apply dithering to
+ * avoid accumulating roundoff errors
+ */
+ flow->deficit += (b->flow_quantum * quantum_div[host_load] +
+ (prandom_u32() >> 16)) >> 16;
+ list_move_tail(&flow->flowchain, &b->old_flows);
+
+ /* Keep all flows with deficits out of the sparse and decaying
+ * rotations. No non-empty flow can go into the decaying
+ * rotation, so they can't get deficits
+ */
+ if (flow->set == CAKE_SET_SPARSE) {
+ if (flow->head) {
+ b->sparse_flow_count--;
+ b->bulk_flow_count++;
+ flow->set = CAKE_SET_BULK;
+ } else {
+ /* we've moved it to the bulk rotation for
+ * correct deficit accounting but we still want
+ * to count it as a sparse flow, not a bulk one.
+ */
+ flow->set = CAKE_SET_SPARSE_WAIT;
+ }
+ }
+ goto retry;
+ }
+
+ /* Retrieve a packet via the AQM */
+ while (1) {
+ skb = cake_dequeue_one(sch);
+ if (!skb) {
+ /* this queue was actually empty */
+ if (cobalt_queue_empty(&flow->cvars, &b->cparams, now))
+ b->unresponsive_flow_count--;
+
+ if (flow->cvars.p_drop || flow->cvars.count ||
+ ktime_before(now, flow->cvars.drop_next)) {
+ /* keep in the flowchain until the state has
+ * decayed to rest
+ */
+ list_move_tail(&flow->flowchain,
+ &b->decaying_flows);
+ if (flow->set == CAKE_SET_BULK) {
+ b->bulk_flow_count--;
+ b->decaying_flow_count++;
+ } else if (flow->set == CAKE_SET_SPARSE ||
+ flow->set == CAKE_SET_SPARSE_WAIT) {
+ b->sparse_flow_count--;
+ b->decaying_flow_count++;
+ }
+ flow->set = CAKE_SET_DECAYING;
+ } else {
+ /* remove empty queue from the flowchain */
+ list_del_init(&flow->flowchain);
+ if (flow->set == CAKE_SET_SPARSE ||
+ flow->set == CAKE_SET_SPARSE_WAIT)
+ b->sparse_flow_count--;
+ else if (flow->set == CAKE_SET_BULK)
+ b->bulk_flow_count--;
+ else
+ b->decaying_flow_count--;
+
+ flow->set = CAKE_SET_NONE;
+ srchost->srchost_refcnt--;
+ dsthost->dsthost_refcnt--;
+ }
+ goto begin;
+ }
+
+ /* Last packet in queue may be marked, shouldn't be dropped */
+ if (!cobalt_should_drop(&flow->cvars, &b->cparams, now, skb) ||
+ !flow->head)
+ break;
+
+ flow->dropped++;
+ b->tin_dropped++;
+ qdisc_tree_reduce_backlog(sch, 1, qdisc_pkt_len(skb));
+ qdisc_qstats_drop(sch);
+ kfree_skb(skb);
+ }
+
+ b->tin_ecn_mark += !!flow->cvars.ecn_marked;
+ qdisc_bstats_update(sch, skb);
+
+ /* collect delay stats */
+ delay = ktime_to_ns(ktime_sub(now, cobalt_get_enqueue_time(skb)));
+ b->avge_delay = cake_ewma(b->avge_delay, delay, 8);
+ b->peak_delay = cake_ewma(b->peak_delay, delay,
+ delay > b->peak_delay ? 2 : 8);
+ b->base_delay = cake_ewma(b->base_delay, delay,
+ delay < b->base_delay ? 2 : 8);
+
+ len = cake_advance_shaper(q, b, skb, now, false);
+ flow->deficit -= len;
+ b->tin_deficit -= len;
+
+ if (ktime_after(q->time_next_packet, now) && sch->q.qlen) {
+ u64 next = min(ktime_to_ns(q->time_next_packet),
+ ktime_to_ns(q->failsafe_next_packet));
+
+ qdisc_watchdog_schedule_ns(&q->watchdog, next);
+ } else if (!sch->q.qlen) {
+ int i;
+
+ for (i = 0; i < q->tin_cnt; i++) {
+ if (q->tins[i].decaying_flow_count) {
+ ktime_t next = \
+ ktime_add_ns(now,
+ q->tins[i].cparams.target);
+
+ qdisc_watchdog_schedule_ns(&q->watchdog,
+ ktime_to_ns(next));
+ break;
+ }
+ }
+ }
+
+ if (q->overflow_timeout)
+ q->overflow_timeout--;
+
+ return skb;
+}
+
+static void cake_reset(struct Qdisc *sch)
+{
+ u32 c;
+
+ for (c = 0; c < CAKE_MAX_TINS; c++)
+ cake_clear_tin(sch, c);
+}
+
+static const struct nla_policy cake_policy[TCA_CAKE_MAX + 1] = {
+ [TCA_CAKE_BASE_RATE64] = { .type = NLA_U64 },
+ [TCA_CAKE_DIFFSERV_MODE] = { .type = NLA_U32 },
+ [TCA_CAKE_ATM] = { .type = NLA_U32 },
+ [TCA_CAKE_FLOW_MODE] = { .type = NLA_U32 },
+ [TCA_CAKE_OVERHEAD] = { .type = NLA_S32 },
+ [TCA_CAKE_RTT] = { .type = NLA_U32 },
+ [TCA_CAKE_TARGET] = { .type = NLA_U32 },
+ [TCA_CAKE_AUTORATE] = { .type = NLA_U32 },
+ [TCA_CAKE_MEMORY] = { .type = NLA_U32 },
+ [TCA_CAKE_NAT] = { .type = NLA_U32 },
+ [TCA_CAKE_RAW] = { .type = NLA_U32 },
+ [TCA_CAKE_WASH] = { .type = NLA_U32 },
+ [TCA_CAKE_MPU] = { .type = NLA_U32 },
+ [TCA_CAKE_INGRESS] = { .type = NLA_U32 },
+ [TCA_CAKE_ACK_FILTER] = { .type = NLA_U32 },
+};
+
+static void cake_set_rate(struct cake_tin_data *b, u64 rate, u32 mtu,
+ u64 target_ns, u64 rtt_est_ns)
+{
+ /* convert byte-rate into time-per-byte
+ * so it will always unwedge in reasonable time.
+ */
+ static const u64 MIN_RATE = 64;
+ u32 byte_target = mtu;
+ u64 byte_target_ns;
+ u8 rate_shft = 0;
+ u64 rate_ns = 0;
+
+ b->flow_quantum = 1514;
+ if (rate) {
+ b->flow_quantum = max(min(rate >> 12, 1514ULL), 300ULL);
+ rate_shft = 34;
+ rate_ns = ((u64)NSEC_PER_SEC) << rate_shft;
+ rate_ns = div64_u64(rate_ns, max(MIN_RATE, rate));
+ while (!!(rate_ns >> 34)) {
+ rate_ns >>= 1;
+ rate_shft--;
+ }
+ } /* else unlimited, ie. zero delay */
+
+ b->tin_rate_bps = rate;
+ b->tin_rate_ns = rate_ns;
+ b->tin_rate_shft = rate_shft;
+
+ byte_target_ns = (byte_target * rate_ns) >> rate_shft;
+
+ b->cparams.target = max((byte_target_ns * 3) / 2, target_ns);
+ b->cparams.interval = max(rtt_est_ns +
+ b->cparams.target - target_ns,
+ b->cparams.target * 2);
+ b->cparams.mtu_time = byte_target_ns;
+ b->cparams.p_inc = 1 << 24; /* 1/256 */
+ b->cparams.p_dec = 1 << 20; /* 1/4096 */
+}
+
+static void cake_reconfigure(struct Qdisc *sch)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct cake_tin_data *b = &q->tins[0];
+ int c, ft = 0;
+
+ q->tin_cnt = 1;
+ cake_set_rate(b, q->rate_bps, psched_mtu(qdisc_dev(sch)),
+ us_to_ns(q->target), us_to_ns(q->interval));
+ b->tin_quantum_band = 65535;
+ b->tin_quantum_prio = 65535;
+
+ for (c = q->tin_cnt; c < CAKE_MAX_TINS; c++) {
+ cake_clear_tin(sch, c);
+ q->tins[c].cparams.mtu_time = q->tins[ft].cparams.mtu_time;
+ }
+
+ q->rate_ns = q->tins[ft].tin_rate_ns;
+ q->rate_shft = q->tins[ft].tin_rate_shft;
+
+ if (q->buffer_config_limit) {
+ q->buffer_limit = q->buffer_config_limit;
+ } else if (q->rate_bps) {
+ u64 t = q->rate_bps * q->interval;
+
+ do_div(t, USEC_PER_SEC / 4);
+ q->buffer_limit = max_t(u32, t, 4U << 20);
+ } else {
+ q->buffer_limit = ~0;
+ }
+
+ sch->flags &= ~TCQ_F_CAN_BYPASS;
+
+ q->buffer_limit = min(q->buffer_limit,
+ max(sch->limit * psched_mtu(qdisc_dev(sch)),
+ q->buffer_config_limit));
+}
+
+static int cake_change(struct Qdisc *sch, struct nlattr *opt,
+ struct netlink_ext_ack *extack)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct nlattr *tb[TCA_CAKE_MAX + 1];
+ int err;
+
+ if (!opt)
+ return -EINVAL;
+
+ err = nla_parse_nested(tb, TCA_CAKE_MAX, opt, cake_policy, extack);
+ if (err < 0)
+ return err;
+
+ if (tb[TCA_CAKE_BASE_RATE64])
+ q->rate_bps = nla_get_u64(tb[TCA_CAKE_BASE_RATE64]);
+
+ if (tb[TCA_CAKE_FLOW_MODE])
+ q->flow_mode = (nla_get_u32(tb[TCA_CAKE_FLOW_MODE]) &
+ CAKE_FLOW_MASK);
+
+ if (tb[TCA_CAKE_RTT]) {
+ q->interval = nla_get_u32(tb[TCA_CAKE_RTT]);
+
+ if (!q->interval)
+ q->interval = 1;
+ }
+
+ if (tb[TCA_CAKE_TARGET]) {
+ q->target = nla_get_u32(tb[TCA_CAKE_TARGET]);
+
+ if (!q->target)
+ q->target = 1;
+ }
+
+ if (tb[TCA_CAKE_MEMORY])
+ q->buffer_config_limit = nla_get_u32(tb[TCA_CAKE_MEMORY]);
+
+ if (q->tins) {
+ sch_tree_lock(sch);
+ cake_reconfigure(sch);
+ sch_tree_unlock(sch);
+ }
+
+ return 0;
+}
+
+static void cake_destroy(struct Qdisc *sch)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+
+ qdisc_watchdog_cancel(&q->watchdog);
+ tcf_block_put(q->block);
+ kvfree(q->tins);
+}
+
+static int cake_init(struct Qdisc *sch, struct nlattr *opt,
+ struct netlink_ext_ack *extack)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ int i, j, err;
+
+ sch->limit = 10240;
+ q->tin_mode = CAKE_DIFFSERV_BESTEFFORT;
+ q->flow_mode = CAKE_FLOW_TRIPLE;
+
+ q->rate_bps = 0; /* unlimited by default */
+
+ q->interval = 100000; /* 100ms default */
+ q->target = 5000; /* 5ms: codel RFC argues
+ * for 5 to 10% of interval
+ */
+
+ q->cur_tin = 0;
+ q->cur_flow = 0;
+
+ if (opt) {
+ int err = cake_change(sch, opt, extack);
+
+ if (err)
+ return err;
+ }
+
+ err = tcf_block_get(&q->block, &q->filter_list, sch, extack);
+ if (err)
+ return err;
+
+ qdisc_watchdog_init(&q->watchdog, sch);
+
+ quantum_div[0] = ~0;
+ for (i = 1; i <= CAKE_QUEUES; i++)
+ quantum_div[i] = 65535 / i;
+
+ q->tins = kvzalloc(CAKE_MAX_TINS * sizeof(struct cake_tin_data),
+ GFP_KERNEL);
+ if (!q->tins)
+ goto nomem;
+
+ for (i = 0; i < CAKE_MAX_TINS; i++) {
+ struct cake_tin_data *b = q->tins + i;
+
+ INIT_LIST_HEAD(&b->new_flows);
+ INIT_LIST_HEAD(&b->old_flows);
+ INIT_LIST_HEAD(&b->decaying_flows);
+ b->sparse_flow_count = 0;
+ b->bulk_flow_count = 0;
+ b->decaying_flow_count = 0;
+
+ for (j = 0; j < CAKE_QUEUES; j++) {
+ struct cake_flow *flow = b->flows + j;
+ u32 k = j * CAKE_MAX_TINS + i;
+
+ INIT_LIST_HEAD(&flow->flowchain);
+ cobalt_vars_init(&flow->cvars);
+
+ q->overflow_heap[k].t = i;
+ q->overflow_heap[k].b = j;
+ b->overflow_idx[j] = k;
+ }
+ }
+
+ cake_reconfigure(sch);
+ q->avg_peak_bandwidth = q->rate_bps;
+ q->min_netlen = ~0;
+ q->min_adjlen = ~0;
+ return 0;
+
+nomem:
+ cake_destroy(sch);
+ return -ENOMEM;
+}
+
+static int cake_dump(struct Qdisc *sch, struct sk_buff *skb)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct nlattr *opts;
+
+ opts = nla_nest_start(skb, TCA_OPTIONS);
+ if (!opts)
+ goto nla_put_failure;
+
+ if (nla_put_u64_64bit(skb, TCA_CAKE_BASE_RATE64, q->rate_bps,
+ TCA_CAKE_PAD))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_FLOW_MODE,
+ q->flow_mode & CAKE_FLOW_MASK))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_RTT, q->interval))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_TARGET, q->target))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_MEMORY, q->buffer_config_limit))
+ goto nla_put_failure;
+
+ return nla_nest_end(skb, opts);
+
+nla_put_failure:
+ return -1;
+}
+
+static int cake_dump_stats(struct Qdisc *sch, struct gnet_dump *d)
+{
+ struct nlattr *stats = nla_nest_start(d->skb, TCA_STATS_APP);
+ struct cake_sched_data *q = qdisc_priv(sch);
+ struct nlattr *tstats, *ts;
+ int i;
+
+ if (!stats)
+ return -1;
+
+#define PUT_STAT_U32(attr, data) do { \
+ if (nla_put_u32(d->skb, TCA_CAKE_STATS_ ## attr, data)) \
+ goto nla_put_failure; \
+ } while (0)
+#define PUT_STAT_U64(attr, data) do { \
+ if (nla_put_u64_64bit(d->skb, TCA_CAKE_STATS_ ## attr, \
+ data, TCA_CAKE_STATS_PAD)) \
+ goto nla_put_failure; \
+ } while (0)
+
+ PUT_STAT_U64(CAPACITY_ESTIMATE64, q->avg_peak_bandwidth);
+ PUT_STAT_U32(MEMORY_LIMIT, q->buffer_limit);
+ PUT_STAT_U32(MEMORY_USED, q->buffer_max_used);
+ PUT_STAT_U32(AVG_NETOFF, ((q->avg_netoff + 0x8000) >> 16));
+ PUT_STAT_U32(MAX_NETLEN, q->max_netlen);
+ PUT_STAT_U32(MAX_ADJLEN, q->max_adjlen);
+ PUT_STAT_U32(MIN_NETLEN, q->min_netlen);
+ PUT_STAT_U32(MIN_ADJLEN, q->min_adjlen);
+
+#undef PUT_STAT_U32
+#undef PUT_STAT_U64
+
+ tstats = nla_nest_start(d->skb, TCA_CAKE_STATS_TIN_STATS);
+ if (!tstats)
+ goto nla_put_failure;
+
+#define PUT_TSTAT_U32(attr, data) do { \
+ if (nla_put_u32(d->skb, TCA_CAKE_TIN_STATS_ ## attr, data)) \
+ goto nla_put_failure; \
+ } while (0)
+#define PUT_TSTAT_U64(attr, data) do { \
+ if (nla_put_u64_64bit(d->skb, TCA_CAKE_TIN_STATS_ ## attr, \
+ data, TCA_CAKE_TIN_STATS_PAD)) \
+ goto nla_put_failure; \
+ } while (0)
+
+ for (i = 0; i < q->tin_cnt; i++) {
+ struct cake_tin_data *b = &q->tins[i];
+
+ ts = nla_nest_start(d->skb, i + 1);
+ if (!ts)
+ goto nla_put_failure;
+
+ PUT_TSTAT_U64(THRESHOLD_RATE64, b->tin_rate_bps);
+ PUT_TSTAT_U64(SENT_BYTES64, b->bytes);
+ PUT_TSTAT_U32(BACKLOG_BYTES, b->tin_backlog);
+
+ PUT_TSTAT_U32(TARGET_US,
+ ktime_to_us(ns_to_ktime(b->cparams.target)));
+ PUT_TSTAT_U32(INTERVAL_US,
+ ktime_to_us(ns_to_ktime(b->cparams.interval)));
+
+ PUT_TSTAT_U32(SENT_PACKETS, b->packets);
+ PUT_TSTAT_U32(DROPPED_PACKETS, b->tin_dropped);
+ PUT_TSTAT_U32(ECN_MARKED_PACKETS, b->tin_ecn_mark);
+ PUT_TSTAT_U32(ACKS_DROPPED_PACKETS, b->ack_drops);
+
+ PUT_TSTAT_U32(PEAK_DELAY_US,
+ ktime_to_us(ns_to_ktime(b->peak_delay)));
+ PUT_TSTAT_U32(AVG_DELAY_US,
+ ktime_to_us(ns_to_ktime(b->avge_delay)));
+ PUT_TSTAT_U32(BASE_DELAY_US,
+ ktime_to_us(ns_to_ktime(b->base_delay)));
+
+ PUT_TSTAT_U32(WAY_INDIRECT_HITS, b->way_hits);
+ PUT_TSTAT_U32(WAY_MISSES, b->way_misses);
+ PUT_TSTAT_U32(WAY_COLLISIONS, b->way_collisions);
+
+ PUT_TSTAT_U32(SPARSE_FLOWS, b->sparse_flow_count +
+ b->decaying_flow_count);
+ PUT_TSTAT_U32(BULK_FLOWS, b->bulk_flow_count);
+ PUT_TSTAT_U32(UNRESPONSIVE_FLOWS, b->unresponsive_flow_count);
+ PUT_TSTAT_U32(MAX_SKBLEN, b->max_skblen);
+
+ PUT_TSTAT_U32(FLOW_QUANTUM, b->flow_quantum);
+ nla_nest_end(d->skb, ts);
+ }
+
+#undef PUT_TSTAT_U32
+#undef PUT_TSTAT_U64
+
+ nla_nest_end(d->skb, tstats);
+ return nla_nest_end(d->skb, stats);
+
+nla_put_failure:
+ nla_nest_cancel(d->skb, stats);
+ return -1;
+}
+
+static struct Qdisc *cake_leaf(struct Qdisc *sch, unsigned long arg)
+{
+ return NULL;
+}
+
+static unsigned long cake_find(struct Qdisc *sch, u32 classid)
+{
+ return 0;
+}
+
+static unsigned long cake_bind(struct Qdisc *sch, unsigned long parent,
+ u32 classid)
+{
+ return 0;
+}
+
+static void cake_unbind(struct Qdisc *q, unsigned long cl)
+{
+}
+
+static struct tcf_block *cake_tcf_block(struct Qdisc *sch, unsigned long cl,
+ struct netlink_ext_ack *extack)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+
+ if (cl)
+ return NULL;
+ return q->block;
+}
+
+static int cake_dump_class(struct Qdisc *sch, unsigned long cl,
+ struct sk_buff *skb, struct tcmsg *tcm)
+{
+ tcm->tcm_handle |= TC_H_MIN(cl);
+ return 0;
+}
+
+static int cake_dump_class_stats(struct Qdisc *sch, unsigned long cl,
+ struct gnet_dump *d)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ const struct cake_flow *flow = NULL;
+ struct gnet_stats_queue qs = { 0 };
+ struct nlattr *stats;
+ u32 idx = cl - 1;
+
+ if (idx < CAKE_QUEUES * q->tin_cnt) {
+ const struct cake_tin_data *b = &q->tins[idx / CAKE_QUEUES];
+ const struct sk_buff *skb;
+
+ flow = &b->flows[idx % CAKE_QUEUES];
+
+ if (flow->head) {
+ sch_tree_lock(sch);
+ skb = flow->head;
+ while (skb) {
+ qs.qlen++;
+ skb = skb->next;
+ }
+ sch_tree_unlock(sch);
+ }
+ qs.backlog = b->backlogs[idx % CAKE_QUEUES];
+ qs.drops = flow->dropped;
+ }
+ if (gnet_stats_copy_queue(d, NULL, &qs, qs.qlen) < 0)
+ return -1;
+ if (flow) {
+ ktime_t now = ktime_get();
+
+ stats = nla_nest_start(d->skb, TCA_STATS_APP);
+ if (!stats)
+ return -1;
+
+#define PUT_STAT_U32(attr, data) do { \
+ if (nla_put_u32(d->skb, TCA_CAKE_STATS_ ## attr, data)) \
+ goto nla_put_failure; \
+ } while (0)
+#define PUT_STAT_S32(attr, data) do { \
+ if (nla_put_s32(d->skb, TCA_CAKE_STATS_ ## attr, data)) \
+ goto nla_put_failure; \
+ } while (0)
+
+ PUT_STAT_S32(DEFICIT, flow->deficit);
+ PUT_STAT_U32(DROPPING, flow->cvars.dropping);
+ PUT_STAT_U32(COBALT_COUNT, flow->cvars.count);
+ PUT_STAT_U32(P_DROP, flow->cvars.p_drop);
+ if (flow->cvars.p_drop) {
+ PUT_STAT_S32(BLUE_TIMER_US,
+ ktime_to_us(
+ ktime_sub(now,
+ flow->cvars.blue_timer)));
+ }
+ if (flow->cvars.dropping) {
+ PUT_STAT_S32(DROP_NEXT_US,
+ ktime_to_us(
+ ktime_sub(now,
+ flow->cvars.drop_next)));
+ }
+
+ if (nla_nest_end(d->skb, stats) < 0)
+ return -1;
+ }
+
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(d->skb, stats);
+ return -1;
+}
+
+static void cake_walk(struct Qdisc *sch, struct qdisc_walker *arg)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ unsigned int i, j;
+
+ if (arg->stop)
+ return;
+
+ for (i = 0; i < q->tin_cnt; i++) {
+ struct cake_tin_data *b = &q->tins[i];
+
+ for (j = 0; j < CAKE_QUEUES; j++) {
+ if (list_empty(&b->flows[j].flowchain) ||
+ arg->count < arg->skip) {
+ arg->count++;
+ continue;
+ }
+ if (arg->fn(sch, i * CAKE_QUEUES + j + 1, arg) < 0) {
+ arg->stop = 1;
+ break;
+ }
+ arg->count++;
+ }
+ }
+}
+
+static const struct Qdisc_class_ops cake_class_ops = {
+ .leaf = cake_leaf,
+ .find = cake_find,
+ .tcf_block = cake_tcf_block,
+ .bind_tcf = cake_bind,
+ .unbind_tcf = cake_unbind,
+ .dump = cake_dump_class,
+ .dump_stats = cake_dump_class_stats,
+ .walk = cake_walk,
+};
+
+static struct Qdisc_ops cake_qdisc_ops __read_mostly = {
+ .cl_ops = &cake_class_ops,
+ .id = "cake",
+ .priv_size = sizeof(struct cake_sched_data),
+ .enqueue = cake_enqueue,
+ .dequeue = cake_dequeue,
+ .peek = qdisc_peek_dequeued,
+ .init = cake_init,
+ .reset = cake_reset,
+ .destroy = cake_destroy,
+ .change = cake_change,
+ .dump = cake_dump,
+ .dump_stats = cake_dump_stats,
+ .owner = THIS_MODULE,
+};
+
+static int __init cake_module_init(void)
+{
+ return register_qdisc(&cake_qdisc_ops);
+}
+
+static void __exit cake_module_exit(void)
+{
+ unregister_qdisc(&cake_qdisc_ops);
+}
+
+module_init(cake_module_init)
+module_exit(cake_module_exit)
+MODULE_AUTHOR("Jonathan Morton");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("The CAKE shaper.");
^ permalink raw reply related
* [PATCH net-next v17 6/8] sch_cake: Add DiffServ handling
From: Toke Høiland-Jørgensen @ 2018-05-28 22:28 UTC (permalink / raw)
To: netdev; +Cc: cake
In-Reply-To: <152754646847.18169.14016621463603092505.stgit@alrua-kau>
This adds support for DiffServ-based priority queueing to CAKE. If the
shaper is in use, each priority tier gets its own virtual clock, which
limits that tier's rate to a fraction of the overall shaped rate, to
discourage trying to game the priority mechanism.
CAKE defaults to a simple, three-tier mode that interprets most code points
as "best effort", but places CS1 traffic into a low-priority "bulk" tier
which is assigned 1/16 of the total rate, and a few code points indicating
latency-sensitive or control traffic (specifically TOS4, VA, EF, CS6, CS7)
into a "latency sensitive" high-priority tier, which is assigned 1/4 rate.
The other supported DiffServ modes are a 4-tier mode matching the 802.11e
precedence rules, as well as two 8-tier modes, one of which implements
strict precedence of the eight priority levels.
This commit also adds an optional DiffServ 'wash' mode, which will zero out
the DSCP fields of any packet passing through CAKE. While this can
technically be done with other mechanisms in the kernel, having the feature
available in CAKE significantly decreases configuration complexity; and the
implementation cost is low on top of the other DiffServ-handling code.
Filters and applications can set the skb->priority field to override the
DSCP-based classification into tiers. If TC_H_MAJ(skb->priority) matches
CAKE's qdisc handle, the minor number will be interpreted as a priority
tier if it is less than or equal to the number of configured priority
tiers.
Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk>
---
net/sched/sch_cake.c | 412 +++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 404 insertions(+), 8 deletions(-)
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index c2cba5f32351..b2760bc1a34f 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -296,6 +296,68 @@ static void cobalt_set_enqueue_time(struct sk_buff *skb,
static u16 quantum_div[CAKE_QUEUES + 1] = {0};
+/* Diffserv lookup tables */
+
+static const u8 precedence[] = {
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 2, 2, 2, 2, 2, 2, 2, 2,
+ 3, 3, 3, 3, 3, 3, 3, 3,
+ 4, 4, 4, 4, 4, 4, 4, 4,
+ 5, 5, 5, 5, 5, 5, 5, 5,
+ 6, 6, 6, 6, 6, 6, 6, 6,
+ 7, 7, 7, 7, 7, 7, 7, 7,
+};
+
+static const u8 diffserv8[] = {
+ 2, 5, 1, 2, 4, 2, 2, 2,
+ 0, 2, 1, 2, 1, 2, 1, 2,
+ 5, 2, 4, 2, 4, 2, 4, 2,
+ 3, 2, 3, 2, 3, 2, 3, 2,
+ 6, 2, 3, 2, 3, 2, 3, 2,
+ 6, 2, 2, 2, 6, 2, 6, 2,
+ 7, 2, 2, 2, 2, 2, 2, 2,
+ 7, 2, 2, 2, 2, 2, 2, 2,
+};
+
+static const u8 diffserv4[] = {
+ 0, 2, 0, 0, 2, 0, 0, 0,
+ 1, 0, 0, 0, 0, 0, 0, 0,
+ 2, 0, 2, 0, 2, 0, 2, 0,
+ 2, 0, 2, 0, 2, 0, 2, 0,
+ 3, 0, 2, 0, 2, 0, 2, 0,
+ 3, 0, 0, 0, 3, 0, 3, 0,
+ 3, 0, 0, 0, 0, 0, 0, 0,
+ 3, 0, 0, 0, 0, 0, 0, 0,
+};
+
+static const u8 diffserv3[] = {
+ 0, 0, 0, 0, 2, 0, 0, 0,
+ 1, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 2, 0, 2, 0,
+ 2, 0, 0, 0, 0, 0, 0, 0,
+ 2, 0, 0, 0, 0, 0, 0, 0,
+};
+
+static const u8 besteffort[] = {
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+};
+
+/* tin priority order for stats dumping */
+
+static const u8 normal_order[] = {0, 1, 2, 3, 4, 5, 6, 7};
+static const u8 bulk_order[] = {1, 0, 2, 3};
+
#define REC_INV_SQRT_CACHE (16)
static u32 cobalt_rec_inv_sqrt_cache[REC_INV_SQRT_CACHE] = {0};
@@ -1382,6 +1444,46 @@ static unsigned int cake_drop(struct Qdisc *sch, struct sk_buff **to_free)
return idx + (tin << 16);
}
+static void cake_wash_diffserv(struct sk_buff *skb)
+{
+ switch (skb->protocol) {
+ case htons(ETH_P_IP):
+ ipv4_change_dsfield(ip_hdr(skb), INET_ECN_MASK, 0);
+ break;
+ case htons(ETH_P_IPV6):
+ ipv6_change_dsfield(ipv6_hdr(skb), INET_ECN_MASK, 0);
+ break;
+ default:
+ break;
+ }
+}
+
+static u8 cake_handle_diffserv(struct sk_buff *skb, u16 wash)
+{
+ u8 dscp;
+
+ switch (skb->protocol) {
+ case htons(ETH_P_IP):
+ dscp = ipv4_get_dsfield(ip_hdr(skb)) >> 2;
+ if (wash && dscp)
+ ipv4_change_dsfield(ip_hdr(skb), INET_ECN_MASK, 0);
+ return dscp;
+
+ case htons(ETH_P_IPV6):
+ dscp = ipv6_get_dsfield(ipv6_hdr(skb)) >> 2;
+ if (wash && dscp)
+ ipv6_change_dsfield(ipv6_hdr(skb), INET_ECN_MASK, 0);
+ return dscp;
+
+ case htons(ETH_P_ARP):
+ return 0x38; /* CS7 - Net Control */
+
+ default:
+ /* If there is no Diffserv field, treat as best-effort */
+ return 0;
+ }
+}
+
static void cake_reconfigure(struct Qdisc *sch);
static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
@@ -1396,7 +1498,26 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct cake_flow *flow;
u32 idx, tin;
- tin = 0;
+ if (TC_H_MAJ(skb->priority) == sch->handle &&
+ TC_H_MIN(skb->priority) > 0 &&
+ TC_H_MIN(skb->priority) <= q->tin_cnt) {
+ tin = TC_H_MIN(skb->priority) - 1;
+
+ if (q->rate_flags & CAKE_FLAG_WASH)
+ cake_wash_diffserv(skb);
+ } else if (q->tin_mode != CAKE_DIFFSERV_BESTEFFORT) {
+ /* extract the Diffserv Precedence field, if it exists */
+ /* and clear DSCP bits if washing */
+ tin = q->tin_index[cake_handle_diffserv(skb,
+ q->rate_flags & CAKE_FLAG_WASH)];
+ if (unlikely(tin >= q->tin_cnt))
+ tin = 0;
+ } else {
+ tin = 0;
+ if (q->rate_flags & CAKE_FLAG_WASH)
+ cake_wash_diffserv(skb);
+ }
+
b = &q->tins[tin];
/* choose flow to insert into */
@@ -1897,18 +2018,275 @@ static void cake_set_rate(struct cake_tin_data *b, u64 rate, u32 mtu,
b->cparams.p_dec = 1 << 20; /* 1/4096 */
}
-static void cake_reconfigure(struct Qdisc *sch)
+static int cake_config_besteffort(struct Qdisc *sch)
{
struct cake_sched_data *q = qdisc_priv(sch);
struct cake_tin_data *b = &q->tins[0];
- int c, ft = 0;
+ u32 mtu = psched_mtu(qdisc_dev(sch));
+ u64 rate = q->rate_bps;
q->tin_cnt = 1;
- cake_set_rate(b, q->rate_bps, psched_mtu(qdisc_dev(sch)),
+
+ q->tin_index = besteffort;
+ q->tin_order = normal_order;
+
+ cake_set_rate(b, rate, mtu,
us_to_ns(q->target), us_to_ns(q->interval));
b->tin_quantum_band = 65535;
b->tin_quantum_prio = 65535;
+ return 0;
+}
+
+static int cake_config_precedence(struct Qdisc *sch)
+{
+ /* convert high-level (user visible) parameters into internal format */
+ struct cake_sched_data *q = qdisc_priv(sch);
+ u32 mtu = psched_mtu(qdisc_dev(sch));
+ u64 rate = q->rate_bps;
+ u32 quantum1 = 256;
+ u32 quantum2 = 256;
+ u32 i;
+
+ q->tin_cnt = 8;
+ q->tin_index = precedence;
+ q->tin_order = normal_order;
+
+ for (i = 0; i < q->tin_cnt; i++) {
+ struct cake_tin_data *b = &q->tins[i];
+
+ cake_set_rate(b, rate, mtu, us_to_ns(q->target),
+ us_to_ns(q->interval));
+
+ b->tin_quantum_prio = max_t(u16, 1U, quantum1);
+ b->tin_quantum_band = max_t(u16, 1U, quantum2);
+
+ /* calculate next class's parameters */
+ rate *= 7;
+ rate >>= 3;
+
+ quantum1 *= 3;
+ quantum1 >>= 1;
+
+ quantum2 *= 7;
+ quantum2 >>= 3;
+ }
+
+ return 0;
+}
+
+/* List of known Diffserv codepoints:
+ *
+ * Least Effort (CS1)
+ * Best Effort (CS0)
+ * Max Reliability & LLT "Lo" (TOS1)
+ * Max Throughput (TOS2)
+ * Min Delay (TOS4)
+ * LLT "La" (TOS5)
+ * Assured Forwarding 1 (AF1x) - x3
+ * Assured Forwarding 2 (AF2x) - x3
+ * Assured Forwarding 3 (AF3x) - x3
+ * Assured Forwarding 4 (AF4x) - x3
+ * Precedence Class 2 (CS2)
+ * Precedence Class 3 (CS3)
+ * Precedence Class 4 (CS4)
+ * Precedence Class 5 (CS5)
+ * Precedence Class 6 (CS6)
+ * Precedence Class 7 (CS7)
+ * Voice Admit (VA)
+ * Expedited Forwarding (EF)
+
+ * Total 25 codepoints.
+ */
+
+/* List of traffic classes in RFC 4594:
+ * (roughly descending order of contended priority)
+ * (roughly ascending order of uncontended throughput)
+ *
+ * Network Control (CS6,CS7) - routing traffic
+ * Telephony (EF,VA) - aka. VoIP streams
+ * Signalling (CS5) - VoIP setup
+ * Multimedia Conferencing (AF4x) - aka. video calls
+ * Realtime Interactive (CS4) - eg. games
+ * Multimedia Streaming (AF3x) - eg. YouTube, NetFlix, Twitch
+ * Broadcast Video (CS3)
+ * Low Latency Data (AF2x,TOS4) - eg. database
+ * Ops, Admin, Management (CS2,TOS1) - eg. ssh
+ * Standard Service (CS0 & unrecognised codepoints)
+ * High Throughput Data (AF1x,TOS2) - eg. web traffic
+ * Low Priority Data (CS1) - eg. BitTorrent
+
+ * Total 12 traffic classes.
+ */
+
+static int cake_config_diffserv8(struct Qdisc *sch)
+{
+/* Pruned list of traffic classes for typical applications:
+ *
+ * Network Control (CS6, CS7)
+ * Minimum Latency (EF, VA, CS5, CS4)
+ * Interactive Shell (CS2, TOS1)
+ * Low Latency Transactions (AF2x, TOS4)
+ * Video Streaming (AF4x, AF3x, CS3)
+ * Bog Standard (CS0 etc.)
+ * High Throughput (AF1x, TOS2)
+ * Background Traffic (CS1)
+ *
+ * Total 8 traffic classes.
+ */
+
+ struct cake_sched_data *q = qdisc_priv(sch);
+ u32 mtu = psched_mtu(qdisc_dev(sch));
+ u64 rate = q->rate_bps;
+ u32 quantum1 = 256;
+ u32 quantum2 = 256;
+ u32 i;
+
+ q->tin_cnt = 8;
+
+ /* codepoint to class mapping */
+ q->tin_index = diffserv8;
+ q->tin_order = normal_order;
+
+ /* class characteristics */
+ for (i = 0; i < q->tin_cnt; i++) {
+ struct cake_tin_data *b = &q->tins[i];
+
+ cake_set_rate(b, rate, mtu, us_to_ns(q->target),
+ us_to_ns(q->interval));
+
+ b->tin_quantum_prio = max_t(u16, 1U, quantum1);
+ b->tin_quantum_band = max_t(u16, 1U, quantum2);
+
+ /* calculate next class's parameters */
+ rate *= 7;
+ rate >>= 3;
+
+ quantum1 *= 3;
+ quantum1 >>= 1;
+
+ quantum2 *= 7;
+ quantum2 >>= 3;
+ }
+
+ return 0;
+}
+
+static int cake_config_diffserv4(struct Qdisc *sch)
+{
+/* Further pruned list of traffic classes for four-class system:
+ *
+ * Latency Sensitive (CS7, CS6, EF, VA, CS5, CS4)
+ * Streaming Media (AF4x, AF3x, CS3, AF2x, TOS4, CS2, TOS1)
+ * Best Effort (CS0, AF1x, TOS2, and those not specified)
+ * Background Traffic (CS1)
+ *
+ * Total 4 traffic classes.
+ */
+
+ struct cake_sched_data *q = qdisc_priv(sch);
+ u32 mtu = psched_mtu(qdisc_dev(sch));
+ u64 rate = q->rate_bps;
+ u32 quantum = 1024;
+
+ q->tin_cnt = 4;
+
+ /* codepoint to class mapping */
+ q->tin_index = diffserv4;
+ q->tin_order = bulk_order;
+
+ /* class characteristics */
+ cake_set_rate(&q->tins[0], rate, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+ cake_set_rate(&q->tins[1], rate >> 4, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+ cake_set_rate(&q->tins[2], rate >> 1, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+ cake_set_rate(&q->tins[3], rate >> 2, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+
+ /* priority weights */
+ q->tins[0].tin_quantum_prio = quantum;
+ q->tins[1].tin_quantum_prio = quantum >> 4;
+ q->tins[2].tin_quantum_prio = quantum << 2;
+ q->tins[3].tin_quantum_prio = quantum << 4;
+
+ /* bandwidth-sharing weights */
+ q->tins[0].tin_quantum_band = quantum;
+ q->tins[1].tin_quantum_band = quantum >> 4;
+ q->tins[2].tin_quantum_band = quantum >> 1;
+ q->tins[3].tin_quantum_band = quantum >> 2;
+
+ return 0;
+}
+
+static int cake_config_diffserv3(struct Qdisc *sch)
+{
+/* Simplified Diffserv structure with 3 tins.
+ * Low Priority (CS1)
+ * Best Effort
+ * Latency Sensitive (TOS4, VA, EF, CS6, CS7)
+ */
+ struct cake_sched_data *q = qdisc_priv(sch);
+ u32 mtu = psched_mtu(qdisc_dev(sch));
+ u64 rate = q->rate_bps;
+ u32 quantum = 1024;
+
+ q->tin_cnt = 3;
+
+ /* codepoint to class mapping */
+ q->tin_index = diffserv3;
+ q->tin_order = bulk_order;
+
+ /* class characteristics */
+ cake_set_rate(&q->tins[0], rate, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+ cake_set_rate(&q->tins[1], rate >> 4, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+ cake_set_rate(&q->tins[2], rate >> 2, mtu,
+ us_to_ns(q->target), us_to_ns(q->interval));
+
+ /* priority weights */
+ q->tins[0].tin_quantum_prio = quantum;
+ q->tins[1].tin_quantum_prio = quantum >> 4;
+ q->tins[2].tin_quantum_prio = quantum << 4;
+
+ /* bandwidth-sharing weights */
+ q->tins[0].tin_quantum_band = quantum;
+ q->tins[1].tin_quantum_band = quantum >> 4;
+ q->tins[2].tin_quantum_band = quantum >> 2;
+
+ return 0;
+}
+
+static void cake_reconfigure(struct Qdisc *sch)
+{
+ struct cake_sched_data *q = qdisc_priv(sch);
+ int c, ft;
+
+ switch (q->tin_mode) {
+ case CAKE_DIFFSERV_BESTEFFORT:
+ ft = cake_config_besteffort(sch);
+ break;
+
+ case CAKE_DIFFSERV_PRECEDENCE:
+ ft = cake_config_precedence(sch);
+ break;
+
+ case CAKE_DIFFSERV_DIFFSERV8:
+ ft = cake_config_diffserv8(sch);
+ break;
+
+ case CAKE_DIFFSERV_DIFFSERV4:
+ ft = cake_config_diffserv4(sch);
+ break;
+
+ case CAKE_DIFFSERV_DIFFSERV3:
+ default:
+ ft = cake_config_diffserv3(sch);
+ break;
+ }
+
for (c = q->tin_cnt; c < CAKE_MAX_TINS; c++) {
cake_clear_tin(sch, c);
q->tins[c].cparams.mtu_time = q->tins[ft].cparams.mtu_time;
@@ -1964,6 +2342,16 @@ static int cake_change(struct Qdisc *sch, struct nlattr *opt,
if (tb[TCA_CAKE_BASE_RATE64])
q->rate_bps = nla_get_u64(tb[TCA_CAKE_BASE_RATE64]);
+ if (tb[TCA_CAKE_DIFFSERV_MODE])
+ q->tin_mode = nla_get_u32(tb[TCA_CAKE_DIFFSERV_MODE]);
+
+ if (tb[TCA_CAKE_WASH]) {
+ if (!!nla_get_u32(tb[TCA_CAKE_WASH]))
+ q->rate_flags |= CAKE_FLAG_WASH;
+ else
+ q->rate_flags &= ~CAKE_FLAG_WASH;
+ }
+
if (tb[TCA_CAKE_FLOW_MODE])
q->flow_mode = (nla_get_u32(tb[TCA_CAKE_FLOW_MODE]) &
CAKE_FLOW_MASK);
@@ -2027,7 +2415,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt,
int i, j, err;
sch->limit = 10240;
- q->tin_mode = CAKE_DIFFSERV_BESTEFFORT;
+ q->tin_mode = CAKE_DIFFSERV_DIFFSERV3;
q->flow_mode = CAKE_FLOW_TRIPLE;
q->rate_bps = 0; /* unlimited by default */
@@ -2137,6 +2525,13 @@ static int cake_dump(struct Qdisc *sch, struct sk_buff *skb)
!!(q->flow_mode & CAKE_FLOW_NAT_FLAG)))
goto nla_put_failure;
+ if (nla_put_u32(skb, TCA_CAKE_DIFFSERV_MODE, q->tin_mode))
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_CAKE_WASH,
+ !!(q->rate_flags & CAKE_FLAG_WASH)))
+ goto nla_put_failure;
+
return nla_nest_end(skb, opts);
nla_put_failure:
@@ -2190,7 +2585,7 @@ static int cake_dump_stats(struct Qdisc *sch, struct gnet_dump *d)
} while (0)
for (i = 0; i < q->tin_cnt; i++) {
- struct cake_tin_data *b = &q->tins[i];
+ struct cake_tin_data *b = &q->tins[q->tin_order[i]];
ts = nla_nest_start(d->skb, i + 1);
if (!ts)
@@ -2289,7 +2684,8 @@ static int cake_dump_class_stats(struct Qdisc *sch, unsigned long cl,
u32 idx = cl - 1;
if (idx < CAKE_QUEUES * q->tin_cnt) {
- const struct cake_tin_data *b = &q->tins[idx / CAKE_QUEUES];
+ const struct cake_tin_data *b = \
+ &q->tins[q->tin_order[idx / CAKE_QUEUES]];
const struct sk_buff *skb;
flow = &b->flows[idx % CAKE_QUEUES];
@@ -2361,7 +2757,7 @@ static void cake_walk(struct Qdisc *sch, struct qdisc_walker *arg)
return;
for (i = 0; i < q->tin_cnt; i++) {
- struct cake_tin_data *b = &q->tins[i];
+ struct cake_tin_data *b = &q->tins[q->tin_order[i]];
for (j = 0; j < CAKE_QUEUES; j++) {
if (list_empty(&b->flows[j].flowchain) ||
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox