Problem with cls_flow nfct-* keys

Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

Hi everyone! I've discovered some issues with cls_flow nfct-* keys. I have a router with NAT and clients behind it. All incoming traffic from WAN interface is redirecting to ifb0 device. With HTB qdisc.
I Am install SFQ qdisc 99:0 with flow classifyer and nfct-dst key on HTB deafault leaf class (1:99) :

    tc qdisc add dev eth0 ingress
    tc filter add dev eth0 parent ffff: protocol all pref 100 u32 match u32 0 0 action mirred egress redirect dev ifb0

    tc qdisc add dev ifb0 root handle 1: htb default 99 r2q 10
    tc class add dev ifb0 parent 1: classid 1:1 htb rate 10Mbit

    tc class add dev ifb0 parent 1:1 classid 1:99 htb rate 2mbit ceil 10Mbit burst 150k prio 7
    tc qdisc add dev ifb0 parent 1:99 handle 99: sfq limit 10240 
    tc filter add dev ifb0 parent 99: protocol all handle 1 flow map key nfct-dst and 0xff divisor 1024

I am trying to achieve equal bandwidth sharing between internal IPs, so that single IP could not get all free bandwidth with Torrent. But it doesn't work. After investigation I've discovered, that all incoming traffic,
that hit default HTB class (1:99), goes to SFQ class 99:1f, which equals to WAN IP of my router *.*.*.30, so I see that nfct-dst key behaves like a simple dst. Is there any chance to fix it?

Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

Hi everyone! I've discovered some issues with cls_flow nfct-* keys. I have a router with NAT and clients behind it. All incoming traffic from WAN interface is redirecting to ifb0 device. With HTB qdisc.
I Am install SFQ qdisc 99:0 with flow classifyer and nfct-dst key on HTB deafault leaf class (1:99) :

šššštc qdisc add dev eth0 ingress
šššštc filter add dev eth0 parent ffff: protocol all pref 100 u32 match u32 0 0 action mirred egress redirect dev ifb0

šššštc qdisc add dev ifb0 root handle 1: htb default 99 r2q 10
šššštc class add dev ifb0 parent 1: classid 1:1 htb rate 10Mbit

šššštc class add dev ifb0 parent 1:1 classid 1:99 htb rate 2mbit ceil 10Mbit burst 150k prio 7
šššštc qdisc add dev ifb0 parent 1:99 handle 99: sfq limit 10240
šššštc filter add dev ifb0 parent 99: protocol all handle 1 flow map key nfct-dst and 0xff divisor 1024

I am trying to achieve equal bandwidth sharing between internal IPs, so that single IP could not get all free bandwidth with Torrent. But it doesn't work. After investigation I've discovered, that all incoming traffic,
that hit default HTB class (1:99), goes to SFQ class 99:1f, which equals to WAN IP of my router *.*.*.30, so I see that nfct-dst key behaves like a simple dst. Is there any chance to fix it?

Re: Problem with cls_flow nfct-* keys

[Andy Furniss]

Гаврилов Игорь wrote:
> Hi everyone! I've discovered some issues with cls_flow nfct-* keys. I have a router with NAT and clients behind it. All incoming traffic from WAN interface is redirecting to ifb0 device. With HTB qdisc.
> I Am install SFQ qdisc 99:0 with flow classifyer and nfct-dst key on HTB deafault leaf class (1:99) :
>
>      tc qdisc add dev eth0 ingress
>      tc filter add dev eth0 parent ffff: protocol all pref 100 u32 match u32 0 0 action mirred egress redirect dev ifb0
>
>      tc qdisc add dev ifb0 root handle 1: htb default 99 r2q 10
>      tc class add dev ifb0 parent 1: classid 1:1 htb rate 10Mbit
>
>      tc class add dev ifb0 parent 1:1 classid 1:99 htb rate 2mbit ceil 10Mbit burst 150k prio 7
>      tc qdisc add dev ifb0 parent 1:99 handle 99: sfq limit 10240
>      tc filter add dev ifb0 parent 99: protocol all handle 1 flow map key nfct-dst and 0xff divisor 1024
>
> I am trying to achieve equal bandwidth sharing between internal IPs, so that single IP could not get all free bandwidth with Torrent. But it doesn't work. After investigation I've discovered, that all incoming traffic,
> that hit default HTB class (1:99), goes to SFQ class 99:1f, which equals to WAN IP of my router *.*.*.30, so I see that nfct-dst key behaves like a simple dst. Is there any chance to fix it?

I am not sure if this should work or not.

If there is no/low incoming traffic to this box then you could shape on 
egress.

Generally I would avoid redirecting protocol all then restricting htb 
default - you may end up dropping arp.

--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

> I am not sure if this should work or not.
In all examples this worked, But I can't test it on old kenels - I use 3.10.

> If there is no/low incoming traffic to this box then you could shape on
> egress.
Unfortunately  this is not an option. The single case, where IFB could be a problem, and you can't shape on egress interface is when you have traffic to/from router.
There is IPSec tunnels from WAN interface, and I need to share bandwidth dynamically between regular internet traffic and IPSec. 

> Generally I would avoid redirecting protocol all then restricting htb
> default - you may end up dropping arp.
I have dedicated class for ARP.

Re: Problem with cls_flow nfct-* keys

[Andy Furniss]

Гаврилов Игорь wrote:
>> I am not sure if this should work or not.
> In all examples this worked, But I can't test it on old kenels - I
> use 3.10.
>
>> If there is no/low incoming traffic to this box then you could
>> shape on egress.
> Unfortunately  this is not an option. The single case, where IFB
> could be a problem, and you can't shape on egress interface is when
> you have traffic to/from router. There is IPSec tunnels from WAN
> interface, and I need to share bandwidth dynamically between regular
> internet traffic and IPSec.
>
>> Generally I would avoid redirecting protocol all then restricting
>> htb default - you may end up dropping arp.
> I have dedicated class for ARP.
>

Ok, I can't easily test old kernels either.

I did a quick test on my desktop with a recent git kernel and it doesn't
work for me either.

The same test (using ifb) does work on egress.

Adding a printk shows a call to nf_ct_get(skb, &ctinfo)

returns NULL which makes flow fallback to dst.

Re: Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

So this problem not related to my setup. Seems that it occurs for a long time. Is there any possibility to fix this, and what is the reason of such behavior.

Re: Problem with cls_flow nfct-* keys

[Florian Westphal]

Гаврилов Игорь <iggorok@yandex.ua> wrote:
> So this problem not related to my setup. Seems that it occurs for a long time. Is there any possibility to fix this, and what is the reason of such behavior.

conntrack is hooked into the ip stack,
thats after the ingress hook.

So by time cls_flow is called in ingress
conntrack was not yet invoked.

You would need to extend cls_flow to do a conntrack lookup.
(similar to act_connmark.c)

Re: Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

So, as far as I understand, on ingress, nfc-* keys have never worked?
About extending cls_flow.c, unfortunately I don't have experience with coding kernel modules. 

03.12.2015, 11:19, "Florian Westphal" <fw@strlen.de>:
> Гаврилов Игорь <iggorok@yandex.ua> wrote:
>>  So this problem not related to my setup. Seems that it occurs for a long time. Is there any possibility to fix this, and what is the reason of such behavior.
>
> conntrack is hooked into the ip stack,
> thats after the ingress hook.
>
> So by time cls_flow is called in ingress
> conntrack was not yet invoked.
>
> You would need to extend cls_flow to do a conntrack lookup.
> (similar to act_connmark.c)
--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with cls_flow nfct-* keys

[Andy Furniss]

Гаврилов Игорь wrote:
> So this problem not related to my setup. Seems that it occurs for a
> long time. Is there any possibility to fix this, and what is the
> reason of such behavior.

As Florian says on ingress tc gets the packet before conntrack.

I was slightly hopeful that I was wrong (in thinking this never worked)
when I saw "nf_ct_get" in the code, but it just looks at the skb which
on ingress won't have the fields filled in.

/* Return conntrack_info and tuple hash for given skb. */
static inline struct nf_conn *
nf_ct_get(const struct sk_buff *skb, enum ip_conntrack_info *ctinfo)
{
         *ctinfo = skb->nfctinfo;
         return (struct nf_conn *)skb->nfct;
}

Historically at least people used imq to get at traffic after netfilter.
It seems it's still alive -

www.linuximq.net

but you need to patch kernel/iptables.

When ifb first appeared it was mentioned that an action could be made to
get netfilter state and Florian mentions act_connmark.c so I guess some
progress was made.

Not really much use for your example I suppose.

Re: Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

After day after the day of trials and errors I finally get working solution - I use code from act_connmark and it's working pretty well. 
#define CTTUPLE(skb, member)                                            \
({                                                                      \
        enum ip_conntrack_info ctinfo;                                  \
        struct nf_conntrack_tuple tuple;                                \
        struct nf_conntrack_zone zone;                                  \
        const struct nf_conntrack_tuple_hash *thash;                    \
        int proto;                                                      \
        const struct nf_conn *ct = nf_ct_get(skb, &ctinfo);             \
        if (ct = NULL){                                                \
                        switch (tc_skb_protocol(skb)) {                 \
                        case htons(ETH_P_IP):                           \
                                proto = NFPROTO_IPV4;                   \
                                break;                                  \
                        case htons(ETH_P_IPV6):                         \
                                proto = NFPROTO_IPV6;                   \
                                break;                                  \
                        default: goto fallback;                         \
        }                                                               \
        if (!nf_ct_get_tuplepr(skb, skb_network_offset(skb), proto, &tuple)) \
                goto fallback;                                          \
        zone.id = NF_CT_DEFAULT_ZONE_ID;                                \
        zone.dir = NF_CT_DEFAULT_ZONE_DIR;                              \
                                                                        \
        thash = nf_conntrack_find_get(dev_net(skb->dev), &zone, &tuple);\
        if (!thash) goto fallback;                                      \
        ct = nf_ct_tuplehash_to_ctrack(thash);}                         \

        ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.member;                 \
})


I can get info about original src/dst on ingress. But there is a little problem in function ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.member;
I can't get valid direction, because I don't get ctinfo. Is any solustion for my situation?

03.12.2015, 12:51, "Andy Furniss" <adf.lists@gmail.com>:
> çÁ×ÒÉÌÏ× éÇÏÒØ wrote:
>> šSo this problem not related to my setup. Seems that it occurs for a
>> šlong time. Is there any possibility to fix this, and what is the
>> šreason of such behavior.
>
> As Florian says on ingress tc gets the packet before conntrack.
>
> I was slightly hopeful that I was wrong (in thinking this never worked)
> when I saw "nf_ct_get" in the code, but it just looks at the skb which
> on ingress won't have the fields filled in.
>
> /* Return conntrack_info and tuple hash for given skb. */
> static inline struct nf_conn *
> nf_ct_get(const struct sk_buff *skb, enum ip_conntrack_info *ctinfo)
> {
> ššššššššš*ctinfo = skb->nfctinfo;
> šššššššššreturn (struct nf_conn *)skb->nfct;
> }
>
> Historically at least people used imq to get at traffic after netfilter.
> It seems it's still alive -
>
> www.linuximq.net
>
> but you need to patch kernel/iptables.
>
> When ifb first appeared it was mentioned that an action could be made to
> get netfilter state and Florian mentions act_connmark.c so I guess some
> progress was made.
>
> Not really much use for your example I suppose.

Re: Problem with cls_flow nfct-* keys

[Andy Furniss]

Гаврилов Игорь wrote:
> After day after the day of trials and errors I finally get working solution - I use code from act_connmark and it's working pretty well.
> #define CTTUPLE(skb, member)                                            \
> ({                                                                      \
>          enum ip_conntrack_info ctinfo;                                  \
>          struct nf_conntrack_tuple tuple;                                \
>          struct nf_conntrack_zone zone;                                  \
>          const struct nf_conntrack_tuple_hash *thash;                    \
>          int proto;                                                      \
>          const struct nf_conn *ct = nf_ct_get(skb, &ctinfo);             \
>          if (ct = NULL){                                                \
>                          switch (tc_skb_protocol(skb)) {                 \
>                          case htons(ETH_P_IP):                           \
>                                  proto = NFPROTO_IPV4;                   \
>                                  break;                                  \
>                          case htons(ETH_P_IPV6):                         \
>                                  proto = NFPROTO_IPV6;                   \
>                                  break;                                  \
>                          default: goto fallback;                         \
>          }                                                               \
>          if (!nf_ct_get_tuplepr(skb, skb_network_offset(skb), proto, &tuple)) \
>                  goto fallback;                                          \
>          zone.id = NF_CT_DEFAULT_ZONE_ID;                                \
>          zone.dir = NF_CT_DEFAULT_ZONE_DIR;                              \
>                                                                          \
>          thash = nf_conntrack_find_get(dev_net(skb->dev), &zone, &tuple);\
>          if (!thash) goto fallback;                                      \
>          ct = nf_ct_tuplehash_to_ctrack(thash);}                         \
>
>          ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.member;                 \
> })
>
>
> I can get info about original src/dst on ingress. But there is a little problem in function ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.member;
> I can't get valid direction, because I don't get ctinfo. Is any solustion for my situation?

I don't know, maybe you could ask on the netfilter list or netdev.


--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with cls_flow nfct-* keys

[Гаврилов Игорь]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Finally I have working patch, but I need somebody to look at it, and help to send it to developers. Can someone?


30.11.2015, 13:40, "Гаврилов Игорь" <iggorok@yandex.ua>:
> Hi everyone! I've discovered some issues with cls_flow nfct-* keys. I have a router with NAT and clients behind it. All incoming traffic from WAN interface is redirecting to ifb0 device. With HTB qdisc.
> I Am install SFQ qdisc 99:0 with flow classifyer and nfct-dst key on HTB deafault leaf class (1:99) :
>
>     tc qdisc add dev eth0 ingress
>     tc filter add dev eth0 parent ffff: protocol all pref 100 u32 match u32 0 0 action mirred egress redirect dev ifb0
>
>     tc qdisc add dev ifb0 root handle 1: htb default 99 r2q 10
>     tc class add dev ifb0 parent 1: classid 1:1 htb rate 10Mbit
>
>     tc class add dev ifb0 parent 1:1 classid 1:99 htb rate 2mbit ceil 10Mbit burst 150k prio 7
>     tc qdisc add dev ifb0 parent 1:99 handle 99: sfq limit 10240
>     tc filter add dev ifb0 parent 99: protocol all handle 1 flow map key nfct-dst and 0xff divisor 1024
>
> I am trying to achieve equal bandwidth sharing between internal IPs, so that single IP could not get all free bandwidth with Torrent. But it doesn't work. After investigation I've discovered, that all incoming traffic,
> that hit default HTB class (1:99), goes to SFQ class 99:1f, which equals to WAN IP of my router *.*.*.30, so I see that nfct-dst key behaves like a simple dst. Is there any chance to fix it?
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cls_flow.patch --]
[-- Type: text/x-diff; name="cls_flow.patch", Size: 3316 bytes --]

--- cls_flow.c	2015-11-02 02:05:25.000000000 +0200
+++ net/sched/cls_flow.c	2015-12-07 12:32:56.767127200 +0200
@@ -30,6 +30,8 @@
 
 #if defined(CONFIG_NF_CONNTRACK) || defined(CONFIG_NF_CONNTRACK_MODULE)
 #include <net/netfilter/nf_conntrack.h>
+#include <net/netfilter/nf_conntrack_core.h>
+#include <net/netfilter/nf_conntrack_zones.h>
 #endif
 
 struct flow_head {
@@ -132,16 +134,42 @@
 }
 
 #if defined(CONFIG_NF_CONNTRACK) || defined(CONFIG_NF_CONNTRACK_MODULE)
-#define CTTUPLE(skb, member)						\
+#define CTTUPLE(skb, direction, member)						\
 ({									\
 	enum ip_conntrack_info ctinfo;					\
+	struct nf_conntrack_tuple tuple;				\
+	struct nf_conntrack_zone zone;					\
+	const struct nf_conntrack_tuple_hash *thash;			\
+	__be32 result;							\
+	int proto;							\
 	const struct nf_conn *ct = nf_ct_get(skb, &ctinfo);		\
-	if (ct == NULL)							\
-		goto fallback;						\
-	ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.member;			\
+	if (ct == NULL){						\
+		        switch (tc_skb_protocol(skb)) {			\
+		        case htons(ETH_P_IP):				\
+                		proto = NFPROTO_IPV4; 			\
+				break;					\
+		        case htons(ETH_P_IPV6):				\
+				proto = NFPROTO_IPV6;			\
+				break;					\
+			default: goto fallback;				\
+	        } 							\
+									\
+	if (!nf_ct_get_tuplepr(skb, skb_network_offset(skb), proto, &tuple)) \
+                goto fallback;						\
+        zone.id = NF_CT_DEFAULT_ZONE_ID;				\
+        zone.dir = NF_CT_DEFAULT_ZONE_DIR;				\
+									\
+        thash = nf_conntrack_find_get(dev_net(skb->dev), &zone, &tuple);\
+        if (!thash) goto fallback;					\
+        ct = nf_ct_tuplehash_to_ctrack(thash);				\
+	result = ct->tuplehash[(thash->tuple.dst.dir == IP_CT_DIR_REPLY) ? IP_CT_DIR_ORIGINAL : IP_CT_DIR_REPLY].tuple.src.member;	\
+	} else {							\
+	result = ct->tuplehash[CTINFO2DIR(ctinfo)].tuple.direction.member;	\
+	}								\
+	result;								\
 })
 #else
-#define CTTUPLE(skb, member)						\
+#define CTTUPLE(skb, direction, member)						\
 ({									\
 	goto fallback;							\
 	0;								\
@@ -152,9 +180,9 @@
 {
 	switch (tc_skb_protocol(skb)) {
 	case htons(ETH_P_IP):
-		return ntohl(CTTUPLE(skb, src.u3.ip));
+		return ntohl(CTTUPLE(skb, src, u3.ip));
 	case htons(ETH_P_IPV6):
-		return ntohl(CTTUPLE(skb, src.u3.ip6[3]));
+		return ntohl(CTTUPLE(skb, src, u3.ip6[3]));
 	}
 fallback:
 	return flow_get_src(skb, flow);
@@ -162,11 +190,12 @@
 
 static u32 flow_get_nfct_dst(const struct sk_buff *skb, const struct flow_keys *flow)
 {
+
 	switch (tc_skb_protocol(skb)) {
 	case htons(ETH_P_IP):
-		return ntohl(CTTUPLE(skb, dst.u3.ip));
+		return ntohl(CTTUPLE(skb, dst, u3.ip));
 	case htons(ETH_P_IPV6):
-		return ntohl(CTTUPLE(skb, dst.u3.ip6[3]));
+		return ntohl(CTTUPLE(skb, dst, u3.ip6[3]));
 	}
 fallback:
 	return flow_get_dst(skb, flow);
@@ -174,14 +203,14 @@
 
 static u32 flow_get_nfct_proto_src(const struct sk_buff *skb, const struct flow_keys *flow)
 {
-	return ntohs(CTTUPLE(skb, src.u.all));
+	return ntohs(CTTUPLE(skb, src, u.all));
 fallback:
 	return flow_get_proto_src(skb, flow);
 }
 
 static u32 flow_get_nfct_proto_dst(const struct sk_buff *skb, const struct flow_keys *flow)
 {
-	return ntohs(CTTUPLE(skb, dst.u.all));
+	return ntohs(CTTUPLE(skb, dst, u.all));
 fallback:
 	return flow_get_proto_dst(skb, flow);
 }

Re: Problem with cls_flow nfct-* keys

[Andy Furniss]

Гаврилов Игорь wrote:
> Finally I have working patch, but I need somebody to look at it, and help to send it to developers. Can someone?

You should subscribe and send it to netdev@vger.kernel.org


--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html