ingress policying for realtime protocol

ingress policying for realtime protocol

[Uwe Kleine-König]

Hello,

I hope this is the/a right list to ask this question. If not, I'd be
happy to learn the right alternative.

For a system that implements the Media Redundancy Protocol (MRP,
http://en.wikipedia.org/wiki/Media_Redundancy_Protocol) I'm faced with
the task to assert that the corresponding ethernet packets are handled
"in time". Currently this fails already if I do

	scp somebigfile mymrpsystem:

The first idea was to use ingress policying to limit all packets that
are not related to MRP to a certain rate.

The problem in practise however is that the rate that has to be used
above is very low (100kbit) so we're looking for alternatives. I wonder
what's missing to get ingress policying more versatile to allow for
example a prio qdisc or if there is some conceptual problem that I don't
see. I'm not sure this would solve our problem, but it's definitely
something I want to play with before trying/resorting to hackish
solutions (like hacking the eth-driver to pipe MRP packets directly to
the network userspace application instead of through the network stack).

Do you have any ideas or hints?

Thanks in advance
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: ingress policying for realtime protocol

[Cong Wang]

(Cc'ing Jamal)

On Tue, May 19, 2015 at 2:11 PM, Uwe Kleine-König
<u.kleine-koenig@pengutronix.de> wrote:
> Hello,
>
> I hope this is the/a right list to ask this question. If not, I'd be
> happy to learn the right alternative.
>
> For a system that implements the Media Redundancy Protocol (MRP,
> http://en.wikipedia.org/wiki/Media_Redundancy_Protocol) I'm faced with
> the task to assert that the corresponding ethernet packets are handled
> "in time". Currently this fails already if I do
>
>         scp somebigfile mymrpsystem:
>
> The first idea was to use ingress policying to limit all packets that
> are not related to MRP to a certain rate.
>
> The problem in practise however is that the rate that has to be used
> above is very low (100kbit) so we're looking for alternatives. I wonder
> what's missing to get ingress policying more versatile to allow for
> example a prio qdisc or if there is some conceptual problem that I don't
> see. I'm not sure this would solve our problem, but it's definitely
> something I want to play with before trying/resorting to hackish
> solutions (like hacking the eth-driver to pipe MRP packets directly to
> the network userspace application instead of through the network stack).
>

There is very little to do on ingress side since there is no queue at all,
not to mention priority, you could try ifb to see if it fits your need.

Re: ingress policying for realtime protocol

[Eric Dumazet]

On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:

> There is very little to do on ingress side since there is no queue at all,
> not to mention priority, you could try ifb to see if it fits your need.

Note that if the need is to police traffic, ifb is not really needed :

TC="tc"
DEV="dev eth0"
IP=10.246.11.51/32
$TC qdisc del $DEV ingress 2>/dev/null
$TC qdisc add $DEV ingress
$TC filter add $DEV parent ffff: protocol ip u32 match ip src $IP \
	police rate 1Mbit burst 10Mbit mtu 66000 action drop/continue

$TC -s filter ls $DEV parent ffff: protocol ip

Re: ingress policying for realtime protocol

[Cong Wang]

On Wed, May 20, 2015 at 5:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
>
>> There is very little to do on ingress side since there is no queue at all,
>> not to mention priority, you could try ifb to see if it fits your need.
>
> Note that if the need is to police traffic, ifb is not really needed :
>

You really need to read Uwe's email before reply,
he mentioned prio qdisc:

"I wonder
what's missing to get ingress policying more versatile to allow for
example a prio qdisc or if there is some conceptual problem that I don't
see."

Re: ingress policying for realtime protocol

[Eric Dumazet]

On Wed, 2015-05-20 at 18:47 -0700, Cong Wang wrote:
> On Wed, May 20, 2015 at 5:30 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
> >
> >> There is very little to do on ingress side since there is no queue at all,
> >> not to mention priority, you could try ifb to see if it fits your need.
> >
> > Note that if the need is to police traffic, ifb is not really needed :
> >
> 
> You really need to read Uwe's email before reply,
> he mentioned prio qdisc:
> 
> "I wonder
> what's missing to get ingress policying more versatile to allow for
> example a prio qdisc or if there is some conceptual problem that I don't
> see."


I did. He used the very precise word of policing, not shaping.

The fact there is a error in "policying" does not allow you to fix this
to "shaping".

It seems very clear to me that Uwe use case does not require ifb at all.

Listen Cong, could you please stop assuming that I am a jerk ?

Thanks !

Re: ingress policying for realtime protocol

[Cong Wang]

On Wed, May 20, 2015 at 6:53 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> Listen Cong, could you please stop assuming that I am a jerk ?

It is at least the second time to tell you: you SHOULD direct my email
to /dev/null, SERIOUSLY, I have no interest to argue any stupid thing
like this with you, PERIOD.

Since you use gmail, add a filter for both cwang@twopensource.com
and xiyou.wangcong@gmail.com

HUGE thanks for doing it! It will me a lot of time!

Re: ingress policying for realtime protocol

[Eric Dumazet]

On Wed, 2015-05-20 at 18:59 -0700, Cong Wang wrote:
> On Wed, May 20, 2015 at 6:53 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> >
> > Listen Cong, could you please stop assuming that I am a jerk ?
> 
> It is at least the second time to tell you: you SHOULD direct my email
> to /dev/null, SERIOUSLY, I have no interest to argue any stupid thing
> like this with you, PERIOD.
> 
> Since you use gmail, add a filter for both cwang@twopensource.com
> and xiyou.wangcong@gmail.com
> 
> HUGE thanks for doing it! It will me a lot of time!

No, I want to block you sending stuff without me having a chance to
comment.

Yeah, I am _that_ stupid.

In case you missed it, there was 'realtime protocol' in the thread
title.

ifb in ingress is adding jitter and nondeterministic behavior.

While a tc policer adds nothing but possibly some cpu cycles that can
easily be accounted for.

I gave a simple working script, because I am tired of people sending
'good advices' but do not care to send a working sample.

Re: ingress policying for realtime protocol

[Uwe Kleine-König]

On Wed, May 20, 2015 at 05:30:40PM -0700, Eric Dumazet wrote:
> On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
> 
> > There is very little to do on ingress side since there is no queue at all,
> > not to mention priority, you could try ifb to see if it fits your need.
> 
> Note that if the need is to police traffic, ifb is not really needed :
> 
> TC="tc"
> DEV="dev eth0"
> IP=10.246.11.51/32
> $TC qdisc del $DEV ingress 2>/dev/null
> $TC qdisc add $DEV ingress
> $TC filter add $DEV parent ffff: protocol ip u32 match ip src $IP \
> 	police rate 1Mbit burst 10Mbit mtu 66000 action drop/continue
> 
> $TC -s filter ls $DEV parent ffff: protocol ip
I have something like that (matching on dst mac addresses instead of src ip):

	tc qdisc add dev eth0 handle ffff: ingress
	tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match ether dst 01:15:4E:00:00:01 police pass
	tc filter add dev eth0 parent ffff: protocol all prio 50 u32 match u32 0 0 at 0 police rate 100kbit burst 10k drop

. So Cong interpreted my question right and probably I just used the
wrong keywords to make you understand the same. I try again to put my
idea in words to make it explicit:

I imagine that it could help in my case if I could assert that MRP
packets are handled priorized over other traffic without throwing away
so many unrelated packets. For egress that works by e.g. using a prio
qdisc. For ingress however only shaping is available.

So the question essentially is: Why doesn't this work for ingress? Cong
wrote "there is no queue at all [for ingress]". Is this by design? Or is
it just not implemented because noone spend the effort to work on that?
Do you think it would help me?

Maybe there is another bottleneck in the application that currently
forces us to use this tight limit on ingress shaping. I will try to work
on that, maybe shaping is good enough then?! I will report back.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: ingress policying for realtime protocol

[Jamal Hadi Salim]

On 05/21/15 03:07, Uwe Kleine-König wrote:
> On Wed, May 20, 2015 at 05:30:40PM -0700, Eric Dumazet wrote:
>> On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
>>
>>> There is very little to do on ingress side since there is no queue at all,
>>> not to mention priority, you could try ifb to see if it fits your need.
>>
>> Note that if the need is to police traffic, ifb is not really needed :
>>
>> TC="tc"
>> DEV="dev eth0"
>> IP=10.246.11.51/32
>> $TC qdisc del $DEV ingress 2>/dev/null
>> $TC qdisc add $DEV ingress
>> $TC filter add $DEV parent ffff: protocol ip u32 match ip src $IP \
>> 	police rate 1Mbit burst 10Mbit mtu 66000 action drop/continue
>>
>> $TC -s filter ls $DEV parent ffff: protocol ip
> I have something like that (matching on dst mac addresses instead of src ip):
>
> 	tc qdisc add dev eth0 handle ffff: ingress
> 	tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match ether dst 01:15:4E:00:00:01 police pass
> 	tc filter add dev eth0 parent ffff: protocol all prio 50 u32 match u32 0 0 at 0 police rate 100kbit burst 10k drop
>
> So Cong interpreted my question right and probably I just used the
> wrong keywords to make you understand the same.

I think both Cong and Eric are right.
You wanted to priotize something thats _realtime_ by using queues, so
Cong answered your question with ifb which will provide you a queue on
ingress.
OTOH, You should really avoid queues of any sort if latency is
important to you - hence what Eric said is correct. Jitter will occur
when it matters the most for you i.e when congestion kicks in; otherwise
it will work (when there is no congestion;->)

So your requirements are conflicting and the result is two talented
people are intepretting things differently;->

So some questions to you:
Why is there a 100Kbps limit for everything else? If it has to be at
100Kbps, what is wrong with the policy you have?

 From my quick reading is it seems this thing has a state machine infact
where sometimes you have to drop all other packets and when the state
machine transitions to a stable state then you just want to accept all
packets but prioritize its protocol packets. Also the state machine
seems to involve more than one port (for path redundancy reasons).
So where is this rate control coming from?

cheers,
jamal

Re: ingress policying for realtime protocol

[Uwe Kleine-König]

Hello,

On Thu, May 21, 2015 at 09:36:28AM -0400, Jamal Hadi Salim wrote:
> On 05/21/15 03:07, Uwe Kleine-König wrote:
> >On Wed, May 20, 2015 at 05:30:40PM -0700, Eric Dumazet wrote:
> >>On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
> >>
> >>>There is very little to do on ingress side since there is no queue at all,
> >>>not to mention priority, you could try ifb to see if it fits your need.
> >>
> >>Note that if the need is to police traffic, ifb is not really needed :
> >>
> >>TC="tc"
> >>DEV="dev eth0"
> >>IP=10.246.11.51/32
> >>$TC qdisc del $DEV ingress 2>/dev/null
> >>$TC qdisc add $DEV ingress
> >>$TC filter add $DEV parent ffff: protocol ip u32 match ip src $IP \
> >>	police rate 1Mbit burst 10Mbit mtu 66000 action drop/continue
> >>
> >>$TC -s filter ls $DEV parent ffff: protocol ip
> >I have something like that (matching on dst mac addresses instead of src ip):
> >
> >	tc qdisc add dev eth0 handle ffff: ingress
> >	tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match ether dst 01:15:4E:00:00:01 police pass
> >	tc filter add dev eth0 parent ffff: protocol all prio 50 u32 match u32 0 0 at 0 police rate 100kbit burst 10k drop
> >
> >So Cong interpreted my question right and probably I just used the
> >wrong keywords to make you understand the same.
> 
> I think both Cong and Eric are right.
> You wanted to priotize something thats _realtime_ by using queues, so
> Cong answered your question with ifb which will provide you a queue on
> ingress.
> OTOH, You should really avoid queues of any sort if latency is
My picture of the network stack might be wrong, but if the ethernet
driver queues say 5 packets to the network stack and the fourth is a MRP
packet than a priorization that makes the fourth packet processed first
would be nice.
If there is no queue and the first packet is processed before the
ethernet driver has a chance to hand over the second obviously there is
no benefit from using a prio queue because it would always only contain
a single packet.

> important to you - hence what Eric said is correct. Jitter will occur
> when it matters the most for you i.e when congestion kicks in; otherwise
> it will work (when there is no congestion;->)
> 
> So your requirements are conflicting and the result is two talented
> people are intepretting things differently;->
:-)

> So some questions to you:
> Why is there a 100Kbps limit for everything else? If it has to be at
> 100Kbps, what is wrong with the policy you have?
The 100kbit limit was found by starting with a higher limit and
decrement while scp still made the MRP hiccup. Now what's wrong: It's
annoying that other traffic is cut down that much.

> From my quick reading is it seems this thing has a state machine infact
> where sometimes you have to drop all other packets and when the state
> machine transitions to a stable state then you just want to accept all
> packets but prioritize its protocol packets. Also the state machine
> seems to involve more than one port (for path redundancy reasons).
> So where is this rate control coming from?
There is only a single port involved but that one is connected to a
Marvell switch. So the packets all come in on eth0 but the userspace
application that handles the MRP stuff still knows on which port of the
switch the packet came in. Also the blocking of a port is done with
configuration of the switch. Does this answer your question?

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: ingress policying for realtime protocol

[Jamal Hadi Salim]

On 05/21/15 14:58, Uwe Kleine-König wrote:
> Hello,
>

> My picture of the network stack might be wrong, but if the ethernet
> driver queues say 5 packets to the network stack and the fourth is a MRP
> packet than a priorization that makes the fourth packet processed first
> would be nice.
 >
> If there is no queue and the first packet is processed before the
> ethernet driver has a chance to hand over the second obviously there is
> no benefit from using a prio queue because it would always only contain
> a single packet.
>

Yes. If these packets arrived at about the same time then i think
good MRP friendly hardware should hand over the MRP packet first ahead
of some of the other ones. Then you really dont need a queue.
Otherwise you are compensating for this hardware not being smart enough.

Only scenario i can think of where a queue would be useful is if
the (host) system is overloaded. I think NAPI would pull up to N
packets and queue them to something like ifb. Later when the tasklet
is run they would be  re-injected back to the stack.

> The 100kbit limit was found by starting with a higher limit and
> decrement while scp still made the MRP hiccup. Now what's wrong: It's
> annoying that other traffic is cut down that much.
>

I think from that perspective I like Eric's script, except i would
modify it to use simple pfifo i.e shaping is not really necessary
but as you experiment you may reach a different conclusion.

> There is only a single port involved but that one is connected to a
> Marvell switch. So the packets all come in on eth0 but the userspace
> application that handles the MRP stuff still knows on which port of the
> switch the packet came in. Also the blocking of a port is done with
> configuration of the switch. Does this answer your question?
>

Hrm. So eth0 is a management/control port? I thought this was a speacial
MRP kind of hardware. And if am not mistaken the switch is just driven
by classical STP state of blocking/forwarding etc. So other than the
control packets showing up on out of band port this looks and smells
like STP. i.e this is just vanilla off the shelf switch ASIC with no
MRP specialization.


cheers,
jamal

Re: ingress policying for realtime protocol

[Eric Dumazet]

On Thu, 2015-05-21 at 09:07 +0200, Uwe Kleine-König wrote:
> On Wed, May 20, 2015 at 05:30:40PM -0700, Eric Dumazet wrote:
> > On Wed, 2015-05-20 at 16:46 -0700, Cong Wang wrote:
> > 
> > > There is very little to do on ingress side since there is no queue at all,
> > > not to mention priority, you could try ifb to see if it fits your need.
> > 
> > Note that if the need is to police traffic, ifb is not really needed :
> > 
> > TC="tc"
> > DEV="dev eth0"
> > IP=10.246.11.51/32
> > $TC qdisc del $DEV ingress 2>/dev/null
> > $TC qdisc add $DEV ingress
> > $TC filter add $DEV parent ffff: protocol ip u32 match ip src $IP \
> > 	police rate 1Mbit burst 10Mbit mtu 66000 action drop/continue
> > 
> > $TC -s filter ls $DEV parent ffff: protocol ip
> I have something like that (matching on dst mac addresses instead of src ip):
> 
> 	tc qdisc add dev eth0 handle ffff: ingress
> 	tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match ether dst 01:15:4E:00:00:01 police pass
> 	tc filter add dev eth0 parent ffff: protocol all prio 50 u32 match u32 0 0 at 0 police rate 100kbit burst 10k drop

Thats a very aggressive policer.

You cannot really prevent 'other traffic' from consuming wire time.
Dropping them at receiver might be too late, as they had already blocked
your realtime traffic.

> 
> . So Cong interpreted my question right and probably I just used the
> wrong keywords to make you understand the same. I try again to put my
> idea in words to make it explicit:
> 
> I imagine that it could help in my case if I could assert that MRP
> packets are handled priorized over other traffic without throwing away
> so many unrelated packets. For egress that works by e.g. using a prio
> qdisc. For ingress however only shaping is available.

Your script actually implements policing.

If you want shaping, then you need to add IFB to get queues and
scheduling of these queues.

> 
> So the question essentially is: Why doesn't this work for ingress? Cong
> wrote "there is no queue at all [for ingress]". Is this by design? Or is
> it just not implemented because noone spend the effort to work on that?
> Do you think it would help me?

Use IFB, and install TBF on top of it.

(But bypass it for your realtime packets)

Adapt following script for example.

ETH=eth0
IFB=ifb0

modprobe ifb
ip link set dev $IFB up

tc qdisc del dev $ETH ingress 2>/dev/null
tc qdisc add dev $ETH ingress 2>/dev/null

tc filter add dev $ETH parent ffff: \
   protocol all prio 10 u32 match ether dst 01:15:4E:00:00:01 police pass

tc filter add dev $ETH parent ffff: \
   protocol ip u32 match u32 0 0 action mirred egress \
   redirect dev $IFB

ethtool -K $ETH gro off lro off 2>/dev/null

ifconfig $IFB txqueuelen 1000

tc qdisc del dev $IFB root 2>/dev/null
tc qdisc add dev $IFB root \
    tbf limit 1000 burst 64000 rate 100kbit