UDP jitter

UDP jitter

[Nebojša Ćosić]

Hi,
I am doing some work on a product running kernel 2.6.33.7.2-rt30.
Applications running on this kernel are a bit specific, meaning that
there are a number of threads running on a different priorities.
For a several months I was haunted with spurious jitter, detected on
UDP messages - multicast UDP messages where received on originating
node without any delay, but on other nodes a delay in range of 10s of
milliseconds was detected. Simply, it looked like a message was stuck
in kernel before finally getting transmitted.
Finally, thanks to LTTng tool, I was able to locate the problem down to
this peace of code in net/sched/sch_generic.c:

int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
                    struct net_device *dev, struct netdev_queue *txq,
                    spinlock_t *root_lock)
{
        int ret = NETDEV_TX_BUSY;

        /* And release qdisc */
        spin_unlock(root_lock);

        HARD_TX_LOCK(dev, txq);

        if (!netif_tx_queue_stopped(txq) && !netif_tx_queue_frozen(txq))
                ret = dev_hard_start_xmit(skb, dev, txq);


        HARD_TX_UNLOCK(dev, txq);

        spin_lock(root_lock);
...

When transmit queue is empty, thread wanting to send a message comes
directly to sch_direct_xmit, without changing context. It then releases
spin lock, and than takes another. So far so good.
If this starting thread is of lower priority, it can be preempted by
another thread, while still being in dev_hard_start_xmit function
This thread will check if HARD_TX_LOCK is taken, and if so, go on and
queue its own message.
If there are enough higher priority tasks, tx can be stalled
indefinitely... Effectively, there is priority inversion.
My temporary workaround was to simply remove lock handling from
sch_direct_xmit, getting priority inheritance on root_lock, and
effectively disabling queuing... Not very good solution, but it works
perfectly well for my particular application.
I see that the same code still exists in latest kernels, so I suppose
that problem exists even there. It is RT specific thou. 
What would be a proper solution to this problem?
Sending first message without switching context is a nice
optimization, but it creates problems...

-- 
Regards
Nebojša
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Carsten Emde]

Hi Nebojša,

> I am doing some work on a product running kernel 2.6.33.7.2-rt30.
> Applications running on this kernel are a bit specific, meaning that
> there are a number of threads running on a different priorities.
> For a several months I was haunted with spurious jitter, detected on
> UDP messages - multicast UDP messages where received on originating
> node without any delay, but on other nodes a delay in range of 10s of
> milliseconds was detected. Simply, it looked like a message was stuck
> in kernel before finally getting transmitted.
> Finally, thanks to LTTng tool, I was able to locate the problem down to
> this peace of code in net/sched/sch_generic.c:
>
> int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
>                      struct net_device *dev, struct netdev_queue *txq,
>                      spinlock_t *root_lock)
> {
>          int ret = NETDEV_TX_BUSY;
>
>          /* And release qdisc */
>          spin_unlock(root_lock);
>
>          HARD_TX_LOCK(dev, txq);
>
>          if (!netif_tx_queue_stopped(txq) && !netif_tx_queue_frozen(txq))
>                  ret = dev_hard_start_xmit(skb, dev, txq);
>
>
>          HARD_TX_UNLOCK(dev, txq);
>
>          spin_lock(root_lock);
> ...
>
> When transmit queue is empty, thread wanting to send a message comes
> directly to sch_direct_xmit, without changing context. It then releases
> spin lock, and than takes another. So far so good.
> If this starting thread is of lower priority, it can be preempted by
> another thread, while still being in dev_hard_start_xmit function
> This thread will check if HARD_TX_LOCK is taken, and if so, go on and
> queue its own message.
> If there are enough higher priority tasks, tx can be stalled
> indefinitely. [..]
Did you increase the priority of the related sirq-net-tx and sirq-net-rx 
kernel threads appropriately? Some more details on enabling real-time 
Ethernet are given here -> https://www.osadl.org/?id=930.

	-Carsten.
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Nebojša Ćosić]

> Hi Nebojša,
Hi Carsten
> 
> > I am doing some work on a product running kernel 2.6.33.7.2-rt30.
> > Applications running on this kernel are a bit specific, meaning that
> > there are a number of threads running on a different priorities.
> > For a several months I was haunted with spurious jitter, detected on
> > UDP messages - multicast UDP messages where received on originating
> > node without any delay, but on other nodes a delay in range of 10s of
> > milliseconds was detected. Simply, it looked like a message was stuck
> > in kernel before finally getting transmitted.
> > Finally, thanks to LTTng tool, I was able to locate the problem down to
> > this peace of code in net/sched/sch_generic.c:
> >
> > int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
> >                      struct net_device *dev, struct netdev_queue *txq,
> >                      spinlock_t *root_lock)
> > {
> >          int ret = NETDEV_TX_BUSY;
> >
> >          /* And release qdisc */
> >          spin_unlock(root_lock);
> >
> >          HARD_TX_LOCK(dev, txq);
> >
> >          if (!netif_tx_queue_stopped(txq) && !netif_tx_queue_frozen(txq))
> >                  ret = dev_hard_start_xmit(skb, dev, txq);
> >
> >
> >          HARD_TX_UNLOCK(dev, txq);
> >
> >          spin_lock(root_lock);
> > ...
> >
> > When transmit queue is empty, thread wanting to send a message comes
> > directly to sch_direct_xmit, without changing context. It then releases
> > spin lock, and than takes another. So far so good.
> > If this starting thread is of lower priority, it can be preempted by
> > another thread, while still being in dev_hard_start_xmit function
> > This thread will check if HARD_TX_LOCK is taken, and if so, go on and
> > queue its own message.
> > If there are enough higher priority tasks, tx can be stalled
> > indefinitely. [..]
> Did you increase the priority of the related sirq-net-tx and sirq-net-rx 
> kernel threads appropriately? Some more details on enabling real-time 
> Ethernet are given here -> https://www.osadl.org/?id=930.
Thanks for the link, I was aware of it.
I did try to increase sirq-net-tx and rx, even to get tx higher than rx
(in case incoming traffic was creating problems), but it didn't make
any difference. 
I was trying to isolate problem by running iperf, but it worked
perfectly well when run on it's own. No wonder, because it generates
traffic from the same priority, and to trigger this behaviour, one need
traffic from at least two levels of priority, and a busy CPU (so that
low priority thread can get blocked in driver for a noticeable period
of time ).
I suppose that running two iperf processes at different priorities can
demonstrate the problem.

> 
> 	-Carsten.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Nebojša
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

AW: UDP jitter

[eg Engleder Gerhard]

Hello Nebojša,

I have a similar problem now with 3.2.51-rt72. Did
you find any solution?

regards, gerhard

> -----Ursprüngliche Nachricht-----
> Von: linux-rt-users-owner@vger.kernel.org 
> [mailto:linux-rt-users-owner@vger.kernel.org] Im Auftrag von 
> Nebojša Cosic
> Gesendet: Dienstag, 30. April 2013 19:27
> An: Carsten Emde
> Cc: linux-rt-users
> Betreff: Re: UDP jitter
> 
> 
> > Hi Nebojša,
> Hi Carsten
> > 
> > > I am doing some work on a product running kernel 2.6.33.7.2-rt30.
> > > Applications running on this kernel are a bit specific, 
> meaning that 
> > > there are a number of threads running on a different priorities.
> > > For a several months I was haunted with spurious jitter, 
> detected on 
> > > UDP messages - multicast UDP messages where received on 
> originating 
> > > node without any delay, but on other nodes a delay in 
> range of 10s 
> > > of milliseconds was detected. Simply, it looked like a 
> message was 
> > > stuck in kernel before finally getting transmitted.
> > > Finally, thanks to LTTng tool, I was able to locate the 
> problem down 
> > > to this peace of code in net/sched/sch_generic.c:
> > >
> > > int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
> > >                      struct net_device *dev, struct 
> netdev_queue *txq,
> > >                      spinlock_t *root_lock) {
> > >          int ret = NETDEV_TX_BUSY;
> > >
> > >          /* And release qdisc */
> > >          spin_unlock(root_lock);
> > >
> > >          HARD_TX_LOCK(dev, txq);
> > >
> > >          if (!netif_tx_queue_stopped(txq) && 
> !netif_tx_queue_frozen(txq))
> > >                  ret = dev_hard_start_xmit(skb, dev, txq);
> > >
> > >
> > >          HARD_TX_UNLOCK(dev, txq);
> > >
> > >          spin_lock(root_lock);
> > > ...
> > >
> > > When transmit queue is empty, thread wanting to send a 
> message comes 
> > > directly to sch_direct_xmit, without changing context. It then 
> > > releases spin lock, and than takes another. So far so good.
> > > If this starting thread is of lower priority, it can be 
> preempted by 
> > > another thread, while still being in dev_hard_start_xmit function 
> > > This thread will check if HARD_TX_LOCK is taken, and if so, go on 
> > > and queue its own message.
> > > If there are enough higher priority tasks, tx can be stalled 
> > > indefinitely. [..]
> > Did you increase the priority of the related sirq-net-tx and 
> > sirq-net-rx kernel threads appropriately? Some more details on 
> > enabling real-time Ethernet are given here -> 
> https://www.osadl.org/?id=930.
> Thanks for the link, I was aware of it.
> I did try to increase sirq-net-tx and rx, even to get tx 
> higher than rx (in case incoming traffic was creating 
> problems), but it didn't make any difference. 
> I was trying to isolate problem by running iperf, but it 
> worked perfectly well when run on it's own. No wonder, 
> because it generates traffic from the same priority, and to 
> trigger this behaviour, one need traffic from at least two 
> levels of priority, and a busy CPU (so that low priority 
> thread can get blocked in driver for a noticeable period of time ).
> I suppose that running two iperf processes at different 
> priorities can demonstrate the problem.
> 
> > 
> > 	-Carsten.
> > --
> > To unsubscribe from this list: send the line "unsubscribe 
> > linux-rt-users" in the body of a message to 
> majordomo@vger.kernel.org 
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> Nebojša
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-rt-users" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> --
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Nebojša Ćosić]

> Hello Nebojša,
> 
> I have a similar problem now with 3.2.51-rt72. Did
> you find any solution?
> 
> regards, gerhard
> 
> > -----Ursprüngliche Nachricht-----
> > Von: linux-rt-users-owner@vger.kernel.org 
> > [mailto:linux-rt-users-owner@vger.kernel.org] Im Auftrag von 
> > Nebojša Cosic
> > Gesendet: Dienstag, 30. April 2013 19:27
> > An: Carsten Emde
> > Cc: linux-rt-users
> > Betreff: Re: UDP jitter
> > 
> > 
> > > Hi Nebojša,
> > Hi Carsten
> > > 
> > > > I am doing some work on a product running kernel 2.6.33.7.2-rt30.
> > > > Applications running on this kernel are a bit specific, 
> > meaning that 
> > > > there are a number of threads running on a different priorities.
> > > > For a several months I was haunted with spurious jitter, 
> > detected on 
> > > > UDP messages - multicast UDP messages where received on 
> > originating 
> > > > node without any delay, but on other nodes a delay in 
> > range of 10s 
> > > > of milliseconds was detected. Simply, it looked like a 
> > message was 
> > > > stuck in kernel before finally getting transmitted.
> > > > Finally, thanks to LTTng tool, I was able to locate the 
> > problem down 
> > > > to this peace of code in net/sched/sch_generic.c:
> > > >
> > > > int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
> > > >                      struct net_device *dev, struct 
> > netdev_queue *txq,
> > > >                      spinlock_t *root_lock) {
> > > >          int ret = NETDEV_TX_BUSY;
> > > >
> > > >          /* And release qdisc */
> > > >          spin_unlock(root_lock);
> > > >
> > > >          HARD_TX_LOCK(dev, txq);
> > > >
> > > >          if (!netif_tx_queue_stopped(txq) && 
> > !netif_tx_queue_frozen(txq))
> > > >                  ret = dev_hard_start_xmit(skb, dev, txq);
> > > >
> > > >
> > > >          HARD_TX_UNLOCK(dev, txq);
> > > >
> > > >          spin_lock(root_lock);
> > > > ...
> > > >
> > > > When transmit queue is empty, thread wanting to send a 
> > message comes 
> > > > directly to sch_direct_xmit, without changing context. It then 
> > > > releases spin lock, and than takes another. So far so good.
> > > > If this starting thread is of lower priority, it can be 
> > preempted by 
> > > > another thread, while still being in dev_hard_start_xmit function 
> > > > This thread will check if HARD_TX_LOCK is taken, and if so, go on 
> > > > and queue its own message.
> > > > If there are enough higher priority tasks, tx can be stalled 
> > > > indefinitely. [..]
> > > Did you increase the priority of the related sirq-net-tx and 
> > > sirq-net-rx kernel threads appropriately? Some more details on 
> > > enabling real-time Ethernet are given here -> 
> > https://www.osadl.org/?id=930.
> > Thanks for the link, I was aware of it.
> > I did try to increase sirq-net-tx and rx, even to get tx 
> > higher than rx (in case incoming traffic was creating 
> > problems), but it didn't make any difference. 
> > I was trying to isolate problem by running iperf, but it 
> > worked perfectly well when run on it's own. No wonder, 
> > because it generates traffic from the same priority, and to 
> > trigger this behaviour, one need traffic from at least two 
> > levels of priority, and a busy CPU (so that low priority 
> > thread can get blocked in driver for a noticeable period of time ).
> > I suppose that running two iperf processes at different 
> > priorities can demonstrate the problem.
> > 
> > > 
> > > 	-Carsten.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe 
> > > linux-rt-users" in the body of a message to 
> > majordomo@vger.kernel.org 
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > --
> > Nebojša
> > --
> > To unsubscribe from this list: send the line "unsubscribe 
> > linux-rt-users" in the body of a message to 
> > majordomo@vger.kernel.org More majordomo info at  
> > http://vger.kernel.org/majordomo-info.html
> > --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

You can try with this patch. I am quite sure that same problem persists
on all newer kernels (I am using 2.6.33), but never had a time to create
simple test to prove it.

Index: net/sched/sch_generic.c
===================================================================
--- net/sched/sch_generic.c	(revision 1709)
+++ net/sched/sch_generic.c	(revision 1710)
@@ -120,16 +120,18 @@
 	int ret = NETDEV_TX_BUSY;
 
 	/* And release qdisc */
-	spin_unlock(root_lock);
+/*	spin_unlock(root_lock);
 
 	HARD_TX_LOCK(dev, txq);
+*/
 	if (!netif_tx_queue_stopped(txq)
&& !netif_tx_queue_frozen(txq)) ret = dev_hard_start_xmit(skb, dev,
txq); 
+/*
 	HARD_TX_UNLOCK(dev, txq);
 
 	spin_lock(root_lock);

Re: UDP jitter

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1812 bytes --]

On Wed, 6 Nov 2013, Nebojša Ćosić wrote:
> You can try with this patch. I am quite sure that same problem persists
> on all newer kernels (I am using 2.6.33), but never had a time to create
> simple test to prove it.

I'm looking forward to your detailed analysis ....

> Index: net/sched/sch_generic.c
> ===================================================================
> --- net/sched/sch_generic.c	(revision 1709)
> +++ net/sched/sch_generic.c	(revision 1710)
> @@ -120,16 +120,18 @@
>  	int ret = NETDEV_TX_BUSY;
>  
>  	/* And release qdisc */
> -	spin_unlock(root_lock);
> +/*	spin_unlock(root_lock);
>  
>  	HARD_TX_LOCK(dev, txq);
> +*/

Do you really think this locking scheme is just for fun? qdisk_lock,
i.e. root_lock and the netif_tx_lock are mutually exclusive for a
reason. Reading the documentation and comments of code is optional,
right?

> > > When transmit queue is empty, thread wanting to send a message
> > > comes directly to sch_direct_xmit, without changing context. It
> > > then releases spin lock, and than takes another. So far so good.
> > > If this starting thread is of lower priority, it can be
> > > preempted by another thread, while still being in
> > > dev_hard_start_xmit function This thread will check if
> > > HARD_TX_LOCK is taken, and if so, go on and queue its own
> > > message.  If there are enough higher priority tasks, tx can be
> > > stalled indefinitely. [..]

Utter nonsense.

The code (sch_direct_xmit) you are modifying is called from exactly
two places:

1) __dev_xmit_skb

2) __qdisc_run()

   Which in turn two call sites:

   1) __dev_xmit_skb() either directly or via qdisc_run()

   2) net_tx_action() which is the NET_TX softirq action

None of those calls does a trylock on the xmit lock. So what the heck
are you talking about?

Thanks,

	tglx

AW: UDP jitter

[eg Engleder Gerhard]

> On Wed, 6 Nov 2013, Nebojša Ćosić wrote:
> > You can try with this patch. I am quite sure that same problem 
> > persists on all newer kernels (I am using 2.6.33), but never had a 
> > time to create simple test to prove it.
> 
> I'm looking forward to your detailed analysis ....
> 
> > Index: net/sched/sch_generic.c
> > ===================================================================
> > --- net/sched/sch_generic.c	(revision 1709)
> > +++ net/sched/sch_generic.c	(revision 1710)
> > @@ -120,16 +120,18 @@
> >  	int ret = NETDEV_TX_BUSY;
> >  
> >  	/* And release qdisc */
> > -	spin_unlock(root_lock);
> > +/*	spin_unlock(root_lock);
> >  
> >  	HARD_TX_LOCK(dev, txq);
> > +*/
> 
> Do you really think this locking scheme is just for fun? 
> qdisk_lock, i.e. root_lock and the netif_tx_lock are mutually 
> exclusive for a reason. Reading the documentation and 
> comments of code is optional, right?
> 
> > > > When transmit queue is empty, thread wanting to send a message 
> > > > comes directly to sch_direct_xmit, without changing context. It 
> > > > then releases spin lock, and than takes another. So far so good.
> > > > If this starting thread is of lower priority, it can be 
> preempted 
> > > > by another thread, while still being in dev_hard_start_xmit 
> > > > function This thread will check if HARD_TX_LOCK is 
> taken, and if 
> > > > so, go on and queue its own message.  If there are 
> enough higher 
> > > > priority tasks, tx can be stalled indefinitely. [..]
> 
> Utter nonsense.

In my situation if a first thread is preempted in dev_hard_start_xmit, then a second
thread which transmits over the same interface sees a none empty queue and
just enqueue the skb in the queue. No transmit (HARD_TX_LOCK) is necessary
because the first thread is already working on the queue. Please correct me if I'm
not right.
For my case the first thread does normal SSH communication and the second
thread does real-time UDP communication. This results in a priority inversion for
the transmission of skb's over the same network interface.
Is there any chance to get rid of this priority inversion? Maybe with priority qdisc?

> The code (sch_direct_xmit) you are modifying is called from 
> exactly two places:
> 
> 1) __dev_xmit_skb
> 
> 2) __qdisc_run()
> 
>    Which in turn two call sites:
> 
>    1) __dev_xmit_skb() either directly or via qdisc_run()
> 
>    2) net_tx_action() which is the NET_TX softirq action
> 
> None of those calls does a trylock on the xmit lock. So what 
> the heck are you talking about?
> 
> Thanks,
> 
> 	tglx

regards, gerhard--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Nebojša Ćosić]

> > On Wed, 6 Nov 2013, Nebojša Ćosić wrote:
> > > You can try with this patch. I am quite sure that same problem 
> > > persists on all newer kernels (I am using 2.6.33), but never had a 
> > > time to create simple test to prove it.
> > 
> > I'm looking forward to your detailed analysis ....
> > 
> > > Index: net/sched/sch_generic.c
> > > ===================================================================
> > > --- net/sched/sch_generic.c	(revision 1709)
> > > +++ net/sched/sch_generic.c	(revision 1710)
> > > @@ -120,16 +120,18 @@
> > >  	int ret = NETDEV_TX_BUSY;
> > >  
> > >  	/* And release qdisc */
> > > -	spin_unlock(root_lock);
> > > +/*	spin_unlock(root_lock);
> > >  
> > >  	HARD_TX_LOCK(dev, txq);
> > > +*/
> > 
> > Do you really think this locking scheme is just for fun? 
> > qdisk_lock, i.e. root_lock and the netif_tx_lock are mutually 
> > exclusive for a reason. Reading the documentation and 
> > comments of code is optional, right?
> > 
> > > > > When transmit queue is empty, thread wanting to send a message 
> > > > > comes directly to sch_direct_xmit, without changing context. It 
> > > > > then releases spin lock, and than takes another. So far so good.
> > > > > If this starting thread is of lower priority, it can be 
> > preempted 
> > > > > by another thread, while still being in dev_hard_start_xmit 
> > > > > function This thread will check if HARD_TX_LOCK is 
> > taken, and if 
> > > > > so, go on and queue its own message.  If there are 
> > enough higher 
> > > > > priority tasks, tx can be stalled indefinitely. [..]
> > 
> > Utter nonsense.
> 
> In my situation if a first thread is preempted in dev_hard_start_xmit, then a second
> thread which transmits over the same interface sees a none empty queue and
> just enqueue the skb in the queue. No transmit (HARD_TX_LOCK) is necessary
> because the first thread is already working on the queue. Please correct me if I'm
> not right.
> For my case the first thread does normal SSH communication and the second
> thread does real-time UDP communication. This results in a priority inversion for
> the transmission of skb's over the same network interface.
> Is there any chance to get rid of this priority inversion? Maybe with priority qdisc?
And this is precisely what this hack is trying to work around. Original
thread was a call for a discussion in order to find real solution to the
problem.
New messages arriving to the queue should somehow raise
priority of thread sitting in dev_hard_start_xmit according to their
own thread priority. Or can priority of a thread currently in
dev_hard_start_xmit temporarily be raised to some configurable value. 
This hack is effectively shutting down queuing of skbs, and instead
forces queuing of threads.
> 
> > The code (sch_direct_xmit) you are modifying is called from 
> > exactly two places:
> > 
> > 1) __dev_xmit_skb
> > 
> > 2) __qdisc_run()
> > 
> >    Which in turn two call sites:
> > 
> >    1) __dev_xmit_skb() either directly or via qdisc_run()
> > 
> >    2) net_tx_action() which is the NET_TX softirq action
> > 
> > None of those calls does a trylock on the xmit lock. So what 
> > the heck are you talking about?
> > 
> > Thanks,
> > 
> > 	tglx
> 
> regards, gerhard--
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Pozdrav,
Nebojša
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4340 bytes --]

On Thu, 7 Nov 2013, Nebojša Ćosić wrote:
> On Thu, 7 Nov 2013 eg Engleder Gerhard wrote:
> > On Wed, 6 Nov 2013, Thomas Gleixner wrote:
> > > Utter nonsense.
> > 
> > In my situation if a first thread is preempted in dev_hard_start_xmit, then a second
> > thread which transmits over the same interface sees a none empty queue and
> > just enqueue the skb in the queue. No transmit (HARD_TX_LOCK) is necessary
> > because the first thread is already working on the queue. Please correct me if I'm
> > not right.

You are right. When the queue is active, there is no contention or
trylock on the xmit lock. And the resulting queuing operation is
documented behaviour.

> > For my case the first thread does normal SSH communication and the second
> > thread does real-time UDP communication. This results in a priority inversion for
> > the transmission of skb's over the same network interface.

That's the nature of a single queue which has very primitive fifo
odering constraints. There is no priority inversion. It's documented
behaviour. Just because you set the prio of your task to uberimportant
does not magically change the behaviour of everything to your favour.

> > Is there any chance to get rid of this priority inversion? Maybe
> > with priority qdisc?

See below.

> And this is precisely what this hack is trying to work around. Original
> thread was a call for a discussion in order to find real solution to the
> problem.

By providing a problem description which is completely useless? And
"solving" the issue by violating the locking rules of the networking
code?

> New messages arriving to the queue should somehow raise
> priority of thread sitting in dev_hard_start_xmit according to their
> own thread priority. Or can priority of a thread currently in
> dev_hard_start_xmit temporarily be raised to some configurable value. 
> This hack is effectively shutting down queuing of skbs, and instead
> forces queuing of threads.

Are you really believing that a generic fifo based queueing mechanism
just needs a random locking/priority tweak to become deterministic?

There is a damned good reason why the networking folks have spent a
lot of time to implement queueing, filtering and traffic shaping
mechanisms which allow you to influence the order and bandwidth
consumed by outgoing packets.

But understanding this seems to be way harder than wasting precious
time to debug the "shortcomings" of RT and babbling about priorities
and priority inversions.

Just a simple example why your reasoning is completely bogus:

 Assume a network device which has a tx ring size of 256, which is
 quite common.

 Now assume the following event flow:

 highprio_app() -> send package()

 	Queues ONE package which is enqueued into the TX ring right away

 lowprio_app() -> sendfile()
 
	The file length results in 255 packages. These are enqueued
	into the TX_RING right away WITHOUT congesting qdisc

 highprio_app() -> send package()

 	Queues ONE package which is enqueued into the TX ring right away


 NOTE: Neither operations are running into your observed scenario!

 Now the package of the high prio app is at the end of the TX ring and
 it has to wait for all other packages queued by the low prio app to
 be sent out over the wire.

 Let's do some rough math:

 100MB: Package size 1542 Bytes

 	time per package = ~15us

	time for 255 packages = ~3ms

 So now, assume that the low prio task is taking less than 1 msec to
 queue 255 packages and your high prio task is required to send every
 1ms with a maximum latency of 0.5 ms.

 How is your bogus hack going to solve that problem?

Not at all. Period. 

And none of your ideas to prevent that alleged priority inversion is
going to solve that.

WHY?

Simply because it has nothing to do with priority inversion. It's just
the nature of a single unmanaged queue. The behaviour is completely
correct.

Just for the record. I'm really frightened by the phrase "UDP
realtime" which was mentioned in this thread more than once. Looking
at the desperation level of these posts I fear, that there are going
to be real world products out already or available in the near future
which are based on the profound lack of understanding of the
technology they are based on.

This just confirms my theory, that most parts of this industry just
work by chance.

Thanks,

	tglx

Re: UDP jitter

[Stanislav Meduna]

On 08.11.2013 03:07, Thomas Gleixner wrote:

> Simply because it has nothing to do with priority inversion. It's just
> the nature of a single unmanaged queue. The behaviour is completely
> correct.

I cannot comment on the code as I did not analyze it myself
(at least yet), but I think Nebojsa is worried by the situation
where the high-prio thread is not able to _queue_ its packets because
of the low-prio thread is sitting in some lock being preempted
by something unrelated.

> Just for the record. I'm really frightened by the phrase "UDP
> realtime" which was mentioned in this thread more than once. Looking
> at the desperation level of these posts I fear, that there are going
> to be real world products out already or available in the near future
> which are based on the profound lack of understanding of the
> technology they are based on.

Yes there are real-world product using real-time ethernet - not
necessarily UDP but for example anything EtherCAT based absolutely
needs to be able to send certain packets cyclically no more than
100 ms (or 10 ms or 2 ms) apart otherwise all hell breaks loose
with real-world connected hardware. The room for jitter is the
limit minus cycle the packets are being sent, which can be pretty
tight.

On the same wire there is a non-rt traffic, usually sent by another
lower-prio thread. The queuing of the packets itself is not a problem -
this is basically a request-response protocol and there will never
be more than several packets before the higher-level one - but
a priority inversion where the first thread is stuck in the network
code because something preempted the low-prio one that is just queuing
a packet would be a big problem.

There is nothing else on the network interface, but there usually
is another ethernet interface for non-realtime traffic. If some
of the locks involved is driver-wise instead of interface-wise
we already lost (I understand that this case would be the problem
of the driver and not the infrastructure).

If I am understanding the Nebojsa's worries wrong or if the
scenario cannot happen, please disregard.

Regards
-- 
                                     Stano

Re: UDP jitter

[Nebojša Ćosić]

> On 08.11.2013 03:07, Thomas Gleixner wrote:
> 
> > Simply because it has nothing to do with priority inversion. It's just
> > the nature of a single unmanaged queue. The behaviour is completely
> > correct.
> 
> I cannot comment on the code as I did not analyze it myself
> (at least yet), but I think Nebojsa is worried by the situation
> where the high-prio thread is not able to _queue_ its packets because
> of the low-prio thread is sitting in some lock being preempted
> by something unrelated.
> 
> > Just for the record. I'm really frightened by the phrase "UDP
> > realtime" which was mentioned in this thread more than once. Looking
> > at the desperation level of these posts I fear, that there are going
> > to be real world products out already or available in the near future
> > which are based on the profound lack of understanding of the
> > technology they are based on.
> 
> Yes there are real-world product using real-time ethernet - not
> necessarily UDP but for example anything EtherCAT based absolutely
> needs to be able to send certain packets cyclically no more than
> 100 ms (or 10 ms or 2 ms) apart otherwise all hell breaks loose
> with real-world connected hardware. The room for jitter is the
> limit minus cycle the packets are being sent, which can be pretty
> tight.
> 
> On the same wire there is a non-rt traffic, usually sent by another
> lower-prio thread. The queuing of the packets itself is not a problem -
> this is basically a request-response protocol and there will never
> be more than several packets before the higher-level one - but
> a priority inversion where the first thread is stuck in the network
> code because something preempted the low-prio one that is just queuing
> a packet would be a big problem.
> 
> There is nothing else on the network interface, but there usually
> is another ethernet interface for non-realtime traffic. If some
> of the locks involved is driver-wise instead of interface-wise
> we already lost (I understand that this case would be the problem
> of the driver and not the infrastructure).
> 
> If I am understanding the Nebojsa's worries wrong or if the
> scenario cannot happen, please disregard.
> 
> Regards

Thanks Stanislav for excellent description.
In my case, bandwidth usage was bellow 5%, and measured delays on udp
messages were in some cases longer than 40ms. Only two nodes where
connected on test bench, and only one network interface per node was
available.
I have to repeat that the code in question worked in my case, which is
very specific, and that reason for discussing it is to try to come to a
proper solution to this problem.
And yes, I do believe that, even if behaviour is documented, it is still
a problem and can be dealt with in a better way.

-- 
Nebojša
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Tom Cook]

On Fri, Nov 8, 2013 at 9:39 AM, Stanislav Meduna <stano@meduna.org> wrote:
>
> On 08.11.2013 03:07, Thomas Gleixner wrote:
> > Just for the record. I'm really frightened by the phrase "UDP
> > realtime" which was mentioned in this thread more than once. Looking
> > at the desperation level of these posts I fear, that there are going
> > to be real world products out already or available in the near future
> > which are based on the profound lack of understanding of the
> > technology they are based on.
>
> Yes there are real-world product using real-time ethernet - not
> necessarily UDP but for example anything EtherCAT based absolutely
> needs to be able to send certain packets cyclically no more than
> 100 ms (or 10 ms or 2 ms) apart otherwise all hell breaks loose
> with real-world connected hardware. The room for jitter is the
> limit minus cycle the packets are being sent, which can be pretty
> tight.

EtherCAT is a rather different case - you're never going to get mixed
RT and non-RT traffic on an EtherCAT link (though you can carry
Ethernet traffic over EtherCAT, but that's different).  An Ethernet
port that's being used for EtherCAT is simply not able to also carry
IP traffic.  Even given those restrictions, there is, IIRC, only one
Ethernet chipset that can be used for deterministic IO interfacing.

ProfiNet is a slightly more on-topic example, though that is only RT
given special hardware and drivers that are able to prioritise the
ProfiNet traffic over non-RT traffic.

UDP realtime is indeed rather a pipe dream unless the whole system is
very carefully controlled - which at minimum means not mixing RT and
non-RT traffic.

T

AW: UDP jitter

[eg Engleder Gerhard]

> -----Ursprüngliche Nachricht-----
> Von: Stanislav Meduna [mailto:stano@meduna.org] 
> Gesendet: Freitag, 08. November 2013 10:40
> An: Thomas Gleixner; Nebojša Ćosić
> Cc: eg Engleder Gerhard; linux-rt-users
> Betreff: Re: UDP jitter
>
> I cannot comment on the code as I did not analyze it myself 
> (at least yet), but I think Nebojsa is worried by the 
> situation where the high-prio thread is not able to _queue_ 
> its packets because of the low-prio thread is sitting in some 
> lock being preempted by something unrelated.

The high-prio thread is able to queue the packet. Thus for the
thread itself everthing is fine, no priority inversion for the thread.
At least in my case. In my situation the transmission of the
packet from the high-prio thread is delayed until the low-prio
thread is scheduled again.

Is there any possibility to transmit high-prio traffic and low-prio
traffic over the same network interface, with limited delay for
the high-prio traffic? The delay of a full ring buffer of the network
device is at least limited with e.g. 10ms, this is acceptable
in my application.

regards, gerhard--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Thomas Gleixner]

On Fri, 8 Nov 2013, Stanislav Meduna wrote:
> On the same wire there is a non-rt traffic, usually sent by another
> lower-prio thread. The queuing of the packets itself is not a problem -
> this is basically a request-response protocol and there will never
> be more than several packets before the higher-level one - but

So you exactly know that the non-rt traffic is never going to queue
enough traffic to disturb your high prio traffic? That might be true
for your particular scenario, but definitely not true for the general
case as I showed with a very trivial example.

> a priority inversion where the first thread is stuck in the network
> code because something preempted the low-prio one that is just queuing
> a packet would be a big problem.

Did you even take the time to read and understand what I wrote? It's
not a priority inversion problem. It's a problem of not understanding
the technology.

The default queue of the network stack is a single fifo queue. So what
are you expecting from a single queue? That it's magically allowing
your high priority task to succeed? And just because you are too lazy
or unable to understand the concepts of advanced networking you join
the choir and shout "priority inversion"?

I explained, that this is an expected and completely correct behaviour
which has absolutely nothing to do with priority inversion.

There are other ways to solve this without touching a single line of
kernel code. The technology is all there, you just have to use it. Of
course that requires to understand it in the first place, though this
seems to be way harder than spending a lot of time tracing the problem
and then yelling "priority inversion" and hope that the RT folks are
going to "fix" it.

Hell no, there is no way to cure blissful ignorance with kernel
patches.

Thanks,

	tglx

Re: UDP jitter

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 738 bytes --]

On Fri, 8 Nov 2013, Nebojša Ćosić wrote:
> I have to repeat that the code in question worked in my case, which is
> very specific, and that reason for discussing it is to try to come to a
> proper solution to this problem.
> And yes, I do believe that, even if behaviour is documented, it is still
> a problem and can be dealt with in a better way.

I have to repeat what the real problem is:

 You are not understanding at all how networking, queueing, traffic
 control and other advanced concepts are working. You are not even
 trying it.

 You just insist that your view of the problem is correct and there
 must be a magic technical solution to it.

Again, there is no technical solution to cure ignorance.

Thanks,

	tglx

AW: UDP jitter

[eg Engleder Gerhard]

 > -----Ursprüngliche Nachricht-----
> Von: Thomas Gleixner [mailto:tglx@linutronix.de] 
> Gesendet: Freitag, 08. November 2013 12:31
> An: Stanislav Meduna
> Cc: Nebojša Ćosić; eg Engleder Gerhard; linux-rt-users
> Betreff: Re: UDP jitter
> 
> There are other ways to solve this without touching a single 
> line of kernel code. The technology is all there, you just 
> have to use it. Of course that requires to understand it in 
> the first place, though this seems to be way harder than 
> spending a lot of time tracing the problem and then yelling 
> "priority inversion" and hope that the RT folks are going to "fix" it.

Can you give us a hint which technology to use? Currently I am
trying to use QoS Multi Band Priority Queueing.

regards, gerhard--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Stanislav Meduna]

On 08.11.2013 12:30, Thomas Gleixner wrote:

> The default queue of the network stack is a single fifo queue. So what
> are you expecting from a single queue? That it's magically allowing
> your high priority task to succeed? And just because you are too lazy
> or unable to understand the concepts of advanced networking you join
> the choir and shout "priority inversion"?

I did not meant the priority inversion in the strict thread running/blocked
sense. If the

  On 08.11.2013 11:51, eg Engleder Gerhard wrote:

  > In my situation the transmission of the packet from the high-prio
  > thread is delayed until the low-prio thread is scheduled again.

is true then to me it is a kind of priority inversion.

It is understood that the default implementation will not change
the order of packets already queued for transmission.

> There are other ways to solve this without touching a single line of
> kernel code. The technology is all there, you just have to use it. Of
> course that requires to understand it in the first place,

Fair enough.

> Hell no, there is no way to cure blissful ignorance with kernel
> patches.

The thing is that the userspace developers assume things. In this
case they assume that if a higher prio thread wants to send
a network packet and the send() returns, the maximum time until
that one packet gets out on the wire is the time all the already
queued packets need to get out. If this is true, I'm perfectly
fine - but from what Nebojsa and Gerhard are writing I have
my doubts and I am a bit scared as it can affect what we are
doing. As my job is not a kernel developer, I have to trust
others on this...

If not, I would not call it ignorance. This is IMHO a reasonable
assumption.

Thanks
-- 
                                          Stano

Re: UDP jitter

[Armin Steinhoff]

Thomas Gleixner wrote:
> On Fri, 8 Nov 2013, Nebojša Ćosić wrote:
>> I have to repeat that the code in question worked in my case, which is
>> very specific, and that reason for discussing it is to try to come to a
>> proper solution to this problem.
>> And yes, I do believe that, even if behaviour is documented, it is still
>> a problem and can be dealt with in a better way.
> I have to repeat what the real problem is:

It's also not possible to solve problems of the x86 based hardware by
kernel patches :)

Is the problem a mirage created by "SMI" ?
Running EtherCAT stable with SMI based delays of ~200-400 us is just an
illusion ...

--Armin

>
>  You are not understanding at all how networking, queueing, traffic
>  control and other advanced concepts are working. You are not even
>  trying it.
>
>  You just insist that your view of the problem is correct and there
>  must be a magic technical solution to it.
>
> Again, there is no technical solution to cure ignorance.
>
> Thanks,
>
> 	tglx
>
>
>
>
>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Thomas Gleixner]

On Fri, 8 Nov 2013, Stanislav Meduna wrote:
> On 08.11.2013 12:30, Thomas Gleixner wrote:
> 
> > The default queue of the network stack is a single fifo queue. So what
> > are you expecting from a single queue? That it's magically allowing
> > your high priority task to succeed? And just because you are too lazy
> > or unable to understand the concepts of advanced networking you join
> > the choir and shout "priority inversion"?
> 
> I did not meant the priority inversion in the strict thread running/blocked
> sense. If the
> 
>   On 08.11.2013 11:51, eg Engleder Gerhard wrote:
> 
>   > In my situation the transmission of the packet from the high-prio
>   > thread is delayed until the low-prio thread is scheduled again.
> 
> is true then to me it is a kind of priority inversion.

It's a simple queueing problem. FIFO queue:

     This principle states that customers are served one at a time and
     that the customer that has been waiting the longest is served
     first. It also implies that the customer who arrived last will
     have to wait until all others before him have been served.
 
That's exactly what happening here. No matter how important you
believe to be, you still have to wait in the line, if there is only a
single one. There is a reason why busses have reserved lanes in big
cities.

> > Hell no, there is no way to cure blissful ignorance with kernel
> > patches.
> 
> The thing is that the userspace developers assume things. In this
> case they assume that if a higher prio thread wants to send
> a network packet and the send() returns, the maximum time until
> that one packet gets out on the wire is the time all the already
> queued packets need to get out. If this is true, I'm perfectly
> fine - but from what Nebojsa and Gerhard are writing I have
> my doubts and I am a bit scared as it can affect what we are
> doing. As my job is not a kernel developer, I have to trust
> others on this...
> 
> If not, I would not call it ignorance. This is IMHO a reasonable
> assumption.

It seems there are really frightening assumptions made when it comes
to real time systems. Stuff does not get magically deterministic just
by throwing a real time kernel at it and by setting priorities.

real time systems always must be carefully designed, and if your
system requires deterministic networking, you have to understand how
networking works. Nobody is asking a user space developer to grok the
kernel internals, but understanding the basic principles of technology
on which an application depends on, is still required.

If people developing real time systems just make uninformed
assumptions and then blame the kernel for getting it wrong, I call
that ignorance.

Thanks,

	tglx

Re: AW: UDP jitter

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 991 bytes --]

On Fri, 8 Nov 2013, eg Engleder Gerhard wrote:
>  > -----Ursprüngliche Nachricht-----
> > Von: Thomas Gleixner [mailto:tglx@linutronix.de] 
> > Gesendet: Freitag, 08. November 2013 12:31
> > An: Stanislav Meduna
> > Cc: Nebojša Ćosić; eg Engleder Gerhard; linux-rt-users
> > Betreff: Re: UDP jitter
> > 
> > There are other ways to solve this without touching a single 
> > line of kernel code. The technology is all there, you just 
> > have to use it. Of course that requires to understand it in 
> > the first place, though this seems to be way harder than 
> > spending a lot of time tracing the problem and then yelling 
> > "priority inversion" and hope that the RT folks are going to "fix" it.
> 
> Can you give us a hint which technology to use? Currently I am
> trying to use QoS Multi Band Priority Queueing.

Well, it really depends on your requirements. Assuming you have a
simple binary decision (Bulk and VIP traffic) a simple 2 band priority
queue should work.

Thanks,

	tglx

Re: UDP jitter

[E-Blokos]

> Again, there is no technical solution to cure ignorance.
yes there is, it's called learning....

Re: UDP jitter

[Nebojša Ćosić]

Дана Fri, 8 Nov 2013 14:35:44 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> написа:

> On Fri, 8 Nov 2013, eg Engleder Gerhard wrote:
> >  > -----Ursprüngliche Nachricht-----
> > > Von: Thomas Gleixner [mailto:tglx@linutronix.de] 
> > > Gesendet: Freitag, 08. November 2013 12:31
> > > An: Stanislav Meduna
> > > Cc: Nebojša Ćosić; eg Engleder Gerhard; linux-rt-users
> > > Betreff: Re: UDP jitter
> > > 
> > > There are other ways to solve this without touching a single 
> > > line of kernel code. The technology is all there, you just 
> > > have to use it. Of course that requires to understand it in 
> > > the first place, though this seems to be way harder than 
> > > spending a lot of time tracing the problem and then yelling 
> > > "priority inversion" and hope that the RT folks are going to "fix" it.
> > 
> > Can you give us a hint which technology to use? Currently I am
> > trying to use QoS Multi Band Priority Queueing.
> 
> Well, it really depends on your requirements. Assuming you have a
> simple binary decision (Bulk and VIP traffic) a simple 2 band priority
> queue should work.
And how exactly is it going to help? You still have fifo, and you still
get messages from higher priority thread stuck waiting for low priority
task to finish sending whatever it has in there (please note how I
avoid mentioning priority inversion here).
By the way, by default you have pfifo_fast in there, which is capable
enough to do the priority based queuing.

> 
> Thanks,
> 
> 	tglx

Excuse my ignorance
Nebojša
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: UDP jitter

[Joe Korty]

On Fri, Nov 08, 2013 at 07:07:41AM -0500, eg Engleder Gerhard wrote:
>  > -----Urspr?ngliche Nachricht-----
> > Von: Thomas Gleixner [mailto:tglx@linutronix.de] 
> > Gesendet: Freitag, 08. November 2013 12:31
> > An: Stanislav Meduna
> > Cc: Neboj?a ?osi?; eg Engleder Gerhard; linux-rt-users
> > Betreff: Re: UDP jitter
> > 
> > There are other ways to solve this without touching a single 
> > line of kernel code. The technology is all there, you just 
> > have to use it. Of course that requires to understand it in 
> > the first place, though this seems to be way harder than 
> > spending a lot of time tracing the problem and then yelling 
> > "priority inversion" and hope that the RT folks are going to "fix" it.
> 
> Can you give us a hint which technology to use? Currently I am
> trying to use QoS Multi Band Priority Queueing.


Look into OpenOnload.  The software is GPLed, tho it
requires a custom Ethernet card that is perhaps 50% more
expensive than other 10GigE cards.

Regards
Joe