Eliminating Packet Latency

Eliminating Packet Latency

[Michael Hewitt]

Folks,

Please forgive me if this is an inappropriate forum for my question. 

We are using a real time Linux kernel (3.10.0) in a network appliance in order to achieve extremely consistent packet delivery times.  Generally, we see packet delivery variations of less than 100 microseconds, which is fabulous.  Occasionally, we see a packet delivery delay in excess of 1000 microseconds.  We are hoping to eliminate these spikes, which occur perhaps 1-2 times in a 24 hour period.

The machine configuration is as follows.  Thread IRQs are enabled, and we have elevated the priority of both the irq threads that service the specific network interface to 55.  We have also elevated the priority of the relevant user space thread to 49.  We are running on a 4 core Intel Xeon E3-1220v3 with an Intel NIC and the igb version 5.3.2 driver.  We disabled interrupt throttling in the Intel driver (rx-usecs = 0, tx-usecs = 0).  SE Linux is disabled, eliminating a huge packet latency spike during login.  We are running CentOS 7.1 tuned for network latency ("tuned-adm profile network-latency").  IRQ balancing is disabled.  BIOS CPU power management is set to maximum performance.

Any help you can provide would be greatly appreciated.

Thanks,
Mike

RE: Eliminating Packet Latency

[Dave Rolenc]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

Michael,

First, I don't know if my personal findings are applicable to your
situation. I am using much older kernels, so the issue I ran into may be
obsolete or at least things may be different.

I found that sometimes when sending network packets, the kernel would hijack
the thread of the sender when the queue was empty. This would cause issues,
since I had no control over the priority of that thread, since it could be
anything. My fix was to never allow the kernel to send in the context of the
sending userspace thread. I suspect that most uses of the real time  kernel
don't expect the real time nature of things to extend across the network,
but I could be wrong.

Best Regards,

Dave  



-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org
[mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Michael Hewitt
Sent: Monday, March 14, 2016 3:02 PM
To: linux-rt-users@vger.kernel.org
Subject: Eliminating Packet Latency

Folks,

Please forgive me if this is an inappropriate forum for my question. 

We are using a real time Linux kernel (3.10.0) in a network appliance in
order to achieve extremely consistent packet delivery times.  Generally, we
see packet delivery variations of less than 100 microseconds, which is
fabulous.  Occasionally, we see a packet delivery delay in excess of 1000
microseconds.  We are hoping to eliminate these spikes, which occur perhaps
1-2 times in a 24 hour period.

The machine configuration is as follows.  Thread IRQs are enabled, and we
have elevated the priority of both the irq threads that service the specific
network interface to 55.  We have also elevated the priority of the relevant
user space thread to 49.  We are running on a 4 core Intel Xeon E3-1220v3
with an Intel NIC and the igb version 5.3.2 driver.  We disabled interrupt
throttling in the Intel driver (rx-usecs = 0, tx-usecs = 0).  SE Linux is
disabled, eliminating a huge packet latency spike during login.  We are
running CentOS 7.1 tuned for network latency ("tuned-adm profile
network-latency").  IRQ balancing is disabled.  BIOS CPU power management is
set to maximum performance.

Any help you can provide would be greatly appreciated.

Thanks,
Mike


--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5694 bytes --]

Re: Eliminating Packet Latency

[Stanislav Meduna]

On 3/14/2016 10:02 PM, Michael Hewitt wrote:

> Generally, we see packet delivery variations of less
> than 100 microseconds, which is fabulous.  Occasionally,
> we see a packet delivery delay in excess of 1000 microseconds.

Maybe this thread applies
http://www.spinics.net/lists/linux-rt-users/msg09887.html
(no straightforward solution there though).

-- 
                                 Stano

RE: Eliminating Packet Latency

[Dave Rolenc]

[-- Attachment #1: Type: text/plain, Size: 2751 bytes --]

Michael,

You can try this. The 3.10.0 code looked a little different, but this is the
gist of what I did on the older code. This is untested, so you are on the
hook for that part. I don't even know if this compiles, but it should:

--- linux-3.10.old/net/core/dev.c       2016-03-15 10:57:11.399668975 -0600
+++ linux-3.10/net/core/dev.c   2016-03-15 11:14:58.566734817 -0600
@@ -2675,28 +2675,6 @@
        if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
                kfree_skb(skb);
                rc = NET_XMIT_DROP;
-       } else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) &&
-                  qdisc_run_begin(q)) {
-               /*
-                * This is a work-conserving queue; there are no old skbs
-                * waiting to be sent out; and the qdisc is not running -
-                * xmit the skb directly.
-                */
-               if (!(dev->priv_flags & IFF_XMIT_DST_RELEASE))
-                       skb_dst_force(skb);
-
-               qdisc_bstats_update(q, skb);
-
-               if (sch_direct_xmit(skb, q, dev, txq, root_lock)) {
-                       if (unlikely(contended)) {
-                               spin_unlock(&q->busylock);
-                               contended = false;
-                       }
-                       __qdisc_run(q);
-               } else
-                       qdisc_run_end(q);
-
-               rc = NET_XMIT_SUCCESS;
        } else {
                skb_dst_force(skb);
                rc = q->enqueue(skb, q) & NET_XMIT_MASK;
@@ -2705,7 +2683,8 @@
                                spin_unlock(&q->busylock);
                                contended = false;
                        }
-                       __qdisc_run(q);
+            __netif_schedule(q);
+            qdisc_run_end(q);
                }
        }
        spin_unlock(root_lock);

Dave

-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org
[mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Stanislav Meduna
Sent: Tuesday, March 15, 2016 2:56 AM
To: Michael Hewitt; linux-rt-users@vger.kernel.org
Subject: Re: Eliminating Packet Latency

On 3/14/2016 10:02 PM, Michael Hewitt wrote:

> Generally, we see packet delivery variations of less than 100 
> microseconds, which is fabulous.  Occasionally, we see a packet 
> delivery delay in excess of 1000 microseconds.

Maybe this thread applies
http://www.spinics.net/lists/linux-rt-users/msg09887.html
(no straightforward solution there though).

-- 
                                 Stano

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5694 bytes --]

Re: Eliminating Packet Latency

[Giuliano Colla]

Il 14/03/2016 22:02, Michael Hewitt ha scritto:
> We are using a real time Linux kernel (3.10.0) in a network appliance in order to achieve extremely consistent packet delivery times.  Generally, we see packet delivery variations of less than 100 microseconds, which is fabulous.  Occasionally, we see a packet delivery delay in excess of 1000 microseconds.  We are hoping to eliminate these spikes, which occur perhaps 1-2 times in a 24 hour period.
>
> The machine configuration is as follows.  Thread IRQs are enabled, and we have elevated the priority of both the irq threads that service the specific network interface to 55.  We have also elevated the priority of the relevant user space thread to 49.  We are running on a 4 core Intel Xeon E3-1220v3 with an Intel NIC and the igb version 5.3.2 driver.  We disabled interrupt throttling in the Intel driver (rx-usecs = 0, tx-usecs = 0).  SE Linux is disabled, eliminating a huge packet latency spike during login.  We are running CentOS 7.1 tuned for network latency ("tuned-adm profile network-latency").  IRQ balancing is disabled.  BIOS CPU power management is set to maximum performance.
>

I'm using a 3.10 line kernel four our real-time applications (actually a 
3.10.10). We have experienced (quite unexpectedly) a better overall 
performance (reduced latency) with the on demand governor than with the 
performance governor. As the performance achieved was sufficient four 
our purposes in both cases we didn't investigate further. Maybe it's 
worth a try.

Giuliano

Re: Eliminating Packet Latency

[Sebastian Andrzej Siewior]

* eg Engleder Gerhard | 2016-03-16 08:34:10 [+0100]:

>> I found that sometimes when sending network packets, the kernel would hijack the
>> thread of the sender when the queue was empty. This would cause issues, since I
>> had no control over the priority of that thread, since it could be anything. My fix
>> was to never allow the kernel to send in the context of the sending userspace
>
>We had a similar problem and solved it with changed locking:

So the following patch should do the job. Can someone confirm?

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Date: Wed, 30 Mar 2016 13:36:29 +0200
Subject: [PATCH] net: dev: always take qdisc's busylock in __dev_xmit_skb()

The root-lock is dropped before dev_hard_start_xmit() is invoked and after
setting the __QDISC___STATE_RUNNING bit. If this task is now pushed away
by a task with a higher priority then the task with the higher priority
won't be able to submit packets to the NIC directly instead they will be
enqueued into the Qdisc. The NIC will remain idle until the task(s) with
higher priority leave the CPU and the task with lower priority gets back
and finishes the job.

If we take always the busylock we ensure that the RT task can boost the
low-prio task and submit the packet.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 net/core/dev.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/core/dev.c b/net/core/dev.c
index cc364be3587b..0e17592adbff 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2891,7 +2891,11 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 	 * This permits __QDISC___STATE_RUNNING owner to get the lock more
 	 * often and dequeue packets faster.
 	 */
+#ifdef CONFIG_PREEMPT_RT_FULL
+	contended = true;
+#else
 	contended = qdisc_is_running(q);
+#endif
 	if (unlikely(contended))
 		spin_lock(&q->busylock);
 
-- 
2.8.0.rc3

Sebastian