* [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
@ 2013-08-29 22:49 Eric Dumazet
2013-08-30 1:47 ` David Miller
2013-09-04 5:26 ` [PATCH v2 net-next] " Jason Wang
0 siblings, 2 replies; 18+ messages in thread
From: Eric Dumazet @ 2013-08-29 22:49 UTC (permalink / raw)
To: David Miller; +Cc: netdev, Yuchung Cheng, Neal Cardwell
From: Eric Dumazet <edumazet@google.com>
- Uses perfect flow match (not stochastic hash like SFQ/FQ_codel)
- Uses the new_flow/old_flow separation from FQ_codel
- New flows get an initial credit allowing IW10 without added delay.
- Special FIFO queue for high prio packets (no need for PRIO + FQ)
- Uses a hash table of RB trees to locate the flows at enqueue() time
- Smart on demand gc (at enqueue() time, RB tree lookup evicts old
unused flows)
- Dynamic memory allocations.
- Designed to allow millions of concurrent flows per Qdisc.
- Small memory footprint : ~8K per Qdisc, and 104 bytes per flow.
- Single high resolution timer for throttled flows (if any).
- One RB tree to link throttled flows.
- Ability to have a max rate per flow. We might add a socket option
to add per socket limitation.
Attempts have been made to add TCP pacing in TCP stack, but this
seems to add complex code to an already complex stack.
TCP pacing is welcomed for flows having idle times, as the cwnd
permits TCP stack to queue a possibly large number of packets.
This removes the 'slow start after idle' choice, hitting badly
large BDP flows, and applications delivering chunks of data
as video streams.
Nicely spaced packets :
Here interface is 10Gbit, but flow bottleneck is ~20Mbit
cwin is big, yet FQ avoids the typical bursts generated by TCP
(as in netperf TCP_RR -- -r 100000,100000)
15:01:23.545279 IP A > B: . 78193:81089(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.545394 IP B > A: . ack 81089 win 3668 <nop,nop,timestamp 11597985 1115>
15:01:23.546488 IP A > B: . 81089:83985(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.546565 IP B > A: . ack 83985 win 3668 <nop,nop,timestamp 11597986 1115>
15:01:23.547713 IP A > B: . 83985:86881(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.547778 IP B > A: . ack 86881 win 3668 <nop,nop,timestamp 11597987 1115>
15:01:23.548911 IP A > B: . 86881:89777(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.548949 IP B > A: . ack 89777 win 3668 <nop,nop,timestamp 11597988 1115>
15:01:23.550116 IP A > B: . 89777:92673(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.550182 IP B > A: . ack 92673 win 3668 <nop,nop,timestamp 11597989 1115>
15:01:23.551333 IP A > B: . 92673:95569(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.551406 IP B > A: . ack 95569 win 3668 <nop,nop,timestamp 11597991 1115>
15:01:23.552539 IP A > B: . 95569:98465(2896) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.552576 IP B > A: . ack 98465 win 3668 <nop,nop,timestamp 11597992 1115>
15:01:23.553756 IP A > B: . 98465:99913(1448) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.554138 IP A > B: P 99913:100001(88) ack 65248 win 3125 <nop,nop,timestamp 1115 11597805>
15:01:23.554204 IP B > A: . ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.554234 IP B > A: . 65248:68144(2896) ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.555620 IP B > A: . 68144:71040(2896) ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.557005 IP B > A: . 71040:73936(2896) ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.558390 IP B > A: . 73936:76832(2896) ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.559773 IP B > A: . 76832:79728(2896) ack 100001 win 3668 <nop,nop,timestamp 11597993 1115>
15:01:23.561158 IP B > A: . 79728:82624(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.562543 IP B > A: . 82624:85520(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.563928 IP B > A: . 85520:88416(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.565313 IP B > A: . 88416:91312(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.566698 IP B > A: . 91312:94208(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.568083 IP B > A: . 94208:97104(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.569467 IP B > A: . 97104:100000(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.570852 IP B > A: . 100000:102896(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.572237 IP B > A: . 102896:105792(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.573639 IP B > A: . 105792:108688(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.575024 IP B > A: . 108688:111584(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.576408 IP B > A: . 111584:114480(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
15:01:23.577793 IP B > A: . 114480:117376(2896) ack 100001 win 3668 <nop,nop,timestamp 11597994 1115>
TCP timestamps show that most packets from B were queued in the same ms
timeframe (TSval 1159799{3,4}), but FQ managed to send them right
in time to avoid a big burst.
In slow start or steady state, very few packets are throttled [1]
FQ gets a bunch of tunables as :
limit : max number of packets on whole Qdisc (default 10000)
flow_limit : max number of packets per flow (default 100)
quantum : the credit per RR round (default is 2 MTU)
initial_quantum : initial credit for new flows (default is 10 MTU)
maxrate : max per flow rate (default : unlimited)
buckets : number of RB trees (default : 1024) in hash table.
(consumes 8 bytes per bucket)
[no]pacing : disable/enable pacing (default is enable)
All of them can be changed on a live qdisc.
$ tc qd add dev eth0 root fq help
Usage: ... fq [ limit PACKETS ] [ flow_limit PACKETS ]
[ quantum BYTES ] [ initial_quantum BYTES ]
[ maxrate RATE ] [ buckets NUMBER ]
[ [no]pacing ]
$ tc -s -d qd
qdisc fq 8002: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 256 quantum 3028 initial_quantum 15140
Sent 216532416 bytes 148395 pkt (dropped 0, overlimits 0 requeues 14)
backlog 0b 0p requeues 14
511 flows, 511 inactive, 0 throttled
110 gc, 0 highprio, 0 retrans, 1143 throttled, 0 flows_plimit
[1] Except if initial srtt is overestimated, as if using
cached srtt in tcp metrics. We'll provide a fix for this issue.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
---
v2: added initial_quantum support
include/uapi/linux/pkt_sched.h | 41 +
net/sched/Kconfig | 14
net/sched/Makefile | 1
net/sched/sch_fq.c | 792 +++++++++++++++++++++++++++++++
4 files changed, 848 insertions(+)
diff --git a/include/uapi/linux/pkt_sched.h b/include/uapi/linux/pkt_sched.h
index 09d62b92..9b82913 100644
--- a/include/uapi/linux/pkt_sched.h
+++ b/include/uapi/linux/pkt_sched.h
@@ -744,4 +744,45 @@ struct tc_fq_codel_xstats {
};
};
+/* FQ */
+
+enum {
+ TCA_FQ_UNSPEC,
+
+ TCA_FQ_PLIMIT, /* limit of total number of packets in queue */
+
+ TCA_FQ_FLOW_PLIMIT, /* limit of packets per flow */
+
+ TCA_FQ_QUANTUM, /* RR quantum */
+
+ TCA_FQ_INITIAL_QUANTUM, /* RR quantum for new flow */
+
+ TCA_FQ_RATE_ENABLE, /* enable/disable rate limiting */
+
+ TCA_FQ_FLOW_DEFAULT_RATE,/* for sockets with unspecified sk_rate,
+ * use the following rate
+ */
+
+ TCA_FQ_FLOW_MAX_RATE, /* per flow max rate */
+
+ TCA_FQ_BUCKETS_LOG, /* log2(number of buckets) */
+ __TCA_FQ_MAX
+};
+
+#define TCA_FQ_MAX (__TCA_FQ_MAX - 1)
+
+struct tc_fq_qd_stats {
+ __u64 gc_flows;
+ __u64 highprio_packets;
+ __u64 tcp_retrans;
+ __u64 throttled;
+ __u64 flows_plimit;
+ __u64 pkts_too_long;
+ __u64 allocation_errors;
+ __s64 time_next_delayed_flow;
+ __u32 flows;
+ __u32 inactive_flows;
+ __u32 throttled_flows;
+ __u32 pad;
+};
#endif
diff --git a/net/sched/Kconfig b/net/sched/Kconfig
index 235e01a..c03a32a 100644
--- a/net/sched/Kconfig
+++ b/net/sched/Kconfig
@@ -272,6 +272,20 @@ config NET_SCH_FQ_CODEL
If unsure, say N.
+config NET_SCH_FQ
+ tristate "Fair Queue"
+ help
+ Say Y here if you want to use the FQ packet scheduling algorithm.
+
+ FQ does flow separation, and is able to respect pacing requirements
+ set by TCP stack into sk->sk_pacing_rate (for localy generated
+ traffic)
+
+ To compile this driver as a module, choose M here: the module
+ will be called sch_fq.
+
+ If unsure, say N.
+
config NET_SCH_INGRESS
tristate "Ingress Qdisc"
depends on NET_CLS_ACT
diff --git a/net/sched/Makefile b/net/sched/Makefile
index 978cbf0..e5f9abe 100644
--- a/net/sched/Makefile
+++ b/net/sched/Makefile
@@ -39,6 +39,7 @@ obj-$(CONFIG_NET_SCH_CHOKE) += sch_choke.o
obj-$(CONFIG_NET_SCH_QFQ) += sch_qfq.o
obj-$(CONFIG_NET_SCH_CODEL) += sch_codel.o
obj-$(CONFIG_NET_SCH_FQ_CODEL) += sch_fq_codel.o
+obj-$(CONFIG_NET_SCH_FQ) += sch_fq.o
obj-$(CONFIG_NET_CLS_U32) += cls_u32.o
obj-$(CONFIG_NET_CLS_ROUTE4) += cls_route.o
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
new file mode 100644
index 0000000..d46d0ec
--- /dev/null
+++ b/net/sched/sch_fq.c
@@ -0,0 +1,792 @@
+/*
+ * net/sched/sch_fq.c Fair Queue Packet Scheduler (per flow pacing)
+ *
+ * Copyright (C) 2013 Eric Dumazet <edumazet@google.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ *
+ * Meant to be mostly used for localy generated traffic :
+ * Fast classification depends on skb->sk being set before reaching us.
+ * If not, (router workload), we use rxhash as fallback, with 32 bits wide hash.
+ * All packets belonging to a socket are considered as a 'flow'.
+ *
+ * Flows are dynamically allocated and stored in a hash table of RB trees
+ * They are also part of one Round Robin 'queues' (new or old flows)
+ *
+ * Burst avoidance (aka pacing) capability :
+ *
+ * Transport (eg TCP) can set in sk->sk_pacing_rate a rate, enqueue a
+ * bunch of packets, and this packet scheduler adds delay between
+ * packets to respect rate limitation.
+ *
+ * enqueue() :
+ * - lookup one RB tree (out of 1024 or more) to find the flow.
+ * If non existent flow, create it, add it to the tree.
+ * Add skb to the per flow list of skb (fifo).
+ * - Use a special fifo for high prio packets
+ *
+ * dequeue() : serves flows in Round Robin
+ * Note : When a flow becomes empty, we do not immediately remove it from
+ * rb trees, for performance reasons (its expected to send additional packets,
+ * or SLAB cache will reuse socket for another flow)
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/jiffies.h>
+#include <linux/string.h>
+#include <linux/in.h>
+#include <linux/errno.h>
+#include <linux/init.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/rbtree.h>
+#include <linux/hash.h>
+#include <net/netlink.h>
+#include <net/pkt_sched.h>
+#include <net/sock.h>
+#include <net/tcp_states.h>
+
+/*
+ * Per flow structure, dynamically allocated
+ */
+struct fq_flow {
+ struct sk_buff *head; /* list of skbs for this flow : first skb */
+ union {
+ struct sk_buff *tail; /* last skb in the list */
+ unsigned long age; /* jiffies when flow was emptied, for gc */
+ };
+ struct rb_node fq_node; /* anchor in fq_root[] trees */
+ struct sock *sk;
+ int qlen; /* number of packets in flow queue */
+ int credit;
+ u32 socket_hash; /* sk_hash */
+ struct fq_flow *next; /* next pointer in RR lists, or &detached */
+
+ struct rb_node rate_node; /* anchor in q->delayed tree */
+ u64 time_next_packet;
+};
+
+struct fq_flow_head {
+ struct fq_flow *first;
+ struct fq_flow *last;
+};
+
+struct fq_sched_data {
+ struct fq_flow_head new_flows;
+
+ struct fq_flow_head old_flows;
+
+ struct rb_root delayed; /* for rate limited flows */
+ u64 time_next_delayed_flow;
+
+ struct fq_flow internal; /* for non classified or high prio packets */
+ u32 quantum;
+ u32 initial_quantum;
+ u32 flow_default_rate;/* rate per flow : bytes per second */
+ u32 flow_max_rate; /* optional max rate per flow */
+ u32 flow_plimit; /* max packets per flow */
+ struct rb_root *fq_root;
+ u8 rate_enable;
+ u8 fq_trees_log;
+
+ u32 flows;
+ u32 inactive_flows;
+ u32 throttled_flows;
+
+ u64 stat_gc_flows;
+ u64 stat_internal_packets;
+ u64 stat_tcp_retrans;
+ u64 stat_throttled;
+ u64 stat_flows_plimit;
+ u64 stat_pkts_too_long;
+ u64 stat_allocation_errors;
+ struct qdisc_watchdog watchdog;
+};
+
+/* special value to mark a detached flow (not on old/new list) */
+static struct fq_flow detached, throttled;
+
+static void fq_flow_set_detached(struct fq_flow *f)
+{
+ f->next = &detached;
+}
+
+static bool fq_flow_is_detached(const struct fq_flow *f)
+{
+ return f->next == &detached;
+}
+
+static void fq_flow_set_throttled(struct fq_sched_data *q, struct fq_flow *f)
+{
+ struct rb_node **p = &q->delayed.rb_node, *parent = NULL;
+
+ while (*p) {
+ struct fq_flow *aux;
+
+ parent = *p;
+ aux = container_of(parent, struct fq_flow, rate_node);
+ if (f->time_next_packet >= aux->time_next_packet)
+ p = &parent->rb_right;
+ else
+ p = &parent->rb_left;
+ }
+ rb_link_node(&f->rate_node, parent, p);
+ rb_insert_color(&f->rate_node, &q->delayed);
+ q->throttled_flows++;
+ q->stat_throttled++;
+
+ f->next = &throttled;
+ if (q->time_next_delayed_flow > f->time_next_packet)
+ q->time_next_delayed_flow = f->time_next_packet;
+}
+
+
+static struct kmem_cache *fq_flow_cachep __read_mostly;
+
+static void fq_flow_add_tail(struct fq_flow_head *head, struct fq_flow *flow)
+{
+ if (head->first)
+ head->last->next = flow;
+ else
+ head->first = flow;
+ head->last = flow;
+ flow->next = NULL;
+}
+
+/* limit number of collected flows per round */
+#define FQ_GC_MAX 8
+#define FQ_GC_AGE (3*HZ)
+
+static bool fq_gc_candidate(const struct fq_flow *f)
+{
+ return fq_flow_is_detached(f) &&
+ time_after(jiffies, f->age + FQ_GC_AGE);
+}
+
+static void fq_gc(struct fq_sched_data *q,
+ struct rb_root *root,
+ struct sock *sk)
+{
+ struct fq_flow *f, *tofree[FQ_GC_MAX];
+ struct rb_node **p, *parent;
+ int fcnt = 0;
+
+ p = &root->rb_node;
+ parent = NULL;
+ while (*p) {
+ parent = *p;
+
+ f = container_of(parent, struct fq_flow, fq_node);
+ if (f->sk == sk)
+ break;
+
+ if (fq_gc_candidate(f)) {
+ tofree[fcnt++] = f;
+ if (fcnt == FQ_GC_MAX)
+ break;
+ }
+
+ if (f->sk > sk)
+ p = &parent->rb_right;
+ else
+ p = &parent->rb_left;
+ }
+
+ q->flows -= fcnt;
+ q->inactive_flows -= fcnt;
+ q->stat_gc_flows += fcnt;
+ while (fcnt) {
+ struct fq_flow *f = tofree[--fcnt];
+
+ rb_erase(&f->fq_node, root);
+ kmem_cache_free(fq_flow_cachep, f);
+ }
+}
+
+static const u8 prio2band[TC_PRIO_MAX + 1] = {
+ 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
+};
+
+static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
+{
+ struct rb_node **p, *parent;
+ struct sock *sk = skb->sk;
+ struct rb_root *root;
+ struct fq_flow *f;
+ int band;
+
+ /* warning: no starvation prevention... */
+ band = prio2band[skb->priority & TC_PRIO_MAX];
+ if (unlikely(band == 0))
+ return &q->internal;
+
+ if (unlikely(!sk)) {
+ /* By forcing low order bit to 1, we make sure to not
+ * collide with a local flow (socket pointers are word aligned)
+ */
+ sk = (struct sock *)(skb_get_rxhash(skb) | 1L);
+ }
+
+ root = &q->fq_root[hash_32((u32)(long)sk, q->fq_trees_log)];
+
+ if (q->flows >= (2U << q->fq_trees_log) &&
+ q->inactive_flows > q->flows/2)
+ fq_gc(q, root, sk);
+
+ p = &root->rb_node;
+ parent = NULL;
+ while (*p) {
+ parent = *p;
+
+ f = container_of(parent, struct fq_flow, fq_node);
+ if (f->sk == sk) {
+ /* socket might have been reallocated, so check
+ * if its sk_hash is the same.
+ * It not, we need to refill credit with
+ * initial quantum
+ */
+ if (unlikely(skb->sk &&
+ f->socket_hash != sk->sk_hash)) {
+ f->credit = q->initial_quantum;
+ f->socket_hash = sk->sk_hash;
+ }
+ return f;
+ }
+ if (f->sk > sk)
+ p = &parent->rb_right;
+ else
+ p = &parent->rb_left;
+ }
+
+ f = kmem_cache_zalloc(fq_flow_cachep, GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!f)) {
+ q->stat_allocation_errors++;
+ return &q->internal;
+ }
+ fq_flow_set_detached(f);
+ f->sk = sk;
+ if (skb->sk)
+ f->socket_hash = sk->sk_hash;
+ f->credit = q->initial_quantum;
+
+ rb_link_node(&f->fq_node, parent, p);
+ rb_insert_color(&f->fq_node, root);
+
+ q->flows++;
+ q->inactive_flows++;
+ return f;
+}
+
+
+/* remove one skb from head of flow queue */
+static struct sk_buff *fq_dequeue_head(struct fq_flow *flow)
+{
+ struct sk_buff *skb = flow->head;
+
+ if (skb) {
+ flow->head = skb->next;
+ skb->next = NULL;
+ flow->qlen--;
+ }
+ return skb;
+}
+
+/* We might add in the future detection of retransmits
+ * For the time being, just return false
+ */
+static bool skb_is_retransmit(struct sk_buff *skb)
+{
+ return false;
+}
+
+/* add skb to flow queue
+ * flow queue is a linked list, kind of FIFO, except for TCP retransmits
+ * We special case tcp retransmits to be transmitted before other packets.
+ * We rely on fact that TCP retransmits are unlikely, so we do not waste
+ * a separate queue or a pointer.
+ * head-> [retrans pkt 1]
+ * [retrans pkt 2]
+ * [ normal pkt 1]
+ * [ normal pkt 2]
+ * [ normal pkt 3]
+ * tail-> [ normal pkt 4]
+ */
+static void flow_queue_add(struct fq_flow *flow, struct sk_buff *skb)
+{
+ struct sk_buff *prev, *head = flow->head;
+
+ skb->next = NULL;
+ if (!head) {
+ flow->head = skb;
+ flow->tail = skb;
+ return;
+ }
+ if (likely(!skb_is_retransmit(skb))) {
+ flow->tail->next = skb;
+ flow->tail = skb;
+ return;
+ }
+
+ /* This skb is a tcp retransmit,
+ * find the last retrans packet in the queue
+ */
+ prev = NULL;
+ while (skb_is_retransmit(head)) {
+ prev = head;
+ head = head->next;
+ if (!head)
+ break;
+ }
+ if (!prev) { /* no rtx packet in queue, become the new head */
+ skb->next = flow->head;
+ flow->head = skb;
+ } else {
+ if (prev == flow->tail)
+ flow->tail = skb;
+ else
+ skb->next = prev->next;
+ prev->next = skb;
+ }
+}
+
+static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ struct fq_flow *f;
+
+ if (unlikely(sch->q.qlen >= sch->limit))
+ return qdisc_drop(skb, sch);
+
+ f = fq_classify(skb, q);
+ if (unlikely(f->qlen >= q->flow_plimit && f != &q->internal)) {
+ q->stat_flows_plimit++;
+ return qdisc_drop(skb, sch);
+ }
+
+ f->qlen++;
+ flow_queue_add(f, skb);
+ if (skb_is_retransmit(skb))
+ q->stat_tcp_retrans++;
+ sch->qstats.backlog += qdisc_pkt_len(skb);
+ if (fq_flow_is_detached(f)) {
+ fq_flow_add_tail(&q->new_flows, f);
+ if (q->quantum > f->credit)
+ f->credit = q->quantum;
+ q->inactive_flows--;
+ qdisc_unthrottled(sch);
+ }
+ if (unlikely(f == &q->internal)) {
+ q->stat_internal_packets++;
+ qdisc_unthrottled(sch);
+ }
+ sch->q.qlen++;
+
+ return NET_XMIT_SUCCESS;
+}
+
+static void fq_check_throttled(struct fq_sched_data *q, u64 now)
+{
+ struct rb_node *p;
+
+ if (q->time_next_delayed_flow > now)
+ return;
+
+ q->time_next_delayed_flow = ~0ULL;
+ while ((p = rb_first(&q->delayed)) != NULL) {
+ struct fq_flow *f = container_of(p, struct fq_flow, rate_node);
+
+ if (f->time_next_packet > now) {
+ q->time_next_delayed_flow = f->time_next_packet;
+ break;
+ }
+ rb_erase(p, &q->delayed);
+ q->throttled_flows--;
+ fq_flow_add_tail(&q->old_flows, f);
+ }
+}
+
+static struct sk_buff *fq_dequeue(struct Qdisc *sch)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ u64 now = ktime_to_ns(ktime_get());
+ struct fq_flow_head *head;
+ struct sk_buff *skb;
+ struct fq_flow *f;
+
+ skb = fq_dequeue_head(&q->internal);
+ if (skb)
+ goto out;
+ fq_check_throttled(q, now);
+begin:
+ head = &q->new_flows;
+ if (!head->first) {
+ head = &q->old_flows;
+ if (!head->first) {
+ if (q->time_next_delayed_flow != ~0ULL)
+ qdisc_watchdog_schedule_ns(&q->watchdog,
+ q->time_next_delayed_flow);
+ return NULL;
+ }
+ }
+ f = head->first;
+
+ if (f->credit <= 0) {
+ f->credit += q->quantum;
+ head->first = f->next;
+ fq_flow_add_tail(&q->old_flows, f);
+ goto begin;
+ }
+
+ if (unlikely(f->head && now < f->time_next_packet)) {
+ head->first = f->next;
+ fq_flow_set_throttled(q, f);
+ goto begin;
+ }
+
+ skb = fq_dequeue_head(f);
+ if (!skb) {
+ head->first = f->next;
+ /* force a pass through old_flows to prevent starvation */
+ if ((head == &q->new_flows) && q->old_flows.first) {
+ fq_flow_add_tail(&q->old_flows, f);
+ } else {
+ fq_flow_set_detached(f);
+ f->age = jiffies;
+ q->inactive_flows++;
+ }
+ goto begin;
+ }
+ f->time_next_packet = now;
+ f->credit -= qdisc_pkt_len(skb);
+
+ if (f->credit <= 0 &&
+ q->rate_enable &&
+ skb->sk && skb->sk->sk_state != TCP_TIME_WAIT) {
+ u32 rate = skb->sk->sk_pacing_rate ?: q->flow_default_rate;
+
+ rate = min(rate, q->flow_max_rate);
+ if (rate) {
+ u64 len = (u64)qdisc_pkt_len(skb) * NSEC_PER_SEC;
+
+ do_div(len, rate);
+ /* Since socket rate can change later,
+ * clamp the delay to 125 ms.
+ * TODO: maybe segment the too big skb, as in commit
+ * e43ac79a4bc ("sch_tbf: segment too big GSO packets")
+ */
+ if (unlikely(len > 125 * NSEC_PER_MSEC)) {
+ len = 125 * NSEC_PER_MSEC;
+ q->stat_pkts_too_long++;
+ }
+
+ f->time_next_packet = now + len;
+ }
+ }
+out:
+ prefetch(&skb->end);
+ sch->qstats.backlog -= qdisc_pkt_len(skb);
+ qdisc_bstats_update(sch, skb);
+ sch->q.qlen--;
+ qdisc_unthrottled(sch);
+ return skb;
+}
+
+static void fq_reset(struct Qdisc *sch)
+{
+ struct sk_buff *skb;
+
+ while ((skb = fq_dequeue(sch)) != NULL)
+ kfree_skb(skb);
+}
+
+static void fq_rehash(struct fq_sched_data *q,
+ struct rb_root *old_array, u32 old_log,
+ struct rb_root *new_array, u32 new_log)
+{
+ struct rb_node *op, **np, *parent;
+ struct rb_root *oroot, *nroot;
+ struct fq_flow *of, *nf;
+ int fcnt = 0;
+ u32 idx;
+
+ for (idx = 0; idx < (1U << old_log); idx++) {
+ oroot = &old_array[idx];
+ while ((op = rb_first(oroot)) != NULL) {
+ rb_erase(op, oroot);
+ of = container_of(op, struct fq_flow, fq_node);
+ if (fq_gc_candidate(of)) {
+ fcnt++;
+ kmem_cache_free(fq_flow_cachep, of);
+ continue;
+ }
+ nroot = &new_array[hash_32((u32)(long)of->sk, new_log)];
+
+ np = &nroot->rb_node;
+ parent = NULL;
+ while (*np) {
+ parent = *np;
+
+ nf = container_of(parent, struct fq_flow, fq_node);
+ BUG_ON(nf->sk == of->sk);
+
+ if (nf->sk > of->sk)
+ np = &parent->rb_right;
+ else
+ np = &parent->rb_left;
+ }
+
+ rb_link_node(&of->fq_node, parent, np);
+ rb_insert_color(&of->fq_node, nroot);
+ }
+ }
+ q->flows -= fcnt;
+ q->inactive_flows -= fcnt;
+ q->stat_gc_flows += fcnt;
+}
+
+static int fq_resize(struct fq_sched_data *q, u32 log)
+{
+ struct rb_root *array;
+ u32 idx;
+
+ if (q->fq_root && log == q->fq_trees_log)
+ return 0;
+
+ array = kmalloc(sizeof(struct rb_root) << log, GFP_KERNEL);
+ if (!array)
+ return -ENOMEM;
+
+ for (idx = 0; idx < (1U << log); idx++)
+ array[idx] = RB_ROOT;
+
+ if (q->fq_root) {
+ fq_rehash(q, q->fq_root, q->fq_trees_log, array, log);
+ kfree(q->fq_root);
+ }
+ q->fq_root = array;
+ q->fq_trees_log = log;
+
+ return 0;
+}
+
+static const struct nla_policy fq_policy[TCA_FQ_MAX + 1] = {
+ [TCA_FQ_PLIMIT] = { .type = NLA_U32 },
+ [TCA_FQ_FLOW_PLIMIT] = { .type = NLA_U32 },
+ [TCA_FQ_QUANTUM] = { .type = NLA_U32 },
+ [TCA_FQ_INITIAL_QUANTUM] = { .type = NLA_U32 },
+ [TCA_FQ_RATE_ENABLE] = { .type = NLA_U32 },
+ [TCA_FQ_FLOW_DEFAULT_RATE] = { .type = NLA_U32 },
+ [TCA_FQ_FLOW_MAX_RATE] = { .type = NLA_U32 },
+ [TCA_FQ_BUCKETS_LOG] = { .type = NLA_U32 },
+};
+
+static int fq_change(struct Qdisc *sch, struct nlattr *opt)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ struct nlattr *tb[TCA_FQ_MAX + 1];
+ int err, drop_count = 0;
+ u32 fq_log;
+
+ if (!opt)
+ return -EINVAL;
+
+ err = nla_parse_nested(tb, TCA_FQ_MAX, opt, fq_policy);
+ if (err < 0)
+ return err;
+
+ sch_tree_lock(sch);
+
+ fq_log = q->fq_trees_log;
+
+ if (tb[TCA_FQ_BUCKETS_LOG]) {
+ u32 nval = nla_get_u32(tb[TCA_FQ_BUCKETS_LOG]);
+
+ if (nval >= 1 && nval <= ilog2(256*1024))
+ fq_log = nval;
+ else
+ err = -EINVAL;
+ }
+ if (tb[TCA_FQ_PLIMIT])
+ sch->limit = nla_get_u32(tb[TCA_FQ_PLIMIT]);
+
+ if (tb[TCA_FQ_FLOW_PLIMIT])
+ q->flow_plimit = nla_get_u32(tb[TCA_FQ_FLOW_PLIMIT]);
+
+ if (tb[TCA_FQ_QUANTUM])
+ q->quantum = nla_get_u32(tb[TCA_FQ_QUANTUM]);
+
+ if (tb[TCA_FQ_INITIAL_QUANTUM])
+ q->quantum = nla_get_u32(tb[TCA_FQ_INITIAL_QUANTUM]);
+
+ if (tb[TCA_FQ_FLOW_DEFAULT_RATE])
+ q->flow_default_rate = nla_get_u32(tb[TCA_FQ_FLOW_DEFAULT_RATE]);
+
+ if (tb[TCA_FQ_FLOW_MAX_RATE])
+ q->flow_max_rate = nla_get_u32(tb[TCA_FQ_FLOW_MAX_RATE]);
+
+ if (tb[TCA_FQ_RATE_ENABLE]) {
+ u32 enable = nla_get_u32(tb[TCA_FQ_RATE_ENABLE]);
+
+ if (enable <= 1)
+ q->rate_enable = enable;
+ else
+ err = -EINVAL;
+ }
+
+ if (!err)
+ err = fq_resize(q, fq_log);
+
+ while (sch->q.qlen > sch->limit) {
+ struct sk_buff *skb = fq_dequeue(sch);
+
+ kfree_skb(skb);
+ drop_count++;
+ }
+ qdisc_tree_decrease_qlen(sch, drop_count);
+
+ sch_tree_unlock(sch);
+ return err;
+}
+
+static void fq_destroy(struct Qdisc *sch)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ struct rb_root *root;
+ struct rb_node *p;
+ unsigned int idx;
+
+ if (q->fq_root) {
+ for (idx = 0; idx < (1U << q->fq_trees_log); idx++) {
+ root = &q->fq_root[idx];
+ while ((p = rb_first(root)) != NULL) {
+ rb_erase(p, root);
+ kmem_cache_free(fq_flow_cachep,
+ container_of(p, struct fq_flow, fq_node));
+ }
+ }
+ kfree(q->fq_root);
+ }
+ qdisc_watchdog_cancel(&q->watchdog);
+}
+
+static int fq_init(struct Qdisc *sch, struct nlattr *opt)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ int err;
+
+ sch->limit = 10000;
+ q->flow_plimit = 100;
+ q->quantum = 2 * psched_mtu(qdisc_dev(sch));
+ q->initial_quantum = 10 * psched_mtu(qdisc_dev(sch));
+ q->flow_default_rate = 0;
+ q->flow_max_rate = ~0U;
+ q->rate_enable = 1;
+ q->new_flows.first = NULL;
+ q->old_flows.first = NULL;
+ q->delayed = RB_ROOT;
+ q->fq_root = NULL;
+ q->fq_trees_log = ilog2(1024);
+ qdisc_watchdog_init(&q->watchdog, sch);
+
+ if (opt)
+ err = fq_change(sch, opt);
+ else
+ err = fq_resize(q, q->fq_trees_log);
+
+ return err;
+}
+
+static int fq_dump(struct Qdisc *sch, struct sk_buff *skb)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ struct nlattr *opts;
+
+ opts = nla_nest_start(skb, TCA_OPTIONS);
+ if (opts == NULL)
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, TCA_FQ_PLIMIT, sch->limit) ||
+ nla_put_u32(skb, TCA_FQ_FLOW_PLIMIT, q->flow_plimit) ||
+ nla_put_u32(skb, TCA_FQ_QUANTUM, q->quantum) ||
+ nla_put_u32(skb, TCA_FQ_INITIAL_QUANTUM, q->initial_quantum) ||
+ nla_put_u32(skb, TCA_FQ_RATE_ENABLE, q->rate_enable) ||
+ nla_put_u32(skb, TCA_FQ_FLOW_DEFAULT_RATE, q->flow_default_rate) ||
+ nla_put_u32(skb, TCA_FQ_FLOW_MAX_RATE, q->flow_max_rate) ||
+ nla_put_u32(skb, TCA_FQ_BUCKETS_LOG, q->fq_trees_log))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, opts);
+ return skb->len;
+
+nla_put_failure:
+ return -1;
+}
+
+static int fq_dump_stats(struct Qdisc *sch, struct gnet_dump *d)
+{
+ struct fq_sched_data *q = qdisc_priv(sch);
+ u64 now = ktime_to_ns(ktime_get());
+ struct tc_fq_qd_stats st = {
+ .gc_flows = q->stat_gc_flows,
+ .highprio_packets = q->stat_internal_packets,
+ .tcp_retrans = q->stat_tcp_retrans,
+ .throttled = q->stat_throttled,
+ .flows_plimit = q->stat_flows_plimit,
+ .pkts_too_long = q->stat_pkts_too_long,
+ .allocation_errors = q->stat_allocation_errors,
+ .flows = q->flows,
+ .inactive_flows = q->inactive_flows,
+ .throttled_flows = q->throttled_flows,
+ .time_next_delayed_flow = q->time_next_delayed_flow - now,
+ };
+
+ return gnet_stats_copy_app(d, &st, sizeof(st));
+}
+
+static struct Qdisc_ops fq_qdisc_ops __read_mostly = {
+ .id = "fq",
+ .priv_size = sizeof(struct fq_sched_data),
+
+ .enqueue = fq_enqueue,
+ .dequeue = fq_dequeue,
+ .peek = qdisc_peek_dequeued,
+ .init = fq_init,
+ .reset = fq_reset,
+ .destroy = fq_destroy,
+ .change = fq_change,
+ .dump = fq_dump,
+ .dump_stats = fq_dump_stats,
+ .owner = THIS_MODULE,
+};
+
+static int __init fq_module_init(void)
+{
+ int ret;
+
+ fq_flow_cachep = kmem_cache_create("fq_flow_cache",
+ sizeof(struct fq_flow),
+ 0, 0, NULL);
+ if (!fq_flow_cachep)
+ return -ENOMEM;
+
+ ret = register_qdisc(&fq_qdisc_ops);
+ if (ret)
+ kmem_cache_destroy(fq_flow_cachep);
+ return ret;
+}
+
+static void __exit fq_module_exit(void)
+{
+ unregister_qdisc(&fq_qdisc_ops);
+ kmem_cache_destroy(fq_flow_cachep);
+}
+
+module_init(fq_module_init)
+module_exit(fq_module_exit)
+MODULE_AUTHOR("Eric Dumazet");
+MODULE_LICENSE("GPL");
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-08-29 22:49 [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler Eric Dumazet
@ 2013-08-30 1:47 ` David Miller
2013-08-30 2:30 ` [PATCH iproute2] " Eric Dumazet
2013-09-04 5:26 ` [PATCH v2 net-next] " Jason Wang
1 sibling, 1 reply; 18+ messages in thread
From: David Miller @ 2013-08-30 1:47 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev, ycheng, ncardwell
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 29 Aug 2013 15:49:55 -0700
> From: Eric Dumazet <edumazet@google.com>
>
> - Uses perfect flow match (not stochastic hash like SFQ/FQ_codel)
> - Uses the new_flow/old_flow separation from FQ_codel
> - New flows get an initial credit allowing IW10 without added delay.
> - Special FIFO queue for high prio packets (no need for PRIO + FQ)
> - Uses a hash table of RB trees to locate the flows at enqueue() time
> - Smart on demand gc (at enqueue() time, RB tree lookup evicts old
> unused flows)
> - Dynamic memory allocations.
> - Designed to allow millions of concurrent flows per Qdisc.
> - Small memory footprint : ~8K per Qdisc, and 104 bytes per flow.
> - Single high resolution timer for throttled flows (if any).
> - One RB tree to link throttled flows.
> - Ability to have a max rate per flow. We might add a socket option
> to add per socket limitation.
>
> Attempts have been made to add TCP pacing in TCP stack, but this
> seems to add complex code to an already complex stack.
>
> TCP pacing is welcomed for flows having idle times, as the cwnd
> permits TCP stack to queue a possibly large number of packets.
>
> This removes the 'slow start after idle' choice, hitting badly
> large BDP flows, and applications delivering chunks of data
> as video streams.
>
> Nicely spaced packets :
> Here interface is 10Gbit, but flow bottleneck is ~20Mbit
>
> cwin is big, yet FQ avoids the typical bursts generated by TCP
> (as in netperf TCP_RR -- -r 100000,100000)
...
> Signed-off-by: Eric Dumazet <edumazet@google.com>
Applied, thanks Eric!
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH iproute2] pkt_sched: fq: Fair Queue packet scheduler
2013-08-30 1:47 ` David Miller
@ 2013-08-30 2:30 ` Eric Dumazet
2013-09-03 15:49 ` Stephen Hemminger
0 siblings, 1 reply; 18+ messages in thread
From: Eric Dumazet @ 2013-08-30 2:30 UTC (permalink / raw)
To: David Miller, Stephen Hemminger; +Cc: netdev, ycheng, ncardwell
From: Eric Dumazet <edumazet@google.com>
Support for FQ packet scheduler
$ tc qd add dev eth0 root fq help
Usage: ... fq [ limit PACKETS ] [ flow_limit PACKETS ]
[ quantum BYTES ] [ initial_quantum BYTES ]
[ maxrate RATE ] [ buckets NUMBER ]
[ [no]pacing ]
$ tc -s -d qd
qdisc fq 8002: dev eth0 root refcnt 32 limit 10000p flow_limit 100p
buckets 256 quantum 3028 initial_quantum 15140
Sent 216532416 bytes 148395 pkt (dropped 0, overlimits 0 requeues 14)
backlog 0b 0p requeues 14
511 flows (511 inactive, 0 throttled)
110 gc, 0 highprio, 0 retrans, 1143 throttled, 0 flows_plimit
limit : max number of packets on whole Qdisc (default 10000)
flow_limit : max number of packets per flow (default 100)
quantum : the max deficit per RR round (default is 2 MTU)
initial_quantum : initial credit for new flows (default is 10 MTU)
maxrate : max per flow rate (default : unlimited)
buckets : number of RB trees (default : 1024) in hash table.
(consumes 8 bytes per bucket)
[no]pacing : disable/enable pacing (default is enable)
Usage :
tc qdisc add dev $ETH root fq
tc qdisc del dev $ETH root 2>/dev/null
tc qdisc add dev $ETH root handle 1: mq
for i in `seq 1 4`
do
tc qdisc add dev $ETH parent 1:$i est 1sec 4sec fq
done
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/linux/pkt_sched.h | 51 ++++++
tc/Makefile | 1
tc/q_fq.c | 279 ++++++++++++++++++++++++++++++++++++
3 files changed, 330 insertions(+), 1 deletion(-)
diff --git a/include/linux/pkt_sched.h b/include/linux/pkt_sched.h
index dbd71b0..9b82913 100644
--- a/include/linux/pkt_sched.h
+++ b/include/linux/pkt_sched.h
@@ -73,9 +73,17 @@ struct tc_estimator {
#define TC_H_ROOT (0xFFFFFFFFU)
#define TC_H_INGRESS (0xFFFFFFF1U)
+/* Need to corrospond to iproute2 tc/tc_core.h "enum link_layer" */
+enum tc_link_layer {
+ TC_LINKLAYER_UNAWARE, /* Indicate unaware old iproute2 util */
+ TC_LINKLAYER_ETHERNET,
+ TC_LINKLAYER_ATM,
+};
+#define TC_LINKLAYER_MASK 0x0F /* limit use to lower 4 bits */
+
struct tc_ratespec {
unsigned char cell_log;
- unsigned char __reserved;
+ __u8 linklayer; /* lower 4 bits */
unsigned short overhead;
short cell_align;
unsigned short mpu;
@@ -736,4 +744,45 @@ struct tc_fq_codel_xstats {
};
};
+/* FQ */
+
+enum {
+ TCA_FQ_UNSPEC,
+
+ TCA_FQ_PLIMIT, /* limit of total number of packets in queue */
+
+ TCA_FQ_FLOW_PLIMIT, /* limit of packets per flow */
+
+ TCA_FQ_QUANTUM, /* RR quantum */
+
+ TCA_FQ_INITIAL_QUANTUM, /* RR quantum for new flow */
+
+ TCA_FQ_RATE_ENABLE, /* enable/disable rate limiting */
+
+ TCA_FQ_FLOW_DEFAULT_RATE,/* for sockets with unspecified sk_rate,
+ * use the following rate
+ */
+
+ TCA_FQ_FLOW_MAX_RATE, /* per flow max rate */
+
+ TCA_FQ_BUCKETS_LOG, /* log2(number of buckets) */
+ __TCA_FQ_MAX
+};
+
+#define TCA_FQ_MAX (__TCA_FQ_MAX - 1)
+
+struct tc_fq_qd_stats {
+ __u64 gc_flows;
+ __u64 highprio_packets;
+ __u64 tcp_retrans;
+ __u64 throttled;
+ __u64 flows_plimit;
+ __u64 pkts_too_long;
+ __u64 allocation_errors;
+ __s64 time_next_delayed_flow;
+ __u32 flows;
+ __u32 inactive_flows;
+ __u32 throttled_flows;
+ __u32 pad;
+};
#endif
diff --git a/tc/Makefile b/tc/Makefile
index f26e764..1eeabd8 100644
--- a/tc/Makefile
+++ b/tc/Makefile
@@ -50,6 +50,7 @@ TCMODULES += em_meta.o
TCMODULES += q_mqprio.o
TCMODULES += q_codel.o
TCMODULES += q_fq_codel.o
+TCMODULES += q_fq.o
ifeq ($(TC_CONFIG_IPSET), y)
ifeq ($(TC_CONFIG_XT), y)
diff --git a/tc/q_fq.c b/tc/q_fq.c
new file mode 100644
index 0000000..c0bcdb9
--- /dev/null
+++ b/tc/q_fq.c
@@ -0,0 +1,279 @@
+/*
+ * Fair Queue
+ *
+ * Copyright (C) 2013 Eric Dumazet <edumazet@google.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions, and the following disclaimer,
+ * without modification.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. The names of the authors may not be used to endorse or promote products
+ * derived from this software without specific prior written permission.
+ *
+ * Alternatively, provided that this notice is retained in full, this
+ * software may be distributed under the terms of the GNU General
+ * Public License ("GPL") version 2, in which case the provisions of the
+ * GPL apply INSTEAD OF those given above.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <syslog.h>
+#include <fcntl.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <arpa/inet.h>
+#include <string.h>
+
+#include "utils.h"
+#include "tc_util.h"
+
+static void explain(void)
+{
+ fprintf(stderr, "Usage: ... fq [ limit PACKETS ] [ flow_limit PACKETS ]\n");
+ fprintf(stderr, " [ quantum BYTES ] [ initial_quantum BYTES ]\n");
+ fprintf(stderr, " [ maxrate RATE ] [ buckets NUMBER ]\n");
+ fprintf(stderr, " [ [no]pacing ]\n");
+}
+
+static unsigned int ilog2(unsigned int val)
+{
+ unsigned int res = 0;
+
+ val--;
+ while (val) {
+ res++;
+ val >>= 1;
+ }
+ return res;
+}
+
+static int fq_parse_opt(struct qdisc_util *qu, int argc, char **argv,
+ struct nlmsghdr *n)
+{
+ unsigned int plimit = ~0U;
+ unsigned int flow_plimit = ~0U;
+ unsigned int quantum = ~0U;
+ unsigned int initial_quantum = ~0U;
+ unsigned int buckets = 0;
+ unsigned int maxrate = ~0U;
+ unsigned int defrate = ~0U;
+ int pacing = -1;
+ struct rtattr *tail;
+
+ while (argc > 0) {
+ if (strcmp(*argv, "limit") == 0) {
+ NEXT_ARG();
+ if (get_unsigned(&plimit, *argv, 0)) {
+ fprintf(stderr, "Illegal \"limit\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "flow_limit") == 0) {
+ NEXT_ARG();
+ if (get_unsigned(&flow_plimit, *argv, 0)) {
+ fprintf(stderr, "Illegal \"flow_limit\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "buckets") == 0) {
+ NEXT_ARG();
+ if (get_unsigned(&buckets, *argv, 0)) {
+ fprintf(stderr, "Illegal \"buckets\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "maxrate") == 0) {
+ NEXT_ARG();
+ if (get_rate(&maxrate, *argv)) {
+ fprintf(stderr, "Illegal \"maxrate\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "defrate") == 0) {
+ NEXT_ARG();
+ if (get_rate(&defrate, *argv)) {
+ fprintf(stderr, "Illegal \"defrate\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "quantum") == 0) {
+ NEXT_ARG();
+ if (get_unsigned(&quantum, *argv, 0)) {
+ fprintf(stderr, "Illegal \"quantum\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "initial_quantum") == 0) {
+ NEXT_ARG();
+ if (get_unsigned(&initial_quantum, *argv, 0)) {
+ fprintf(stderr, "Illegal \"initial_quantum\"\n");
+ return -1;
+ }
+ } else if (strcmp(*argv, "pacing") == 0) {
+ pacing = 1;
+ } else if (strcmp(*argv, "nopacing") == 0) {
+ pacing = 0;
+ } else if (strcmp(*argv, "help") == 0) {
+ explain();
+ return -1;
+ } else {
+ fprintf(stderr, "What is \"%s\"?\n", *argv);
+ explain();
+ return -1;
+ }
+ argc--; argv++;
+ }
+
+ tail = NLMSG_TAIL(n);
+ addattr_l(n, 1024, TCA_OPTIONS, NULL, 0);
+ if (buckets) {
+ unsigned int log = ilog2(buckets);
+
+ addattr_l(n, 1024, TCA_FQ_BUCKETS_LOG,
+ &log, sizeof(log));
+ }
+ if (plimit != ~0U)
+ addattr_l(n, 1024, TCA_FQ_PLIMIT,
+ &plimit, sizeof(plimit));
+ if (flow_plimit != ~0U)
+ addattr_l(n, 1024, TCA_FQ_FLOW_PLIMIT,
+ &flow_plimit, sizeof(flow_plimit));
+ if (quantum != ~0U)
+ addattr_l(n, 1024, TCA_FQ_QUANTUM, &quantum, sizeof(quantum));
+ if (initial_quantum != ~0U)
+ addattr_l(n, 1024, TCA_FQ_INITIAL_QUANTUM,
+ &initial_quantum, sizeof(initial_quantum));
+ if (pacing != -1)
+ addattr_l(n, 1024, TCA_FQ_RATE_ENABLE,
+ &pacing, sizeof(pacing));
+ if (maxrate != ~0U)
+ addattr_l(n, 1024, TCA_FQ_FLOW_MAX_RATE,
+ &maxrate, sizeof(maxrate));
+ if (defrate != ~0U)
+ addattr_l(n, 1024, TCA_FQ_FLOW_DEFAULT_RATE,
+ &defrate, sizeof(defrate));
+ tail->rta_len = (void *) NLMSG_TAIL(n) - (void *) tail;
+ return 0;
+}
+
+static int fq_print_opt(struct qdisc_util *qu, FILE *f, struct rtattr *opt)
+{
+ struct rtattr *tb[TCA_FQ_MAX + 1];
+ unsigned int plimit, flow_plimit;
+ unsigned int buckets_log;
+ int pacing;
+ unsigned int rate, quantum;
+ SPRINT_BUF(b1);
+
+ if (opt == NULL)
+ return 0;
+
+ parse_rtattr_nested(tb, TCA_FQ_MAX, opt);
+
+ if (tb[TCA_FQ_PLIMIT] &&
+ RTA_PAYLOAD(tb[TCA_FQ_PLIMIT]) >= sizeof(__u32)) {
+ plimit = rta_getattr_u32(tb[TCA_FQ_PLIMIT]);
+ fprintf(f, "limit %up ", plimit);
+ }
+ if (tb[TCA_FQ_FLOW_PLIMIT] &&
+ RTA_PAYLOAD(tb[TCA_FQ_FLOW_PLIMIT]) >= sizeof(__u32)) {
+ flow_plimit = rta_getattr_u32(tb[TCA_FQ_FLOW_PLIMIT]);
+ fprintf(f, "flow_limit %up ", flow_plimit);
+ }
+ if (tb[TCA_FQ_BUCKETS_LOG] &&
+ RTA_PAYLOAD(tb[TCA_FQ_BUCKETS_LOG]) >= sizeof(__u32)) {
+ buckets_log = rta_getattr_u32(tb[TCA_FQ_BUCKETS_LOG]);
+ fprintf(f, "buckets %u ", 1U << buckets_log);
+ }
+ if (tb[TCA_FQ_RATE_ENABLE] &&
+ RTA_PAYLOAD(tb[TCA_FQ_RATE_ENABLE]) >= sizeof(int)) {
+ pacing = rta_getattr_u32(tb[TCA_FQ_RATE_ENABLE]);
+ if (pacing == 0)
+ fprintf(f, "nopacing ");
+ }
+ if (tb[TCA_FQ_QUANTUM] &&
+ RTA_PAYLOAD(tb[TCA_FQ_QUANTUM]) >= sizeof(__u32)) {
+ quantum = rta_getattr_u32(tb[TCA_FQ_QUANTUM]);
+ fprintf(f, "quantum %u ", quantum);
+ }
+ if (tb[TCA_FQ_INITIAL_QUANTUM] &&
+ RTA_PAYLOAD(tb[TCA_FQ_INITIAL_QUANTUM]) >= sizeof(__u32)) {
+ quantum = rta_getattr_u32(tb[TCA_FQ_INITIAL_QUANTUM]);
+ fprintf(f, "initial_quantum %u ", quantum);
+ }
+ if (tb[TCA_FQ_FLOW_MAX_RATE] &&
+ RTA_PAYLOAD(tb[TCA_FQ_FLOW_MAX_RATE]) >= sizeof(__u32)) {
+ rate = rta_getattr_u32(tb[TCA_FQ_FLOW_MAX_RATE]);
+
+ if (rate != ~0U)
+ fprintf(f, "maxrate %s ", sprint_rate(rate, b1));
+ }
+ if (tb[TCA_FQ_FLOW_DEFAULT_RATE] &&
+ RTA_PAYLOAD(tb[TCA_FQ_FLOW_DEFAULT_RATE]) >= sizeof(__u32)) {
+ rate = rta_getattr_u32(tb[TCA_FQ_FLOW_DEFAULT_RATE]);
+
+ if (rate != 0)
+ fprintf(f, "defrate %s ", sprint_rate(rate, b1));
+ }
+
+ return 0;
+}
+
+static int fq_print_xstats(struct qdisc_util *qu, FILE *f,
+ struct rtattr *xstats)
+{
+ struct tc_fq_qd_stats *st;
+
+ if (xstats == NULL)
+ return 0;
+
+ if (RTA_PAYLOAD(xstats) < sizeof(*st))
+ return -1;
+
+ st = RTA_DATA(xstats);
+
+ fprintf(f, " %u flows (%u inactive, %u throttled)",
+ st->flows, st->inactive_flows, st->throttled_flows);
+
+ if (st->time_next_delayed_flow > 0)
+ fprintf(f, ", next packet delay %llu ns", st->time_next_delayed_flow);
+
+ fprintf(f, "\n %llu gc, %llu highprio",
+ st->gc_flows, st->highprio_packets);
+
+ if (st->tcp_retrans)
+ fprintf(f, ", %llu retrans", st->tcp_retrans);
+
+ fprintf(f, ", %llu throttled", st->throttled);
+
+ if (st->flows_plimit)
+ fprintf(f, ", %llu flows_plimit", st->flows_plimit);
+
+ if (st->pkts_too_long || st->allocation_errors)
+ fprintf(f, "\n %llu too long pkts, %llu alloc errors\n",
+ st->pkts_too_long, st->allocation_errors);
+
+ return 0;
+}
+
+struct qdisc_util fq_qdisc_util = {
+ .id = "fq",
+ .parse_qopt = fq_parse_opt,
+ .print_qopt = fq_print_opt,
+ .print_xstats = fq_print_xstats,
+};
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH iproute2] pkt_sched: fq: Fair Queue packet scheduler
2013-08-30 2:30 ` [PATCH iproute2] " Eric Dumazet
@ 2013-09-03 15:49 ` Stephen Hemminger
0 siblings, 0 replies; 18+ messages in thread
From: Stephen Hemminger @ 2013-09-03 15:49 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David Miller, netdev, ycheng, ncardwell
On Thu, 29 Aug 2013 19:30:36 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:
> From: Eric Dumazet <edumazet@google.com>
>
> Support for FQ packet scheduler
Applied to net-next-3.11 branch, with some minor cleanups.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-08-29 22:49 [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler Eric Dumazet
2013-08-30 1:47 ` David Miller
@ 2013-09-04 5:26 ` Jason Wang
2013-09-04 5:59 ` Eric Dumazet
1 sibling, 1 reply; 18+ messages in thread
From: Jason Wang @ 2013-09-04 5:26 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 08/30/2013 06:49 AM, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
>
> - Uses perfect flow match (not stochastic hash like SFQ/FQ_codel)
> - Uses the new_flow/old_flow separation from FQ_codel
> - New flows get an initial credit allowing IW10 without added delay.
> - Special FIFO queue for high prio packets (no need for PRIO + FQ)
> - Uses a hash table of RB trees to locate the flows at enqueue() time
> - Smart on demand gc (at enqueue() time, RB tree lookup evicts old
> unused flows)
> - Dynamic memory allocations.
> - Designed to allow millions of concurrent flows per Qdisc.
> - Small memory footprint : ~8K per Qdisc, and 104 bytes per flow.
> - Single high resolution timer for throttled flows (if any).
> - One RB tree to link throttled flows.
> - Ability to have a max rate per flow. We might add a socket option
> to add per socket limitation.
>
> Attempts have been made to add TCP pacing in TCP stack, but this
> seems to add complex code to an already complex stack.
>
> TCP pacing is welcomed for flows having idle times, as the cwnd
> permits TCP stack to queue a possibly large number of packets.
>
[...]
>
> FQ gets a bunch of tunables as :
>
> limit : max number of packets on whole Qdisc (default 10000)
>
> flow_limit : max number of packets per flow (default 100)
>
> quantum : the credit per RR round (default is 2 MTU)
>
> initial_quantum : initial credit for new flows (default is 10 MTU)
>
> maxrate : max per flow rate (default : unlimited)
>
> buckets : number of RB trees (default : 1024) in hash table.
> (consumes 8 bytes per bucket)
>
> [no]pacing : disable/enable pacing (default is enable)
>
> All of them can be changed on a live qdisc.
>
> $ tc qd add dev eth0 root fq help
> Usage: ... fq [ limit PACKETS ] [ flow_limit PACKETS ]
> [ quantum BYTES ] [ initial_quantum BYTES ]
> [ maxrate RATE ] [ buckets NUMBER ]
> [ [no]pacing ]
>
> $ tc -s -d qd
> qdisc fq 8002: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 256 quantum 3028 initial_quantum 15140
> Sent 216532416 bytes 148395 pkt (dropped 0, overlimits 0 requeues 14)
> backlog 0b 0p requeues 14
> 511 flows, 511 inactive, 0 throttled
> 110 gc, 0 highprio, 0 retrans, 1143 throttled, 0 flows_plimit
>
>
> [1] Except if initial srtt is overestimated, as if using
> cached srtt in tcp metrics. We'll provide a fix for this issue.
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Yuchung Cheng <ycheng@google.com>
> Cc: Neal Cardwell <ncardwell@google.com>
> ---
> v2: added initial_quantum support
I see both degradation and jitter when using fq with virtio-net. Guest
to guest performance drops from 8Gb/s to 3Gb/s-7Gb/s. Guest to local
host drops from 8Gb/s to 4Gb/s-6Gb/s. Guest to external host with ixgbe
drops from 9Gb/s to 7Gb/s
I didn't meet the issue when using sfq or disabling pacing.
So it looks like it was caused by the inaccuracy and jitter of the
pacing estimation in a virt guest?
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 5:26 ` [PATCH v2 net-next] " Jason Wang
@ 2013-09-04 5:59 ` Eric Dumazet
2013-09-04 6:30 ` Jason Wang
0 siblings, 1 reply; 18+ messages in thread
From: Eric Dumazet @ 2013-09-04 5:59 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Wed, 2013-09-04 at 13:26 +0800, Jason Wang wrote:
> I see both degradation and jitter when using fq with virtio-net. Guest
> to guest performance drops from 8Gb/s to 3Gb/s-7Gb/s. Guest to local
> host drops from 8Gb/s to 4Gb/s-6Gb/s. Guest to external host with ixgbe
> drops from 9Gb/s to 7Gb/s
>
> I didn't meet the issue when using sfq or disabling pacing.
>
> So it looks like it was caused by the inaccuracy and jitter of the
> pacing estimation in a virt guest?
Well, using virtio-net means you use FQ without pacing.
Make sure you do not have reorders because of a bug in queue selection.
TCP stack has the ooo_okay thing, I do not think a VM can get it.
nstat >/dev/null ; <your test> ; nstat
And tcpdump would certainly help ;)
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 5:59 ` Eric Dumazet
@ 2013-09-04 6:30 ` Jason Wang
2013-09-04 10:30 ` Eric Dumazet
0 siblings, 1 reply; 18+ messages in thread
From: Jason Wang @ 2013-09-04 6:30 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
[-- Attachment #1: Type: text/plain, Size: 3124 bytes --]
On 09/04/2013 01:59 PM, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 13:26 +0800, Jason Wang wrote:
>
>> I see both degradation and jitter when using fq with virtio-net. Guest
>> to guest performance drops from 8Gb/s to 3Gb/s-7Gb/s. Guest to local
>> host drops from 8Gb/s to 4Gb/s-6Gb/s. Guest to external host with ixgbe
>> drops from 9Gb/s to 7Gb/s
>>
>> I didn't meet the issue when using sfq or disabling pacing.
>>
>> So it looks like it was caused by the inaccuracy and jitter of the
>> pacing estimation in a virt guest?
> Well, using virtio-net means you use FQ without pacing.
>
> Make sure you do not have reorders because of a bug in queue selection.
I test with only one queue enabled, so should not have this problem.
>
> TCP stack has the ooo_okay thing, I do not think a VM can get it.
>
> nstat >/dev/null ; <your test> ; nstat
This is result of guest to guest:
using sfq:
nstat > /dev/null; netperf -H 192.168.100.5; nstat
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.100.5 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.01 9155.88
#kernel
IpInReceives 130989 0.0
IpInDelivers 130989 0.0
IpOutRequests 176518 0.0
TcpActiveOpens 2 0.0
TcpInSegs 130989 0.0
TcpOutSegs 7908396 0.0
TcpExtDelayedACKs 1 0.0
TcpExtTCPPureAcks 60997 0.0
TcpExtTCPHPAcks 69985 0.0
IpExtInOctets 6813412 0.0
IpExtOutOctets 11460499000 0.0
IpExtInNoECTPkts 130989 0.0
using fq:
nstat > /dev/null; netperf -H 192.168.100.5; nstat
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.100.5 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 6340.29
#kernel
IpInReceives 121595 0.0
IpInDelivers 121595 0.0
IpOutRequests 121763 0.0
TcpActiveOpens 2 0.0
TcpInSegs 121595 0.0
TcpOutSegs 5474944 0.0
TcpExtTW 2 0.0
TcpExtDelayedACKs 1 0.0
TcpExtTCPPureAcks 50946 0.0
TcpExtTCPHPAcks 70642 0.0
IpExtInOctets 6324924 0.0
IpExtOutOctets 7934016612 0.0
IpExtInNoECTPkts 121595 0.0
>
> And tcpdump would certainly help ;)
See attachment.
Thanks
[-- Attachment #2: fq_out --]
[-- Type: text/plain, Size: 47989 bytes --]
14:09:54.296328 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 268750855, win 16256, options [nop,nop,TS val 222137 ecr 230249], length 0
14:09:54.296378 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 130321:195481, ack 0, win 229, options [nop,nop,TS val 230249 ecr 222137], length 65160
14:09:54.296477 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 195481:260641, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.296493 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 65161, win 16290, options [nop,nop,TS val 222137 ecr 230249], length 0
14:09:54.296560 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 260641:325801, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.296576 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 130321, win 16256, options [nop,nop,TS val 222137 ecr 230249], length 0
14:09:54.296651 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 325801:390961, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.296734 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 390961:456121, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.296748 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 195481, win 16290, options [nop,nop,TS val 222137 ecr 230249], length 0
14:09:54.296753 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 260641, win 16290, options [nop,nop,TS val 222137 ecr 230250], length 0
14:09:54.296755 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 325801, win 16256, options [nop,nop,TS val 222137 ecr 230250], length 0
14:09:54.296819 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 456121:521281, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.296905 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 521281:586441, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.297013 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 586441:651601, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.297029 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 390961, win 16290, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297034 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 456121, win 16256, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297085 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 651601:716761, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.297100 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 521281, win 16256, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297166 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 716761:781921, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.297273 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 781921:847081, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222137], length 65160
14:09:54.297289 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 586441, win 16290, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297345 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 847081:912241, ack 0, win 229, options [nop,nop,TS val 230250 ecr 222138], length 65160
14:09:54.297367 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 651601, win 16290, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297371 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 716761, win 16256, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297420 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 912241:977401, ack 0, win 229, options [nop,nop,TS val 230251 ecr 222138], length 65160
14:09:54.297507 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 977401:1042561, ack 0, win 229, options [nop,nop,TS val 230251 ecr 222138], length 65160
14:09:54.297598 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 1042561:1107721, ack 0, win 229, options [nop,nop,TS val 230251 ecr 222138], length 65160
14:09:54.297613 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 781921, win 16256, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297619 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 847081, win 16290, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297622 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 912241, win 16256, options [nop,nop,TS val 222138 ecr 230250], length 0
14:09:54.297681 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 1107721:1172881, ack 0, win 229, options [nop,nop,TS val 230251 ecr 222138], length 65160
14:09:54.309333 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 9643681:9708841, ack 0, win 229, options [nop,nop,TS val 230262 ecr 222150], length 65160
14:09:54.309504 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 9774001:9839161, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222150], length 65160
14:09:54.309597 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 9839161:9904321, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222150], length 65160
14:09:54.309613 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 9643681, win 16290, options [nop,nop,TS val 222150 ecr 230262], length 0
14:09:54.309709 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 9708841, win 16290, options [nop,nop,TS val 222150 ecr 230262], length 0
14:09:54.309713 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 9774001, win 16256, options [nop,nop,TS val 222150 ecr 230262], length 0
14:09:54.309942 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10099801:10164961, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222150], length 65160
14:09:54.310020 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10034641, win 16256, options [nop,nop,TS val 222151 ecr 230263], length 0
14:09:54.310069 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10099801, win 15997, options [nop,nop,TS val 222151 ecr 230263], length 0
14:09:54.310125 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10230121:10295281, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222150], length 65160
14:09:54.310206 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10295281:10360441, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222151], length 65160
14:09:54.310304 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10360441:10425601, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222151], length 65160
14:09:54.310398 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10425601:10490761, ack 0, win 229, options [nop,nop,TS val 230263 ecr 222151], length 65160
14:09:54.310414 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10164961, win 16290, options [nop,nop,TS val 222151 ecr 230263], length 0
14:09:54.310420 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10230121, win 16256, options [nop,nop,TS val 222151 ecr 230263], length 0
14:09:54.310493 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10490761:10555921, ack 0, win 229, options [nop,nop,TS val 230264 ecr 222151], length 65160
14:09:54.310664 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10621081:10686241, ack 0, win 229, options [nop,nop,TS val 230264 ecr 222151], length 65160
14:09:54.310753 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 10686241:10751401, ack 0, win 229, options [nop,nop,TS val 230264 ecr 222151], length 65160
14:09:54.310769 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10425601, win 16290, options [nop,nop,TS val 222151 ecr 230263], length 0
14:09:54.310964 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10686241, win 16290, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.310968 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10751401, win 16256, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.311289 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11077201:11142361, ack 0, win 229, options [nop,nop,TS val 230264 ecr 222152], length 65160
14:09:54.311374 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11142361:11207521, ack 0, win 229, options [nop,nop,TS val 230264 ecr 222152], length 65160
14:09:54.311390 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 10946881, win 16290, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.311395 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11012041, win 16290, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.311453 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11207521:11272681, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.311583 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11142361, win 16290, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.311635 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11337841:11403001, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.311722 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11403001:11468161, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.311738 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11207521, win 16256, options [nop,nop,TS val 222152 ecr 230264], length 0
14:09:54.311812 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11468161:11533321, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.311828 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11272681, win 16290, options [nop,nop,TS val 222152 ecr 230265], length 0
14:09:54.311832 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11337841, win 16256, options [nop,nop,TS val 222152 ecr 230265], length 0
14:09:54.311945 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11403001, win 16290, options [nop,nop,TS val 222153 ecr 230265], length 0
14:09:54.311949 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11468161, win 16256, options [nop,nop,TS val 222153 ecr 230265], length 0
14:09:54.311981 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11598481:11663641, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.312080 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11663641:11728801, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.312168 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11728801:11793961, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222152], length 65160
14:09:54.312185 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11533321, win 16290, options [nop,nop,TS val 222153 ecr 230265], length 0
14:09:54.312190 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11598481, win 16256, options [nop,nop,TS val 222153 ecr 230265], length 0
14:09:54.312324 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 11859121:11924281, ack 0, win 229, options [nop,nop,TS val 230265 ecr 222153], length 65160
14:09:54.312542 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 11793961, win 16256, options [nop,nop,TS val 222153 ecr 230265], length 0
14:09:54.312708 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12119761:12184921, ack 0, win 229, options [nop,nop,TS val 230266 ecr 222153], length 65160
14:09:54.312970 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12315241:12380401, ack 0, win 229, options [nop,nop,TS val 230266 ecr 222153], length 65160
14:09:54.313058 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12380401:12445561, ack 0, win 229, options [nop,nop,TS val 230266 ecr 222153], length 65160
14:09:54.313072 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12119761, win 16290, options [nop,nop,TS val 222153 ecr 230266], length 0
14:09:54.313077 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12184921, win 16290, options [nop,nop,TS val 222154 ecr 230266], length 0
14:09:54.313374 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12380401, win 16290, options [nop,nop,TS val 222154 ecr 230266], length 0
14:09:54.313536 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12510721, win 16256, options [nop,nop,TS val 222154 ecr 230266], length 0
14:09:54.313540 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12575881, win 16290, options [nop,nop,TS val 222154 ecr 230266], length 0
14:09:54.313584 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12771361:12836521, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222154], length 65160
14:09:54.313682 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12836521:12901681, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222154], length 65160
14:09:54.313698 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12641041, win 16290, options [nop,nop,TS val 222154 ecr 230266], length 0
14:09:54.313814 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12771361, win 16290, options [nop,nop,TS val 222154 ecr 230267], length 0
14:09:54.313860 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 12966841:13032001, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222154], length 65160
14:09:54.313943 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13032001:13097161, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222154], length 65160
14:09:54.313960 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12836521, win 16290, options [nop,nop,TS val 222154 ecr 230267], length 0
14:09:54.313983 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12901681, win 16256, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314133 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13162321:13227481, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222154], length 65160
14:09:54.314225 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13227481:13292641, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222155], length 65160
14:09:54.314236 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 12966841, win 16256, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314241 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13032001, win 16290, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314243 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13097161, win 16256, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314246 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13162321, win 16256, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314304 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13292641:13357801, ack 0, win 229, options [nop,nop,TS val 230267 ecr 222155], length 65160
14:09:54.314393 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13357801:13422961, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314482 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13422961:13488121, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314573 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13488121:13553281, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314591 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13227481, win 16290, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314597 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13292641, win 16256, options [nop,nop,TS val 222155 ecr 230267], length 0
14:09:54.314657 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13553281:13618441, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314748 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13618441:13683601, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314853 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13683601:13748761, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.314868 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13488121, win 16290, options [nop,nop,TS val 222155 ecr 230268], length 0
14:09:54.314929 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13748761:13813921, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.315094 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13879081:13944241, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.315112 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13618441, win 16256, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315190 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 13944241:14009401, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222155], length 65160
14:09:54.315206 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13683601, win 16256, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315211 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13748761, win 16290, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315213 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13813921, win 16256, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315295 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14009401:14074561, ack 0, win 229, options [nop,nop,TS val 230268 ecr 222156], length 65160
14:09:54.315308 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13879081, win 16256, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315356 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14074561:14139721, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.315451 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14139721:14204881, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.315468 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 13944241, win 16290, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315537 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14204881:14270041, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.315552 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14009401, win 16290, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315573 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14074561, win 16256, options [nop,nop,TS val 222156 ecr 230268], length 0
14:09:54.315709 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14335201:14400361, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.315836 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14204881, win 16290, options [nop,nop,TS val 222156 ecr 230269], length 0
14:09:54.315852 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14270041, win 16256, options [nop,nop,TS val 222156 ecr 230269], length 0
14:09:54.315858 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14335201, win 16256, options [nop,nop,TS val 222156 ecr 230269], length 0
14:09:54.315896 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14465521:14530681, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.315982 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14530681:14595841, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.316069 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14595841:14661001, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.316339 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14791321:14856481, ack 0, win 229, options [nop,nop,TS val 230269 ecr 222156], length 65160
14:09:54.316518 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 14921641:14986801, ack 0, win 229, options [nop,nop,TS val 230270 ecr 222157], length 65160
14:09:54.316656 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14726161, win 16290, options [nop,nop,TS val 222157 ecr 230269], length 0
14:09:54.316660 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 14791321, win 16256, options [nop,nop,TS val 222157 ecr 230269], length 0
14:09:54.316695 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15051961:15117121, ack 0, win 229, options [nop,nop,TS val 230270 ecr 222157], length 65160
14:09:54.316822 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15051961, win 16256, options [nop,nop,TS val 222157 ecr 230270], length 0
14:09:54.317047 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15315633:15331561, ack 0, win 229, options [nop,nop,TS val 230270 ecr 222157], length 15928
14:09:54.317214 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15414097:15430025, ack 0, win 229, options [nop,nop,TS val 230270 ecr 222158], length 15928
14:09:54.317240 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15430025:15445953, ack 0, win 229, options [nop,nop,TS val 230270 ecr 222158], length 15928
14:09:54.317435 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15511329:15528705, ack 0, win 229, options [nop,nop,TS val 230271 ecr 222158], length 17376
14:09:54.317643 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15610089:15626017, ack 0, win 229, options [nop,nop,TS val 230271 ecr 222158], length 15928
14:09:54.317651 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15544633, win 16290, options [nop,nop,TS val 222158 ecr 230271], length 0
14:09:54.317821 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15643393, win 16290, options [nop,nop,TS val 222158 ecr 230271], length 0
14:09:54.317826 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15659321, win 16290, options [nop,nop,TS val 222158 ecr 230271], length 0
14:09:54.317830 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15675249, win 16290, options [nop,nop,TS val 222158 ecr 230271], length 0
14:09:54.317853 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15708849:15724777, ack 0, win 229, options [nop,nop,TS val 230271 ecr 222158], length 15928
14:09:54.318024 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15708849, win 16290, options [nop,nop,TS val 222159 ecr 230271], length 0
14:09:54.318027 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15724777, win 16290, options [nop,nop,TS val 222159 ecr 230271], length 0
14:09:54.318234 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 15905617:15921545, ack 0, win 229, options [nop,nop,TS val 230271 ecr 222159], length 15928
14:09:54.318469 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16003841:16019769, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 15928
14:09:54.318481 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 15921545, win 16290, options [nop,nop,TS val 222159 ecr 230271], length 0
14:09:54.318676 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16086137:16102065, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 15928
14:09:54.318701 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16102065:16117993, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 15928
14:09:54.318711 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16036529, win 16290, options [nop,nop,TS val 222159 ecr 230272], length 0
14:09:54.318822 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16167225:16183153, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 15928
14:09:54.318858 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16183153:16200529, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 17376
14:09:54.318867 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16052457, win 16290, options [nop,nop,TS val 222159 ecr 230272], length 0
14:09:54.318871 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16068385, win 16290, options [nop,nop,TS val 222159 ecr 230272], length 0
14:09:54.318919 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16200825:16216753, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222159], length 15928
14:09:54.319045 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16265985:16281913, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222160], length 15928
14:09:54.319128 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16299129:16315057, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222160], length 15928
14:09:54.319277 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16364289:16380217, ack 0, win 229, options [nop,nop,TS val 230272 ecr 222160], length 15928
14:09:54.319372 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16396977:16412905, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319403 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16412905:16428833, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319409 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16364289, win 16290, options [nop,nop,TS val 222160 ecr 230272], length 0
14:09:54.319470 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16446129:16462057, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319496 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16462057:16477985, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319511 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16396977, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319535 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16477985:16495361, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 17376
14:09:54.319544 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16412905, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319607 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16511289:16527217, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319616 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16428833, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319619 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16446129, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319622 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16462057, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319625 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16477985, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319664 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16527217:16544593, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 17376
14:09:54.319677 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16544593:16544889, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 296
14:09:54.319705 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16544889:16560817, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319734 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16560817:16576745, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319764 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16576745:16592673, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319794 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16592673:16610049, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 17376
14:09:54.319810 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16495361, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319813 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16511289, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319816 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16527217, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319843 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16610049:16625977, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319852 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16544593, win 16290, options [nop,nop,TS val 222160 ecr 230273], length 0
14:09:54.319879 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16625977:16641905, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 15928
14:09:54.319907 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16641905:16659281, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222160], length 17376
14:09:54.319918 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16560817, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.319926 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16659281:16659577, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 296
14:09:54.319929 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16576745, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.319933 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16592673, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.319957 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16659577:16675505, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.319999 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16675505:16691433, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.320028 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16691433:16707361, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.320042 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16610049, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320046 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16625977, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320063 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16641905, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320089 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16707361:16724737, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 17376
14:09:54.320098 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16659281, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320101 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16675505, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320112 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16724737:16725113, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 376
14:09:54.320142 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16725113:16741041, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.320155 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16691433, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320159 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16707361, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320189 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16741041:16756969, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.320220 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16756969:16772897, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 15928
14:09:54.320273 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16772897:16790273, ack 0, win 229, options [nop,nop,TS val 230273 ecr 222161], length 17376
14:09:54.320308 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16790273:16806201, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320317 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16724737, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320321 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16741041, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320331 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16806201:16807033, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 832
14:09:54.320345 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16756969, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320363 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16807033:16822961, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320394 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16822961:16838889, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320432 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16838889:16854817, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320440 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16772897, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320444 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16790273, win 16290, options [nop,nop,TS val 222161 ecr 230273], length 0
14:09:54.320446 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16806201, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320449 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16807033, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320459 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16854817:16856185, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 1368
14:09:54.320492 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16856185:16872113, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320523 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16872113:16888041, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320557 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16888041:16903969, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320565 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16822961, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320569 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16838889, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320571 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16854817, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320574 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16856185, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320584 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16903969:16905337, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 1368
14:09:54.320617 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16905337:16921265, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320653 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16921265:16937193, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320689 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16937193:16953121, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320697 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16872113, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320701 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16888041, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320705 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16903969, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320707 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16905337, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320718 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 16953121:16954489, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 1368
14:09:54.320751 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16954489:16970417, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320783 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16970417:16986345, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320818 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 16986345:17002273, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320826 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16921265, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320830 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16937193, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320832 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16953121, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320835 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16954489, win 16290, options [nop,nop,TS val 222161 ecr 230274], length 0
14:09:54.320845 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17002273:17003641, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 1368
14:09:54.320875 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17003641:17019569, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320907 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17019569:17035497, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320941 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17035497:17051425, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222161], length 15928
14:09:54.320947 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16970417, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.320950 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 16986345, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.320953 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17002273, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.320956 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17003641, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.320967 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17051425:17052793, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 1368
14:09:54.321010 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17052793:17068721, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321043 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17068721:17084649, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321074 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17084649:17100577, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321106 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17100577:17117953, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 17376
14:09:54.321135 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17117953:17133881, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321147 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17019569, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321150 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17035497, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321168 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17051425, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321172 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17052793, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321174 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17068721, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321177 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17084649, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321191 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17133881:17149809, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321219 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17100577, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321228 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17149809:17151097, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 1288
14:09:54.321304 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17151097:17167025, ack 0, win 229, options [nop,nop,TS val 230274 ecr 222162], length 15928
14:09:54.321313 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17117953, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321318 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17133881, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321397 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17149809, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321401 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17151097, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321452 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17167025, win 16290, options [nop,nop,TS val 222162 ecr 230274], length 0
14:09:54.321492 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17167025:17167481, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 456
14:09:54.321530 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17167481:17183409, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321559 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17183409:17199337, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321587 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17199337:17215265, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321616 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17215265:17232641, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 17376
14:09:54.321643 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17232641:17248569, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321663 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17167481, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321667 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17183409, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321684 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17248569:17264497, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321698 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17264497:17265785, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 1288
14:09:54.321726 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17265785:17281713, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321735 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17199337, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321738 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17215265, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321770 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17281713:17297641, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321779 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17232641, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321783 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17248569, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321810 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17297641:17313569, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321844 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17313569:17330945, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 17376
14:09:54.321853 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17264497, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321857 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17265785, win 16290, options [nop,nop,TS val 222162 ecr 230275], length 0
14:09:54.321865 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [P.], seq 17330945:17331321, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 376
14:09:54.321887 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17331321:17347249, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321914 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17347249:17363177, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321942 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17363177:17379105, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 15928
14:09:54.321958 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17281713, win 16290, options [nop,nop,TS val 222163 ecr 230275], length 0
14:09:54.321962 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17297641, win 16290, options [nop,nop,TS val 222163 ecr 230275], length 0
14:09:54.321996 IP 192.168.100.4.47649 > 192.168.100.5.48748: Flags [.], seq 17379105:17396481, ack 0, win 229, options [nop,nop,TS val 230275 ecr 222162], length 17376
14:09:54.322004 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17313569, win 16290, options [nop,nop,TS val 222163 ecr 230275], length 0
14:09:54.322008 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17330945, win 16290, options [nop,nop,TS val 222163 ecr 230275], length 0
14:09:54.322030 IP 192.168.100.5.48748 > 192.168.100.4.47649: Flags [.], ack 17347249, win 16290, options [nop,nop,TS val 222163 ecr 230275], length 0
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 6:30 ` Jason Wang
@ 2013-09-04 10:30 ` Eric Dumazet
2013-09-04 11:27 ` Eric Dumazet
2013-09-05 3:07 ` Jason Wang
0 siblings, 2 replies; 18+ messages in thread
From: Eric Dumazet @ 2013-09-04 10:30 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
> > And tcpdump would certainly help ;)
>
> See attachment.
>
Nothing obvious on tcpdump (only that lot of frames are missing)
1) Are you capturing part of the payload only (like tcpdump -s 128)
2) What is the setup.
3) tc -s -d qdisc
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 10:30 ` Eric Dumazet
@ 2013-09-04 11:27 ` Eric Dumazet
2013-09-04 11:59 ` Daniel Borkmann
` (2 more replies)
2013-09-05 3:07 ` Jason Wang
1 sibling, 3 replies; 18+ messages in thread
From: Eric Dumazet @ 2013-09-04 11:27 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>
> > > And tcpdump would certainly help ;)
> >
> > See attachment.
> >
>
> Nothing obvious on tcpdump (only that lot of frames are missing)
>
> 1) Are you capturing part of the payload only (like tcpdump -s 128)
>
> 2) What is the setup.
>
> 3) tc -s -d qdisc
If you use FQ in the guest, then it could be that high resolution timers
have high latency ?
So FQ arms short timers, but effective duration could be much longer.
Here I get a smooth latency of up to ~3 us
lpq83:~# ./netperf -H lpq84 ; ./tc -s -d qd ; dmesg | tail -n1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpq84.prod.google.com () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 9410.82
qdisc fq 8005: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
Sent 50545633991 bytes 33385894 pkt (dropped 0, overlimits 0 requeues 19)
rate 9258Mbit 764335pps backlog 0b 0p requeues 19
117 flow, 115 inactive, 0 throttled
0 gc, 0 highprio, 0 retrans, 96861 throttled, 0 flows_plimit
[ 572.551664] latency = 3035 ns
What do you get with this debugging patch ?
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index 32ad015..c1312a0 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -103,6 +103,7 @@ struct fq_sched_data {
u64 stat_internal_packets;
u64 stat_tcp_retrans;
u64 stat_throttled;
+ s64 slatency;
u64 stat_flows_plimit;
u64 stat_pkts_too_long;
u64 stat_allocation_errors;
@@ -393,6 +394,7 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
static void fq_check_throttled(struct fq_sched_data *q, u64 now)
{
struct rb_node *p;
+ bool first = true;
if (q->time_next_delayed_flow > now)
return;
@@ -405,6 +407,13 @@ static void fq_check_throttled(struct fq_sched_data *q, u64 now)
q->time_next_delayed_flow = f->time_next_packet;
break;
}
+ if (first) {
+ s64 delay = now - f->time_next_packet;
+
+ first = false;
+ delay -= q->slatency >> 3;
+ q->slatency += delay;
+ }
rb_erase(p, &q->delayed);
q->throttled_flows--;
fq_flow_add_tail(&q->old_flows, f);
@@ -711,6 +720,7 @@ static int fq_dump(struct Qdisc *sch, struct sk_buff *skb)
if (opts == NULL)
goto nla_put_failure;
+ pr_err("latency = %lld ns\n", q->slatency >> 3);
if (nla_put_u32(skb, TCA_FQ_PLIMIT, sch->limit) ||
nla_put_u32(skb, TCA_FQ_FLOW_PLIMIT, q->flow_plimit) ||
nla_put_u32(skb, TCA_FQ_QUANTUM, q->quantum) ||
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 11:27 ` Eric Dumazet
@ 2013-09-04 11:59 ` Daniel Borkmann
2013-09-05 3:39 ` Jason Wang
2013-09-05 0:50 ` Eric Dumazet
2013-09-05 3:34 ` Jason Wang
2 siblings, 1 reply; 18+ messages in thread
From: Daniel Borkmann @ 2013-09-04 11:59 UTC (permalink / raw)
To: Eric Dumazet
Cc: Jason Wang, David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 09/04/2013 01:27 PM, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
>> On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>>
>>>> And tcpdump would certainly help ;)
>>>
>>> See attachment.
>>>
>>
>> Nothing obvious on tcpdump (only that lot of frames are missing)
>>
>> 1) Are you capturing part of the payload only (like tcpdump -s 128)
>>
>> 2) What is the setup.
>>
>> 3) tc -s -d qdisc
>
> If you use FQ in the guest, then it could be that high resolution timers
> have high latency ?
Probably they internally switch to a lower resolution clock event source if
there's no hardware support available:
The [source event] management layer provides interfaces for hrtimers to
implement high resolution timers [...] [and it] supports these more advanced
functions only when appropriate clock event sources have been registered,
otherwise the traditional periodic tick based behaviour is retained. [1]
[1] https://www.kernel.org/doc/ols/2006/ols2006v1-pages-333-346.pdf
> So FQ arms short timers, but effective duration could be much longer.
>
> Here I get a smooth latency of up to ~3 us
>
> lpq83:~# ./netperf -H lpq84 ; ./tc -s -d qd ; dmesg | tail -n1
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpq84.prod.google.com () port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 9410.82
> qdisc fq 8005: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
> Sent 50545633991 bytes 33385894 pkt (dropped 0, overlimits 0 requeues 19)
> rate 9258Mbit 764335pps backlog 0b 0p requeues 19
> 117 flow, 115 inactive, 0 throttled
> 0 gc, 0 highprio, 0 retrans, 96861 throttled, 0 flows_plimit
> [ 572.551664] latency = 3035 ns
>
>
> What do you get with this debugging patch ?
>
> diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
> index 32ad015..c1312a0 100644
> --- a/net/sched/sch_fq.c
> +++ b/net/sched/sch_fq.c
> @@ -103,6 +103,7 @@ struct fq_sched_data {
> u64 stat_internal_packets;
> u64 stat_tcp_retrans;
> u64 stat_throttled;
> + s64 slatency;
> u64 stat_flows_plimit;
> u64 stat_pkts_too_long;
> u64 stat_allocation_errors;
> @@ -393,6 +394,7 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
> static void fq_check_throttled(struct fq_sched_data *q, u64 now)
> {
> struct rb_node *p;
> + bool first = true;
>
> if (q->time_next_delayed_flow > now)
> return;
> @@ -405,6 +407,13 @@ static void fq_check_throttled(struct fq_sched_data *q, u64 now)
> q->time_next_delayed_flow = f->time_next_packet;
> break;
> }
> + if (first) {
> + s64 delay = now - f->time_next_packet;
> +
> + first = false;
> + delay -= q->slatency >> 3;
> + q->slatency += delay;
> + }
> rb_erase(p, &q->delayed);
> q->throttled_flows--;
> fq_flow_add_tail(&q->old_flows, f);
> @@ -711,6 +720,7 @@ static int fq_dump(struct Qdisc *sch, struct sk_buff *skb)
> if (opts == NULL)
> goto nla_put_failure;
>
> + pr_err("latency = %lld ns\n", q->slatency >> 3);
> if (nla_put_u32(skb, TCA_FQ_PLIMIT, sch->limit) ||
> nla_put_u32(skb, TCA_FQ_FLOW_PLIMIT, q->flow_plimit) ||
> nla_put_u32(skb, TCA_FQ_QUANTUM, q->quantum) ||
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 11:27 ` Eric Dumazet
2013-09-04 11:59 ` Daniel Borkmann
@ 2013-09-05 0:50 ` Eric Dumazet
2013-09-05 1:23 ` Eric Dumazet
2013-09-05 3:43 ` Jason Wang
2013-09-05 3:34 ` Jason Wang
2 siblings, 2 replies; 18+ messages in thread
From: Eric Dumazet @ 2013-09-05 0:50 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Wed, 2013-09-04 at 04:27 -0700, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
> > On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
> >
> > > > And tcpdump would certainly help ;)
> > >
> > > See attachment.
> > >
> >
> > Nothing obvious on tcpdump (only that lot of frames are missing)
> >
> > 1) Are you capturing part of the payload only (like tcpdump -s 128)
> >
> > 2) What is the setup.
> >
> > 3) tc -s -d qdisc
>
> If you use FQ in the guest, then it could be that high resolution timers
> have high latency ?
>
> So FQ arms short timers, but effective duration could be much longer.
>
> Here I get a smooth latency of up to ~3 us
>
> lpq83:~# ./netperf -H lpq84 ; ./tc -s -d qd ; dmesg | tail -n1
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpq84.prod.google.com () port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 9410.82
> qdisc fq 8005: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
> Sent 50545633991 bytes 33385894 pkt (dropped 0, overlimits 0 requeues 19)
> rate 9258Mbit 764335pps backlog 0b 0p requeues 19
> 117 flow, 115 inactive, 0 throttled
> 0 gc, 0 highprio, 0 retrans, 96861 throttled, 0 flows_plimit
> [ 572.551664] latency = 3035 ns
>
>
> What do you get with this debugging patch ?
>
> diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
> index 32ad015..c1312a0 100644
> --- a/net/sched/sch_fq.c
> +++ b/net/sched/sch_fq.c
> @@ -103,6 +103,7 @@ struct fq_sched_data {
> u64 stat_internal_packets;
> u64 stat_tcp_retrans;
> u64 stat_throttled;
> + s64 slatency;
> u64 stat_flows_plimit;
> u64 stat_pkts_too_long;
> u64 stat_allocation_errors;
> @@ -393,6 +394,7 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
> static void fq_check_throttled(struct fq_sched_data *q, u64 now)
> {
> struct rb_node *p;
> + bool first = true;
>
> if (q->time_next_delayed_flow > now)
> return;
> @@ -405,6 +407,13 @@ static void fq_check_throttled(struct fq_sched_data *q, u64 now)
> q->time_next_delayed_flow = f->time_next_packet;
> break;
> }
> + if (first) {
> + s64 delay = now - f->time_next_packet;
> +
> + first = false;
> + delay -= q->slatency >> 3;
> + q->slatency += delay;
> + }
> rb_erase(p, &q->delayed);
> q->throttled_flows--;
> fq_flow_add_tail(&q->old_flows, f);
> @@ -711,6 +720,7 @@ static int fq_dump(struct Qdisc *sch, struct sk_buff *skb)
> if (opts == NULL)
> goto nla_put_failure;
>
> + pr_err("latency = %lld ns\n", q->slatency >> 3);
> if (nla_put_u32(skb, TCA_FQ_PLIMIT, sch->limit) ||
> nla_put_u32(skb, TCA_FQ_FLOW_PLIMIT, q->flow_plimit) ||
> nla_put_u32(skb, TCA_FQ_QUANTUM, q->quantum) ||
>
BTW what is your HZ value ?
We have a problem in TCP stack, because srtt is in HZ units.
Before we change to us units, I guess tcp_update_pacing_rate() should be
changed a bit if HZ=250
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-05 0:50 ` Eric Dumazet
@ 2013-09-05 1:23 ` Eric Dumazet
2013-09-05 3:43 ` Jason Wang
1 sibling, 0 replies; 18+ messages in thread
From: Eric Dumazet @ 2013-09-05 1:23 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Wed, 2013-09-04 at 17:50 -0700, Eric Dumazet wrote:
>
> BTW what is your HZ value ?
>
> We have a problem in TCP stack, because srtt is in HZ units.
>
> Before we change to us units, I guess tcp_update_pacing_rate() should be
> changed a bit if HZ=250
Oh well, I feel dumb, please try this fix (If you have HZ != 1000) :
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 93d7e9d..fd96d8e 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -700,7 +700,7 @@ static void tcp_update_pacing_rate(struct sock *sk)
u64 rate;
/* set sk_pacing_rate to 200 % of current rate (mss * cwnd / srtt) */
- rate = (u64)tp->mss_cache * 2 * (HZ << 3);
+ rate = (u64)tp->mss_cache * 2 * ((USEC_PER_SEC/HZ) << 3);
rate *= max(tp->snd_cwnd, tp->packets_out);
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 10:30 ` Eric Dumazet
2013-09-04 11:27 ` Eric Dumazet
@ 2013-09-05 3:07 ` Jason Wang
2013-09-05 3:41 ` Eric Dumazet
1 sibling, 1 reply; 18+ messages in thread
From: Jason Wang @ 2013-09-05 3:07 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 09/04/2013 06:30 PM, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>
>>> And tcpdump would certainly help ;)
>> See attachment.
>>
> Nothing obvious on tcpdump (only that lot of frames are missing)
>
> 1) Are you capturing part of the payload only (like tcpdump -s 128)
No, I use something like tcpdump -i eth0 -w fq -c 300 after the netperf
start.
>
> 2) What is the setup.
Tow kvm guest with virtio-net and vhost enabled. Only one queue is
enabled and the guest were connected with bridge. Both host and guest
were net-next.git
>
> 3) tc -s -d qdisc
tc -s -d qdisc
qdisc fq 8001: dev eth0 root refcnt 2 [Unknown qdisc, optlen=64]
Sent 6680760347 bytes 4431855 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
btw, is the fq support for tc merged into iproute2? Looks like I can't
find them.
>
>
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 11:27 ` Eric Dumazet
2013-09-04 11:59 ` Daniel Borkmann
2013-09-05 0:50 ` Eric Dumazet
@ 2013-09-05 3:34 ` Jason Wang
2 siblings, 0 replies; 18+ messages in thread
From: Jason Wang @ 2013-09-05 3:34 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 09/04/2013 07:27 PM, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
>> > On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>> >
>>>> > > > And tcpdump would certainly help ;)
>>> > >
>>> > > See attachment.
>>> > >
>> >
>> > Nothing obvious on tcpdump (only that lot of frames are missing)
>> >
>> > 1) Are you capturing part of the payload only (like tcpdump -s 128)
>> >
>> > 2) What is the setup.
>> >
>> > 3) tc -s -d qdisc
> If you use FQ in the guest, then it could be that high resolution timers
> have high latency ?
Not sure, but it should not affect so much. And I'm using kvm-clock in
guest whose overhead should be very small.
>
> So FQ arms short timers, but effective duration could be much longer.
>
> Here I get a smooth latency of up to ~3 us
>
> lpq83:~# ./netperf -H lpq84 ; ./tc -s -d qd ; dmesg | tail -n1
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpq84.prod.google.com () port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 87380 16384 16384 10.00 9410.82
> qdisc fq 8005: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
> Sent 50545633991 bytes 33385894 pkt (dropped 0, overlimits 0 requeues 19)
> rate 9258Mbit 764335pps backlog 0b 0p requeues 19
> 117 flow, 115 inactive, 0 throttled
> 0 gc, 0 highprio, 0 retrans, 96861 throttled, 0 flows_plimit
> [ 572.551664] latency = 3035 ns
>
>
> What do you get with this debugging patch ?
I'm getting about 13us-19us, one run like:
netperf -H 192.168.100.5; tc -s -d qd; dmesg | tail -n1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.100.5 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 4542.09
qdisc fq 8001: dev eth0 root refcnt 2 [Unknown qdisc, optlen=64]
Sent 53652327205 bytes 35580150 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
[ 201.320565] latency = 14905 ns
One interesting thing is if I switch from kvm-clock to acpi_pm which has
much more overhead, the latency increase to about 50ns, and the
throughput drops very quickly.
netperf -H 192.168.100.5; tc -s -d qd; dmesg | tail -n1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.100.5 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 2262.46
qdisc fq 8001: dev eth0 root refcnt 2 [Unknown qdisc, optlen=64]
Sent 56611533075 bytes 37550429 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
[ 474.121689] latency = 51841 ns
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-04 11:59 ` Daniel Borkmann
@ 2013-09-05 3:39 ` Jason Wang
0 siblings, 0 replies; 18+ messages in thread
From: Jason Wang @ 2013-09-05 3:39 UTC (permalink / raw)
To: Daniel Borkmann
Cc: Eric Dumazet, David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin, KVM
On 09/04/2013 07:59 PM, Daniel Borkmann wrote:
> On 09/04/2013 01:27 PM, Eric Dumazet wrote:
>> On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
>>> On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>>>
>>>>> And tcpdump would certainly help ;)
>>>>
>>>> See attachment.
>>>>
>>>
>>> Nothing obvious on tcpdump (only that lot of frames are missing)
>>>
>>> 1) Are you capturing part of the payload only (like tcpdump -s 128)
>>>
>>> 2) What is the setup.
>>>
>>> 3) tc -s -d qdisc
>>
>> If you use FQ in the guest, then it could be that high resolution timers
>> have high latency ?
>
> Probably they internally switch to a lower resolution clock event
> source if
> there's no hardware support available:
>
> The [source event] management layer provides interfaces for hrtimers to
> implement high resolution timers [...] [and it] supports these more
> advanced
> functions only when appropriate clock event sources have been
> registered,
> otherwise the traditional periodic tick based behaviour is retained.
> [1]
>
> [1] https://www.kernel.org/doc/ols/2006/ols2006v1-pages-333-346.pdf
Maybe, AFAIK, kvm-clock does not provide a clock event, only a pv
clocksource were provided.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-05 3:07 ` Jason Wang
@ 2013-09-05 3:41 ` Eric Dumazet
2013-09-05 5:16 ` Jason Wang
0 siblings, 1 reply; 18+ messages in thread
From: Eric Dumazet @ 2013-09-05 3:41 UTC (permalink / raw)
To: Jason Wang
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On Thu, 2013-09-05 at 11:07 +0800, Jason Wang wrote:
> tc -s -d qdisc
> qdisc fq 8001: dev eth0 root refcnt 2 [Unknown qdisc, optlen=64]
> Sent 6680760347 bytes 4431855 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
> btw, is the fq support for tc merged into iproute2? Looks like I can't
> find them.
Yep its there on net-next-3.11 branch
# git branch -a
master
* net-next-3.11
remotes/origin/HEAD -> origin/master
remotes/origin/iproute-3.5.1
remotes/origin/master
remotes/origin/net-next-3.11
# cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
[branch "net-next-3.11"]
remote = origin
merge = refs/heads/net-next-3.11
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-05 0:50 ` Eric Dumazet
2013-09-05 1:23 ` Eric Dumazet
@ 2013-09-05 3:43 ` Jason Wang
1 sibling, 0 replies; 18+ messages in thread
From: Jason Wang @ 2013-09-05 3:43 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 09/05/2013 08:50 AM, Eric Dumazet wrote:
> On Wed, 2013-09-04 at 04:27 -0700, Eric Dumazet wrote:
>> On Wed, 2013-09-04 at 03:30 -0700, Eric Dumazet wrote:
>>> On Wed, 2013-09-04 at 14:30 +0800, Jason Wang wrote:
>>>
>>>>> And tcpdump would certainly help ;)
>>>> See attachment.
>>>>
>>> Nothing obvious on tcpdump (only that lot of frames are missing)
>>>
>>> 1) Are you capturing part of the payload only (like tcpdump -s 128)
>>>
>>> 2) What is the setup.
>>>
>>> 3) tc -s -d qdisc
>> If you use FQ in the guest, then it could be that high resolution timers
>> have high latency ?
>>
>> So FQ arms short timers, but effective duration could be much longer.
>>
>> Here I get a smooth latency of up to ~3 us
>>
>> lpq83:~# ./netperf -H lpq84 ; ./tc -s -d qd ; dmesg | tail -n1
>> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpq84.prod.google.com () port 0 AF_INET
>> Recv Send Send
>> Socket Socket Message Elapsed
>> Size Size Size Time Throughput
>> bytes bytes bytes secs. 10^6bits/sec
>>
>> 87380 16384 16384 10.00 9410.82
>> qdisc fq 8005: dev eth0 root refcnt 32 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140
>> Sent 50545633991 bytes 33385894 pkt (dropped 0, overlimits 0 requeues 19)
>> rate 9258Mbit 764335pps backlog 0b 0p requeues 19
>> 117 flow, 115 inactive, 0 throttled
>> 0 gc, 0 highprio, 0 retrans, 96861 throttled, 0 flows_plimit
>> [ 572.551664] latency = 3035 ns
>>
>>
>> What do you get with this debugging patch ?
>>
>> diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
>> index 32ad015..c1312a0 100644
>> --- a/net/sched/sch_fq.c
>> +++ b/net/sched/sch_fq.c
>> @@ -103,6 +103,7 @@ struct fq_sched_data {
>> u64 stat_internal_packets;
>> u64 stat_tcp_retrans;
>> u64 stat_throttled;
>> + s64 slatency;
>> u64 stat_flows_plimit;
>> u64 stat_pkts_too_long;
>> u64 stat_allocation_errors;
>> @@ -393,6 +394,7 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch)
>> static void fq_check_throttled(struct fq_sched_data *q, u64 now)
>> {
>> struct rb_node *p;
>> + bool first = true;
>>
>> if (q->time_next_delayed_flow > now)
>> return;
>> @@ -405,6 +407,13 @@ static void fq_check_throttled(struct fq_sched_data *q, u64 now)
>> q->time_next_delayed_flow = f->time_next_packet;
>> break;
>> }
>> + if (first) {
>> + s64 delay = now - f->time_next_packet;
>> +
>> + first = false;
>> + delay -= q->slatency >> 3;
>> + q->slatency += delay;
>> + }
>> rb_erase(p, &q->delayed);
>> q->throttled_flows--;
>> fq_flow_add_tail(&q->old_flows, f);
>> @@ -711,6 +720,7 @@ static int fq_dump(struct Qdisc *sch, struct sk_buff *skb)
>> if (opts == NULL)
>> goto nla_put_failure;
>>
>> + pr_err("latency = %lld ns\n", q->slatency >> 3);
>> if (nla_put_u32(skb, TCA_FQ_PLIMIT, sch->limit) ||
>> nla_put_u32(skb, TCA_FQ_FLOW_PLIMIT, q->flow_plimit) ||
>> nla_put_u32(skb, TCA_FQ_QUANTUM, q->quantum) ||
>>
>
> BTW what is your HZ value ?
Guest HZ is 1000.
>
> We have a problem in TCP stack, because srtt is in HZ units.
>
> Before we change to us units, I guess tcp_update_pacing_rate() should be
> changed a bit if HZ=250
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler
2013-09-05 3:41 ` Eric Dumazet
@ 2013-09-05 5:16 ` Jason Wang
0 siblings, 0 replies; 18+ messages in thread
From: Jason Wang @ 2013-09-05 5:16 UTC (permalink / raw)
To: Eric Dumazet
Cc: David Miller, netdev, Yuchung Cheng, Neal Cardwell,
Michael S. Tsirkin
On 09/05/2013 11:41 AM, Eric Dumazet wrote:
> On Thu, 2013-09-05 at 11:07 +0800, Jason Wang wrote:
>
>> > tc -s -d qdisc
>> > qdisc fq 8001: dev eth0 root refcnt 2 [Unknown qdisc, optlen=64]
>> > Sent 6680760347 bytes 4431855 pkt (dropped 0, overlimits 0 requeues 0)
>> > backlog 0b 0p requeues 0
>> >
>> > btw, is the fq support for tc merged into iproute2? Looks like I can't
>> > find them.
> Yep its there on net-next-3.11 branch
Thanks. With the tc in this branch, after test I get:
./tc -s -d qdisc
qdisc fq 8001: dev eth0 root refcnt 2 limit 10000p flow_limit 100p
buckets 1024 quantum 3028 initial_quantum 15140
Sent 3545374119 bytes 2341785 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
5 flows (0 inactive, 0 throttled)
0 gc, 37 highprio, 51805 throttled
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2013-09-05 5:16 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-08-29 22:49 [PATCH v2 net-next] pkt_sched: fq: Fair Queue packet scheduler Eric Dumazet
2013-08-30 1:47 ` David Miller
2013-08-30 2:30 ` [PATCH iproute2] " Eric Dumazet
2013-09-03 15:49 ` Stephen Hemminger
2013-09-04 5:26 ` [PATCH v2 net-next] " Jason Wang
2013-09-04 5:59 ` Eric Dumazet
2013-09-04 6:30 ` Jason Wang
2013-09-04 10:30 ` Eric Dumazet
2013-09-04 11:27 ` Eric Dumazet
2013-09-04 11:59 ` Daniel Borkmann
2013-09-05 3:39 ` Jason Wang
2013-09-05 0:50 ` Eric Dumazet
2013-09-05 1:23 ` Eric Dumazet
2013-09-05 3:43 ` Jason Wang
2013-09-05 3:34 ` Jason Wang
2013-09-05 3:07 ` Jason Wang
2013-09-05 3:41 ` Eric Dumazet
2013-09-05 5:16 ` Jason Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).