* RE: [Intel-wired-lan] [next-queue PATCH v7 09/10] igb: Add the skeletons for tc-flower offloading
From: Brown, Aaron F @ 2018-04-14 2:23 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-10-vinicius.gomes@intel.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Sanchez-Palencia, Jesus <jesus.sanchez-
> palencia@intel.com>
> Subject: [Intel-wired-lan] [next-queue PATCH v7 09/10] igb: Add the
> skeletons for tc-flower offloading
>
> This adds basic functions needed to implement offloading for filters
> created by tc-flower.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 66
> +++++++++++++++++++++++
> 1 file changed, 66 insertions(+)
>
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [next-queue PATCH v7 08/10] igb: Add MAC address support for ethtool nftuple filters
From: Brown, Aaron F @ 2018-04-14 2:22 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: Gomes, Vinicius, Kirsher, Jeffrey T, netdev@vger.kernel.org,
Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-9-vinicius.gomes@intel.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: Gomes, Vinicius <vinicius.gomes@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; netdev@vger.kernel.org; Sanchez-Palencia,
> Jesus <jesus.sanchez-palencia@intel.com>
> Subject: [next-queue PATCH v7 08/10] igb: Add MAC address support for
> ethtool nftuple filters
>
> This adds the capability of configuring the queue steering of arriving
> packets based on their source and destination MAC addresses.
>
> Source address steering (i.e. driving traffic to a specific queue),
> for the i210, does not work, but filtering does (i.e. accepting
> traffic based on the source address). So, trying to add a filter
> specifying only a source address will be an error.
>
> In practical terms this adds support for the following use cases,
> characterized by these examples:
>
> $ ethtool -N eth0 flow-type ether dst aa:aa:aa:aa:aa:aa action 0
> (this will direct packets with destination address "aa:aa:aa:aa:aa:aa"
> to the RX queue 0)
>
> $ ethtool -N eth0 flow-type ether src 44:44:44:44:44:44 \
> proto 0x22f0 action 3
> (this will direct packets with source address "44:44:44:44:44:44" and
> ethertype 0x22f0 to the RX queue 3)
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_ethtool.c | 43 ++++++++++++++++++--
> 1 file changed, 39 insertions(+), 4 deletions(-)
>
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [next-queue PATCH v7 07/10] igb: Enable nfc filters to specify MAC addresses
From: Brown, Aaron F @ 2018-04-14 2:22 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: Gomes, Vinicius, Kirsher, Jeffrey T, netdev@vger.kernel.org,
Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-8-vinicius.gomes@intel.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: Gomes, Vinicius <vinicius.gomes@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; netdev@vger.kernel.org; Sanchez-Palencia,
> Jesus <jesus.sanchez-palencia@intel.com>
> Subject: [next-queue PATCH v7 07/10] igb: Enable nfc filters to specify MAC
> addresses
>
> This allows igb_add_filter()/igb_erase_filter() to work on filters
> that include MAC addresses (both source and destination).
>
> For now, this only exposes the functionality, the next commit glues
> ethtool into this. Later in this series, these APIs are used to allow
> offloading of cls_flower filters.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb.h | 4 +++
> drivers/net/ethernet/intel/igb/igb_ethtool.c | 28 ++++++++++++++++++++
> 2 files changed, 32 insertions(+)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [Intel-wired-lan] [next-queue PATCH v7 06/10] igb: Allow filters to be added for the local MAC address
From: Brown, Aaron F @ 2018-04-14 2:20 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-7-vinicius.gomes@intel.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Sanchez-Palencia, Jesus <jesus.sanchez-
> palencia@intel.com>
> Subject: [Intel-wired-lan] [next-queue PATCH v7 06/10] igb: Allow filters to
> be added for the local MAC address
>
> Users expect that when adding a steering filter for the local MAC
> address, that all the traffic directed to that address will go to some
> queue.
>
> Currently, it's not possible to configure entries in the "in use"
> state, which is the normal state of the local MAC address entry (it is
> the default), this patch allows to override the steering configuration
> of "in use" entries, if the filter to be added match the address and
> address type (source or destination) of an existing entry.
>
> There is a bit of a special handling for entries referring to the
> local MAC address, when they are removed, only the steering
> configuration is reset.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 40 ++++++++++++++++++++---
> 1 file changed, 36 insertions(+), 4 deletions(-)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [Intel-wired-lan] [next-queue PATCH v7 05/10] igb: Add support for enabling queue steering in filters
From: Brown, Aaron F @ 2018-04-14 2:19 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-6-vinicius.gomes@intel.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Sanchez-Palencia, Jesus <jesus.sanchez-
> palencia@intel.com>
> Subject: [Intel-wired-lan] [next-queue PATCH v7 05/10] igb: Add support for
> enabling queue steering in filters
>
> On some igb models (82575 and i210) the MAC address filters can
> control to which queue the packet will be assigned.
>
> This extends the 'state' with one more state to signify that queue
> selection should be enabled for that filter.
>
> As 82575 parts are no longer easily obtained (and this was developed
> against i210), only support for the i210 model is enabled.
>
> These functions are exported and will be used in the next patch.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> .../net/ethernet/intel/igb/e1000_defines.h | 1 +
> drivers/net/ethernet/intel/igb/igb.h | 6 +++++
> drivers/net/ethernet/intel/igb/igb_main.c | 26 +++++++++++++++++++
> 3 files changed, 33 insertions(+)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [next-queue PATCH v7 04/10] igb: Add support for MAC address filters specifying source addresses
From: Brown, Aaron F @ 2018-04-14 2:19 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: Gomes, Vinicius, Kirsher, Jeffrey T, netdev@vger.kernel.org,
Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-5-vinicius.gomes@intel.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: Gomes, Vinicius <vinicius.gomes@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; netdev@vger.kernel.org; Sanchez-Palencia,
> Jesus <jesus.sanchez-palencia@intel.com>
> Subject: [next-queue PATCH v7 04/10] igb: Add support for MAC address
> filters specifying source addresses
>
> Makes it possible to direct packets to queues based on their source
> address. Documents the expected usage of the 'flags' parameter.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> .../net/ethernet/intel/igb/e1000_defines.h | 1 +
> drivers/net/ethernet/intel/igb/igb.h | 1 +
> drivers/net/ethernet/intel/igb/igb_main.c | 40 ++++++++++++++++---
> 3 files changed, 37 insertions(+), 5 deletions(-)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [Intel-wired-lan] [next-queue PATCH v7 03/10] igb: Enable the hardware traffic class feature bit for igb models
From: Brown, Aaron F @ 2018-04-14 2:18 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-4-vinicius.gomes@intel.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Sanchez-Palencia, Jesus <jesus.sanchez-
> palencia@intel.com>
> Subject: [Intel-wired-lan] [next-queue PATCH v7 03/10] igb: Enable the
> hardware traffic class feature bit for igb models
>
> This will allow functionality depending on the hardware being traffic
> class aware to work. In particular the tc-flower offloading checks
> verifies that this bit is set.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 3 +++
> 1 file changed, 3 insertions(+)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [Intel-wired-lan] [next-queue PATCH v7 02/10] igb: Fix queue selection on MAC filters on i210
From: Brown, Aaron F @ 2018-04-14 2:17 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-3-vinicius.gomes@intel.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; Sanchez-Palencia, Jesus <jesus.sanchez-
> palencia@intel.com>
> Subject: [Intel-wired-lan] [next-queue PATCH v7 02/10] igb: Fix queue
> selection on MAC filters on i210
>
> On the RAH registers there are semantic differences on the meaning of
> the "queue" parameter for traffic steering depending on the controller
> model: there is the 82575 meaning, which "queue" means a RX Hardware
> Queue, and the i350 meaning, where it is a reception pool.
>
> The previous behaviour was having no effect for i210 based controllers
> because the QSEL bit of the RAH register wasn't being set.
>
> This patch separates the condition in discrete cases, so the different
> handling is clearer.
>
> Fixes: 83c21335c876 ("igb: improve MAC filter handling")
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 13 +++++++++----
> 1 file changed, 9 insertions(+), 4 deletions(-)
>
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [next-queue PATCH v7 01/10] igb: Fix not adding filter elements to the list
From: Brown, Aaron F @ 2018-04-14 2:16 UTC (permalink / raw)
To: Gomes, Vinicius, intel-wired-lan@lists.osuosl.org
Cc: Gomes, Vinicius, Kirsher, Jeffrey T, netdev@vger.kernel.org,
Sanchez-Palencia, Jesus
In-Reply-To: <20180410174959.18757-2-vinicius.gomes@intel.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Vinicius Costa Gomes
> Sent: Tuesday, April 10, 2018 10:50 AM
> To: intel-wired-lan@lists.osuosl.org
> Cc: Gomes, Vinicius <vinicius.gomes@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; netdev@vger.kernel.org; Sanchez-Palencia,
> Jesus <jesus.sanchez-palencia@intel.com>
> Subject: [next-queue PATCH v7 01/10] igb: Fix not adding filter elements to
> the list
>
> Because the order of the parameters passes to 'hlist_add_behind()' was
> inverted, the 'parent' node was added "behind" the 'input', as input
> is not in the list, this causes the 'input' node to be lost.
>
> Fixes: 0e71def25281 ("igb: add support of RX network flow classification")
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_ethtool.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
Tested by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* Re: tcp hang when socket fills up ?
From: Dominique Martinet @ 2018-04-14 1:55 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Michal Kubecek, netdev
In-Reply-To: <863dc88d-bc91-d3b6-70c6-5e279451f66d@gmail.com>
Eric Dumazet wrote on Fri, Apr 13, 2018:
> Ah sorry, your trace was truncated, we need more packets _before_ the excerpt.
Ah, sorry as well. I tried to go far back enough to include the replayed
packets, but I see it didn't include the original ack for that packet.
I'm resending both full traces from connect to fin this time.
> That might be caused by some TS val/ecr breakage :
>
> Many acks were received by the server tcpdump,
> but none of them was accepted by TCP stack, for some reason.
>
> Try to disable TCP timestamps, it will give some hint if bug does not reproduce.
I unfortunately cannot reproduce on a local network, but will give this
a try on Monday and report. (sysctl net.ipv4.tcp_timestamps=0, correct?)
Here's the traces again, server-capture first:
16:49:22.374452 IP <server local ip>.13317 > <client public ip>.31872: Flags [S], seq 2026966826, win 29200, options [mss 1460,sackOK,TS val 1313933281 ecr 0,nop,wscale 7], length 0
16:49:22.700553 IP <client public ip>.31872 > <server local ip>.13317: Flags [S], seq 882075258, win 29200, options [mss 1386,sackOK,TS val 1617125446 ecr 0,nop,wscale 7], length 0
16:49:22.700577 IP <server local ip>.13317 > <client public ip>.31872: Flags [S.], seq 2026966826, ack 882075259, win 29200, options [mss 1460,sackOK,TS val 1313933607 ecr 1617125446,nop,wscale 7], length 0
16:49:22.727131 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 0
16:49:22.729193 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 21
16:49:22.729230 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 22, win 229, options [nop,nop,TS val 1313933635 ecr 1617125472], length 0
16:49:22.732726 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1:22, ack 22, win 229, options [nop,nop,TS val 1313933639 ecr 1617125472], length 21
16:49:22.759809 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 22, win 229, options [nop,nop,TS val 1617125503 ecr 1313933639], length 0
16:49:22.759835 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 22:734, ack 22, win 229, options [nop,nop,TS val 1313933666 ecr 1617125503], length 712
16:49:22.761643 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 22:1270, ack 22, win 229, options [nop,nop,TS val 1617125504 ecr 1313933639], length 1248
16:49:22.801860 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1270, win 248, options [nop,nop,TS val 1313933708 ecr 1617125504], length 0
16:49:22.826902 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 734, win 240, options [nop,nop,TS val 1617125572 ecr 1313933666], length 0
16:49:22.827576 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1270:1318, ack 734, win 240, options [nop,nop,TS val 1617125573 ecr 1313933708], length 48
16:49:22.827600 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1318, win 248, options [nop,nop,TS val 1313933734 ecr 1617125573], length 0
16:49:22.833028 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 734:1122, ack 1318, win 248, options [nop,nop,TS val 1313933739 ecr 1617125573], length 388
16:49:22.858455 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1122, win 251, options [nop,nop,TS val 1617125604 ecr 1313933739], length 0
16:49:22.865866 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1318:1334, ack 1122, win 251, options [nop,nop,TS val 1617125612 ecr 1313933739], length 16
16:49:22.906865 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1334, win 248, options [nop,nop,TS val 1313933813 ecr 1617125612], length 0
16:49:22.944474 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1334:1386, ack 1122, win 251, options [nop,nop,TS val 1617125678 ecr 1313933813], length 52
16:49:22.944497 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 0
16:49:22.944747 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1122:1174, ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 52
16:49:22.971083 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1386:1454, ack 1174, win 251, options [nop,nop,TS val 1617125716 ecr 1313933851], length 68
16:49:22.971607 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1174:1258, ack 1454, win 248, options [nop,nop,TS val 1313933878 ecr 1617125716], length 84
16:49:22.998201 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 1454:2082, ack 1258, win 251, options [nop,nop,TS val 1617125742 ecr 1313933878], length 628
16:49:22.998987 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1258:1838, ack 2082, win 268, options [nop,nop,TS val 1313933905 ecr 1617125742], length 580
16:49:23.065102 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1838, win 262, options [nop,nop,TS val 1617125810 ecr 1313933905], length 0
16:49:24.811297 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 2082:3254, ack 1838, win 262, options [nop,nop,TS val 1617127527 ecr 1313933905], length 1172
16:49:24.812562 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1838:1874, ack 3254, win 287, options [nop,nop,TS val 1313935719 ecr 1617127527], length 36
16:49:24.838210 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 1874, win 262, options [nop,nop,TS val 1617127583 ecr 1313935719], length 0
16:49:24.838236 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 1874:2534, ack 3254, win 287, options [nop,nop,TS val 1313935744 ecr 1617127583], length 660
16:49:24.839175 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3254:3374, ack 1874, win 262, options [nop,nop,TS val 1617127584 ecr 1313935719], length 120
16:49:24.879860 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 3374, win 287, options [nop,nop,TS val 1313935786 ecr 1617127584], length 0
16:49:24.906889 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2534, win 273, options [nop,nop,TS val 1617127652 ecr 1313935744], length 0
16:49:24.906916 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2534:2586, ack 3374, win 287, options [nop,nop,TS val 1313935813 ecr 1617127652], length 52
16:49:24.932283 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 0
16:49:24.933286 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3374:3902, ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 528
16:49:24.933314 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], ack 3902, win 307, options [nop,nop,TS val 1313935839 ecr 1617127678], length 0
16:49:24.934315 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2586:2694, ack 3902, win 307, options [nop,nop,TS val 1313935840 ecr 1617127678], length 108
16:49:24.999902 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2694, win 273, options [nop,nop,TS val 1617127745 ecr 1313935840], length 0
16:49:24.999927 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2694:2862, ack 3902, win 307, options [nop,nop,TS val 1313935906 ecr 1617127745], length 168
16:49:25.025121 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 2862, win 284, options [nop,nop,TS val 1617127771 ecr 1313935906], length 0
16:49:25.732946 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3902:3938, ack 2862, win 284, options [nop,nop,TS val 1617128391 ecr 1313935906], length 36
16:49:25.733531 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2862:2898, ack 3938, win 307, options [nop,nop,TS val 1313936640 ecr 1617128391], length 36
16:49:25.763138 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3938:3974, ack 2898, win 284, options [nop,nop,TS val 1617128507 ecr 1313936640], length 36
16:49:25.763721 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2898:2934, ack 3974, win 307, options [nop,nop,TS val 1313936670 ecr 1617128507], length 36
16:49:25.832807 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 3974:4010, ack 2934, win 284, options [nop,nop,TS val 1617128576 ecr 1313936670], length 36
16:49:25.833392 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2934:2970, ack 4010, win 307, options [nop,nop,TS val 1313936740 ecr 1617128576], length 36
16:49:25.873065 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4010:4046, ack 2970, win 284, options [nop,nop,TS val 1617128617 ecr 1313936740], length 36
16:49:25.873608 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 2970:3006, ack 4046, win 307, options [nop,nop,TS val 1313936780 ecr 1617128617], length 36
16:49:25.942195 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3006, win 284, options [nop,nop,TS val 1617128687 ecr 1313936780], length 0
16:49:26.244880 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4046:4082, ack 3006, win 284, options [nop,nop,TS val 1617128954 ecr 1313936780], length 36
16:49:26.245455 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3006:3042, ack 4082, win 307, options [nop,nop,TS val 1313937152 ecr 1617128954], length 36
16:49:26.270697 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3042, win 284, options [nop,nop,TS val 1617129016 ecr 1313937152], length 0
16:49:26.289989 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4082:4118, ack 3042, win 284, options [nop,nop,TS val 1617129035 ecr 1313937152], length 36
16:49:26.290569 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3042:3078, ack 4118, win 307, options [nop,nop,TS val 1313937197 ecr 1617129035], length 36
16:49:26.357222 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3078, win 284, options [nop,nop,TS val 1617129102 ecr 1313937197], length 0
16:49:26.409189 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4118:4154, ack 3078, win 284, options [nop,nop,TS val 1617129154 ecr 1313937197], length 36
16:49:26.409773 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3078:3114, ack 4154, win 307, options [nop,nop,TS val 1313937316 ecr 1617129154], length 36
16:49:26.435224 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3114, win 284, options [nop,nop,TS val 1617129181 ecr 1313937316], length 0
16:49:26.654517 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4154:4190, ack 3114, win 284, options [nop,nop,TS val 1617129290 ecr 1313937316], length 36
16:49:26.654979 IP <server local ip>.13317 > <client public ip>.31872: Flags [P.], seq 3114:3150, ack 4190, win 307, options [nop,nop,TS val 1313937561 ecr 1617129290], length 36
16:49:26.664510 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 3150:4524, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664538 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 4524:5898, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664645 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 5898:7272, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.664648 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 7272:8646, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.665977 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 8646:10020, ack 4190, win 307, options [nop,nop,TS val 1313937572 ecr 1617129290], length 1374
16:49:26.666414 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 10020:11394, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.666424 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 11394:12768, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.666427 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 12768:14142, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.667536 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 14142:15516, ack 4190, win 307, options [nop,nop,TS val 1313937574 ecr 1617129290], length 1374
16:49:26.684298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 3150, win 284, options [nop,nop,TS val 1617129430 ecr 1313937561], length 0
16:49:26.684314 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 15516:16890, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.684320 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 16890:18264, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.691620 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 4524, win 307, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.691635 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 18264:19638, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.691642 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 19638:21012, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692236 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 5898, win 329, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.692250 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 21012:22386, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692254 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 22386:23760, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.692551 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 7272, win 352, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.692954 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 23760:25134, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.692958 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 25134:26508, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.693801 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 8646, win 375, options [nop,nop,TS val 1617129438 ecr 1313937571], length 0
16:49:26.694488 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 10020, win 397, options [nop,nop,TS val 1617129439 ecr 1313937572], length 0
16:49:26.694920 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 26508:27882, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.694925 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 27882:29256, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.695593 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 29256:30630, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.695597 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 30630:32004, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.696500 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 11394, win 420, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.697258 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 12768, win 443, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.697576 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 14142, win 465, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.698057 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 15516, win 488, options [nop,nop,TS val 1617129440 ecr 1313937574], length 0
16:49:26.700531 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.700536 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 33378:34752, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.701255 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 34752:36126, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.701260 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 36126:37500, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.702122 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 37500:38874, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.702128 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 38874:40248, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.703142 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.703148 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.710201 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 16890, win 510, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.710214 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.710218 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.710568 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 18264, win 533, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.710577 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.710580 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.720991 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 19638, win 556, options [nop,nop,TS val 1617129464 ecr 1313937598], length 0
16:49:26.721006 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.721011 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.722681 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 21012, win 578, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.722692 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.722696 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723268 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 22386, win 601, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.723280 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723284 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.723597 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 23760, win 624, options [nop,nop,TS val 1617129467 ecr 1313937598], length 0
16:49:26.724171 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.724175 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.724457 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 25134, win 646, options [nop,nop,TS val 1617129467 ecr 1313937599], length 0
16:49:26.726081 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 26508, win 669, options [nop,nop,TS val 1617129471 ecr 1313937599], length 0
16:49:26.726391 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 27882, win 691, options [nop,nop,TS val 1617129471 ecr 1313937601], length 0
16:49:26.726962 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 29256, win 714, options [nop,nop,TS val 1617129472 ecr 1313937601], length 0
16:49:26.727614 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 30630, win 737, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.728084 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.728507 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.728511 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.729531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.730002 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.730340 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.730344 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.731398 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.731402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.731634 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.732955 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.732963 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.733956 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 37500, win 850, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.734242 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 38874, win 872, options [nop,nop,TS val 1617129477 ecr 1313937608], length 0
16:49:26.734653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 40248, win 895, options [nop,nop,TS val 1617129478 ecr 1313937608], length 0
16:49:26.735042 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735046 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.735334 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736005 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.736402 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.736408 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.738561 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739539 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.739882 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740255 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.746756 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.747923 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.749083 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.750171 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.750808 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.754648 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.755985 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.758513 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.759096 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759421 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.759755 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.760653 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.761453 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.762199 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.763547 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.763553 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.764298 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.764676 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.807754 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.876467 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.051791 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.076444 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.371182 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.519862 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.547662 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.883372 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.511861 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.538891 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.907197 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.431864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.459127 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.955388 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.207879 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.235726 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.256285 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.143864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.171531 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.448262 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
16:49:57.503875 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313968410 ecr 1617129473], length 1374
16:49:57.533167 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617160279 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:59.832712 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617162530 ecr 1313937714], length 36
16:50:27.711882 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313998617 ecr 1617129473], length 1374
16:50:27.745163 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617190488 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:50:34.649363 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617197346 ecr 1313937714], length 36
16:51:29.151870 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314060057 ecr 1617129473], length 1374
16:51:29.186204 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617251927 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:51:40.186590 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617262882 ecr 1313937714], length 36
16:53:32.031867 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314182936 ecr 1617129473], length 1374
16:53:32.062310 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617374807 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:53:43.016872 IP <client public ip>.31872 > <server local ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617385762 ecr 1313937714], length 36
16:54:30.276051 IP <client public ip>.31872 > <server local ip>.13317: Flags [FP.], seq 4226:5390, ack 77346, win 1444, options [nop,nop,TS val 1617432931 ecr 1313937714], length 1164
16:55:34.911864 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314305815 ecr 1617129473], length 1374
16:55:34.941093 IP <client public ip>.31872 > <server local ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617497688 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
and client side
16:49:22.693949 IP <client local ip>.31872 > <server public ip>.13317: Flags [S], seq 882075258, win 29200, options [mss 1460,sackOK,TS val 1617125446 ecr 0,nop,wscale 7], length 0
16:49:22.720391 IP <server public ip>.13317 > <client local ip>.31872: Flags [S.], seq 2026966826, ack 882075259, win 29200, options [mss 1386,sackOK,TS val 1313933607 ecr 1617125446,nop,wscale 7], length 0
16:49:22.720437 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 0
16:49:22.720502 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1617125472 ecr 1313933607], length 21
16:49:22.748570 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 22, win 229, options [nop,nop,TS val 1313933635 ecr 1617125472], length 0
16:49:22.751807 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1:22, ack 22, win 229, options [nop,nop,TS val 1313933639 ecr 1617125472], length 21
16:49:22.751823 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 22, win 229, options [nop,nop,TS val 1617125503 ecr 1313933639], length 0
16:49:22.752235 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 22:1270, ack 22, win 229, options [nop,nop,TS val 1617125504 ecr 1313933639], length 1248
16:49:22.779858 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 22:734, ack 22, win 229, options [nop,nop,TS val 1313933666 ecr 1617125503], length 712
16:49:22.820776 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 734, win 240, options [nop,nop,TS val 1617125572 ecr 1313933666], length 0
16:49:22.820910 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1270, win 248, options [nop,nop,TS val 1313933708 ecr 1617125504], length 0
16:49:22.820923 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1270:1318, ack 734, win 240, options [nop,nop,TS val 1617125573 ecr 1313933708], length 48
16:49:22.846614 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1318, win 248, options [nop,nop,TS val 1313933734 ecr 1617125573], length 0
16:49:22.852230 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 734:1122, ack 1318, win 248, options [nop,nop,TS val 1313933739 ecr 1617125573], length 388
16:49:22.852249 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1122, win 251, options [nop,nop,TS val 1617125604 ecr 1313933739], length 0
16:49:22.859964 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1318:1334, ack 1122, win 251, options [nop,nop,TS val 1617125612 ecr 1313933739], length 16
16:49:22.926230 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1334, win 248, options [nop,nop,TS val 1313933813 ecr 1617125612], length 0
16:49:22.926256 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1334:1386, ack 1122, win 251, options [nop,nop,TS val 1617125678 ecr 1313933813], length 52
16:49:22.964007 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 0
16:49:22.964240 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1122:1174, ack 1386, win 248, options [nop,nop,TS val 1313933851 ecr 1617125678], length 52
16:49:22.964367 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1386:1454, ack 1174, win 251, options [nop,nop,TS val 1617125716 ecr 1313933851], length 68
16:49:22.990580 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1174:1258, ack 1454, win 248, options [nop,nop,TS val 1313933878 ecr 1617125716], length 84
16:49:22.990705 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 1454:2082, ack 1258, win 251, options [nop,nop,TS val 1617125742 ecr 1313933878], length 628
16:49:23.018404 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1258:1838, ack 2082, win 268, options [nop,nop,TS val 1313933905 ecr 1617125742], length 580
16:49:23.058783 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1838, win 262, options [nop,nop,TS val 1617125810 ecr 1313933905], length 0
16:49:24.775119 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 2082:3254, ack 1838, win 262, options [nop,nop,TS val 1617127527 ecr 1313933905], length 1172
16:49:24.831833 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1838:1874, ack 3254, win 287, options [nop,nop,TS val 1313935719 ecr 1617127527], length 36
16:49:24.831862 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 1874, win 262, options [nop,nop,TS val 1617127583 ecr 1313935719], length 0
16:49:24.832031 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3254:3374, ack 1874, win 262, options [nop,nop,TS val 1617127584 ecr 1313935719], length 120
16:49:24.859918 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 1874:2534, ack 3254, win 287, options [nop,nop,TS val 1313935744 ecr 1617127583], length 660
16:49:24.898914 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 3374, win 287, options [nop,nop,TS val 1313935786 ecr 1617127584], length 0
16:49:24.900776 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2534, win 273, options [nop,nop,TS val 1617127652 ecr 1313935744], length 0
16:49:24.926271 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2534:2586, ack 3374, win 287, options [nop,nop,TS val 1313935813 ecr 1617127652], length 52
16:49:24.926284 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 0
16:49:24.926452 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3374:3902, ack 2586, win 273, options [nop,nop,TS val 1617127678 ecr 1313935813], length 528
16:49:24.952442 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], ack 3902, win 307, options [nop,nop,TS val 1313935839 ecr 1617127678], length 0
16:49:24.953230 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2586:2694, ack 3902, win 307, options [nop,nop,TS val 1313935840 ecr 1617127678], length 108
16:49:24.993782 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2694, win 273, options [nop,nop,TS val 1617127745 ecr 1313935840], length 0
16:49:25.019012 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2694:2862, ack 3902, win 307, options [nop,nop,TS val 1313935906 ecr 1617127745], length 168
16:49:25.019033 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 2862, win 284, options [nop,nop,TS val 1617127771 ecr 1313935906], length 0
16:49:25.639817 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3902:3938, ack 2862, win 284, options [nop,nop,TS val 1617128391 ecr 1313935906], length 36
16:49:25.754886 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2862:2898, ack 3938, win 307, options [nop,nop,TS val 1313936640 ecr 1617128391], length 36
16:49:25.754915 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3938:3974, ack 2898, win 284, options [nop,nop,TS val 1617128507 ecr 1313936640], length 36
16:49:25.782855 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2898:2934, ack 3974, win 307, options [nop,nop,TS val 1313936670 ecr 1617128507], length 36
16:49:25.824544 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 3974:4010, ack 2934, win 284, options [nop,nop,TS val 1617128576 ecr 1313936670], length 36
16:49:25.852650 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2934:2970, ack 4010, win 307, options [nop,nop,TS val 1313936740 ecr 1617128576], length 36
16:49:25.865736 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4010:4046, ack 2970, win 284, options [nop,nop,TS val 1617128617 ecr 1313936740], length 36
16:49:25.893292 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 2970:3006, ack 4046, win 307, options [nop,nop,TS val 1313936780 ecr 1617128617], length 36
16:49:25.935768 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3006, win 284, options [nop,nop,TS val 1617128687 ecr 1313936780], length 0
16:49:26.202625 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4046:4082, ack 3006, win 284, options [nop,nop,TS val 1617128954 ecr 1313936780], length 36
16:49:26.264448 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3006:3042, ack 4082, win 307, options [nop,nop,TS val 1313937152 ecr 1617128954], length 36
16:49:26.264480 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3042, win 284, options [nop,nop,TS val 1617129016 ecr 1313937152], length 0
16:49:26.282882 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4082:4118, ack 3042, win 284, options [nop,nop,TS val 1617129035 ecr 1313937152], length 36
16:49:26.309893 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3042:3078, ack 4118, win 307, options [nop,nop,TS val 1313937197 ecr 1617129035], length 36
16:49:26.350729 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3078, win 284, options [nop,nop,TS val 1617129102 ecr 1313937197], length 0
16:49:26.402501 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4118:4154, ack 3078, win 284, options [nop,nop,TS val 1617129154 ecr 1313937197], length 36
16:49:26.428995 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3078:3114, ack 4154, win 307, options [nop,nop,TS val 1313937316 ecr 1617129154], length 36
16:49:26.429016 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3114, win 284, options [nop,nop,TS val 1617129181 ecr 1313937316], length 0
16:49:26.538794 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4154:4190, ack 3114, win 284, options [nop,nop,TS val 1617129290 ecr 1313937316], length 36
16:49:26.678342 IP <server public ip>.13317 > <client local ip>.31872: Flags [P.], seq 3114:3150, ack 4190, win 307, options [nop,nop,TS val 1313937561 ecr 1617129290], length 36
16:49:26.678376 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 3150, win 284, options [nop,nop,TS val 1617129430 ecr 1313937561], length 0
16:49:26.685030 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 3150:4524, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685061 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 4524, win 307, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.685337 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 4524:5898, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685356 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 5898, win 329, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.685709 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 5898:7272, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.685736 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 7272, win 352, options [nop,nop,TS val 1617129437 ecr 1313937571], length 0
16:49:26.686323 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 7272:8646, ack 4190, win 307, options [nop,nop,TS val 1313937571 ecr 1617129290], length 1374
16:49:26.686345 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 8646, win 375, options [nop,nop,TS val 1617129438 ecr 1313937571], length 0
16:49:26.687216 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 8646:10020, ack 4190, win 307, options [nop,nop,TS val 1313937572 ecr 1617129290], length 1374
16:49:26.687225 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 10020, win 397, options [nop,nop,TS val 1617129439 ecr 1313937572], length 0
16:49:26.688495 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 10020:11394, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688507 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 11394, win 420, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688517 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 11394:12768, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688525 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 12768, win 443, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688530 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 12768:14142, ack 4190, win 307, options [nop,nop,TS val 1313937573 ecr 1617129290], length 1374
16:49:26.688535 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 14142, win 465, options [nop,nop,TS val 1617129440 ecr 1313937573], length 0
16:49:26.688834 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 14142:15516, ack 4190, win 307, options [nop,nop,TS val 1313937574 ecr 1617129290], length 1374
16:49:26.688846 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 15516, win 488, options [nop,nop,TS val 1617129440 ecr 1313937574], length 0
16:49:26.704046 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 15516:16890, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.704071 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 16890, win 510, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.704631 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 16890:18264, ack 4190, win 307, options [nop,nop,TS val 1313937590 ecr 1617129430], length 1374
16:49:26.704644 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 18264, win 533, options [nop,nop,TS val 1617129456 ecr 1313937590], length 0
16:49:26.712707 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 18264:19638, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.712738 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 19638, win 556, options [nop,nop,TS val 1617129464 ecr 1313937598], length 0
16:49:26.714041 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 19638:21012, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.714067 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 21012, win 578, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.714639 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 21012:22386, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.714660 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 22386, win 601, options [nop,nop,TS val 1617129466 ecr 1313937598], length 0
16:49:26.714997 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 22386:23760, ack 4190, win 307, options [nop,nop,TS val 1313937598 ecr 1617129437], length 1374
16:49:26.715020 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 23760, win 624, options [nop,nop,TS val 1617129467 ecr 1313937598], length 0
16:49:26.715438 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 23760:25134, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.715462 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 25134, win 646, options [nop,nop,TS val 1617129467 ecr 1313937599], length 0
16:49:26.718975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 25134:26508, ack 4190, win 307, options [nop,nop,TS val 1313937599 ecr 1617129437], length 1374
16:49:26.718992 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 26508, win 669, options [nop,nop,TS val 1617129471 ecr 1313937599], length 0
16:49:26.719636 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 26508:27882, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.719648 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 27882, win 691, options [nop,nop,TS val 1617129471 ecr 1313937601], length 0
16:49:26.720637 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 27882:29256, ack 4190, win 307, options [nop,nop,TS val 1313937601 ecr 1617129439], length 1374
16:49:26.720649 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 29256, win 714, options [nop,nop,TS val 1617129472 ecr 1313937601], length 0
16:49:26.720960 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 29256:30630, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.720972 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 30630, win 737, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.721446 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 30630:32004, ack 4190, win 307, options [nop,nop,TS val 1313937602 ecr 1617129439], length 1374
16:49:26.721459 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 32004, win 759, options [nop,nop,TS val 1617129473 ecr 1313937602], length 0
16:49:26.723143 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.723155 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 33378, win 782, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.723687 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 33378:34752, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.723699 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 34752, win 805, options [nop,nop,TS val 1617129475 ecr 1313937607], length 0
16:49:26.724294 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 34752:36126, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.724306 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 36126, win 827, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.724835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 36126:37500, ack 4190, win 307, options [nop,nop,TS val 1313937607 ecr 1617129440], length 1374
16:49:26.724847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 37500, win 850, options [nop,nop,TS val 1617129476 ecr 1313937607], length 0
16:49:26.724859 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 37500:38874, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.724867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 38874, win 872, options [nop,nop,TS val 1617129477 ecr 1313937608], length 0
16:49:26.726775 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 38874:40248, ack 4190, win 307, options [nop,nop,TS val 1313937608 ecr 1617129440], length 1374
16:49:26.726785 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 40248, win 895, options [nop,nop,TS val 1617129478 ecr 1313937608], length 0
16:49:26.726793 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726798 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.726802 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726807 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.730611 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730619 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.730835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.732371 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732383 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.732393 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732400 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740896 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.740910 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.742220 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.742232 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.743463 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.743476 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.744478 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744490 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.744975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744982 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.748722 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.748733 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.749615 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.749627 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.750248 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.750260 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.750968 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.750980 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.751486 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.751498 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.752095 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752107 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.752920 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752932 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.754573 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.754585 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.755260 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.755271 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.755493 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.756076 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756391 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756402 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.757020 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757349 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.757878 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758457 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758465 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.828268 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.868867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.010733 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.070035 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.070062 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.266769 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.540998 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.541022 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.778687 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.532876 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.532907 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.802879 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.452830 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.452861 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.850734 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.229420 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.229447 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.202854 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.165044 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.165073 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.394773 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
16:49:57.527155 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313968410 ecr 1617129473], length 1374
16:49:57.527184 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617160279 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:59.778790 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617162530 ecr 1313937714], length 36
16:50:27.736331 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313998617 ecr 1617129473], length 1374
16:50:27.736360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617190488 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:50:34.594866 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617197346 ecr 1313937714], length 36
16:51:29.175879 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314060057 ecr 1617129473], length 1374
16:51:29.175917 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617251927 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:51:40.130722 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617262882 ecr 1313937714], length 36
16:53:32.055922 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314182936 ecr 1617129473], length 1374
16:53:32.055963 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617374807 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:53:43.010868 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617385762 ecr 1313937714], length 36
16:54:30.179770 IP <client local ip>.31872 > <server public ip>.13317: Flags [FP.], seq 4226:5390, ack 77346, win 1444, options [nop,nop,TS val 1617432931 ecr 1313937714], length 1164
16:55:34.936179 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1314305815 ecr 1617129473], length 1374
16:55:34.936205 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617497688 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
Thanks,
--
Dominique Martinet | Asmadeus
^ permalink raw reply
* Re: tcp hang when socket fills up ?
From: Eric Dumazet @ 2018-04-14 1:39 UTC (permalink / raw)
To: Dominique Martinet, Michal Kubecek, Eric Dumazet; +Cc: netdev
In-Reply-To: <20180414010907.GA11526@nautica>
On 04/13/2018 06:09 PM, Dominique Martinet wrote:
> Thank you for the replies,
>
> Eric Dumazet wrote on Fri, Apr 13, 2018:
>> There is no way a regular TCP stack (including linux) could send the following frame.
>>
>>> 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
>>
>> So something is mangling the packet, maybe NAT or something.
>
>
> The pcap was produced on the server which emitted the frame, so it
> should be exactly as the server intended it to be without any mangling
> involved?
> If you could point at what strikes you as odd I can check if the same
> happened on other hang traces I might still have. Is it just that it
> replays a very old seq?
> (if it's odd wrt client packets, the same trace as captured on client is
> at the end of this mail)
>
Ah sorry, your trace was truncated, we need more packets _before_ the excerpt.
That might be caused by some TS val/ecr breakage :
Many acks were received by the server tcpdump,
but none of them was accepted by TCP stack, for some reason.
Try to disable TCP timestamps, it will give some hint if bug does not reproduce.
^ permalink raw reply
* Re: Crash due to NULL dereference in tcp_rearm_rto
From: Eric Dumazet @ 2018-04-14 1:14 UTC (permalink / raw)
To: Subash Abhinov Kasiviswanathan, netdev, soheil, eric.dumazet
In-Reply-To: <0e7dbdef0ecf0ee0562a144050165453@codeaurora.org>
On 04/13/2018 05:39 PM, Subash Abhinov Kasiviswanathan wrote:
> We are seeing a warning followed by a crash on an ARM64 device with
> Android 4.14 based kernel.
>
> It looks like both sk->sk_write_queue and sk->sk_send_head are NULL.
> Since the sk->sk_write_queue is NULL and is dereferenced in tcp_rto_delta_us()
> to get the skb->skb_mstamp, there is crash observed.
>
> Since this is 4.14.32, it already has commit ("tcp: reset sk_send_head in tcp_write_queue_purge")
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.14.34&id=dbbf2d1e4077bab0c65ece2765d3fc69cf7d610f
>
> 12876.013077: <6> WARNING: CPU: 5 PID: 14828 at net/ipv4/tcp_output.c:2469 tcp_send_loss_probe+0x198/0x1b8
> 12876.038939: <6> task: ffffffe73f7e5a80 task.stack: ffffff801b068000
> 12876.038941: <2> PC is at tcp_send_loss_probe+0x198/0x1b8
> 12876.038942: <2> LR is at tcp_send_loss_probe+0x28/0x1b8
> 12876.038944: <2> pc : [<ffffff8dc8db16f8>] lr : [<ffffff8dc8db1588>] pstate: 60400145
> 12876.038944: <2> sp : ffffff800802bd30
> 12876.038945: <2> x29: ffffff800802bd50 x28: ffffff8dc9d83eb8
> 12876.038948: <2> x27: ffffff800802be08 x26: ffffff8dc9737000
> 12876.038950: <2> x25: 0000000000000001 x24: ffffffe744ea1728
> 12876.038952: <2> x23: ffffffe73f7e5a80 x22: 0000000000000558
> 12876.038954: <2> x21: 0000000000000000 x20: 0000000001080020
> 12876.038956: <2> x19: ffffffe73d06e440 x18: 0000000000000020
> 12876.038958: <2> x17: 0000000000000014 x16: 0000000000000030
> 12876.038960: <2> x15: 0000000000000000 x14: 0000000000000000
> 12876.038962: <2> x13: 0000000013af314c x12: 0000002773f8a550
> 12876.038965: <2> x11: 0000000000000538 x10: 0000000000000000
> 12876.038967: <2> x9 : 0000000000000020 x8 : ffffffe73d06e5f0
> 12876.038969: <2> x7 : 0000000000924278 x6 : ffffffe76fe9ed80
> 12876.038971: <2> x5 : ffffffe76fe9ed80 x4 : 000000000f500458
> 12876.038973: <2> x3 : ffffff800802bce0 x2 : ffffff800802bce8
> 12876.038975: <2> x1 : 0000000000000000 x0 : 0000000000000558
> 12876.039082: <2> [<ffffff8dc8db16f8>] tcp_send_loss_probe+0x198/0x1b8
> 12876.039084: <2> [<ffffff8dc8db6698>] tcp_write_timer_handler+0xf8/0x1c4
> 12876.039086: <2> [<ffffff8dc8db68cc>] tcp_write_timer+0x5c/0x98
> 12876.039089: <2> [<ffffff8dc8144f10>] call_timer_fn+0xc0/0x1b4
> 12876.039091: <2> [<ffffff8dc8143f68>] run_timer_softirq+0x230/0x850
> 12876.039094: <2> [<ffffff8dc8081b74>] __do_softirq+0x1dc/0x3a4
> 12876.039096: <2> [<ffffff8dc80b8fec>] irq_exit+0xc8/0xd4
> 12876.039098: <2> [<ffffff8dc812a004>] __handle_domain_irq+0x8c/0xc4
> 12876.039099: <2> [<ffffff8dc8081940>] gic_handle_irq+0x164/0x1bc
>
> [net/ipv4/tcp_output.c]
> void tcp_send_loss_probe(struct sock *sk)
> {
> struct tcp_sock *tp = tcp_sk(sk);
> struct sk_buff *skb;
> int pcount;
> int mss = tcp_current_mss(sk);
>
> ...
>
> /* Retransmit last segment. */
> if (WARN_ON(!skb))
> goto rearm_timer;
>
> 12876.043967: <6> Unable to handle kernel NULL pointer dereference at virtual address 00000010
> 12876.091600: <6> Internal error: Oops: 96000005 [#1] PREEMPT SMP
> 12876.152597: <2> PC is at tcp_rearm_rto+0x48/0x90
> 12876.156979: <2> LR is at tcp_send_loss_probe+0x178/0x1b8
> 12876.162077: <2> pc : [<ffffff8dc8da76c4>] lr : [<ffffff8dc8db16d8>] pstate: 60400145
> 12876.169657: <2> sp : ffffff800802bd10
> 12876.173056: <2> x29: ffffff800802bd20 x28: ffffff8dc9d83eb8
> 12876.178511: <2> x27: ffffff800802be08 x26: ffffff8dc9737000
> 12876.183967: <2> x25: 0000000000000001 x24: ffffffe744ea1728
> 12876.189418: <2> x23: ffffffe73f7e5a80 x22: 0000000000000558
> 12876.194863: <2> x21: 0000000000000000 x20: 0000000001080020
> 12876.200312: <2> x19: ffffffe73d06e440 x18: 0000000000000020
> 12876.205758: <2> x17: 0000000000000014 x16: 0000000000000030
> 12876.211212: <2> x15: 0000000000000000 x14: 0000000000000000
> 12876.216660: <2> x13: 0000000013af314c x12: 0000002773f8a550
> 12876.222108: <2> x11: 0000000000000538 x10: 0000000000000000
> 12876.227561: <2> x9 : 0000000000000000 x8 : ffffffe73d06e5f0
> 12876.233008: <2> x7 : 0000000000924278 x6 : ffffffe76fe9ed80
> 12876.238455: <2> x5 : ffffffe76fe9ed80 x4 : 000000000f500458
> 12876.243907: <2> x3 : ffffff800802bce0 x2 : ffffff800802bce8
> 12876.249360: <2> x1 : 0000000000000000 x0 : 0000000000000867
> 12876.473522: <2> [<ffffff8dc8da76c4>] tcp_rearm_rto+0x48/0x90
> 12876.478971: <2> [<ffffff8dc8db16d8>] tcp_send_loss_probe+0x178/0x1b8
> 12876.485131: <2> [<ffffff8dc8db6698>] tcp_write_timer_handler+0xf8/0x1c4
> 12876.491557: <2> [<ffffff8dc8db68cc>] tcp_write_timer+0x5c/0x98
> 12876.497189: <2> [<ffffff8dc8144f10>] call_timer_fn+0xc0/0x1b4
> 12876.502731: <2> [<ffffff8dc8143f68>] run_timer_softirq+0x230/0x850
> 12876.508716: <2> [<ffffff8dc8081b74>] __do_softirq+0x1dc/0x3a4
> 12876.514260: <2> [<ffffff8dc80b8fec>] irq_exit+0xc8/0xd4
> 12876.519261: <2> [<ffffff8dc812a004>] __handle_domain_irq+0x8c/0xc4
> 12876.525245: <2> [<ffffff8dc8081940>] gic_handle_irq+0x164/0x1bc
>
> [net/ipv4/tcp_input.c]
> void tcp_rearm_rto(struct sock *sk)
> {
> ...
> inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS);
> } else {
> u32 rto = inet_csk(sk)->icsk_rto;
> /* Offset the time elapsed after installing regular RTO */
> if (icsk->icsk_pending == ICSK_TIME_REO_TIMEOUT ||
> icsk->icsk_pending == ICSK_TIME_LOSS_PROBE) {
> s64 delta_us = tcp_rto_delta_us(sk);
> /* delta_us may not be positive if the socket is locked
> * when the retrans timer fires and is rescheduled.
> */
> rto = usecs_to_jiffies(max_t(int, delta_us, 1));
>
>
> [include/net/tcp.h]
> static inline s64 tcp_rto_delta_us(const struct sock *sk)
> {
> const struct sk_buff *skb = tcp_write_queue_head(sk);
> u32 rto = inet_csk(sk)->icsk_rto;
> u64 rto_time_stamp_us = skb->skb_mstamp + jiffies_to_usecs(rto);
>
> return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
> }
>
We probably need to clear tp->packets_out :/
^ permalink raw reply
* Re: tcp hang when socket fills up ?
From: Dominique Martinet @ 2018-04-14 1:09 UTC (permalink / raw)
To: Michal Kubecek, Eric Dumazet; +Cc: netdev
In-Reply-To: <20180413163254.ucyvnkqpb4psluea@unicorn.suse.cz>
Thank you for the replies,
Eric Dumazet wrote on Fri, Apr 13, 2018:
> There is no way a regular TCP stack (including linux) could send the following frame.
>
> > 16:49:27.048760 IP <server local ip>.13317 > <client public ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
>
> So something is mangling the packet, maybe NAT or something.
The pcap was produced on the server which emitted the frame, so it
should be exactly as the server intended it to be without any mangling
involved?
If you could point at what strikes you as odd I can check if the same
happened on other hang traces I might still have. Is it just that it
replays a very old seq?
(if it's odd wrt client packets, the same trace as captured on client is
at the end of this mail)
Michal Kubecek wrote on Fri, Apr 13, 2018:
> The way I read this, server doesn't see anything sent by client since
> some point shortly before the dump shown here starts (about 5ms). It
> keeps sending data until 16:49:26.807754 (seq 77346) and then keeps
> resending first (from its point of view) unacknowledged segment
> (32004:33378) in exponentially growing intervals and ignores replies
> from the client. Client apparently receives these retransmits and
> replies with dupack (with D-SACK for 32004:33378) and retransmits of its
> own first unacknowledged segment (4190:4226).
I have the same understanding.
> As we can see the client packets in the dump (which was taken on
> server), it would mean they are dropped after the point where packet
> socket would pass them to libpcap. That might be e.g. netfilter
> (conntrack?) or the IP/TCP code detecting them to be invalid for some
> reason (which is not obvious to me from the dump above).
In my second mail, I got as far as `tcp_ack`, where the replays didn't
get considered for ack because it was seen as having the same ack number
as the lastest seen. I'm not sure why it didn't get considered the first
time, I need to add more instrumentation.
> There are two strange points:
>
> 1. While client apparently responds to all server retransmits, it does
> so with TSecr=1313937714 (matching server packet from 16:49:26.807754)
> rather than TSval of the packets it dupacks (1313937955 through
> 1313953050). This doesn't seem to follow the rules of RFC 7323
> Section 4.3.
I have a pcap of the same sequence on the client as well, and it does
get the packets with higher val= unchanged, so that might be something
to look at.
I'll post a similarily formated tcpdump output at the end of this mail,
the timestamps and verious numbers in the traces match with my original
mail.
> 2. Window size values in acks from client grow with each acked packet by
> 22-23 (which might be ~1400 with scaling factor of 64). I would rather
> expect advertised receive window to go down by 1374 with each received
> segment and to grow by bigger steps with each read()/recv() call from
> application.
I have a very poor understanding of this part of the protocol/code, but
I think the window grows when a socket starts receiving a bulk of data.
Given this hangs very fast, there were only individual small packets up
till this trace where data comes in quickly, and I think this might just
be window scaling?
I might very well be wrong though..
> We might get more insight if we saw the same connection on both sides.
> From what was presented here, my guess is that
>
> (1) received packets are dropped somewhere on server side (after they
> are cloned for the packet socket)
I'd agree for received replays, but I don't get why the first time that
ack was received didn't work.
> (2) there is something wrong either on client side or between the two
> hosts (there is at least a NAT, IIUC)
My minimal reproducer involves the internet, so there is no telling what
is in the middle, but there is at least two NAT, and the connection is
established through two connect() calls with carefully crafted
source/destination ports if that makes a difference.
I've done a bit more fiddling with netem/tbf and a single nat but I
still can't reproduce it on a local network; I need to spend more time
trying as this is very frustrating because I don't control any of the
two NATs in the real case.
I don't think the NATs are culprits, though - looking at the traces I
don't see any reordering within a direction, missing packets or fiddling
with values so this seems fairly straightforward.
Here's the client-side traces. pcap files still available on request, if
you prefer to look at it in wireshark or another tool.
16:49:26.726793 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 40248:41622, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726798 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 41622, win 918, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.726802 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 41622:42996, ack 4190, win 307, options [nop,nop,TS val 1313937609 ecr 1617129440], length 1374
16:49:26.726807 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 42996, win 940, options [nop,nop,TS val 1617129478 ecr 1313937609], length 0
16:49:26.730611 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 42996:44370, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730619 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 44370, win 963, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.730835 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 44370:45744, ack 4190, win 307, options [nop,nop,TS val 1313937616 ecr 1617129456], length 1374
16:49:26.730847 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 45744, win 986, options [nop,nop,TS val 1617129482 ecr 1313937616], length 0
16:49:26.732371 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 45744:47118, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732383 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 47118, win 1008, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.732393 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 47118:48492, ack 4190, win 307, options [nop,nop,TS val 1313937617 ecr 1617129456], length 1374
16:49:26.732400 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 48492, win 1031, options [nop,nop,TS val 1617129484 ecr 1313937617], length 0
16:49:26.740896 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 48492:49866, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.740910 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 49866, win 1053, options [nop,nop,TS val 1617129493 ecr 1313937627], length 0
16:49:26.742220 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 49866:51240, ack 4190, win 307, options [nop,nop,TS val 1313937627 ecr 1617129464], length 1374
16:49:26.742232 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 51240, win 1076, options [nop,nop,TS val 1617129494 ecr 1313937627], length 0
16:49:26.743463 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 51240:52614, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.743476 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 52614, win 1099, options [nop,nop,TS val 1617129495 ecr 1313937629], length 0
16:49:26.744478 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 52614:53988, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744490 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 53988, win 1121, options [nop,nop,TS val 1617129496 ecr 1313937629], length 0
16:49:26.744975 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 53988:55362, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.744982 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 55362, win 1144, options [nop,nop,TS val 1617129497 ecr 1313937629], length 0
16:49:26.748722 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 55362:56736, ack 4190, win 307, options [nop,nop,TS val 1313937629 ecr 1617129466], length 1374
16:49:26.748733 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 56736, win 1167, options [nop,nop,TS val 1617129500 ecr 1313937629], length 0
16:49:26.749615 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 56736:58110, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.749627 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 58110, win 1189, options [nop,nop,TS val 1617129501 ecr 1313937630], length 0
16:49:26.750248 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 58110:59484, ack 4190, win 307, options [nop,nop,TS val 1313937630 ecr 1617129467], length 1374
16:49:26.750260 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 59484, win 1212, options [nop,nop,TS val 1617129502 ecr 1313937630], length 0
16:49:26.750968 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 59484:60858, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.750980 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 60858, win 1234, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.751486 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 60858:62232, ack 4190, win 307, options [nop,nop,TS val 1313937635 ecr 1617129473], length 1374
16:49:26.751498 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 62232, win 1257, options [nop,nop,TS val 1617129503 ecr 1313937635], length 0
16:49:26.752095 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 62232:63606, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752107 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 63606, win 1280, options [nop,nop,TS val 1617129504 ecr 1313937636], length 0
16:49:26.752920 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 63606:64980, ack 4190, win 307, options [nop,nop,TS val 1313937636 ecr 1617129473], length 1374
16:49:26.752932 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 64980, win 1302, options [nop,nop,TS val 1617129505 ecr 1313937636], length 0
16:49:26.754573 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 64980:66354, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.754585 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 66354, win 1325, options [nop,nop,TS val 1617129506 ecr 1313937638], length 0
16:49:26.755260 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 66354:67728, ack 4190, win 307, options [nop,nop,TS val 1313937638 ecr 1617129473], length 1374
16:49:26.755271 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 0
16:49:26.755493 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 67728, win 1348, options [nop,nop,TS val 1617129507 ecr 1313937638], length 36
16:49:26.756076 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 67728:69102, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756391 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 69102:70476, ack 4190, win 307, options [nop,nop,TS val 1313937639 ecr 1617129473], length 1374
16:49:26.756402 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 70476, win 1393, options [nop,nop,TS val 1617129508 ecr 1313937639], length 0
16:49:26.757020 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 70476:71850, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757349 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 71850:73224, ack 4190, win 307, options [nop,nop,TS val 1313937641 ecr 1617129473], length 1374
16:49:26.757360 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 73224, win 1438, options [nop,nop,TS val 1617129509 ecr 1313937641], length 0
16:49:26.757878 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 73224:74598, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758457 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 74598:75972, ack 4190, win 307, options [nop,nop,TS val 1313937643 ecr 1617129473], length 1374
16:49:26.758465 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 75972, win 1444, options [nop,nop,TS val 1617129510 ecr 1313937643], length 0
16:49:26.828268 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 75972:77346, ack 4190, win 307, options [nop,nop,TS val 1313937714 ecr 1617129473], length 1374
16:49:26.868867 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129620 ecr 1313937714], length 0
16:49:27.010733 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617129762 ecr 1313937714], length 36
16:49:27.070035 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313937955 ecr 1617129473], length 1374
16:49:27.070062 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617129822 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.266769 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130018 ecr 1313937714], length 36
16:49:27.540998 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313938426 ecr 1617129473], length 1374
16:49:27.541022 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617130293 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:27.778687 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617130530 ecr 1313937714], length 36
16:49:28.532876 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313939418 ecr 1617129473], length 1374
16:49:28.532907 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617131285 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:28.802879 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617131554 ecr 1313937714], length 36
16:49:30.452830 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313941338 ecr 1617129473], length 1374
16:49:30.452861 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617133204 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:30.850734 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617133602 ecr 1313937714], length 36
16:49:34.229420 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313945114 ecr 1617129473], length 1374
16:49:34.229447 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617136981 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:35.202854 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617137954 ecr 1313937714], length 36
16:49:42.165044 IP <server public ip>.13317 > <client local ip>.31872: Flags [.], seq 32004:33378, ack 4190, win 307, options [nop,nop,TS val 1313953050 ecr 1617129473], length 1374
16:49:42.165073 IP <client local ip>.31872 > <server public ip>.13317: Flags [.], ack 77346, win 1444, options [nop,nop,TS val 1617144917 ecr 1313937714,nop,nop,sack 1 {32004:33378}], length 0
16:49:43.394773 IP <client local ip>.31872 > <server public ip>.13317: Flags [P.], seq 4190:4226, ack 77346, win 1444, options [nop,nop,TS val 1617146146 ecr 1313937714], length 36
Once again, thank you both.
--
Dominique Martinet | Asmadeus
^ permalink raw reply
* Re: Creating FOU tunnels to the same destination IP but different port
From: Tom Herbert @ 2018-04-14 1:07 UTC (permalink / raw)
To: Kostas Peletidis; +Cc: Linux Kernel Network Developers
In-Reply-To: <CAK+mtNMh_CS_sysRM3GPr4QAfRdpXC58FpuciNv_3+0GyXiyfQ@mail.gmail.com>
On Fri, Apr 13, 2018 at 9:57 AM, Kostas Peletidis <kpeletidis@gmail.com> wrote:
> Hello,
>
> I am having trouble with a particular case of setting up a fou tunnel
> and I would really appreciate your help.
>
> I have a remote multihomed host behind a NAT box and I want to create
> a fou tunnel for each of its IP addresses, from my machine.
>
> A typical case would be something like that (output from the local machine):
>
> # ip tun
> ipudp09602: ip/ip remote 135.196.22.100 local 172.31.0.140 ttl 225
> ipudp00101: ip/ip remote 148.252.129.30 local 172.31.0.140 ttl 225
> ipudp09604: ip/ip remote 77.247.11.249 local 172.31.0.140 ttl 225
> tunl0: any/ip remote any local any ttl inherit nopmtudisc
> ipudp00102: ip/ip remote 213.205.194.18 local 172.31.0.140 ttl 225
>
> However, if the remote end has the same IP address with the remote end
> of an existing tunnel (but a different remote port)
> tunnel creation fails. In this example there is already a tunnel to
> 135.196.22.100:32270 and I wanted to create a new tunnel
> to 135.196.22.100:24822 as below:
>
> # ip link add name ipudp09603 mtu 1356 type ipip \
> remote 135.196.22.100 \
> local 172.31.0.140 \
> ttl 225 \
> encap fou \
> encap-sport 4500 \
> encap-dport 24822
>
> RTNETLINK answers: File exists
>
> The remote IP addresses in this case are identical because there is a
> NAT box in the way, but the port numbers are different. The source
> address and port are the same in all cases.
>
> I noticed that ip_tunnel_find() does not check port numbers - being IP
> and all - so I am thinking that a not-so-elegant way to do it is to
> get the port numbers from the netlink request and have
> ip_tunnel_find() compare them against encap.{sport, dport} of existing
> tunnels.
>
> Is there a better way to create a second fou tunnel to the same IP
> address but a different port? Use of keys as unique tunnel IDs maybe?
> Any feedback is appreciated. Thank you.
>
Hi Kostas,
This is an interesting problem, thanks for reporting it! FOU in this
case is being used as modified ipip tunnel so the check of uniqueness
is only based on local and remote addresses for an IP tunnel. As you
point out, the port information does provide more specific information
that could be be used to distinguish between the tunnels (especially
on receive). Using the information is tricky since the FOU and ipip
layers are pretty much independent. The keys approach might be
possible. I'll try to take a closer look.
Tom
>
> Regards,
> Kostas
^ permalink raw reply
* Crash due to NULL dereference in tcp_rearm_rto
From: Subash Abhinov Kasiviswanathan @ 2018-04-14 0:39 UTC (permalink / raw)
To: netdev, soheil, eric.dumazet
We are seeing a warning followed by a crash on an ARM64 device with
Android 4.14 based kernel.
It looks like both sk->sk_write_queue and sk->sk_send_head are NULL.
Since the sk->sk_write_queue is NULL and is dereferenced in
tcp_rto_delta_us()
to get the skb->skb_mstamp, there is crash observed.
Since this is 4.14.32, it already has commit ("tcp: reset sk_send_head
in tcp_write_queue_purge")
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.14.34&id=dbbf2d1e4077bab0c65ece2765d3fc69cf7d610f
12876.013077: <6> WARNING: CPU: 5 PID: 14828 at
net/ipv4/tcp_output.c:2469 tcp_send_loss_probe+0x198/0x1b8
12876.038939: <6> task: ffffffe73f7e5a80 task.stack: ffffff801b068000
12876.038941: <2> PC is at tcp_send_loss_probe+0x198/0x1b8
12876.038942: <2> LR is at tcp_send_loss_probe+0x28/0x1b8
12876.038944: <2> pc : [<ffffff8dc8db16f8>] lr : [<ffffff8dc8db1588>]
pstate: 60400145
12876.038944: <2> sp : ffffff800802bd30
12876.038945: <2> x29: ffffff800802bd50 x28: ffffff8dc9d83eb8
12876.038948: <2> x27: ffffff800802be08 x26: ffffff8dc9737000
12876.038950: <2> x25: 0000000000000001 x24: ffffffe744ea1728
12876.038952: <2> x23: ffffffe73f7e5a80 x22: 0000000000000558
12876.038954: <2> x21: 0000000000000000 x20: 0000000001080020
12876.038956: <2> x19: ffffffe73d06e440 x18: 0000000000000020
12876.038958: <2> x17: 0000000000000014 x16: 0000000000000030
12876.038960: <2> x15: 0000000000000000 x14: 0000000000000000
12876.038962: <2> x13: 0000000013af314c x12: 0000002773f8a550
12876.038965: <2> x11: 0000000000000538 x10: 0000000000000000
12876.038967: <2> x9 : 0000000000000020 x8 : ffffffe73d06e5f0
12876.038969: <2> x7 : 0000000000924278 x6 : ffffffe76fe9ed80
12876.038971: <2> x5 : ffffffe76fe9ed80 x4 : 000000000f500458
12876.038973: <2> x3 : ffffff800802bce0 x2 : ffffff800802bce8
12876.038975: <2> x1 : 0000000000000000 x0 : 0000000000000558
12876.039082: <2> [<ffffff8dc8db16f8>] tcp_send_loss_probe+0x198/0x1b8
12876.039084: <2> [<ffffff8dc8db6698>]
tcp_write_timer_handler+0xf8/0x1c4
12876.039086: <2> [<ffffff8dc8db68cc>] tcp_write_timer+0x5c/0x98
12876.039089: <2> [<ffffff8dc8144f10>] call_timer_fn+0xc0/0x1b4
12876.039091: <2> [<ffffff8dc8143f68>] run_timer_softirq+0x230/0x850
12876.039094: <2> [<ffffff8dc8081b74>] __do_softirq+0x1dc/0x3a4
12876.039096: <2> [<ffffff8dc80b8fec>] irq_exit+0xc8/0xd4
12876.039098: <2> [<ffffff8dc812a004>] __handle_domain_irq+0x8c/0xc4
12876.039099: <2> [<ffffff8dc8081940>] gic_handle_irq+0x164/0x1bc
[net/ipv4/tcp_output.c]
void tcp_send_loss_probe(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
struct sk_buff *skb;
int pcount;
int mss = tcp_current_mss(sk);
...
/* Retransmit last segment. */
if (WARN_ON(!skb))
goto rearm_timer;
12876.043967: <6> Unable to handle kernel NULL pointer dereference at
virtual address 00000010
12876.091600: <6> Internal error: Oops: 96000005 [#1] PREEMPT SMP
12876.152597: <2> PC is at tcp_rearm_rto+0x48/0x90
12876.156979: <2> LR is at tcp_send_loss_probe+0x178/0x1b8
12876.162077: <2> pc : [<ffffff8dc8da76c4>] lr : [<ffffff8dc8db16d8>]
pstate: 60400145
12876.169657: <2> sp : ffffff800802bd10
12876.173056: <2> x29: ffffff800802bd20 x28: ffffff8dc9d83eb8
12876.178511: <2> x27: ffffff800802be08 x26: ffffff8dc9737000
12876.183967: <2> x25: 0000000000000001 x24: ffffffe744ea1728
12876.189418: <2> x23: ffffffe73f7e5a80 x22: 0000000000000558
12876.194863: <2> x21: 0000000000000000 x20: 0000000001080020
12876.200312: <2> x19: ffffffe73d06e440 x18: 0000000000000020
12876.205758: <2> x17: 0000000000000014 x16: 0000000000000030
12876.211212: <2> x15: 0000000000000000 x14: 0000000000000000
12876.216660: <2> x13: 0000000013af314c x12: 0000002773f8a550
12876.222108: <2> x11: 0000000000000538 x10: 0000000000000000
12876.227561: <2> x9 : 0000000000000000 x8 : ffffffe73d06e5f0
12876.233008: <2> x7 : 0000000000924278 x6 : ffffffe76fe9ed80
12876.238455: <2> x5 : ffffffe76fe9ed80 x4 : 000000000f500458
12876.243907: <2> x3 : ffffff800802bce0 x2 : ffffff800802bce8
12876.249360: <2> x1 : 0000000000000000 x0 : 0000000000000867
12876.473522: <2> [<ffffff8dc8da76c4>] tcp_rearm_rto+0x48/0x90
12876.478971: <2> [<ffffff8dc8db16d8>] tcp_send_loss_probe+0x178/0x1b8
12876.485131: <2> [<ffffff8dc8db6698>]
tcp_write_timer_handler+0xf8/0x1c4
12876.491557: <2> [<ffffff8dc8db68cc>] tcp_write_timer+0x5c/0x98
12876.497189: <2> [<ffffff8dc8144f10>] call_timer_fn+0xc0/0x1b4
12876.502731: <2> [<ffffff8dc8143f68>] run_timer_softirq+0x230/0x850
12876.508716: <2> [<ffffff8dc8081b74>] __do_softirq+0x1dc/0x3a4
12876.514260: <2> [<ffffff8dc80b8fec>] irq_exit+0xc8/0xd4
12876.519261: <2> [<ffffff8dc812a004>] __handle_domain_irq+0x8c/0xc4
12876.525245: <2> [<ffffff8dc8081940>] gic_handle_irq+0x164/0x1bc
[net/ipv4/tcp_input.c]
void tcp_rearm_rto(struct sock *sk)
{
...
inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS);
} else {
u32 rto = inet_csk(sk)->icsk_rto;
/* Offset the time elapsed after installing regular RTO */
if (icsk->icsk_pending == ICSK_TIME_REO_TIMEOUT ||
icsk->icsk_pending == ICSK_TIME_LOSS_PROBE) {
s64 delta_us = tcp_rto_delta_us(sk);
/* delta_us may not be positive if the socket is locked
* when the retrans timer fires and is rescheduled.
*/
rto = usecs_to_jiffies(max_t(int, delta_us, 1));
[include/net/tcp.h]
static inline s64 tcp_rto_delta_us(const struct sock *sk)
{
const struct sk_buff *skb = tcp_write_queue_head(sk);
u32 rto = inet_csk(sk)->icsk_rto;
u64 rto_time_stamp_us = skb->skb_mstamp + jiffies_to_usecs(rto);
return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
}
--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
^ permalink raw reply
* RE: [Resend Patch 3/3] Storvsc: Select channel based on available percentage of ring buffer to write
From: Long Li @ 2018-04-14 0:25 UTC (permalink / raw)
To: Michael Kelley (EOSG), KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, James E . J . Bottomley, Martin K . Petersen,
devel@linuxdriverproject.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org
In-Reply-To: <DM5PR2101MB1030EEAC92D8CF826480A6B2DCB30@DM5PR2101MB1030.namprd21.prod.outlook.com>
> Subject: RE: [Resend Patch 3/3] Storvsc: Select channel based on available
> percentage of ring buffer to write
>
> > -----Original Message-----
> > From: linux-kernel-owner@vger.kernel.org
> > <linux-kernel-owner@vger.kernel.org> On Behalf Of Long Li
> > Sent: Tuesday, March 27, 2018 5:49 PM
> > To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> > <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>;
> > James E . J . Bottomley <JBottomley@odin.com>; Martin K . Petersen
> > <martin.petersen@oracle.com>; devel@linuxdriverproject.org; linux-
> > scsi@vger.kernel.org; linux-kernel@vger.kernel.org;
> > netdev@vger.kernel.org
> > Cc: Long Li <longli@microsoft.com>
> > Subject: [Resend Patch 3/3] Storvsc: Select channel based on available
> > percentage of ring buffer to write
> >
> > From: Long Li <longli@microsoft.com>
> >
> > This is a best effort for estimating on how busy the ring buffer is
> > for that channel, based on available buffer to write in percentage. It
> > is still possible that at the time of actual ring buffer write, the
> > space may not be available due to other processes may be writing at the
> time.
> >
> > Selecting a channel based on how full it is can reduce the possibility
> > that a ring buffer write will fail, and avoid the situation a channel
> > is over busy.
> >
> > Now it's possible that storvsc can use a smaller ring buffer size
> > (e.g. 40k bytes) to take advantage of cache locality.
> >
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> > drivers/scsi/storvsc_drv.c | 62
> > +++++++++++++++++++++++++++++++++++++---------
> > 1 file changed, 50 insertions(+), 12 deletions(-)
> >
> > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > index a2ec0bc9e9fa..b1a87072b3ab 100644
> > --- a/drivers/scsi/storvsc_drv.c
> > +++ b/drivers/scsi/storvsc_drv.c
> > @@ -395,6 +395,12 @@ MODULE_PARM_DESC(storvsc_ringbuffer_size,
> "Ring
> > buffer size (bytes)");
> >
> > module_param(storvsc_vcpus_per_sub_channel, int, S_IRUGO);
> > MODULE_PARM_DESC(storvsc_vcpus_per_sub_channel, "Ratio of VCPUs
> to
> > subchannels");
> > +
> > +static int ring_avail_percent_lowater = 10;
>
> Reserving 10% of each ring buffer by default seems like more than is needed
> in the storvsc driver. That would be about 4Kbytes for the 40K ring buffer
> you suggest, and even more for a ring buffer of 128K. Each outgoing record is
> only about 344 bytes (I'd have to check exactly). With the new channel
> selection algorithm below, the only time we use a channel that is already
> below the low water mark is when no channel could be found that is above
> the low water mark. There could be a case of two or more threads deciding
> that a channel is above the low water mark at the same time and both
> choosing it, but that's likely to be rare. So it seems like we could set the
It's not rare for two processes checking on the same channel at the same time, when running multiple processes I/O workload. The CPU to channel is not 1:1 mapping.
> default low water mark to 5 percent or even 3 percent, which will let more of
> the ring buffer be used, and let a channel be assigned according to the
> algorithm, rather than falling through to the default because all channels
> appear to be "full".
It seems it's not about how big ring buffer is, e.g. even you have a ring buffer of infinite size, it won't help with performance if it's getting queued all the time, while other ring buffers are near empty. It's more about how multiple ring buffers are getting utilized in a reasonable and balanced way. Testing shows 10 is a good choice, while 3 is prone to return BUSY and trigger block layer retry.
>
> > +module_param(ring_avail_percent_lowater, int, S_IRUGO);
> > +MODULE_PARM_DESC(ring_avail_percent_lowater,
> > + "Select a channel if available ring size > this in percent");
> > +
> > /*
> > * Timeout in seconds for all devices managed by this driver.
> > */
> > @@ -1285,9 +1291,9 @@ static int storvsc_do_io(struct hv_device
> > *device, {
> > struct storvsc_device *stor_device;
> > struct vstor_packet *vstor_packet;
> > - struct vmbus_channel *outgoing_channel;
> > + struct vmbus_channel *outgoing_channel, *channel;
> > int ret = 0;
> > - struct cpumask alloced_mask;
> > + struct cpumask alloced_mask, other_numa_mask;
> > int tgt_cpu;
> >
> > vstor_packet = &request->vstor_packet; @@ -1301,22 +1307,53 @@
> > static int storvsc_do_io(struct hv_device *device,
> > /*
> > * Select an an appropriate channel to send the request out.
> > */
> > -
> > if (stor_device->stor_chns[q_num] != NULL) {
> > outgoing_channel = stor_device->stor_chns[q_num];
> > - if (outgoing_channel->target_cpu == smp_processor_id()) {
> > + if (outgoing_channel->target_cpu == q_num) {
> > /*
> > * Ideally, we want to pick a different channel if
> > * available on the same NUMA node.
> > */
> > cpumask_and(&alloced_mask, &stor_device-
> >alloced_cpus,
> >
> cpumask_of_node(cpu_to_node(q_num)));
> > - for_each_cpu_wrap(tgt_cpu, &alloced_mask,
> > - outgoing_channel->target_cpu + 1) {
> > - if (tgt_cpu != outgoing_channel->target_cpu)
> {
> > - outgoing_channel =
> > - stor_device->stor_chns[tgt_cpu];
> > - break;
> > +
> > + for_each_cpu_wrap(tgt_cpu, &alloced_mask,
> q_num + 1) {
> > + if (tgt_cpu == q_num)
> > + continue;
> > + channel = stor_device->stor_chns[tgt_cpu];
> > + if (hv_get_avail_to_write_percent(
> > + &channel->outbound)
> > + > ring_avail_percent_lowater)
> {
> > + outgoing_channel = channel;
> > + goto found_channel;
> > + }
> > + }
> > +
> > + /*
> > + * All the other channels on the same NUMA node
> are
> > + * busy. Try to use the channel on the current CPU
> > + */
> > + if (hv_get_avail_to_write_percent(
> > + &outgoing_channel-
> >outbound)
> > + > ring_avail_percent_lowater)
> > + goto found_channel;
> > +
> > + /*
> > + * If we reach here, all the channels on the current
> > + * NUMA node are busy. Try to find a channel in
> > + * other NUMA nodes
> > + */
> > + cpumask_andnot(&other_numa_mask,
> > + &stor_device->alloced_cpus,
> > +
> cpumask_of_node(cpu_to_node(q_num)));
> > +
> > + for_each_cpu(tgt_cpu, &other_numa_mask) {
> > + channel = stor_device->stor_chns[tgt_cpu];
> > + if (hv_get_avail_to_write_percent(
> > + &channel->outbound)
> > + > ring_avail_percent_lowater)
> {
> > + outgoing_channel = channel;
> > + goto found_channel;
> > }
> > }
> > }
> > @@ -1324,7 +1361,7 @@ static int storvsc_do_io(struct hv_device *device,
> > outgoing_channel = get_og_chn(stor_device, q_num);
> > }
> >
> > -
> > +found_channel:
> > vstor_packet->flags |= REQUEST_COMPLETION_FLAG;
> >
> > vstor_packet->vm_srb.length = (sizeof(struct vmscsi_request) - @@
> > -1733,7 +1770,8 @@ static int storvsc_probe(struct hv_device *device,
> > }
> >
> > scsi_driver.can_queue = (max_outstanding_req_per_channel *
> > - (max_sub_channels + 1));
> > + (max_sub_channels + 1)) *
> > + (100 - ring_avail_percent_lowater) / 100;
>
> A minor nit, but the use of parentheses here is inconsistent. There's a set of
> parens around the first two expressions to explicitly code the associativity,
> but not a set to encompass the third term, which must be processed before
> the fourth one is. C does multiplication and division with left to right
> associativity, so the result is as intended.
> But if we're depending on C's default associativity, then that set of parens
> around the first two expression really isn't needed, and one wonders why
> they are there.
>
> Michael
>
> >
> > host = scsi_host_alloc(&scsi_driver,
> > sizeof(struct hv_host_device));
> > --
> > 2.14.1
^ permalink raw reply
* RE: [Resend Patch 2/3] Netvsc: Use the vmbus functiton to calculate ring buffer percentage
From: Michael Kelley (EOSG) @ 2018-04-13 23:39 UTC (permalink / raw)
To: Long Li, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
James E . J . Bottomley, Martin K . Petersen,
devel@linuxdriverproject.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org
In-Reply-To: <20180328004840.22787-2-longli@linuxonhyperv.com>
> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org <linux-kernel-owner@vger.kernel.org> On Behalf
> Of Long Li
> Sent: Tuesday, March 27, 2018 5:49 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Stephen
> Hemminger <sthemmin@microsoft.com>; James E . J . Bottomley <JBottomley@odin.com>;
> Martin K . Petersen <martin.petersen@oracle.com>; devel@linuxdriverproject.org; linux-
> scsi@vger.kernel.org; linux-kernel@vger.kernel.org; netdev@vger.kernel.org
> Cc: Long Li <longli@microsoft.com>
> Subject: [Resend Patch 2/3] Netvsc: Use the vmbus functiton to calculate ring buffer
> percentage
>
> From: Long Li <longli@microsoft.com>
>
> In Vmbus, we have defined a function to calculate available ring buffer
> percentage to write.
>
> Use that function and remove netvsc's private version.
>
> Signed-off-by: Long Li <longli@microsoft.com>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
> ---
> drivers/net/hyperv/hyperv_net.h | 1 -
> drivers/net/hyperv/netvsc.c | 17 +++--------------
> drivers/net/hyperv/netvsc_drv.c | 3 ---
> 3 files changed, 3 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
> index cd538d5a7986..a0199ab13d67 100644
> --- a/drivers/net/hyperv/hyperv_net.h
> +++ b/drivers/net/hyperv/hyperv_net.h
> @@ -189,7 +189,6 @@ struct netvsc_device;
> struct net_device_context;
>
> extern u32 netvsc_ring_bytes;
> -extern struct reciprocal_value netvsc_ring_reciprocal;
>
> struct netvsc_device *netvsc_device_add(struct hv_device *device,
> const struct netvsc_device_info *info);
> diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
> index 0265d703eb03..8af0069e4d8c 100644
> --- a/drivers/net/hyperv/netvsc.c
> +++ b/drivers/net/hyperv/netvsc.c
> @@ -31,7 +31,6 @@
> #include <linux/vmalloc.h>
> #include <linux/rtnetlink.h>
> #include <linux/prefetch.h>
> -#include <linux/reciprocal_div.h>
>
> #include <asm/sync_bitops.h>
>
> @@ -590,17 +589,6 @@ void netvsc_device_remove(struct hv_device *device)
> #define RING_AVAIL_PERCENT_HIWATER 20
> #define RING_AVAIL_PERCENT_LOWATER 10
>
> -/*
> - * Get the percentage of available bytes to write in the ring.
> - * The return value is in range from 0 to 100.
> - */
> -static u32 hv_ringbuf_avail_percent(const struct hv_ring_buffer_info *ring_info)
> -{
> - u32 avail_write = hv_get_bytes_to_write(ring_info);
> -
> - return reciprocal_divide(avail_write * 100, netvsc_ring_reciprocal);
> -}
> -
> static inline void netvsc_free_send_slot(struct netvsc_device *net_device,
> u32 index)
> {
> @@ -649,7 +637,8 @@ static void netvsc_send_tx_complete(struct netvsc_device
> *net_device,
> wake_up(&net_device->wait_drain);
>
> if (netif_tx_queue_stopped(netdev_get_tx_queue(ndev, q_idx)) &&
> - (hv_ringbuf_avail_percent(&channel->outbound) >
> RING_AVAIL_PERCENT_HIWATER ||
> + (hv_get_avail_to_write_percent(&channel->outbound) >
> + RING_AVAIL_PERCENT_HIWATER ||
> queue_sends < 1)) {
> netif_tx_wake_queue(netdev_get_tx_queue(ndev, q_idx));
> ndev_ctx->eth_stats.wake_queue++;
> @@ -757,7 +746,7 @@ static inline int netvsc_send_pkt(
> struct netdev_queue *txq = netdev_get_tx_queue(ndev, packet->q_idx);
> u64 req_id;
> int ret;
> - u32 ring_avail = hv_ringbuf_avail_percent(&out_channel->outbound);
> + u32 ring_avail = hv_get_avail_to_write_percent(&out_channel->outbound);
>
> nvmsg.hdr.msg_type = NVSP_MSG1_TYPE_SEND_RNDIS_PKT;
> if (skb)
> diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
> index faea0be18924..b0b1c2fd2b7b 100644
> --- a/drivers/net/hyperv/netvsc_drv.c
> +++ b/drivers/net/hyperv/netvsc_drv.c
> @@ -35,7 +35,6 @@
> #include <linux/slab.h>
> #include <linux/rtnetlink.h>
> #include <linux/netpoll.h>
> -#include <linux/reciprocal_div.h>
>
> #include <net/arp.h>
> #include <net/route.h>
> @@ -55,7 +54,6 @@ static unsigned int ring_size __ro_after_init = 128;
> module_param(ring_size, uint, S_IRUGO);
> MODULE_PARM_DESC(ring_size, "Ring buffer size (# of pages)");
> unsigned int netvsc_ring_bytes __ro_after_init;
> -struct reciprocal_value netvsc_ring_reciprocal __ro_after_init;
>
> static const u32 default_msg = NETIF_MSG_DRV | NETIF_MSG_PROBE |
> NETIF_MSG_LINK | NETIF_MSG_IFUP |
> @@ -2186,7 +2184,6 @@ static int __init netvsc_drv_init(void)
> ring_size);
> }
> netvsc_ring_bytes = ring_size * PAGE_SIZE;
> - netvsc_ring_reciprocal = reciprocal_value(netvsc_ring_bytes);
>
> ret = vmbus_driver_register(&netvsc_drv);
> if (ret)
> --
> 2.14.1
^ permalink raw reply
* RE: [Resend Patch 3/3] Storvsc: Select channel based on available percentage of ring buffer to write
From: Michael Kelley (EOSG) @ 2018-04-13 23:28 UTC (permalink / raw)
To: Long Li, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
James E . J . Bottomley, Martin K . Petersen,
devel@linuxdriverproject.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org
In-Reply-To: <20180328004840.22787-3-longli@linuxonhyperv.com>
> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org <linux-kernel-owner@vger.kernel.org> On Behalf
> Of Long Li
> Sent: Tuesday, March 27, 2018 5:49 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Stephen
> Hemminger <sthemmin@microsoft.com>; James E . J . Bottomley <JBottomley@odin.com>;
> Martin K . Petersen <martin.petersen@oracle.com>; devel@linuxdriverproject.org; linux-
> scsi@vger.kernel.org; linux-kernel@vger.kernel.org; netdev@vger.kernel.org
> Cc: Long Li <longli@microsoft.com>
> Subject: [Resend Patch 3/3] Storvsc: Select channel based on available percentage of ring
> buffer to write
>
> From: Long Li <longli@microsoft.com>
>
> This is a best effort for estimating on how busy the ring buffer is for
> that channel, based on available buffer to write in percentage. It is still
> possible that at the time of actual ring buffer write, the space may not be
> available due to other processes may be writing at the time.
>
> Selecting a channel based on how full it is can reduce the possibility that
> a ring buffer write will fail, and avoid the situation a channel is over
> busy.
>
> Now it's possible that storvsc can use a smaller ring buffer size
> (e.g. 40k bytes) to take advantage of cache locality.
>
> Signed-off-by: Long Li <longli@microsoft.com>
> ---
> drivers/scsi/storvsc_drv.c | 62 +++++++++++++++++++++++++++++++++++++---------
> 1 file changed, 50 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> index a2ec0bc9e9fa..b1a87072b3ab 100644
> --- a/drivers/scsi/storvsc_drv.c
> +++ b/drivers/scsi/storvsc_drv.c
> @@ -395,6 +395,12 @@ MODULE_PARM_DESC(storvsc_ringbuffer_size, "Ring buffer size
> (bytes)");
>
> module_param(storvsc_vcpus_per_sub_channel, int, S_IRUGO);
> MODULE_PARM_DESC(storvsc_vcpus_per_sub_channel, "Ratio of VCPUs to subchannels");
> +
> +static int ring_avail_percent_lowater = 10;
Reserving 10% of each ring buffer by default seems like more than is needed
in the storvsc driver. That would be about 4Kbytes for the 40K ring buffer
you suggest, and even more for a ring buffer of 128K. Each outgoing record
is only about 344 bytes (I'd have to check exactly). With the new channel
selection algorithm below, the only time we use a channel that is already
below the low water mark is when no channel could be found that is above
the low water mark. There could be a case of two or more threads deciding
that a channel is above the low water mark at the same time and both
choosing it, but that's likely to be rare. So it seems like we could set the
default low water mark to 5 percent or even 3 percent, which will let more
of the ring buffer be used, and let a channel be assigned according to the
algorithm, rather than falling through to the default because all channels
appear to be "full".
> +module_param(ring_avail_percent_lowater, int, S_IRUGO);
> +MODULE_PARM_DESC(ring_avail_percent_lowater,
> + "Select a channel if available ring size > this in percent");
> +
> /*
> * Timeout in seconds for all devices managed by this driver.
> */
> @@ -1285,9 +1291,9 @@ static int storvsc_do_io(struct hv_device *device,
> {
> struct storvsc_device *stor_device;
> struct vstor_packet *vstor_packet;
> - struct vmbus_channel *outgoing_channel;
> + struct vmbus_channel *outgoing_channel, *channel;
> int ret = 0;
> - struct cpumask alloced_mask;
> + struct cpumask alloced_mask, other_numa_mask;
> int tgt_cpu;
>
> vstor_packet = &request->vstor_packet;
> @@ -1301,22 +1307,53 @@ static int storvsc_do_io(struct hv_device *device,
> /*
> * Select an an appropriate channel to send the request out.
> */
> -
> if (stor_device->stor_chns[q_num] != NULL) {
> outgoing_channel = stor_device->stor_chns[q_num];
> - if (outgoing_channel->target_cpu == smp_processor_id()) {
> + if (outgoing_channel->target_cpu == q_num) {
> /*
> * Ideally, we want to pick a different channel if
> * available on the same NUMA node.
> */
> cpumask_and(&alloced_mask, &stor_device->alloced_cpus,
> cpumask_of_node(cpu_to_node(q_num)));
> - for_each_cpu_wrap(tgt_cpu, &alloced_mask,
> - outgoing_channel->target_cpu + 1) {
> - if (tgt_cpu != outgoing_channel->target_cpu) {
> - outgoing_channel =
> - stor_device->stor_chns[tgt_cpu];
> - break;
> +
> + for_each_cpu_wrap(tgt_cpu, &alloced_mask, q_num + 1) {
> + if (tgt_cpu == q_num)
> + continue;
> + channel = stor_device->stor_chns[tgt_cpu];
> + if (hv_get_avail_to_write_percent(
> + &channel->outbound)
> + > ring_avail_percent_lowater) {
> + outgoing_channel = channel;
> + goto found_channel;
> + }
> + }
> +
> + /*
> + * All the other channels on the same NUMA node are
> + * busy. Try to use the channel on the current CPU
> + */
> + if (hv_get_avail_to_write_percent(
> + &outgoing_channel->outbound)
> + > ring_avail_percent_lowater)
> + goto found_channel;
> +
> + /*
> + * If we reach here, all the channels on the current
> + * NUMA node are busy. Try to find a channel in
> + * other NUMA nodes
> + */
> + cpumask_andnot(&other_numa_mask,
> + &stor_device->alloced_cpus,
> + cpumask_of_node(cpu_to_node(q_num)));
> +
> + for_each_cpu(tgt_cpu, &other_numa_mask) {
> + channel = stor_device->stor_chns[tgt_cpu];
> + if (hv_get_avail_to_write_percent(
> + &channel->outbound)
> + > ring_avail_percent_lowater) {
> + outgoing_channel = channel;
> + goto found_channel;
> }
> }
> }
> @@ -1324,7 +1361,7 @@ static int storvsc_do_io(struct hv_device *device,
> outgoing_channel = get_og_chn(stor_device, q_num);
> }
>
> -
> +found_channel:
> vstor_packet->flags |= REQUEST_COMPLETION_FLAG;
>
> vstor_packet->vm_srb.length = (sizeof(struct vmscsi_request) -
> @@ -1733,7 +1770,8 @@ static int storvsc_probe(struct hv_device *device,
> }
>
> scsi_driver.can_queue = (max_outstanding_req_per_channel *
> - (max_sub_channels + 1));
> + (max_sub_channels + 1)) *
> + (100 - ring_avail_percent_lowater) / 100;
A minor nit, but the use of parentheses here is inconsistent. There's a
set of parens around the first two expressions to explicitly code the
associativity, but not a set to encompass the third term, which must
be processed before the fourth one is. C does multiplication and
division with left to right associativity, so the result is as intended.
But if we're depending on C's default associativity, then that set of
parens around the first two expression really isn't needed, and one
wonders why they are there.
Michael
>
> host = scsi_host_alloc(&scsi_driver,
> sizeof(struct hv_host_device));
> --
> 2.14.1
^ permalink raw reply
* Re: SRIOV switchdev mode BoF minutes
From: Samudrala, Sridhar @ 2018-04-13 23:03 UTC (permalink / raw)
To: Or Gerlitz
Cc: David Miller, Anjali Singhai Jain, Andy Gospodarek, Michael Chan,
Simon Horman, Jakub Kicinski, John Fastabend, Saeed Mahameed,
Jiri Pirko, Rony Efraim, Linux Netdev List
In-Reply-To: <CAJ3xEMiDundB9p0CvnQ33mLm=7OvG4Yvd5ASH9uR+TkCd_uJhg@mail.gmail.com>
On 4/13/2018 1:16 PM, Or Gerlitz wrote:
> On Fri, Apr 13, 2018 at 7:49 PM, Samudrala, Sridhar
> <sridhar.samudrala@intel.com> wrote:
>> On 4/13/2018 1:57 AM, Or Gerlitz wrote:
>
>>>> in overlay networks scheme, the uplink rep has the VTEP ip and is not connected
>>>> to the bridge, e.g you use ovs you have vf reps and vxlan ports connected
>>>> to ovs and the ip stack routes through the uplink rep
>> This changes the legacy mode behavior of configuring vtep ip on the pf
>> netdev. How does host to host traffic expected to work when vtep ip is moved to uplink rep?
> What do you mean host to host traffic, is that two VFs on the same host?
> control plane SWs (such as OVS) don't apply encapsulation within the same host
I meant between PFs on 2 compute nodes.
>
>>>>> What about pf-rep?
>> Are you planning to create a pf-rep too? Is pf also treated similar to vf in
>> switchdev mode?
>> All pf traffic goes to pf-rep and pf-rep traffic goes to pf by default
>> without any rules programmed?
> @ the sriov switchdev ARCH level, pf/pf-rep would work indeed as you described.
>
> We will have pf rep for smartnic schemes where the the pf on the host
> is not the manager of the eswitch but rather the smartnic driver instance.
>
> on non smart env, there are some challenges to address for the pf
> nic to be fully functional for the slow path (what you described), we
> will get there down the road if there is a real need.
So on non-smart env, are you planning to only expose uplink rep and vf reps as netdevs.
By smartnic env, i guess you are referring to OVS control plane also running on the NIC.
I will look forward to your patches.
Thanks
Sridhar
^ permalink raw reply
* Re: [PATCH iproute2-next 3/3] treewide: Use addattr_nest()/addattr_nest_end() to handle nested attributes
From: Vinicius Costa Gomes @ 2018-04-13 22:57 UTC (permalink / raw)
To: Serhey Popovych, netdev
In-Reply-To: <1517386508-10915-4-git-send-email-serhe.popovych@gmail.com>
Hi,
Serhey Popovych <serhe.popovych@gmail.com> writes:
[...]
> diff --git a/tc/q_mqprio.c b/tc/q_mqprio.c
> index 89b4600..207d644 100644
> --- a/tc/q_mqprio.c
> +++ b/tc/q_mqprio.c
> @@ -173,8 +173,7 @@ static int mqprio_parse_opt(struct qdisc_util *qu, int argc,
> argc--; argv++;
> }
>
> - tail = NLMSG_TAIL(n);
> - addattr_l(n, 1024, TCA_OPTIONS, &opt, sizeof(opt));
> + tail = addattr_nest_compat(n, 1024, TCA_OPTIONS, &opt, sizeof(opt));
>
> if (flags & TC_MQPRIO_F_MODE)
> addattr_l(n, 1024, TCA_MQPRIO_MODE,
> @@ -209,7 +208,7 @@ static int mqprio_parse_opt(struct qdisc_util *qu, int argc,
> addattr_nest_end(n, start);
> }
>
> - tail->rta_len = (void *)NLMSG_TAIL(n) - (void *)tail;
> + addattr_nest_compat_end(n, tail);
>
> return 0;
> }
Sorry if I am too late, but this breaks mqprio, i.e. something like
this:
$ tc qdisc replace dev enp2s0 handle 100: parent root mqprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 hw 0
that used to work, now doesn't.
This patch looks right, so I thought that it could be possible that mqprio
(in the kernel side) was making some wrong assumptions about the format
of the messages.
And after some investigation, what seems to be happening is something
like this (not too familiar with netlink protocol internals, I may be
missing something).
In the "wire", after this patch, the mqprio part of message may be
represented as:
/* The message format is [ len | type | payload ] */
[ S | 2 | <S bytes> ]
[ 0 | 2 | ]
Some notes:
- S is the aligned value of sizeof(opt);
- The value of TCA_OPTIONS is 2;
Before this patch, I think it was something like:
[ S | 2 | <S bytes> ]
The problem is that mqprio defines an internal type with the same value
as TCA_OPTIONS (2), and that finalizing (empty) is interpreted as the
"internal" field instead of indicating the end of TCA_OPTIONS, which
causes a size mismatch with 'mqprio_policy', causing the command to
create a mqprio qdisc to fail.
In short, I think that replacing the "open coded" version with
addattr_nest_compat() is not a functionally equivalent change.
Cheers,
^ permalink raw reply
* Cavium Octeon III network driver.
From: Steven J. Hill @ 2018-04-13 22:43 UTC (permalink / raw)
To: netdev
Patches for Cavium's Octeon III network driver were submitted by
David Daney back on 20180222. David has since left the company and
I am now responsible for the upstreaming effort. When looking at
<pachwork.ozlabs.org> they are marked as "Not Applicable". What
steps do I take next? Thanks.
Steve
^ permalink raw reply
* [PATCH iproute2-next 1/1] tc: jsonify ife action
From: Roman Mashak @ 2018-04-13 21:40 UTC (permalink / raw)
To: dsahern; +Cc: stephen, netdev, kernel, jhs, xiyou.wangcong, jiri, Roman Mashak
Signed-off-by: Roman Mashak <mrv@mojatatu.com>
---
tc/m_ife.c | 54 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 32 insertions(+), 22 deletions(-)
diff --git a/tc/m_ife.c b/tc/m_ife.c
index d7e61703f666..15d09a167450 100644
--- a/tc/m_ife.c
+++ b/tc/m_ife.c
@@ -240,22 +240,24 @@ static int print_ife(struct action_util *au, FILE *f, struct rtattr *arg)
parse_rtattr_nested(tb, TCA_IFE_MAX, arg);
if (tb[TCA_IFE_PARMS] == NULL) {
- fprintf(f, "[NULL ife parameters]");
+ print_string(PRINT_FP, NULL, "%s", "[NULL ife parameters]");
return -1;
}
p = RTA_DATA(tb[TCA_IFE_PARMS]);
- fprintf(f, "ife %s ", p->flags & IFE_ENCODE ? "encode" : "decode");
+ print_string(PRINT_ANY, "kind", "%s ", "ife");
+ print_string(PRINT_ANY, "mode", "%s",
+ p->flags & IFE_ENCODE ? "encode" : "decode");
print_action_control(f, "action ", p->action, " ");
if (tb[TCA_IFE_TYPE]) {
ife_type = rta_getattr_u16(tb[TCA_IFE_TYPE]);
has_optional = 1;
- fprintf(f, "type 0x%X ", ife_type);
+ print_0xhex(PRINT_ANY, "type", "type 0x%X ", ife_type);
}
if (has_optional)
- fprintf(f, "\n\t ");
+ print_string(PRINT_FP, NULL, "%s\t", _SL_);
if (tb[TCA_IFE_METALST]) {
struct rtattr *metalist[IFE_META_MAX + 1];
@@ -268,9 +270,11 @@ static int print_ife(struct action_util *au, FILE *f, struct rtattr *arg)
len = RTA_PAYLOAD(metalist[IFE_META_SKBMARK]);
if (len) {
mmark = rta_getattr_u32(metalist[IFE_META_SKBMARK]);
- fprintf(f, "use mark %u ", mmark);
+ print_uint(PRINT_ANY, "mark", "use mark %u ",
+ mmark);
} else
- fprintf(f, "allow mark ");
+ print_string(PRINT_ANY, "mark", "%s mark ",
+ "allow");
}
if (metalist[IFE_META_TCINDEX]) {
@@ -278,41 +282,47 @@ static int print_ife(struct action_util *au, FILE *f, struct rtattr *arg)
if (len) {
mtcindex =
rta_getattr_u16(metalist[IFE_META_TCINDEX]);
- fprintf(f, "use tcindex %d ", mtcindex);
+ print_uint(PRINT_ANY, "tcindex",
+ "use tcindex %u ", mtcindex);
} else
- fprintf(f, "allow tcindex ");
+ print_string(PRINT_ANY, "tcindex",
+ "%s tcindex ", "allow");
}
if (metalist[IFE_META_PRIO]) {
len = RTA_PAYLOAD(metalist[IFE_META_PRIO]);
if (len) {
mprio = rta_getattr_u32(metalist[IFE_META_PRIO]);
- fprintf(f, "use prio %u ", mprio);
+ print_uint(PRINT_ANY, "prio", "use prio %u ",
+ mprio);
} else
- fprintf(f, "allow prio ");
+ print_string(PRINT_ANY, "prio", "%s prio ",
+ "allow");
}
}
if (tb[TCA_IFE_DMAC]) {
has_optional = 1;
- fprintf(f, "dst %s ",
- ll_addr_n2a(RTA_DATA(tb[TCA_IFE_DMAC]),
- RTA_PAYLOAD(tb[TCA_IFE_DMAC]), 0, b2,
- sizeof(b2)));
-
+ print_string(PRINT_ANY, "dst", "dst %s ",
+ ll_addr_n2a(RTA_DATA(tb[TCA_IFE_DMAC]),
+ RTA_PAYLOAD(tb[TCA_IFE_DMAC]), 0, b2,
+ sizeof(b2)));
}
if (tb[TCA_IFE_SMAC]) {
has_optional = 1;
- fprintf(f, "src %s ",
- ll_addr_n2a(RTA_DATA(tb[TCA_IFE_SMAC]),
- RTA_PAYLOAD(tb[TCA_IFE_SMAC]), 0, b2,
- sizeof(b2)));
+ print_string(PRINT_ANY, "src", "src %s ",
+ ll_addr_n2a(RTA_DATA(tb[TCA_IFE_SMAC]),
+ RTA_PAYLOAD(tb[TCA_IFE_SMAC]), 0, b2,
+ sizeof(b2)));
}
- fprintf(f, "\n\t index %u ref %d bind %d", p->index, p->refcnt,
- p->bindcnt);
+ print_string(PRINT_FP, NULL, "%s", _SL_);
+ print_uint(PRINT_ANY, "index", "\t index %u", p->index);
+ print_int(PRINT_ANY, "ref", " ref %d", p->refcnt);
+ print_int(PRINT_ANY, "bind", " bind %d", p->bindcnt);
+
if (show_stats) {
if (tb[TCA_IFE_TM]) {
struct tcf_t *tm = RTA_DATA(tb[TCA_IFE_TM]);
@@ -321,7 +331,7 @@ static int print_ife(struct action_util *au, FILE *f, struct rtattr *arg)
}
}
- fprintf(f, "\n");
+ print_string(PRINT_FP, NULL, "%s", _SL_);
return 0;
}
--
2.7.4
^ permalink raw reply related
* Panic since 4.15.15 in tcp_retransmit_timer when doing ss -K
From: Sami Farin @ 2018-04-13 20:46 UTC (permalink / raw)
To: Linux Networking Mailing List
I started getting this since 4.15.15. It's easy to trigger,
for example I get new IP address via dhcp (NetworkManager),
then ss -K the_old_ip_address .
Happens on Ryzen and SandyBridge systems.
My guess of the cause: commit 960058fe196397aecb16bb14e64980e265d2bc5e
(didn't try reverting)
BUG: unable to handle kernel NULL pointer dereference at 000030
IP: tcp_retransmit_skb+0x57/0xc0
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.15.17+ #42
Hardware name: To Be Filled By O.E.M./To Be Filled By O.E.M./X370 Taichi, BIOS P4.60 03/02/2018
RIP: 0010:tcp_retransmit_skb+0x57/0xc0
RSP: 0018:ffff95b1dea03e00 EFLAGS: 00010206
RAX: 00000000fffffff5 RBX: ffff95b15876d000 RCX: 5
RDX: 5 RSI: 0 RDI: ffff95b115876d000
...
Call Trace:
<IRQ>
tcp_retransmit_timer
tcp_write_timer_handler
tcp_write_timer
? tcp_write_timer_handler
expire_timers
run_timer_softirq
sched_clock
sched_clock
sched_clock_cpu
irqtime_account_irq
__do_softirq
sched_clock
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
</IRQ>
--
Do what you love because life is too short for anything else.
^ permalink raw reply
* Re: [PATCH] x86/cpufeature: guard asm_volatile_goto usage with CC_HAVE_ASM_GOTO
From: Alexei Starovoitov @ 2018-04-13 20:42 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Yonghong Song, mingo, daniel, linux-kernel, x86, kernel-team,
Thomas Gleixner, netdev, Jesper Dangaard Brouer
In-Reply-To: <20180413181924.GT4064@hirez.programming.kicks-ass.net>
On 4/13/18 11:19 AM, Peter Zijlstra wrote:
> On Tue, Apr 10, 2018 at 02:28:04PM -0700, Alexei Starovoitov wrote:
>> Instead of
>> #ifdef CC_HAVE_ASM_GOTO
>> we can replace it with
>> #ifndef __BPF__
>> or some other name,
>
> I would prefer the BPF specific hack; otherwise we might be encouraging
> people to build the kernel proper without asm-goto.
>
I don't understand this concern.
1.
arch/x86/Makefile does
ifndef CC_HAVE_ASM_GOTO
$(error Compiler lacks asm-goto support.)
endif
which is pretty strong statement of the kernel direction.
2.
Even with this patch that adds #ifdef CC_HAVE_ASM_GOTO back
the x86 arch still needs asm-goto in the compiler to be built.
As far as I can see there are other places where asm-goto
is open coded.
So there is no 'encouraging'.
Whereas if we do bpf specific marco we'd need to explain that
to all bpf users and they would need to fix their user space scripts.
Amount of user space breakage is unknown at this point.
^ permalink raw reply
* Re: v6/sit tunnels and VRFs
From: David Ahern @ 2018-04-13 20:31 UTC (permalink / raw)
To: Jeff Barnhill; +Cc: netdev
In-Reply-To: <CAL6e_pdihSXvYFQQ0MAXFtw=eUjJPfR9kS9j5QQNSqCNzdQteg@mail.gmail.com>
On 4/13/18 2:23 PM, Jeff Barnhill wrote:
> It seems that the ENETUNREACH response is still desirable in the VRF
> case since the only difference (when using VRF vs. not) is that the
> lookup should be restrained to a specific VRF.
VRF is just policy routing to a table. If the table wants the lookup to
stop, then it needs a default route. What you are referring to is the
lookup goes through all tables and does not find an answer so it fails
with -ENETUNREACH. I do not know of any way to make that happen with the
existing default route options and in the past 2+ years we have not hit
any s/w that discriminates -ENETUNREACH from -EHOSTUNREACH.
I take it this is code from your internal code base. Why does it care
between those two failures?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox