Re: tcp hang when socket fills up ?

[Dominique Martinet <asmadeus@codewreck.org>]

[-- Attachment #1: Type: text/plain, Size: 8673 bytes --]

Thank you for the hints. Out of order reply.

Florian Westphal wrote on Mon, Apr 16, 2018:
> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
> 
> which stops conntrack from marking packets with out-of-window
> acks as invalid.

That worked, the connection stays established with this set.


> echo 6 > /proc/sys/net/netfilter/nf_conntrack_log_invalid

Took me a while to figure how to log that properly (why do distros not
ship ulogd?!) but that is helpful.

I have attached the full logs to this mail as text attachments in case I
missed something in my excerpts


Specifically, there are many outbound invalid packets, but as Marcelo
pointed out there is no conntrack netfilter for outbound packets...
The first invalid incoming packet is that one (json output seemed to be
the most convenient as it has tcp seq/ackseq and the oob.prefix message,
feel free to ask for some other format)

{"timestamp": "2018-04-17T10:29:14.956485", "dvc": "Netfilter",
"raw.pktlen": 52, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: ACK is
over the upper bound (ACKed data not seen yet) ", "oob.time.sec":
1523928554, "oob.time.usec": 956485, "oob.mark": 0, "oob.hook": 0,
"oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "ip.protocol": 6,
"ip.tos": 0, "ip.ttl": 52, "ip.totlen": 52, "ip.ihl": 5, "ip.csum":
50909, "ip.id": 30328, "ip.fragoff": 16384, "src_port": 30280,
"dest_port": 29543, "tcp.seq": 4048786673, "tcp.ackseq": 3024392506,
"tcp.window": 781, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0,
"tcp.ack": 1, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0,
"tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 17752, "oob.in": "",
"oob.out": "", "src_ip": "client", "dest_ip": "server"}

which is the first ack that isn't seen alright (sent packet, blocked
ack, replays):
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
...
10:29:14.956485 IP client.30280 > server.29543: Flags [.], ack 3024392506, win 781, options [nop,nop,TS val 1634528369 ecr 681965809], length 0
...
10:29:15.255489 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966138 ecr 1634528369], length 1374
10:29:15.281581 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528697 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.555466 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528869 ecr 681965904], length 36
10:29:15.719510 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966602 ecr 1634528369], length 1374

So now I guess the question is why netfilter think this is over the
upper bound.
..And surely enough the answer is within the invalid outbound packets
that aren't considered in the conntrack state machine, the first of whih
is this one:
{"timestamp": "2018-04-17T10:29:14.926143", "dvc": "Netfilter",
"raw.pktlen": 2800, "raw.pktcount": 1, "oob.prefix": "nf_ct_tcp: SEQ is
over the upper bound (over the window of the receiver)", "oob.time.sec":
1523928554, "oob.time.usec": 926143, "oob.mark": 0, "oob.hook": 0,
"oob.family": 2, "oob.protocol": 2048, "oob.uid": 1000, "oob.gid": 100,
"raw.label": 0, "ip.protocol": 6, "ip.tos": 8, "ip.ttl": 64,
"ip.totlen": 2800, "ip.ihl": 5, "ip.csum": 47600, "ip.id": 27809,
"ip.fragoff": 16384, "src_port": 29543, "dest_port": 30280, "tcp.seq":
3024391132, "tcp.ackseq": 4048786673, "tcp.window": 307, "tcp.offset":
0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 1, "tcp.psh": 0,
"tcp.rst": 0, "tcp.syn": 0, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0,
"tcp.csum": 5201, "oob.in": "", "oob.out": "", "src_ip": "server",
"dest_ip": "client"}

which sequence matches the start seq of the first packet we don't ack,
so that's why conntrack doesn't recongnize that as a valid ack.


Here's some context before that packet:
10:29:14.921305 IP server.29543 > client.30280: Flags [.], seq 3024378314:3024379688, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.921311 IP server.29543 > client.30280: Flags [.], seq 3024379688:3024381062, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.922166 IP client.30280 > server.29543: Flags [.], ack 3024365948, win 329, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.922178 IP server.29543 > client.30280: Flags [.], seq 3024381062:3024382436, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.922182 IP server.29543 > client.30280: Flags [.], seq 3024382436:3024383810, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.923848 IP client.30280 > server.29543: Flags [.], ack 3024367322, win 352, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.923860 IP server.29543 > client.30280: Flags [.], seq 3024383810:3024385184, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.923863 IP server.29543 > client.30280: Flags [.], seq 3024385184:3024386558, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924175 IP client.30280 > server.29543: Flags [.], ack 3024368696, win 375, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.924187 IP server.29543 > client.30280: Flags [.], seq 3024386558:3024387932, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924190 IP server.29543 > client.30280: Flags [.], seq 3024387932:3024389306, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.925410 IP client.30280 > server.29543: Flags [.], ack 3024370070, win 397, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.925422 IP server.29543 > client.30280: Flags [.], seq 3024389306:3024390680, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 1374
10:29:14.925425 IP server.29543 > client.30280: Flags [P.], seq 3024390680:3024391132, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 452
10:29:14.926030 IP client.30280 > server.29543: Flags [.], ack 3024371444, win 420, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374

But the way I see it, (3024392506-3024371444) = 21064 is smaller than
420*64 = 26880.
Even looking at the previous (ack 3024370070, win 397) we get 22436 and
25408 which still work, same with ack 3024368696, win 375 (23810 <
24000)
So that means conntrack is looking at an ack at least as old as ack
3024367322, win 352?...
Is there any way to confirm that? In the code it looks like we match
! before(seq, sender->td_maxend + 1)

I turned pr_debug on in tcp_in_window() for another try and it's a bit
mangled because the information on multiple lines and the function is
called in parallel but it looks like I do have some seq > maxend +1

Although it's weird, the maxend was set WAY earlier apparently?
Apr 17 11:13:14 res=1 sender end=1913287798 maxend=1913316998 maxwin=29312 receiver end=505004284 maxend=505033596 maxwin=29200
then window decreased drastically e.g. previous ack just before refusal:
Apr 17 11:13:53 seq=1913292311 ack=505007789+(0) sack=505007789+(0) win=284 end=1913292311
Apr 17 11:13:53 sender end=1913292311 maxend=1913331607 maxwin=284 scale=0 receiver end=505020155 maxend=505033596 maxwin=39296 scale=7

first res=0 (refused) packet:
Apr 17 11:13:53 odin kernel: seq=505042139 ack=1913292311+(0) ack=1913292311+(0) win=307 end=505044887
Apr 17 11:13:53 res=0 sender end=505033895 maxend=505033596 maxwin=39296 receiver end=1913292311 maxend=1913331607 maxwin=674


> (Earlier email implies this is related to timestamps, but unfortunately
>  to best of my knowledge conntrack doesn't look at tcp timestamps).

tcp_timestamp introduces other changes in behaviour so it might still be
relateed.



Recap of attachments:
 - new tcpdump output of a failed attempt
 - matching invalid packet dumped in json format
 - logs of tcp_in_window function unfortunately of a different attempt

Happy to provide new/different logs if it would be useful. Once again
thanks for the extra eyes and useful commands.

-- 
Dominique Martinet | Asmadeus

[-- Attachment #2: tcpdump --]
[-- Type: text/plain, Size: 27334 bytes --]

10:29:13.429532 IP server.29543 > client.30280: Flags [S], seq 3024360198, win 29200, options [mss 1460,sackOK,TS val 681964312 ecr 0,nop,wscale 7], length 0
10:29:13.712771 IP client.30280 > server.29543: Flags [S], seq 4048782683, win 29200, options [mss 1386,sackOK,TS val 1634527123 ecr 0,nop,wscale 7], length 0
10:29:13.712791 IP server.29543 > client.30280: Flags [S.], seq 3024360198, ack 4048782684, win 29200, options [mss 1460,sackOK,TS val 681964596 ecr 1634527123,nop,wscale 7], length 0
10:29:13.738441 IP client.30280 > server.29543: Flags [.], ack 3024360199, win 229, options [nop,nop,TS val 1634527153 ecr 681964596], length 0
10:29:13.740587 IP client.30280 > server.29543: Flags [P.], seq 4048782684:4048782705, ack 3024360199, win 229, options [nop,nop,TS val 1634527154 ecr 681964596], length 21
10:29:13.740610 IP server.29543 > client.30280: Flags [.], ack 4048782705, win 229, options [nop,nop,TS val 681964623 ecr 1634527154], length 0
10:29:13.744143 IP server.29543 > client.30280: Flags [P.], seq 3024360199:3024360220, ack 4048782705, win 229, options [nop,nop,TS val 681964627 ecr 1634527154], length 21
10:29:13.769430 IP client.30280 > server.29543: Flags [.], ack 3024360220, win 229, options [nop,nop,TS val 1634527185 ecr 681964627], length 0
10:29:13.769458 IP server.29543 > client.30280: Flags [P.], seq 3024360220:3024360932, ack 4048782705, win 229, options [nop,nop,TS val 681964652 ecr 1634527185], length 712
10:29:13.772401 IP client.30280 > server.29543: Flags [P.], seq 4048782705:4048783953, ack 3024360220, win 229, options [nop,nop,TS val 1634527187 ecr 681964627], length 1248
10:29:13.813512 IP server.29543 > client.30280: Flags [.], ack 4048783953, win 248, options [nop,nop,TS val 681964696 ecr 1634527187], length 0
10:29:13.835970 IP client.30280 > server.29543: Flags [.], ack 3024360932, win 240, options [nop,nop,TS val 1634527251 ecr 681964652], length 0
10:29:13.841601 IP client.30280 > server.29543: Flags [P.], seq 4048783953:4048784001, ack 3024360932, win 240, options [nop,nop,TS val 1634527255 ecr 681964696], length 48
10:29:13.841626 IP server.29543 > client.30280: Flags [.], ack 4048784001, win 248, options [nop,nop,TS val 681964724 ecr 1634527255], length 0
10:29:13.847085 IP server.29543 > client.30280: Flags [P.], seq 3024360932:3024361320, ack 4048784001, win 248, options [nop,nop,TS val 681964730 ecr 1634527255], length 388
10:29:13.872177 IP client.30280 > server.29543: Flags [.], ack 3024361320, win 251, options [nop,nop,TS val 1634527287 ecr 681964730], length 0
10:29:13.906359 IP client.30280 > server.29543: Flags [P.], seq 4048784001:4048784017, ack 3024361320, win 251, options [nop,nop,TS val 1634527321 ecr 681964730], length 16
10:29:13.946509 IP server.29543 > client.30280: Flags [.], ack 4048784017, win 248, options [nop,nop,TS val 681964829 ecr 1634527321], length 0
10:29:13.979888 IP client.30280 > server.29543: Flags [P.], seq 4048784017:4048784069, ack 3024361320, win 251, options [nop,nop,TS val 1634527387 ecr 681964829], length 52
10:29:13.979896 IP server.29543 > client.30280: Flags [.], ack 4048784069, win 248, options [nop,nop,TS val 681964863 ecr 1634527387], length 0
10:29:13.980189 IP server.29543 > client.30280: Flags [P.], seq 3024361320:3024361372, ack 4048784069, win 248, options [nop,nop,TS val 681964863 ecr 1634527387], length 52
10:29:14.007950 IP client.30280 > server.29543: Flags [P.], seq 4048784069:4048784137, ack 3024361372, win 251, options [nop,nop,TS val 1634527422 ecr 681964863], length 68
10:29:14.008365 IP server.29543 > client.30280: Flags [P.], seq 3024361372:3024361456, ack 4048784137, win 248, options [nop,nop,TS val 681964891 ecr 1634527422], length 84
10:29:14.035877 IP client.30280 > server.29543: Flags [P.], seq 4048784137:4048784765, ack 3024361456, win 251, options [nop,nop,TS val 1634527449 ecr 681964891], length 628
10:29:14.036743 IP server.29543 > client.30280: Flags [P.], seq 3024361456:3024362036, ack 4048784765, win 268, options [nop,nop,TS val 681964920 ecr 1634527449], length 580
10:29:14.075014 IP client.30280 > server.29543: Flags [P.], seq 4048784765:4048785937, ack 3024362036, win 262, options [nop,nop,TS val 1634527489 ecr 681964920], length 1172
10:29:14.076317 IP server.29543 > client.30280: Flags [P.], seq 3024362036:3024362072, ack 4048785937, win 287, options [nop,nop,TS val 681964959 ecr 1634527489], length 36
10:29:14.102624 IP client.30280 > server.29543: Flags [P.], seq 4048785937:4048786057, ack 3024362072, win 262, options [nop,nop,TS val 1634527517 ecr 681964959], length 120
10:29:14.102857 IP server.29543 > client.30280: Flags [P.], seq 3024362072:3024362732, ack 4048786057, win 287, options [nop,nop,TS val 681964986 ecr 1634527517], length 660
10:29:14.171874 IP client.30280 > server.29543: Flags [.], ack 3024362732, win 273, options [nop,nop,TS val 1634527584 ecr 681964986], length 0
10:29:14.171890 IP server.29543 > client.30280: Flags [P.], seq 3024362732:3024362784, ack 4048786057, win 287, options [nop,nop,TS val 681965055 ecr 1634527584], length 52
10:29:14.199156 IP client.30280 > server.29543: Flags [.], ack 3024362784, win 273, options [nop,nop,TS val 1634527614 ecr 681965055], length 0
10:29:14.200359 IP client.30280 > server.29543: Flags [P.], seq 4048786057:4048786585, ack 3024362784, win 273, options [nop,nop,TS val 1634527615 ecr 681965055], length 528
10:29:14.201431 IP server.29543 > client.30280: Flags [P.], seq 3024362784:3024362892, ack 4048786585, win 307, options [nop,nop,TS val 681965084 ecr 1634527615], length 108
10:29:14.227696 IP client.30280 > server.29543: Flags [P.], seq 4048786585:4048786637, ack 3024362892, win 273, options [nop,nop,TS val 1634527643 ecr 681965084], length 52
10:29:14.227737 IP server.29543 > client.30280: Flags [P.], seq 3024362892:3024362992, ack 4048786637, win 307, options [nop,nop,TS val 681965111 ecr 1634527643], length 100
10:29:14.296663 IP client.30280 > server.29543: Flags [.], ack 3024362992, win 273, options [nop,nop,TS val 1634527709 ecr 681965111], length 0
10:29:14.296678 IP server.29543 > client.30280: Flags [P.], seq 3024362992:3024363164, ack 4048786637, win 307, options [nop,nop,TS val 681965179 ecr 1634527709], length 172
10:29:14.322190 IP client.30280 > server.29543: Flags [.], ack 3024363164, win 284, options [nop,nop,TS val 1634527738 ecr 681965179], length 0
10:29:14.891087 IP client.30280 > server.29543: Flags [P.], seq 4048786637:4048786673, ack 3024363164, win 284, options [nop,nop,TS val 1634528306 ecr 681965179], length 36
10:29:14.891605 IP server.29543 > client.30280: Flags [P.], seq 3024363164:3024363200, ack 4048786673, win 307, options [nop,nop,TS val 681965774 ecr 1634528306], length 36
10:29:14.895150 IP server.29543 > client.30280: Flags [.], seq 3024363200:3024364574, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895201 IP server.29543 > client.30280: Flags [.], seq 3024364574:3024365948, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895329 IP server.29543 > client.30280: Flags [.], seq 3024365948:3024367322, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.895334 IP server.29543 > client.30280: Flags [.], seq 3024367322:3024368696, ack 4048786673, win 307, options [nop,nop,TS val 681965778 ecr 1634528306], length 1374
10:29:14.896045 IP server.29543 > client.30280: Flags [.], seq 3024368696:3024370070, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896056 IP server.29543 > client.30280: Flags [.], seq 3024370070:3024371444, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896058 IP server.29543 > client.30280: Flags [.], seq 3024371444:3024372818, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896380 IP server.29543 > client.30280: Flags [.], seq 3024372818:3024374192, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.896384 IP server.29543 > client.30280: Flags [.], seq 3024374192:3024375566, ack 4048786673, win 307, options [nop,nop,TS val 681965779 ecr 1634528306], length 1374
10:29:14.917086 IP client.30280 > server.29543: Flags [.], ack 3024363200, win 284, options [nop,nop,TS val 1634528332 ecr 681965774], length 0
10:29:14.917104 IP server.29543 > client.30280: Flags [.], seq 3024375566:3024376940, ack 4048786673, win 307, options [nop,nop,TS val 681965800 ecr 1634528332], length 1374
10:29:14.917110 IP server.29543 > client.30280: Flags [.], seq 3024376940:3024378314, ack 4048786673, win 307, options [nop,nop,TS val 681965800 ecr 1634528332], length 1374
10:29:14.921290 IP client.30280 > server.29543: Flags [.], ack 3024364574, win 307, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.921305 IP server.29543 > client.30280: Flags [.], seq 3024378314:3024379688, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.921311 IP server.29543 > client.30280: Flags [.], seq 3024379688:3024381062, ack 4048786673, win 307, options [nop,nop,TS val 681965804 ecr 1634528336], length 1374
10:29:14.922166 IP client.30280 > server.29543: Flags [.], ack 3024365948, win 329, options [nop,nop,TS val 1634528336 ecr 681965778], length 0
10:29:14.922178 IP server.29543 > client.30280: Flags [.], seq 3024381062:3024382436, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.922182 IP server.29543 > client.30280: Flags [.], seq 3024382436:3024383810, ack 4048786673, win 307, options [nop,nop,TS val 681965805 ecr 1634528336], length 1374
10:29:14.923848 IP client.30280 > server.29543: Flags [.], ack 3024367322, win 352, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.923860 IP server.29543 > client.30280: Flags [.], seq 3024383810:3024385184, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.923863 IP server.29543 > client.30280: Flags [.], seq 3024385184:3024386558, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924175 IP client.30280 > server.29543: Flags [.], ack 3024368696, win 375, options [nop,nop,TS val 1634528338 ecr 681965778], length 0
10:29:14.924187 IP server.29543 > client.30280: Flags [.], seq 3024386558:3024387932, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.924190 IP server.29543 > client.30280: Flags [.], seq 3024387932:3024389306, ack 4048786673, win 307, options [nop,nop,TS val 681965807 ecr 1634528338], length 1374
10:29:14.925410 IP client.30280 > server.29543: Flags [.], ack 3024370070, win 397, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.925422 IP server.29543 > client.30280: Flags [.], seq 3024389306:3024390680, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 1374
10:29:14.925425 IP server.29543 > client.30280: Flags [P.], seq 3024390680:3024391132, ack 4048786673, win 307, options [nop,nop,TS val 681965808 ecr 1634528339], length 452
10:29:14.926030 IP client.30280 > server.29543: Flags [.], ack 3024371444, win 420, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926115 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
10:29:14.926122 IP server.29543 > client.30280: Flags [.], seq 3024392506:3024393880, ack 4048786673, win 307, options [nop,nop,TS val 681965809 ecr 1634528339], length 1374
10:29:14.926353 IP client.30280 > server.29543: Flags [.], ack 3024372818, win 443, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.926924 IP client.30280 > server.29543: Flags [.], ack 3024374192, win 465, options [nop,nop,TS val 1634528339 ecr 681965779], length 0
10:29:14.927677 IP client.30280 > server.29543: Flags [.], ack 3024375566, win 488, options [nop,nop,TS val 1634528340 ecr 681965779], length 0
10:29:14.928109 IP server.29543 > client.30280: Flags [.], seq 3024393880:3024395254, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928113 IP server.29543 > client.30280: Flags [.], seq 3024395254:3024396628, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928701 IP server.29543 > client.30280: Flags [.], seq 3024396628:3024398002, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.928705 IP server.29543 > client.30280: Flags [.], seq 3024398002:3024399376, ack 4048786673, win 307, options [nop,nop,TS val 681965811 ecr 1634528340], length 1374
10:29:14.929817 IP server.29543 > client.30280: Flags [.], seq 3024399376:3024400750, ack 4048786673, win 307, options [nop,nop,TS val 681965813 ecr 1634528340], length 1374
10:29:14.929821 IP server.29543 > client.30280: Flags [.], seq 3024400750:3024402124, ack 4048786673, win 307, options [nop,nop,TS val 681965813 ecr 1634528340], length 1374
10:29:14.942186 IP client.30280 > server.29543: Flags [.], ack 3024376940, win 510, options [nop,nop,TS val 1634528358 ecr 681965800], length 0
10:29:14.942203 IP server.29543 > client.30280: Flags [.], seq 3024402124:3024403498, ack 4048786673, win 307, options [nop,nop,TS val 681965825 ecr 1634528358], length 1374
10:29:14.942208 IP server.29543 > client.30280: Flags [.], seq 3024403498:3024404872, ack 4048786673, win 307, options [nop,nop,TS val 681965825 ecr 1634528358], length 1374
10:29:14.942782 IP client.30280 > server.29543: Flags [.], ack 3024378314, win 533, options [nop,nop,TS val 1634528358 ecr 681965800], length 0
10:29:14.942797 IP server.29543 > client.30280: Flags [.], seq 3024404872:3024406246, ack 4048786673, win 307, options [nop,nop,TS val 681965826 ecr 1634528358], length 1374
10:29:14.942800 IP server.29543 > client.30280: Flags [.], seq 3024406246:3024407620, ack 4048786673, win 307, options [nop,nop,TS val 681965826 ecr 1634528358], length 1374
10:29:14.946310 IP client.30280 > server.29543: Flags [.], ack 3024379688, win 556, options [nop,nop,TS val 1634528362 ecr 681965804], length 0
10:29:14.946326 IP server.29543 > client.30280: Flags [.], seq 3024407620:3024408994, ack 4048786673, win 307, options [nop,nop,TS val 681965829 ecr 1634528362], length 1374
10:29:14.946332 IP server.29543 > client.30280: Flags [.], seq 3024408994:3024410368, ack 4048786673, win 307, options [nop,nop,TS val 681965829 ecr 1634528362], length 1374
10:29:14.946922 IP client.30280 > server.29543: Flags [.], ack 3024381062, win 578, options [nop,nop,TS val 1634528363 ecr 681965804], length 0
10:29:14.946937 IP server.29543 > client.30280: Flags [.], seq 3024410368:3024411742, ack 4048786673, win 307, options [nop,nop,TS val 681965830 ecr 1634528363], length 1374
10:29:14.946940 IP server.29543 > client.30280: Flags [.], seq 3024411742:3024413116, ack 4048786673, win 307, options [nop,nop,TS val 681965830 ecr 1634528363], length 1374
10:29:14.947854 IP client.30280 > server.29543: Flags [.], ack 3024382436, win 601, options [nop,nop,TS val 1634528363 ecr 681965805], length 0
10:29:14.947870 IP server.29543 > client.30280: Flags [.], seq 3024413116:3024414490, ack 4048786673, win 307, options [nop,nop,TS val 681965831 ecr 1634528363], length 1374
10:29:14.947873 IP server.29543 > client.30280: Flags [.], seq 3024414490:3024415864, ack 4048786673, win 307, options [nop,nop,TS val 681965831 ecr 1634528363], length 1374
10:29:14.948437 IP client.30280 > server.29543: Flags [.], ack 3024383810, win 624, options [nop,nop,TS val 1634528364 ecr 681965805], length 0
10:29:14.949374 IP server.29543 > client.30280: Flags [.], seq 3024415864:3024417238, ack 4048786673, win 307, options [nop,nop,TS val 681965832 ecr 1634528364], length 1374
10:29:14.949378 IP server.29543 > client.30280: Flags [.], seq 3024417238:3024418612, ack 4048786673, win 307, options [nop,nop,TS val 681965832 ecr 1634528364], length 1374
10:29:14.950039 IP client.30280 > server.29543: Flags [.], ack 3024385184, win 646, options [nop,nop,TS val 1634528365 ecr 681965807], length 0
10:29:14.951071 IP server.29543 > client.30280: Flags [.], seq 3024418612:3024419986, ack 4048786673, win 307, options [nop,nop,TS val 681965834 ecr 1634528365], length 1374
10:29:14.951075 IP server.29543 > client.30280: Flags [.], seq 3024419986:3024421360, ack 4048786673, win 307, options [nop,nop,TS val 681965834 ecr 1634528365], length 1374
10:29:14.951334 IP client.30280 > server.29543: Flags [.], ack 3024386558, win 669, options [nop,nop,TS val 1634528366 ecr 681965807], length 0
10:29:14.951824 IP client.30280 > server.29543: Flags [.], ack 3024387932, win 691, options [nop,nop,TS val 1634528366 ecr 681965807], length 0
10:29:14.952159 IP client.30280 > server.29543: Flags [.], ack 3024389306, win 714, options [nop,nop,TS val 1634528367 ecr 681965807], length 0
10:29:14.953216 IP server.29543 > client.30280: Flags [.], seq 3024421360:3024422734, ack 4048786673, win 307, options [nop,nop,TS val 681965836 ecr 1634528367], length 1374
10:29:14.953220 IP server.29543 > client.30280: Flags [.], seq 3024422734:3024424108, ack 4048786673, win 307, options [nop,nop,TS val 681965836 ecr 1634528367], length 1374
10:29:14.953473 IP client.30280 > server.29543: Flags [.], ack 3024390680, win 737, options [nop,nop,TS val 1634528368 ecr 681965808], length 0
10:29:14.954411 IP server.29543 > client.30280: Flags [.], seq 3024424108:3024425482, ack 4048786673, win 307, options [nop,nop,TS val 681965837 ecr 1634528368], length 1374
10:29:14.954415 IP server.29543 > client.30280: Flags [.], seq 3024425482:3024426856, ack 4048786673, win 307, options [nop,nop,TS val 681965837 ecr 1634528368], length 1374
10:29:14.955336 IP client.30280 > server.29543: Flags [.], ack 3024391132, win 758, options [nop,nop,TS val 1634528369 ecr 681965808], length 0
10:29:14.955646 IP server.29543 > client.30280: Flags [.], seq 3024426856:3024428230, ack 4048786673, win 307, options [nop,nop,TS val 681965838 ecr 1634528369], length 1374
10:29:14.955650 IP server.29543 > client.30280: Flags [.], seq 3024428230:3024429604, ack 4048786673, win 307, options [nop,nop,TS val 681965838 ecr 1634528369], length 1374
10:29:14.956485 IP client.30280 > server.29543: Flags [.], ack 3024392506, win 781, options [nop,nop,TS val 1634528369 ecr 681965809], length 0
10:29:14.956836 IP server.29543 > client.30280: Flags [.], seq 3024429604:3024430978, ack 4048786673, win 307, options [nop,nop,TS val 681965840 ecr 1634528369], length 1374
10:29:14.956841 IP server.29543 > client.30280: Flags [.], seq 3024430978:3024432352, ack 4048786673, win 307, options [nop,nop,TS val 681965840 ecr 1634528369], length 1374
10:29:14.957123 IP client.30280 > server.29543: Flags [.], ack 3024393880, win 803, options [nop,nop,TS val 1634528370 ecr 681965809], length 0
10:29:14.958055 IP client.30280 > server.29543: Flags [.], ack 3024395254, win 826, options [nop,nop,TS val 1634528371 ecr 681965811], length 0
10:29:14.958541 IP client.30280 > server.29543: Flags [.], ack 3024396628, win 849, options [nop,nop,TS val 1634528371 ecr 681965811], length 0
10:29:14.958931 IP client.30280 > server.29543: Flags [.], ack 3024398002, win 871, options [nop,nop,TS val 1634528372 ecr 681965811], length 0
10:29:14.959265 IP client.30280 > server.29543: Flags [.], ack 3024399376, win 894, options [nop,nop,TS val 1634528373 ecr 681965811], length 0
10:29:14.960799 IP client.30280 > server.29543: Flags [.], ack 3024400750, win 917, options [nop,nop,TS val 1634528375 ecr 681965813], length 0
10:29:14.961117 IP client.30280 > server.29543: Flags [.], ack 3024402124, win 939, options [nop,nop,TS val 1634528376 ecr 681965813], length 0
10:29:14.961443 IP server.29543 > client.30280: Flags [.], seq 3024432352:3024433726, ack 4048786673, win 307, options [nop,nop,TS val 681965844 ecr 1634528369], length 1374
10:29:14.961449 IP server.29543 > client.30280: Flags [.], seq 3024433726:3024435100, ack 4048786673, win 307, options [nop,nop,TS val 681965844 ecr 1634528369], length 1374
10:29:14.969116 IP client.30280 > server.29543: Flags [.], ack 3024403498, win 962, options [nop,nop,TS val 1634528384 ecr 681965825], length 0
10:29:14.969924 IP client.30280 > server.29543: Flags [.], ack 3024404872, win 984, options [nop,nop,TS val 1634528384 ecr 681965825], length 0
10:29:14.970246 IP client.30280 > server.29543: Flags [.], ack 3024406246, win 1007, options [nop,nop,TS val 1634528384 ecr 681965826], length 0
10:29:14.970652 IP client.30280 > server.29543: Flags [.], ack 3024407620, win 1030, options [nop,nop,TS val 1634528385 ecr 681965826], length 0
10:29:14.971939 IP client.30280 > server.29543: Flags [.], ack 3024408994, win 1052, options [nop,nop,TS val 1634528388 ecr 681965829], length 0
10:29:14.973628 IP client.30280 > server.29543: Flags [.], ack 3024410368, win 1075, options [nop,nop,TS val 1634528389 ecr 681965829], length 0
10:29:14.974381 IP client.30280 > server.29543: Flags [.], ack 3024411742, win 1098, options [nop,nop,TS val 1634528390 ecr 681965830], length 0
10:29:14.975803 IP client.30280 > server.29543: Flags [.], ack 3024413116, win 1120, options [nop,nop,TS val 1634528391 ecr 681965830], length 0
10:29:14.976415 IP client.30280 > server.29543: Flags [.], ack 3024414490, win 1143, options [nop,nop,TS val 1634528392 ecr 681965831], length 0
10:29:14.977895 IP client.30280 > server.29543: Flags [.], ack 3024415864, win 1165, options [nop,nop,TS val 1634528393 ecr 681965831], length 0
10:29:14.978221 IP client.30280 > server.29543: Flags [.], ack 3024417238, win 1188, options [nop,nop,TS val 1634528393 ecr 681965832], length 0
10:29:14.978864 IP client.30280 > server.29543: Flags [.], ack 3024418612, win 1211, options [nop,nop,TS val 1634528394 ecr 681965832], length 0
10:29:14.979653 IP client.30280 > server.29543: Flags [.], ack 3024419986, win 1233, options [nop,nop,TS val 1634528395 ecr 681965834], length 0
10:29:14.979976 IP client.30280 > server.29543: Flags [.], ack 3024421360, win 1256, options [nop,nop,TS val 1634528396 ecr 681965834], length 0
10:29:14.980864 IP client.30280 > server.29543: Flags [.], ack 3024422734, win 1279, options [nop,nop,TS val 1634528396 ecr 681965836], length 0
10:29:14.981489 IP client.30280 > server.29543: Flags [.], ack 3024424108, win 1301, options [nop,nop,TS val 1634528397 ecr 681965836], length 0
10:29:14.981877 IP client.30280 > server.29543: Flags [.], ack 3024425482, win 1324, options [nop,nop,TS val 1634528397 ecr 681965837], length 0
10:29:14.985230 IP client.30280 > server.29543: Flags [.], ack 3024426856, win 1346, options [nop,nop,TS val 1634528401 ecr 681965837], length 0
10:29:14.986222 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024426856, win 1346, options [nop,nop,TS val 1634528401 ecr 681965837], length 36
10:29:14.987058 IP client.30280 > server.29543: Flags [.], ack 3024429604, win 1392, options [nop,nop,TS val 1634528402 ecr 681965838], length 0
10:29:14.988349 IP client.30280 > server.29543: Flags [.], ack 3024432352, win 1437, options [nop,nop,TS val 1634528403 ecr 681965840], length 0
10:29:14.989103 IP client.30280 > server.29543: Flags [.], ack 3024435100, win 1444, options [nop,nop,TS val 1634528404 ecr 681965844], length 0
10:29:15.021406 IP server.29543 > client.30280: Flags [.], seq 3024435100:3024436474, ack 4048786673, win 307, options [nop,nop,TS val 681965904 ecr 1634528369], length 1374
10:29:15.087837 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528503 ecr 681965904], length 0
10:29:15.242785 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528637 ecr 681965904], length 36
10:29:15.255489 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966138 ecr 1634528369], length 1374
10:29:15.281581 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634528697 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.555466 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634528869 ecr 681965904], length 36
10:29:15.719510 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681966602 ecr 1634528369], length 1374
10:29:15.746624 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634529162 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:15.918363 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634529333 ecr 681965904], length 36
10:29:16.695512 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681967578 ecr 1634528369], length 1374
10:29:16.737481 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634530143 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:16.886594 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634530301 ecr 681965904], length 36
10:29:18.551512 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681969434 ecr 1634528369], length 1374
10:29:18.578152 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634531993 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:18.742471 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634532157 ecr 681965904], length 36
10:29:22.263513 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681973146 ecr 1634528369], length 1374
10:29:22.298045 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634535706 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:22.456055 IP client.30280 > server.29543: Flags [P.], seq 4048786673:4048786709, ack 3024436474, win 1444, options [nop,nop,TS val 1634535869 ecr 681965904], length 36
10:29:22.698033 IP client.30280 > server.29543: Flags [FP.], seq 4048786709:4048786849, ack 3024436474, win 1444, options [nop,nop,TS val 1634536113 ecr 681965904], length 140
10:29:29.943515 IP server.29543 > client.30280: Flags [.], seq 3024391132:3024392506, ack 4048786673, win 307, options [nop,nop,TS val 681980826 ecr 1634528369], length 1374
10:29:29.976496 IP client.30280 > server.29543: Flags [.], ack 3024436474, win 1444, options [nop,nop,TS val 1634543386 ecr 681965904,nop,nop,sack 1 {3024391132:3024392506}], length 0
10:29:30.326520 IP client.30280 > server.29543: Flags [FP.], seq 4048786673:4048786849, ack 3024436474, win 1444, options [nop,nop,TS val 1634543741 ecr 681965904], length 176

[-- Attachment #3: ulogd.json --]
[-- Type: application/json, Size: 48712 bytes --]

[-- Attachment #4: dmesg.xz --]
[-- Type: application/octet-stream, Size: 500772 bytes --]