Re: [BUG] Kernel recieves DNS reply, but doesn't deliver it to a waiting application

[Andrew Savchenko <bircoph@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 5420 bytes --]

Hello,

On Sun, 21 Oct 2012 14:52:21 +0200 Eric Dumazet wrote:
> On Sun, 2012-10-21 at 03:25 +0400, Andrew Savchenko wrote:
[...]
> > This bug is back again on kernel 3.4.14, but this time I was able to
> > get debug data and to recover running kernel without reboot.
> > 
> > Drowpatch showed that DNS UDP replies are always dropped here:
> > 1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
> > 
> > Another observations:
> > - only UDP replies are lost, TCP works fine;
> > - if network load is dropped dramatically (ip_forward disabled, most
> > network daemons are stopped) UDP DNS queries work again; but with
> > gradual load increase replies became first slow and than cease at all.
> > - CPU load is very low (uptime is below 0.05), so this shouldn't be
> > an insufficient computing power issue.
> > 
> > I found __udp_queue_rcv_skb function in net/ipv4/udp.c. From the code
> > and observations above it follows that this is likely to be a ENOMEM
> > condition leading to a packet loss.
> > 
> > This is a memory data after bug happened:
> > # cat /proc/meminfo
> > MemTotal:        1021576 kB
> > MemFree:           32056 kB
> > Buffers:          105204 kB
> > Cached:           646716 kB
> > SwapCached:          236 kB
> > Active:           205932 kB
> > Inactive:         587156 kB
> > Active(anon):      20636 kB
> > Inactive(anon):    22488 kB
> > Active(file):     185296 kB
> > Inactive(file):   564668 kB
> > Unevictable:        2152 kB
> > Mlocked:            2152 kB
> > SwapTotal:        995992 kB
> > SwapFree:         995020 kB
> > Dirty:                 0 kB
> > Writeback:             0 kB
> > AnonPages:         43120 kB
> > Mapped:             7504 kB
> > Shmem:               148 kB
> > Slab:             176004 kB
> > SReclaimable:     118636 kB
> > SUnreclaim:        57368 kB
> > KernelStack:         688 kB
> > PageTables:         2948 kB
> > NFS_Unstable:          0 kB
> > Bounce:                0 kB
> > WritebackTmp:          0 kB
> > CommitLimit:     1506780 kB
> > Committed_AS:      62708 kB
> > VmallocTotal:   34359738367 kB
> > VmallocUsed:      262732 kB
> > VmallocChunk:   34359474615 kB
> > AnonHugePages:         0 kB
> > DirectMap4k:       33536 kB
> > DirectMap2M:     1013760 kB
> > 
> > # sysctl -a | grep mem
> > net.core.optmem_max = 20480
> > net.core.rmem_default = 229376
> > net.core.rmem_max = 131071
> > net.core.wmem_default = 229376
> > net.core.wmem_max = 131071
> > net.ipv4.igmp_max_memberships = 20
> > net.ipv4.tcp_mem = 22350        29801   44700
> > net.ipv4.tcp_rmem = 4096        87380   6291456
> > net.ipv4.tcp_wmem = 4096        16384   4194304
> > net.ipv4.udp_mem = 24150        32202   48300
> > net.ipv4.udp_rmem_min = 4096
> > net.ipv4.udp_wmem_min = 4096
> > vm.lowmem_reserve_ratio = 256   256     32
> > vm.overcommit_memory = 0
> > 
> > Sysctl memory parameters are system defaults, I haven't changed them
> > via sysctl or /proc interfaces.
> > 
> > I tried to increase udm_mem values to the following:
> > net.ipv4.udp_mem = 100000       150000  200000
> > 
> > This solved my issue, at least for a while: DNS queries are working
> > fine now.
> > 
> > But I suspect that there is some memory loss in the kernel UDP stack,
> > because this issue never happens after reboot and always after about
> > a week of network operation. So this memory increase should help only
> > for a month or so, if memory loss is linear.
> > 
> > If you need some memory debug information, let me know which one and
> > what tools will be needed.
> 
> If drop is in __udp_queue_rcv_skb(), its not because you dont have
> enough memory. Frame was already received and handled by IP stack.
> 
> Thats because sock_queue_rcv_skb() said : there are already enough
> frames in socket receive buffer, I dont want to add another frame.

Actually there was a lot of dropwatch output. I decided that drops
are in __udp_queue_rcv_skb() by comparison between dropwatch outputs
under normal operation and under bug condition. I'm attaching both of
them.

> You forgot to attach :
> 
> cat /proc/net/udp
> netstat -s

Now bug is perfectly reproducible using net.ipv4.udp_mem values as
described in my previous mail. I reproduced reproducing bug
conditions and attaching these outputs.

> By the way, I suspect you are hit by skb recycling.
> (skb truesize is too big after few iterations)
> 
> We removed skb recycling in linux-3.7-rc1
> 
> If so, linux-3.7-rc1 or linux-3.7-rc2 should be fine.

I'll try 3.7 branch in a week or so. New kernel will require some
reconfiguration.

> What NIC are you using ?

This host has four NICs:
2x Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (10ec:8139)
2x D-Link System Inc DGE-528T Gigabit Ethernet Adapter (1462:223c)

One D-Link card is operating on 1Gbit/FD with mtu 7000, other cards
are used as 100Mbit/FD mtu 1500.

2 D-Link and 1 Realtek card are forming a bridge and remaining Realtek
card is used for an uplink. This host serves as a NAT between
the bridge and the uplink (both MASQUERADE and DNAT are used). Also it
has several ipsec tunnels for multiple hosts (mostly AH), l2tp tunnel
(independent from ipsec) and serves as a multicast router using
mrouted. Rather sophisticated ebtables, iptables and ipset setup is
used.

Best regards,
Andrew Savchenko

[-- Attachment #1.2: dropwatch.bug --]
[-- Type: application/octet-stream, Size: 6228 bytes --]

dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at netlink_unicast+1ae (0xffffffff8136a7cc)
891 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1043 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at unix_stream_connect+1b8 (0xffffffff813f479b)
4 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
2 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
861 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
3 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
3 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
760 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
610 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
3 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
915 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
737 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
1 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
592 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
624 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
629 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
2 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
506 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
573 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
2 drops at icmp_rcv+11d (0xffffffff813c3f39)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
506 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
4 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
574 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp_queue_rcv_skb+61 (0xffffffff813bd670)
571 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ppp_read+13c (0xffffffff812dcb86)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
878 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
707 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
548 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
735 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
684 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
702 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
560 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
668 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
824 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
762 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
3 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
857 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1096 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at ip_mr_forward+79 (0xffffffff813d816e)
1 drops at icmp_rcv+11d (0xffffffff813c3f39)
1170 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
^CGot a stop message


[-- Attachment #1.3: dropwatch.normal --]
[-- Type: application/octet-stream, Size: 3613 bytes --]

dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at netlink_unicast+1ae (0xffffffff8136a7cc)
1393 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1751 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
3 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1345 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
5 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
2116 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
6 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
4 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2098 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
7 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1978 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
7 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2299 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
6 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
2123 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
6 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
4 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
2082 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
5 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2136 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
5 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
2 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at ppp_read+13c (0xffffffff812dcb86)
1710 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
3 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1676 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
4 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1599 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
5 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
1989 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
7 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
5 drops at nf_hook_slow+107 (0xffffffff8136cac8)
3 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
2001 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
6 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
4 drops at nf_hook_slow+107 (0xffffffff8136cac8)
2 drops at ppp_read+13c (0xffffffff812dcb86)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
2189 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
7 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
3 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1805 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
3 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
5 drops at skb_free_datagram_locked+0 (0xffffffff8134200b)
1 drops at br_stp_rcv+77 (0xffffffff81438201)
1788 drops at packet_rcv_spkt+39 (0xffffffff8142aeb9)
2 drops at ipv6_rcv+147 (0xffffffff813fb9cd)
1 drops at nf_hook_slow+107 (0xffffffff8136cac8)
1 drops at __udp4_lib_rcv+4a1 (0xffffffff813c0d0c)
^CGot a stop message

[-- Attachment #1.4: netstat-s --]
[-- Type: application/octet-stream, Size: 3429 bytes --]

$ netstat -s
Ip:
    1258376226 total packets received
    68 with invalid headers
    8729 with invalid addresses
    986336591 forwarded
    0 incoming packets discarded
    226645746 incoming packets delivered
    1397083323 requests sent out
    4220 fragments dropped after timeout
    101092287 reassemblies required
    50310250 packets reassembled ok
    471302 packet reassembles failed
    156043507 fragments received ok
    13056 fragments failed
    312087425 fragments created
Icmp:
    16951 ICMP messages received
    10501 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 16908
        echo requests: 12
        echo replies: 31
    81582 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 78532
        time exceeded: 3006
        echo request: 32
        echo replies: 12
IcmpMsg:
        InType0: 31
        InType3: 16908
        InType8: 12
        OutType0: 12
        OutType3: 78532
        OutType8: 32
        OutType11: 3006
Tcp:
    5903 active connections openings
    3298 passive connection openings
    5697 failed connection attempts
    22 connection resets received
    3 connections established
    17065674 segments received
    6067567 segments send out
    11794 segments retransmited
    0 bad segments received.
    36 resets sent
Udp:
    198891130 packets received
    40714 packets to unknown port received.
    845 packet receive errors
    315052 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    18 resets received for embryonic SYN_RECV sockets
    15 packets pruned from receive queue because of socket buffer overrun
    9 ICMP packets dropped because they were out-of-window
    2769 TCP sockets finished time wait in fast timer
    2090 delayed acks sent
    1204 delayed acks further delayed because of locked socket
    Quick ack mode was activated 2 times
    1181909 packets directly queued to recvmsg prequeue.
    2172536151 bytes directly in process context from backlog
    1992120562 bytes directly received in process context from prequeue
    12959268 packet headers predicted
    2159999 packets header predicted and directly queued to user
    62159 acknowledgments not containing data payload received
    1855367 predicted acknowledgments
    90 times recovered from packet loss by selective acknowledgements
    12 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 2
    889 fast retransmits
    11 forward retransmits
    5694 other TCP timeouts
    1 SACK retransmits failed
    6 times receiver scheduled too late for direct processing
    1444 packets collapsed in receive queue due to low socket buffer
    3 DSACKs sent for old packets
    28 DSACKs received
    3 connections reset due to unexpected data
    40 connections aborted due to timeout
    TCPDSACKIgnoredOld: 1
    TCPDSACKIgnoredNoUndo: 3
    TCPSpuriousRTOs: 1
    TCPSackShifted: 4881
    TCPSackMerged: 1465
    TCPSackShiftFallback: 1217
    TCPBacklogDrop: 5000
    TCPDeferAcceptDrop: 2939
    IPReversePathFilter: 64181
    TCPRcvCoalesce: 4
IpExt:
    InMcastPkts: 532105
    OutMcastPkts: 310186
    InBcastPkts: 110406
    OutBcastPkts: 25354
    InOctets: 953399407223
    OutOctets: 1130740735729
    InMcastOctets: 88775757
    OutMcastOctets: 24850249
    InBcastOctets: 12840305
    OutBcastOctets: 2949734

[-- Attachment #1.5: proc_net_udp --]
[-- Type: application/octet-stream, Size: 4073 bytes --]

$ cat /proc/net/udp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
   13: 0100007F:2745 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253469 2 ffff88000423c380 0
   44: 00000000:2D64 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 251478 2 ffff88003d581180 0
   70: 00000000:857E 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253765 2 ffff880039aa1880 0
  171: 00000000:A9E3 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 4859 2 ffff88003d4ece00 0
  188: 7E0F330A:01F4 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5332 2 ffff880039aa0380 0
  199: 0100007F:03FF 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 4853 2 ffff88003d4eca80 0
  201: 00000000:0801 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253755 2 ffff880039aa0e00 0
  253: 010013AC:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253582 2 ffff880039aa0a80 0
  253: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253581 2 ffff880039aa0700 0
  267: 00000000:0043 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 251500 2 ffff88003d580e00 0
  288: 7E0F330A:9458 1200320A:06A5 01 00000000:00000000 00:00000000 00000000     0        0 254288 4 ffff88003b9b2e00 0
  311: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 4550 2 ffff88003d4ec000 0
  319: 00000000:0277 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253883 2 ffff880039aa1180 0
  323: 074A070A:007B 00000000:0000 07 FFFDF700:00000000 00:00000000 00000000   123        0 254469 2 ffff88003d581880 0
  323: 010213AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5189 2 ffff880039aa0000 0
  323: 010013AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5188 2 ffff880039993c00 0
  323: 00FCA8C0:007B 00000000:0000 07 FFFFF900:00000000 00:00000000 00000000     0        0 5187 2 ffff880039993880 0
  323: 7E0F330A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5186 2 ffff880039993500 0
  323: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5185 2 ffff880039993180 0
  323: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 5178 2 ffff880039992a80 0
  337: FF0013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253948 2 ffff88003d580380 0
  337: 010013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253947 2 ffff880039992e00 0
  337: FF0213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253944 2 ffff88000423ca80 0
  337: 010213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253943 2 ffff88000423ce00 0
  337: 00000000:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253940 2 ffff880039aa1c00 0
  338: FF0013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253950 2 ffff88003d581c00 0
  338: 010013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253949 2 ffff88003d580a80 0
  338: FF0213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253946 2 ffff88003d4edc00 0
  338: 010213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253945 2 ffff88000420e700 0
  338: 00000000:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253941 2 ffff88000423d180 0
  365: 00000000:06A5 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 254281 2 ffff88003b9b2380 0
  454: 00000000:90FE 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 253718 2 ffff880039aa1500 0
  477: 00000000:0315 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 4569 2 ffff88003d4ec380 0

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]