ipv6 issues after an DDoS for kernel 4.6.3

ipv6 issues after an DDoS for kernel 4.6.3

[Toralf Förster]

I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at that system.

The IPv6 monitoring from my ISP told my that the to be monitored services (80, 443, 52222) weren't reachable any longer at ipv6 (at ipv4 there was no issue). Restarting the NIC brought back green lights for the services at the ipv6 ports too.

The log gave just :

Jul  7 15:36:28 ms-magpie kernel: ------------[ cut here ]------------
Jul  7 15:36:28 ms-magpie kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:306 dev_watchdog+0x243/0x260
Jul  7 15:36:28 ms-magpie kernel: NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out
Jul  7 15:36:28 ms-magpie kernel: Modules linked in: af_packet nf_log_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables i2c_i801 i2c_core tpm_tis tpm thermal processor atkbd button x86_pkg_temp_thermal
Jul  7 15:36:28 ms-magpie kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.6.3-hardened #1
Jul  7 15:36:28 ms-magpie kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 9002 05/30/2014
Jul  7 15:36:28 ms-magpie kernel:  0000000000000000 ffff88041fa03db8 ffffffffbb3d655b 0000000000000007
Jul  7 15:36:28 ms-magpie kernel:  ffff88041fa03e08 0000000000000000 ffff88041fa03df8 ffffffffbb07f7dd
Jul  7 15:36:28 ms-magpie kernel:  000001321fa11640 0000000000000000 ffff88040d354080 0000000000000000
Jul  7 15:36:28 ms-magpie kernel: Call Trace:
Jul  7 15:36:28 ms-magpie kernel:  <IRQ>  [<ffffffffbb3d655b>] dump_stack+0x4e/0x83
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb07f7dd>] __warn+0xcd/0x100
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb07f85a>] warn_slowpath_fmt+0x4a/0x70
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb59d633>] dev_watchdog+0x243/0x260
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb59d3f0>] ? dev_deactivate_queue+0x80/0x80
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb0db7b3>] call_timer_fn.isra.24+0x33/0xa0
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb59d3f0>] ? dev_deactivate_queue+0x80/0x80
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb0dba52>] run_timer_softirq+0x232/0x3c0
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb0eb188>] ? clockevents_program_event+0x98/0x160
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb08444d>] __do_softirq+0xfd/0x210
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb0846d0>] irq_exit+0x80/0xa0
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb03e9a4>] smp_apic_timer_interrupt+0x54/0x80
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb67805b>] apic_timer_interrupt+0x8b/0x90
Jul  7 15:36:28 ms-magpie kernel:  <EOI>  [<ffffffffbb53fa75>] ? cpuidle_enter_state+0x185/0x240
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb53fb82>] cpuidle_enter+0x12/0x30
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb0c0530>] cpu_startup_entry+0x1d0/0x220
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe13120>] ? early_idt_handler_array+0x120/0x120
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbb6701f5>] rest_init+0x6d/0x88
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe14c6c>] start_kernel+0x64c/0x692
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe13120>] ? early_idt_handler_array+0x120/0x120
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe7c7ff>] ? memblock_reserve+0x76/0x9c
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe136d7>] x86_64_start_reservations+0x53/0x75
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe136d7>] ? x86_64_start_reservations+0x53/0x75
Jul  7 15:36:28 ms-magpie kernel:  [<ffffffffbbe1382d>] x86_64_start_kernel+0x134/0x16f
Jul  7 15:36:28 ms-magpie kernel: ---[ end trace b779686b40691d67 ]---
Jul  7 15:36:28 ms-magpie kernel: r8169 0000:03:00.0 enp3s0: link up 

I did not try to restart just the firewall or so.
WHat let me wonder were why just the IPv6 had a problem, whereas ipV4 worked smoothly.

-- 
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Eric Dumazet]

On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
> that system.
> 
> The IPv6 monitoring from my ISP told my that the to be monitored
> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
> ipv4 there was no issue). Restarting the NIC brought back green lights
> for the services at the ipv6 ports too.

Hard to tell without knowing DDOS details, but IPv6 lacks some
scalability improvements found in IPv4.

IPv4 no longer has a routing cache, but IPv6 still has one.

Are you sure conntrack is needed at all ?

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Loganaden Velvindron]

On Fri, Jul 8, 2016 at 6:14 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>> that system.
>>
>> The IPv6 monitoring from my ISP told my that the to be monitored
>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>> ipv4 there was no issue). Restarting the NIC brought back green lights
>> for the services at the ipv6 ports too.
>
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.
>
> IPv4 no longer has a routing cache, but IPv6 still has one.
>

Any pointers as to which part of the kernel to look for to implement
one for IPv6 ?


> Are you sure conntrack is needed at all ?
>
>

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Toralf Förster]

On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> Are you sure conntrack is needed at all ?

Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.

It is my understanding that conntrack is best practise, right ?

-- 
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Eric Dumazet]

On Fri, 2016-07-08 at 18:17 +0400, Loganaden Velvindron wrote:
> On Fri, Jul 8, 2016 at 6:14 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
> >> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
> >> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
> >> that system.
> >>
> >> The IPv6 monitoring from my ISP told my that the to be monitored
> >> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
> >> ipv4 there was no issue). Restarting the NIC brought back green lights
> >> for the services at the ipv6 ports too.
> >
> > Hard to tell without knowing DDOS details, but IPv6 lacks some
> > scalability improvements found in IPv4.
> >
> > IPv4 no longer has a routing cache, but IPv6 still has one.
> >
> 
> Any pointers as to which part of the kernel to look for to implement
> one for IPv6 ?

net/ipv6/route.c and net/ipv4/route.c ?

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Toralf Förster]

On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.

Well, not too much I got from my ISP :

On 07 Jul 15:42, flow-save@traffic1.core.hetzner.de wrote:
> Direction IN
> Internal 5.9.158.75
> Threshold Packets 300.000 packets/s
> Sum 100.982.000 packets/300s (336.606 packets/s), 100.769 flows/300s (335 flows/s), 6,093 GByte/300s (166 MBit/s)
> External 84.1.57.68, 824.000 packets/300s (2.746 packets/s), 823 flows/300s (2 flows/s), 0,046 GByte/300s (1 MBit/s)
> External 78.189.187.159, 748.000 packets/300s (2.493 packets/s), 748 flows/300s (2 flows/s), 0,042 GByte/300s (1 MBit/s)
> External 85.105.165.143, 725.000 packets/300s (2.416 packets/s), 725 flows/300s (2 flows/s), 0,041 GByte/300s (1 MBit/s)
> External 88.250.158.160, 715.000 packets/300s (2.383 packets/s), 715 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
> External 88.248.51.37, 714.000 packets/300s (2.380 packets/s), 714 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
> External 78.186.151.2, 708.000 packets/300s (2.360 packets/s), 708 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)


-- 
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Hannes Frederic Sowa]

On 08.07.2016 10:14, Eric Dumazet wrote:
> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>> that system.
>>
>> The IPv6 monitoring from my ISP told my that the to be monitored
>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>> ipv4 there was no issue). Restarting the NIC brought back green lights
>> for the services at the ipv6 ports too.
> 
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.
> 
> IPv4 no longer has a routing cache, but IPv6 still has one.

The difference is that routing exceptions are stored in "the" trie
instead of hash tables in the fib nodes. IPv4 limits that by the size of
the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
low.

Only redirects and mtu updates could potentially increase its size.
Redirects are limited to the same L2 network, MTU updates must hit the
socket to be acted upon appropriately. All limited to max_size, so I
currently see a major problem in the routing code.

Unfortunately your report has not enough details to pinpoint a specific
problem in the kernel

Bye,
Hannes

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Eric Dumazet]

On Fri, 2016-07-08 at 11:28 -0400, Hannes Frederic Sowa wrote:
> On 08.07.2016 10:14, Eric Dumazet wrote:
> > On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
> >> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
> >> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
> >> that system.
> >>
> >> The IPv6 monitoring from my ISP told my that the to be monitored
> >> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
> >> ipv4 there was no issue). Restarting the NIC brought back green lights
> >> for the services at the ipv6 ports too.
> > 
> > Hard to tell without knowing DDOS details, but IPv6 lacks some
> > scalability improvements found in IPv4.
> > 
> > IPv4 no longer has a routing cache, but IPv6 still has one.
> 
> The difference is that routing exceptions are stored in "the" trie
> instead of hash tables in the fib nodes. IPv4 limits that by the size of
> the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
> low.
> 
> Only redirects and mtu updates could potentially increase its size.
> Redirects are limited to the same L2 network, MTU updates must hit the
> socket to be acted upon appropriately. All limited to max_size, so I
> currently see a major problem in the routing code.
> 
> Unfortunately your report has not enough details to pinpoint a specific
> problem in the kernel

Well, the typical DDOS simply use SYN flood, right ? ;)

With IPv4, a server can typically absorb 10 Mpps SYN without major
disruption on linux-4.6

With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps

    30.44%  [kernel]  [k] ip6_pol_route.isra.49             
    12.93%  [kernel]  [k] fib6_lookup                       
    12.35%  [kernel]  [k] fib6_get_table                    
    10.36%  [kernel]  [k] _raw_read_lock_bh                 
     8.29%  [kernel]  [k] _raw_read_unlock_bh               
     2.02%  [kernel]  [k] dst_release                       
     1.86%  [kernel]  [k] memcpy_erms       

I guess that switching to plain spinlock could help a bit, before major
surgery.

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Eric Dumazet]

On Fri, 2016-07-08 at 16:34 +0200, Toralf Förster wrote:
> On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> > Are you sure conntrack is needed at all ?
> 
> Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.
> 
> It is my understanding that conntrack is best practise, right ?

It depends what you want to protect ?

linux TCP stack should work quite well without conntrack.

If you are aware of any known defect, we should fix TCP stack instead of
working around by adding a very expensive framework.

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Hannes Frederic Sowa]

On 08.07.2016 11:38, Eric Dumazet wrote:
> On Fri, 2016-07-08 at 11:28 -0400, Hannes Frederic Sowa wrote:
>> On 08.07.2016 10:14, Eric Dumazet wrote:
>>> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>>>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>>>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>>>> that system.
>>>>
>>>> The IPv6 monitoring from my ISP told my that the to be monitored
>>>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>>>> ipv4 there was no issue). Restarting the NIC brought back green lights
>>>> for the services at the ipv6 ports too.
>>>
>>> Hard to tell without knowing DDOS details, but IPv6 lacks some
>>> scalability improvements found in IPv4.
>>>
>>> IPv4 no longer has a routing cache, but IPv6 still has one.
>>
>> The difference is that routing exceptions are stored in "the" trie
>> instead of hash tables in the fib nodes. IPv4 limits that by the size of
>> the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
>> low.
>>
>> Only redirects and mtu updates could potentially increase its size.
>> Redirects are limited to the same L2 network, MTU updates must hit the
>> socket to be acted upon appropriately. All limited to max_size, so I
>> currently see a major problem in the routing code.
>>
>> Unfortunately your report has not enough details to pinpoint a specific
>> problem in the kernel
> 
> Well, the typical DDOS simply use SYN flood, right ? ;)

Exactly, I can very well imagine that the stack becomes unresponsive
during DDoS, but after the DDoS I don't see a reason why services should
come up like in IPv4.

> With IPv4, a server can typically absorb 10 Mpps SYN without major
> disruption on linux-4.6
> 
> With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps
> 
>     30.44%  [kernel]  [k] ip6_pol_route.isra.49             
>     12.93%  [kernel]  [k] fib6_lookup                       
>     12.35%  [kernel]  [k] fib6_get_table                    
>     10.36%  [kernel]  [k] _raw_read_lock_bh                 
>      8.29%  [kernel]  [k] _raw_read_unlock_bh               
>      2.02%  [kernel]  [k] dst_release                       
>      1.86%  [kernel]  [k] memcpy_erms       
> 
> I guess that switching to plain spinlock could help a bit, before major
> surgery.

Hmm, interesting idea.

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Eric Dumazet]

On Fri, 2016-07-08 at 11:43 -0400, Hannes Frederic Sowa wrote:

> Exactly, I can very well imagine that the stack becomes unresponsive
> during DDoS, but after the DDoS I don't see a reason why services should
> come up like in IPv4.

If the service uses different listeners, one for IPV4, one (or more) for
IPv6, it is possible that an IPV6 flood leaves the IPV6 part in a sad
state (this can also be an application bug, dealing with some backlog,
like DNS requests)

Re: ipv6 issues after an DDoS for kernel 4.6.3

[Toralf Förster]

On 07/08/2016 05:38 PM, Eric Dumazet wrote:
> With IPv4, a server can typically absorb 10 Mpps SYN without major
> disruption on linux-4.6
Well, this particular server even survived >900 MBit/sec w/o any service disruption at IPv4 ([1])
but yesterday with a much more less attack the IPv6 issue was bothering me.

[1] https://www.zwiebeltoralf.de/torserver/ddos_sysstat_example.txt

-- 
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7