* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:14 ` Eric Dumazet
@ 2016-07-08 14:17 ` Loganaden Velvindron
2016-07-08 14:53 ` Eric Dumazet
2016-07-08 14:34 ` Toralf Förster
` (2 subsequent siblings)
3 siblings, 1 reply; 12+ messages in thread
From: Loganaden Velvindron @ 2016-07-08 14:17 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Toralf Förster, netdev
On Fri, Jul 8, 2016 at 6:14 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>> that system.
>>
>> The IPv6 monitoring from my ISP told my that the to be monitored
>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>> ipv4 there was no issue). Restarting the NIC brought back green lights
>> for the services at the ipv6 ports too.
>
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.
>
> IPv4 no longer has a routing cache, but IPv6 still has one.
>
Any pointers as to which part of the kernel to look for to implement
one for IPv6 ?
> Are you sure conntrack is needed at all ?
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:17 ` Loganaden Velvindron
@ 2016-07-08 14:53 ` Eric Dumazet
0 siblings, 0 replies; 12+ messages in thread
From: Eric Dumazet @ 2016-07-08 14:53 UTC (permalink / raw)
To: Loganaden Velvindron; +Cc: Toralf Förster, netdev
On Fri, 2016-07-08 at 18:17 +0400, Loganaden Velvindron wrote:
> On Fri, Jul 8, 2016 at 6:14 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
> >> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
> >> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
> >> that system.
> >>
> >> The IPv6 monitoring from my ISP told my that the to be monitored
> >> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
> >> ipv4 there was no issue). Restarting the NIC brought back green lights
> >> for the services at the ipv6 ports too.
> >
> > Hard to tell without knowing DDOS details, but IPv6 lacks some
> > scalability improvements found in IPv4.
> >
> > IPv4 no longer has a routing cache, but IPv6 still has one.
> >
>
> Any pointers as to which part of the kernel to look for to implement
> one for IPv6 ?
net/ipv6/route.c and net/ipv4/route.c ?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:14 ` Eric Dumazet
2016-07-08 14:17 ` Loganaden Velvindron
@ 2016-07-08 14:34 ` Toralf Förster
2016-07-08 15:40 ` Eric Dumazet
2016-07-08 15:03 ` Toralf Förster
2016-07-08 15:28 ` Hannes Frederic Sowa
3 siblings, 1 reply; 12+ messages in thread
From: Toralf Förster @ 2016-07-08 14:34 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> Are you sure conntrack is needed at all ?
Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.
It is my understanding that conntrack is best practise, right ?
--
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:34 ` Toralf Förster
@ 2016-07-08 15:40 ` Eric Dumazet
0 siblings, 0 replies; 12+ messages in thread
From: Eric Dumazet @ 2016-07-08 15:40 UTC (permalink / raw)
To: Toralf Förster; +Cc: netdev
On Fri, 2016-07-08 at 16:34 +0200, Toralf Förster wrote:
> On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> > Are you sure conntrack is needed at all ?
>
> Erm, I didn't mention conntrack - but yes, I do have in the firewall rules.
>
> It is my understanding that conntrack is best practise, right ?
It depends what you want to protect ?
linux TCP stack should work quite well without conntrack.
If you are aware of any known defect, we should fix TCP stack instead of
working around by adding a very expensive framework.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:14 ` Eric Dumazet
2016-07-08 14:17 ` Loganaden Velvindron
2016-07-08 14:34 ` Toralf Förster
@ 2016-07-08 15:03 ` Toralf Förster
2016-07-08 15:28 ` Hannes Frederic Sowa
3 siblings, 0 replies; 12+ messages in thread
From: Toralf Förster @ 2016-07-08 15:03 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On 07/08/2016 04:14 PM, Eric Dumazet wrote:
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.
Well, not too much I got from my ISP :
On 07 Jul 15:42, flow-save@traffic1.core.hetzner.de wrote:
> Direction IN
> Internal 5.9.158.75
> Threshold Packets 300.000 packets/s
> Sum 100.982.000 packets/300s (336.606 packets/s), 100.769 flows/300s (335 flows/s), 6,093 GByte/300s (166 MBit/s)
> External 84.1.57.68, 824.000 packets/300s (2.746 packets/s), 823 flows/300s (2 flows/s), 0,046 GByte/300s (1 MBit/s)
> External 78.189.187.159, 748.000 packets/300s (2.493 packets/s), 748 flows/300s (2 flows/s), 0,042 GByte/300s (1 MBit/s)
> External 85.105.165.143, 725.000 packets/300s (2.416 packets/s), 725 flows/300s (2 flows/s), 0,041 GByte/300s (1 MBit/s)
> External 88.250.158.160, 715.000 packets/300s (2.383 packets/s), 715 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
> External 88.248.51.37, 714.000 packets/300s (2.380 packets/s), 714 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
> External 78.186.151.2, 708.000 packets/300s (2.360 packets/s), 708 flows/300s (2 flows/s), 0,040 GByte/300s (1 MBit/s)
--
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 14:14 ` Eric Dumazet
` (2 preceding siblings ...)
2016-07-08 15:03 ` Toralf Förster
@ 2016-07-08 15:28 ` Hannes Frederic Sowa
2016-07-08 15:38 ` Eric Dumazet
3 siblings, 1 reply; 12+ messages in thread
From: Hannes Frederic Sowa @ 2016-07-08 15:28 UTC (permalink / raw)
To: Eric Dumazet, Toralf Förster; +Cc: netdev
On 08.07.2016 10:14, Eric Dumazet wrote:
> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>> that system.
>>
>> The IPv6 monitoring from my ISP told my that the to be monitored
>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>> ipv4 there was no issue). Restarting the NIC brought back green lights
>> for the services at the ipv6 ports too.
>
> Hard to tell without knowing DDOS details, but IPv6 lacks some
> scalability improvements found in IPv4.
>
> IPv4 no longer has a routing cache, but IPv6 still has one.
The difference is that routing exceptions are stored in "the" trie
instead of hash tables in the fib nodes. IPv4 limits that by the size of
the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
low.
Only redirects and mtu updates could potentially increase its size.
Redirects are limited to the same L2 network, MTU updates must hit the
socket to be acted upon appropriately. All limited to max_size, so I
currently see a major problem in the routing code.
Unfortunately your report has not enough details to pinpoint a specific
problem in the kernel
Bye,
Hannes
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 15:28 ` Hannes Frederic Sowa
@ 2016-07-08 15:38 ` Eric Dumazet
2016-07-08 15:43 ` Hannes Frederic Sowa
2016-07-08 16:10 ` Toralf Förster
0 siblings, 2 replies; 12+ messages in thread
From: Eric Dumazet @ 2016-07-08 15:38 UTC (permalink / raw)
To: Hannes Frederic Sowa; +Cc: Toralf Förster, netdev
On Fri, 2016-07-08 at 11:28 -0400, Hannes Frederic Sowa wrote:
> On 08.07.2016 10:14, Eric Dumazet wrote:
> > On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
> >> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
> >> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
> >> that system.
> >>
> >> The IPv6 monitoring from my ISP told my that the to be monitored
> >> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
> >> ipv4 there was no issue). Restarting the NIC brought back green lights
> >> for the services at the ipv6 ports too.
> >
> > Hard to tell without knowing DDOS details, but IPv6 lacks some
> > scalability improvements found in IPv4.
> >
> > IPv4 no longer has a routing cache, but IPv6 still has one.
>
> The difference is that routing exceptions are stored in "the" trie
> instead of hash tables in the fib nodes. IPv4 limits that by the size of
> the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
> low.
>
> Only redirects and mtu updates could potentially increase its size.
> Redirects are limited to the same L2 network, MTU updates must hit the
> socket to be acted upon appropriately. All limited to max_size, so I
> currently see a major problem in the routing code.
>
> Unfortunately your report has not enough details to pinpoint a specific
> problem in the kernel
Well, the typical DDOS simply use SYN flood, right ? ;)
With IPv4, a server can typically absorb 10 Mpps SYN without major
disruption on linux-4.6
With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps
30.44% [kernel] [k] ip6_pol_route.isra.49
12.93% [kernel] [k] fib6_lookup
12.35% [kernel] [k] fib6_get_table
10.36% [kernel] [k] _raw_read_lock_bh
8.29% [kernel] [k] _raw_read_unlock_bh
2.02% [kernel] [k] dst_release
1.86% [kernel] [k] memcpy_erms
I guess that switching to plain spinlock could help a bit, before major
surgery.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 15:38 ` Eric Dumazet
@ 2016-07-08 15:43 ` Hannes Frederic Sowa
2016-07-08 15:51 ` Eric Dumazet
2016-07-08 16:10 ` Toralf Förster
1 sibling, 1 reply; 12+ messages in thread
From: Hannes Frederic Sowa @ 2016-07-08 15:43 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Toralf Förster, netdev
On 08.07.2016 11:38, Eric Dumazet wrote:
> On Fri, 2016-07-08 at 11:28 -0400, Hannes Frederic Sowa wrote:
>> On 08.07.2016 10:14, Eric Dumazet wrote:
>>> On Fri, 2016-07-08 at 15:51 +0200, Toralf Förster wrote:
>>>> I do run a 4.6.3 hardened Gentoo kernel at a commodity i7 server. A
>>>> DDoS with about 300 MBit/sec over 5 mins resulted an issue for ipv6 at
>>>> that system.
>>>>
>>>> The IPv6 monitoring from my ISP told my that the to be monitored
>>>> services (80, 443, 52222) weren't reachable any longer at ipv6 (at
>>>> ipv4 there was no issue). Restarting the NIC brought back green lights
>>>> for the services at the ipv6 ports too.
>>>
>>> Hard to tell without knowing DDOS details, but IPv6 lacks some
>>> scalability improvements found in IPv4.
>>>
>>> IPv4 no longer has a routing cache, but IPv6 still has one.
>>
>> The difference is that routing exceptions are stored in "the" trie
>> instead of hash tables in the fib nodes. IPv4 limits that by the size of
>> the hash tables, in IPv6 we grow to ipv6/route/max_size, which is pretty
>> low.
>>
>> Only redirects and mtu updates could potentially increase its size.
>> Redirects are limited to the same L2 network, MTU updates must hit the
>> socket to be acted upon appropriately. All limited to max_size, so I
>> currently see a major problem in the routing code.
>>
>> Unfortunately your report has not enough details to pinpoint a specific
>> problem in the kernel
>
> Well, the typical DDOS simply use SYN flood, right ? ;)
Exactly, I can very well imagine that the stack becomes unresponsive
during DDoS, but after the DDoS I don't see a reason why services should
come up like in IPv4.
> With IPv4, a server can typically absorb 10 Mpps SYN without major
> disruption on linux-4.6
>
> With IPv6, kernel hits the route rwlock quite hard, at less than 2 Mpps
>
> 30.44% [kernel] [k] ip6_pol_route.isra.49
> 12.93% [kernel] [k] fib6_lookup
> 12.35% [kernel] [k] fib6_get_table
> 10.36% [kernel] [k] _raw_read_lock_bh
> 8.29% [kernel] [k] _raw_read_unlock_bh
> 2.02% [kernel] [k] dst_release
> 1.86% [kernel] [k] memcpy_erms
>
> I guess that switching to plain spinlock could help a bit, before major
> surgery.
Hmm, interesting idea.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 15:43 ` Hannes Frederic Sowa
@ 2016-07-08 15:51 ` Eric Dumazet
0 siblings, 0 replies; 12+ messages in thread
From: Eric Dumazet @ 2016-07-08 15:51 UTC (permalink / raw)
To: Hannes Frederic Sowa; +Cc: Toralf Förster, netdev
On Fri, 2016-07-08 at 11:43 -0400, Hannes Frederic Sowa wrote:
> Exactly, I can very well imagine that the stack becomes unresponsive
> during DDoS, but after the DDoS I don't see a reason why services should
> come up like in IPv4.
If the service uses different listeners, one for IPV4, one (or more) for
IPv6, it is possible that an IPV6 flood leaves the IPV6 part in a sad
state (this can also be an application bug, dealing with some backlog,
like DNS requests)
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: ipv6 issues after an DDoS for kernel 4.6.3
2016-07-08 15:38 ` Eric Dumazet
2016-07-08 15:43 ` Hannes Frederic Sowa
@ 2016-07-08 16:10 ` Toralf Förster
1 sibling, 0 replies; 12+ messages in thread
From: Toralf Förster @ 2016-07-08 16:10 UTC (permalink / raw)
To: Eric Dumazet, Hannes Frederic Sowa; +Cc: netdev
On 07/08/2016 05:38 PM, Eric Dumazet wrote:
> With IPv4, a server can typically absorb 10 Mpps SYN without major
> disruption on linux-4.6
Well, this particular server even survived >900 MBit/sec w/o any service disruption at IPv4 ([1])
but yesterday with a much more less attack the IPv6 issue was bothering me.
[1] https://www.zwiebeltoralf.de/torserver/ddos_sysstat_example.txt
--
Toralf
PGP: C4EACDDE 0076E94E, OTR: 420E74C8 30246EE7
^ permalink raw reply [flat|nested] 12+ messages in thread