High number of concurrent connections causes 2.6.4 to go berserk

High number of concurrent connections causes 2.6.4 to go berserk

[Christopher Chan]

KERNEL: assertion (flags & MSG_PEEK) failed a[t net/ipv4/tcp.c (1571)
KERNEL: assertion (tp->copied_seq == tp->rcv_nxt || (flags & (MSG_PEEK | 
MSG_TRUNC))) failed at net/ipv4/tcp.c (1635)

These are repeated in the logs during kernel berserk period

postfix is running on this box.

It is set to spawn max 750 smtpd processes to handle incoming smtp sessions.

When tcp_max_syn_backlog is set to 4096 and
route.max_size = 131072
route.gc_thresh = 65536

the box goes berserk almost immediately after bootup due to a bounce 
flood hitting the box.

The below values seem to workaround whatever problem the kernel has.

tcp_max_syn_backlog = 2048
route.max_size = 1048576
route.gc_thresh = 65536

Re: High number of concurrent connections causes 2.6.4 to go berserk

[Nivedita Singhvi]

Christopher Chan wrote:

> When tcp_max_syn_backlog is set to 4096 and
> route.max_size = 131072
> route.gc_thresh = 65536
> 
> the box goes berserk almost immediately after bootup due to a bounce 
> flood hitting the box.

> The below values seem to workaround whatever problem the kernel has.
> 
> tcp_max_syn_backlog = 2048
> route.max_size = 1048576
> route.gc_thresh = 65536

Did you try just one of the first two and see which
one actually made a difference? I'm assuming the first,
but that might not be the case..

thanks,
Nivedita

Re: High number of concurrent connections causes 2.6.4 to go berserk

[Christopher Chan]

Nivedita Singhvi wrote:
> Christopher Chan wrote:
> 
>> When tcp_max_syn_backlog is set to 4096 and
>> route.max_size = 131072
>> route.gc_thresh = 65536
>>
>> the box goes berserk almost immediately after bootup due to a bounce 
>> flood hitting the box.
> 
> 
>> The below values seem to workaround whatever problem the kernel has.
>>
>> tcp_max_syn_backlog = 2048
>> route.max_size = 1048576
>> route.gc_thresh = 65536
> 
> 
> Did you try just one of the first two and see which
> one actually made a difference? I'm assuming the first,
> but that might not be the case..

Another box that has less traffic but sometimes gets ddosed by bounces 
has default settings

tcp_max_syn_backlog = 1024
route.max_size = 131072
route.gc_thresh = 8192

This box's 2.6.4 also went berserk during the ddos.

I'll try your suggestion and get back to you.

> 
> thanks,
> Nivedita
> 
> 
> 

Re: High number of concurrent connections causes 2.6.4 to go berserk

[Christopher Chan]

>> The below values seem to workaround whatever problem the kernel has.
>>
>> tcp_max_syn_backlog = 2048
>> route.max_size = 1048576
>> route.gc_thresh = 65536

I take that back. Under peak hours, the kernel starts spewing its stuff 
onto serial console and network connectivity is lost.

What it spews is not human readable anymore.

eg:

3->g:asag6A &| <ilfart/tv4d_>rKEt d n SE1PERUELnet p-15 ( t bL:(D3laEED1 
f))sspv/ipi
3->g:asag6A &| <3leaititv4d_>r<3_npiti (q G__TRNtd  (c p. 
=msRN||E1(f_P33KNC: t/ne>c7116p-ug ala7gsK 2
a) sev4ipie<6)
: ssgsA &  M3>ediliocp/tsre3>nxieio(M E_PTRNE na(t (.c==sgNE| 
1DflPE3D)C) a/ietco1)63->g:asag6A  |
ai fer4/pved6>
cvcses  s MMS>Kd leonp.tceqc>KxtedonMSE1PERUELnet p(1c = g EL (D3laEED1 
f)asipt/op)
3->g:asag6A  |
ai fer4/pved6>
cvcoes  s MMS>Kd leonp.tceqc>KxtedonMSE1PERUELnet p-1c = g EL (D3laEED1 
f))sipt/op)
3->g:asag6A &| <ilfart/tv4d_>r<3_npiti eqSGG_ERatd  (ccpq vmER | E 
(G_D3EKNC:et n->57(1tpbu: fl37gsK12fa) sev4ipie<6)
c csesA &  M3>ediliocp/tseec>nxieio(M E_PTRNE nattp(.c==sgNE| 1DflPE3D) 
))asietco1)63->g:asag6A &| <ilfart/tv4d_>r<3_npiti eqSGG_ERatd  (c p. 
=msRN| E (G_D3EKNC: t/ne>c711tpbu: fl37gsK 2
a) sev4ipi