Out of socket memory? (2.4.0-test11)

Out of socket memory? (2.4.0-test11)

[Daniel Walton]

Hello,

I've been having a problem with a high volume Linux web server.  This 
particular web server used to be a FreeBSD machine and I've been trying to 
successfully make the switch for some time now.  I've been trying the 2.4 
development kernels as they come out and I've been tweaking the /proc 
filesystem variables but so far nothing seems to have fixed the 
problem.  The problem is that I get "Out of socket memory" errors and the 
networking locks up.  Sometimes the server will go for weeks without 
running into the problem and other times it'll last 30 minutes.  The 
hardware in question is an 1Ghz Athalon system with 256Mb of ram and an IDE 
hard disk.  I've tried every 2.4 test kernel to date.  The web server is a 
specialized web server running about 10 million hits a day.  Of the 256Mb 
of ram the web server uses 40Mb and there are no other significant memory 
consuming processes on the system.  Currently I am using the following 
/proc modifications in the rc.local file.

echo "7168 11776 16384" > /proc/sys/net/ipv4/tcp_mem
echo 32768 > /proc/sys/net/ipv4/tcp_max_orphans

What am I doing wrong?  Is this a kernel problem or a configuration 
problem?  Is there any way I can get runtime information from the kernel on 
things like amount of socket memory used and amount available?  Am I using 
the right variables to increase available socket memory and just not giving 
it enough yet?

I appreciate any help provided.

Thank you,
Daniel Walton


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Out of socket memory? (2.4.0-test11)

[Daniel Walton]

I'm not quite clear how the settings under /proc/sys/vm/* would effect the 
problem.  I neglected to mention in my previous post that all web content 
is served directly from the memory of the web server (no file 
accesses).  The only file accesses that happen are from a MySQL server 
which gets queried about once a second.

Here's the output of /proc/meminfo.  I'm not sure how helpful it is.  I was 
kinda hoping for something that would allow me to see how much memory had 
been allocated for sockets and what the max was.

[root@s4 /proc]# cat meminfo
         total:    used:    free:  shared: buffers:  cached:
Mem:  261742592 122847232 138895360        0  1757184 88633344
Swap: 271392768        0 271392768
MemTotal:       255608 kB
MemFree:        135640 kB
MemShared:           0 kB
Buffers:          1716 kB
Cached:          86556 kB
Active:          15684 kB
Inact_dirty:     72588 kB
Inact_clean:         0 kB
Inact_target:       68 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       255608 kB
LowFree:        135640 kB
SwapTotal:      265032 kB
SwapFree:       265032 kB


-Dan




At 12:30 AM 12/7/2000 -0500, you wrote:

>backlog queue?  tuning /proc/sys/vm/*?
>
> > problem?  Is there any way I can get runtime information from the 
> kernel on
> > things like amount of socket memory used and amount available?  Am I using
>
>/proc/meminfo?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Out of socket memory? (2.4.0-test11)

[Dan Kegel]

Daniel Walton (zwwe@opti.cgi.net) wrote:
> I've been having a problem with a high volume Linux web server. This 
> particular web server used to be a FreeBSD machine and I've been trying to 
> successfully make the switch for some time now. I've been trying the 2.4 
> development kernels as they come out and I've been tweaking the /proc 
> filesystem variables but so far nothing seems to have fixed the 
> problem. The problem is that I get "Out of socket memory" errors and the 
> networking locks up...

FWIW, the code that prints this is in ipv4/tcp_timer.c 
and it looks like it's a DoS attack countermeasure.
Looks like you can tune it with sysctl_tcp_max_orphans, among
other things.  Here's the code from test11-pre4:

/* Do not allow orphaned sockets to eat all our resources.
 * This is direct violation of TCP specs, but it is required
 * to prevent DoS attacks. It is called when a retransmission timeout
 * or zero probe timeout occurs on orphaned socket.
 *
 * Criterium is still not confirmed experimentally and may change.
 * We kill the socket, if:
 * 1. If number of orphaned sockets exceeds an administratively configured
 *    limit.
 * 2. If we have strong memory pressure.
 */
static int tcp_out_of_resources(struct sock *sk, int do_reset)
{
    struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp);
    int orphans = atomic_read(&tcp_orphan_count);
 
    /* If peer does not open window for long time, or did not transmit
     * anything for long time, penalize it. */
    if ((s32)(tcp_time_stamp - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset)
        orphans <<= 1;
 
    /* If some dubious ICMP arrived, penalize even more. */
    if (sk->err_soft)
        orphans <<= 1;
 
    if (orphans >= sysctl_tcp_max_orphans ||
        (sk->wmem_queued > SOCK_MIN_SNDBUF &&
         atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
        if (net_ratelimit())
            printk(KERN_INFO "Out of socket memory\n");
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Out of socket memory? (2.4.0-test11)

[Andi Kleen]

On Wed, Dec 06, 2000 at 10:56:32PM -0600, Daniel Walton wrote:
> 
> Hello,
> 
> I've been having a problem with a high volume Linux web server.  This 
> particular web server used to be a FreeBSD machine and I've been trying to 
> successfully make the switch for some time now.  I've been trying the 2.4 
> development kernels as they come out and I've been tweaking the /proc 
> filesystem variables but so far nothing seems to have fixed the 
> problem.  The problem is that I get "Out of socket memory" errors and the 
> networking locks up.  Sometimes the server will go for weeks without 

You should probably first find out what and why is generating these messages.
It isn't the kernel. When the web server is generating when a send blocks
then it is badly broken.


-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Out of socket memory? (2.4.0-test11)

[Dan Kegel]

Dan Kegel wrote:
> Daniel Walton (zwwe@opti.cgi.net) wrote:
> > I've been having a problem with a high volume Linux web server. This 
> > particular web server used to be a FreeBSD machine and I've been trying to 
> > successfully make the switch for some time now. I've been trying the 2.4 
> > development kernels as they come out and I've been tweaking the /proc 
> > filesystem variables but so far nothing seems to have fixed the 
> > problem. The problem is that I get "Out of socket memory" errors and the 
> > networking locks up...
>
> FWIW, the code that prints this is in ipv4/tcp_timer.c
> and it looks like it's a DoS attack countermeasure.
>
> /* Do not allow orphaned sockets to eat all our resources.
>  * This is direct violation of TCP specs, but it is required
>  * to prevent DoS attacks. It is called when a retransmission timeout
>  * or zero probe timeout occurs on orphaned socket.
> ...
>     if (orphans >= sysctl_tcp_max_orphans ||
>         (sk->wmem_queued > SOCK_MIN_SNDBUF &&
>          atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
>         if (net_ratelimit())
>             printk(KERN_INFO "Out of socket memory\n");

See Documentation/networking/ip-sysctl.txt for more info and tuning
suggestions, esp. tcp_orphan_retries and tcp_max_orphans.
An orphaned socket is one that has at least partially
opened, but has not been fully accepted.  You might try reducing
tcp_orphan_retries.

It looks like you can view the current number of orphans at
/proc/net/sockstat or something like that.
 
/*
 *  Report socket allocation statistics [mea@utu.fi]
 */
int afinet_get_info(char *buffer, char **start, off_t offset, int length)
{
    /* From  net/socket.c  */
    extern int socket_get_info(char *, char **, off_t, int);
 
    int len  = socket_get_info(buffer,start,offset,length);
 
    len += sprintf(buffer+len,"TCP: inuse %d orphan %d tw %d alloc %d mem %d\n",
               fold_prot_inuse(&tcp_prot),
               atomic_read(&tcp_orphan_count), tcp_tw_count,
               atomic_read(&tcp_sockets_allocated),
               atomic_read(&tcp_memory_allocated));     

- Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Out of socket memory? (2.4.0-test11)

[kuznet]

Hello!

> What am I doing wrong?

You change parameters without investigating why failure happened.
This approach cannot succeed, of course.


> problem?  Is there any way I can get runtime information from the kernel on 
> things like amount of socket memory used and amount available?

cat /proc/net/sockstat
cat /proc/net/netstat
cat /proc/net/snmp
cat /proc/slabinfo
cat /proc/net/tcp

Next time when you will see these messages, do this, pack the result
and send to me pointopoint.

Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/