TCP memory pressure question

TCP memory pressure question

[David Schwartz]

	When a Linux machine has reached the tcp_mem limit, what will happen to 
'write's on non-blocking sockets? Will they block until more TCP memory is 
available? Will they return an error code? ENOMEM?

	If it varies by kernel version, details about different versions would be 
extremely helpful. I'm most interested in late 2.4 kernels.

	Thanks in advance.

	DS

-- 
David Schwartz
<davids@webmaster.com>

Re: TCP memory pressure question

[Nivedita Singhvi]

>         When a Linux machine has reached the tcp_mem limit, what will happen to 
> 'write's on non-blocking sockets? Will they block until more TCP memory is 
> available? Will they return an error code? ENOMEM?
> 
>         If it varies by kernel version, details about different versions would be 
> extremely helpful. I'm most interested in late 2.4 kernels.
> 
>         Thanks in advance.
> 

Returns EAGAIN.
Fairly static ~late 2.4.

thanks,
Nivedita

Re: TCP memory pressure question

[Alex Riesen]

On Thu, Nov 21, 2002 at 05:27:45PM -0800, Nivedita Singhvi wrote:
> > When a Linux machine has reached the tcp_mem limit, what will happen to 
> > 'write's on non-blocking sockets? Will they block until more TCP memory is 
> > available? Will they return an error code? ENOMEM?
> > 
> > If it varies by kernel version, details about different versions would be 
> > extremely helpful. I'm most interested in late 2.4 kernels.
> > 
> >  Thanks in advance.
> Returns EAGAIN.
> Fairly static ~late 2.4.

returns number of bytes sent and sets errno to EAGAIN.

-alex

Re: TCP memory pressure question

[Gianni Tedesco]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu, 2002-11-21 at 21:34, David Schwartz wrote:
> 	When a Linux machine has reached the tcp_mem limit, what will happen to 
> 'write's on non-blocking sockets? Will they block until more TCP memory is 
> available? Will they return an error code? ENOMEM?

from write(2) man page.

EAGAIN Non-blocking I/O has been selected using O_NONBLOCK and the write
would block.

-- 
// Gianni Tedesco (gianni at ecsc dot co dot uk)
lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: TCP memory pressure question

[David Schwartz]

On 22 Nov 2002 12:06:29 +0000, Gianni Tedesco wrote:

>On Thu, 2002-11-21 at 21:34, David Schwartz wrote:

>>    When a Linux machine has reached the tcp_mem limit, what will happen to
>>'write's on non-blocking sockets? Will they block until more TCP memory is
>>available? Will they return an error code? ENOMEM?

>from write(2) man page.

>EAGAIN Non-blocking I/O has been selected using O_NONBLOCK and the write
>would block.

	So this would be a case where 'poll' or 'select' would return a write hit 
for a socket but 'write' would return -1 and set errno to EAGAIN.

	DS

Re: TCP memory pressure question

[Jamie Lokier]

David Schwartz wrote:
> 	So this would be a case where 'poll' or 'select' would return
> a write hit for a socket but 'write' would return -1 and set errno
> to EAGAIN.

Is this really true?  It would livelock several servers I've worked on...

-- Jamie

Re: TCP memory pressure question

[David Schwartz]

On Sat, 23 Nov 2002 00:06:17 +0000, Jamie Lokier wrote:

>David Schwartz wrote:

>>So this would be a case where 'poll' or 'select' would return
>>a write hit for a socket but 'write' would return -1 and set errno
>>to EAGAIN.

>Is this really true?  It would livelock several servers I've worked on...
>
>-- Jamie

	We're now getting close to my motivation for asking this question.

	If it does in fact return EAGAIN, then a poll/select loop program that is 
trying to write could get into trouble if another process was responsible for 
large receive queues. It would spin uselessly burning its timeslice as more 
data comes in and it delays the execution of the other process that might be 
able to drain receive queues.

	On the other hand, if send blocked, there would be a disaster if a 
select/poll loop process were the only TCP user on the box. In this case, it 
would deadlock. The process is blocked waiting for memory but the only way to 
get that memory is for the process to get around to reading from other 
connections, which it can't do because it's blocked.

	My motivation in assessing what happens is to develop a sane application 
strategy to detect and handle this condition. If we could detect it, we could 
try not to do TCP writes and try to do reads to relieve the memory problem.

	Some strategies I was considering were this:

	1) For applications that probably aren't using most of the TCP memory: In 
the select/poll loop, we keep track of whether we were able to do any work. 
If we never successfully wrote or read, but did get a hit on at least one 
socket, we sleep for a few milliseconds. (We hope that while we sleep other 
applications can free up more memory for us by draining their receive 
queues.)

	2) For applications that probably are using all or a significant fraction of 
the TCP memory: If a 'write' for a socket that we got a write hit on returns 
EAGAIN, for the rest of this poll/select cycle, we do not attempt any writes, 
only reads. (We hope that this will relieve the memory pressure scenario and 
allow the system to function normally as quickly as possible.)

	3) For applications that have control over how much load they take: If 
writes begin returning EAGAIN under suspicious circumstances, treat this as 
an overload condition and trigger your handling logic. Refuse expensive 
commands or refuse to handle new connections.

	Any comments or suggestions are appreciated. I've found that when we hit TCP 
memory pressure, many applications become very badly behaved.

	DS

RE: TCP memory pressure question

[Folkert van Heusden]

> 	Any comments or suggestions are appreciated. I've found that when we hit
TCP
> memory pressure, many applications become very badly behaved.

What about:

int WRITE(int handle, char *whereto, int len)
{
        int cnt=len;

        while(len>0)
        {
                int rc;

                rc = write(handle, whereto, len);

                if (rc == -1)
                {
                        if (errno == EINTR)
				{
					/* just try again */
				}
				else if (errno == EAGAIN)
				{
					/* give up time-slice */
					if (sched_yield() == -1)
					{
						/* BIG troubles */
                              	syslog(LOG_DEBUG, "WRITE(), during EAGAIN
handling: sched_yield failed! [%d - %s]", errno, strerror(errno));
                               	return -1;
					}
				}
				else
                        {
                                syslog(LOG_DEBUG, "WRITE(): io-error [%d -
%s]", errno, strerror(errno));
                                return -1;
                        }
                }
                else if (rc == 0)
                {
                        return 0;
                }
                else
                {
                        whereto += rc;
                        len -= rc;
                }
        }

        return cnt;
}

RE: TCP memory pressure question

[David Schwartz]

On Sat, 23 Nov 2002 23:34:15 +0100, Folkert van Heusden wrote:

>What about:

>int WRITE(int handle, char *whereto, int len)
>{
>int cnt=len;
>
>while(len>0)
>{
>int rc;
>
>rc = write(handle, whereto, len);
>
>if (rc == -1)
>{
>if (errno == EINTR)
>                {
>                    /* just try again */
>                }
>                else if (errno == EAGAIN)
>                {
>                    /* give up time-slice */
>                    if (sched_yield() == -1)
>                    {

	By yielding your time slice, you prevent yourself from performing any 
'read's. So what's going to fix the memory pressure?

	There are two cases where this could work:

	1) For an application that uses very little TCP memory. By yielding, perhaps 
some other process will come along, a more heavy TCP memory user, and fix the 
problem.

	2) For an application that uses a thread-per-connection architecture. By 
yielding, we give other threads a chance to run and hope that they will 
relieve the memory pressure.

	But for a poll/select loop application that is heavily using TCP memory, 
this makes things worse. By stopping our own execution, we delay the time 
when we'll do 'read's. So we extend the TCP memory pressure problem for even 
longer.

	DS