Memory issue with kernel sctp_connectx

Memory issue with kernel sctp_connectx

[Danny Smit]

Hi all,

I'm look into a possible memory issue when using sctp_connectx. This
week I've asked to following on the CentOS mailing list:

------------------------------
I'm running into a possible memory issue with the SCTP implementation
in CentOS 6, using lksctp-tools. So far I've not seen the problem with
other distributions yet.

The problem is that whenever a SCTP client connection is initiated to
a second host and port at which no SCTP server application is
listening

The scenario is simple:
- call socket() to create a socket.
- call sctp_connectx() to establish a connection.

The last call is repeated periodically to retry to establish a connection.

When looking on the wire using wireshark it shows that an SCTP INIT
packet is sent to the second host, which replies with an SCTP ABORT.

This is exactly according to the SCTP specification. However it
appears ABORT isn't propagated into to application layer that calls
sctp_connectx().

Furthermore, because reconnect attempts are made, a steady memory
increasement occurs. Looking at /proc/meminfo, the increasement occurs
in SUnreclaim, the kernel slab cache.

Can you advice if this can be an issue with the SCTP implementation in
the kernel? Maybe an issue solved upstream, but not (yet) backported
into centos 6?

It applies to the following package versions (also to older kernels):

- kernel-2.6.32-504.8.1.el6.x86_64
- lksctp-tools-1.0.10-7.el6.x86_64
------------------------------

Note that the API is used in non-blocking mode.

However, I've now seen the problem also with other distributions and
came to realize that it may not be a CentOS specific problem. Can
someone advice whether this may be an issue in the linux kernel sctp
implementation?

Kind regards,

Danny Smit

Memory issue with kernel sctp_connectx

[Danny Smit]

Hi all,

I'm look into a possible memory issue when using sctp_connectx. This
week I've asked to following on the CentOS mailing list:

------------------------------
I'm running into a possible memory issue with the SCTP implementation
in CentOS 6, using lksctp-tools. So far I've not seen the problem with
other distributions yet.

The problem is that whenever a SCTP client connection is initiated to
a second host and port at which no SCTP server application is
listening

The scenario is simple:
- call socket() to create a socket.
- call sctp_connectx() to establish a connection.

The last call is repeated periodically to retry to establish a connection.

When looking on the wire using wireshark it shows that an SCTP INIT
packet is sent to the second host, which replies with an SCTP ABORT.

This is exactly according to the SCTP specification. However it
appears ABORT isn't propagated into to application layer that calls
sctp_connectx().

Furthermore, because reconnect attempts are made, a steady memory
increasement occurs. Looking at /proc/meminfo, the increasement occurs
in SUnreclaim, the kernel slab cache.

Can you advice if this can be an issue with the SCTP implementation in
the kernel? Maybe an issue solved upstream, but not (yet) backported
into centos 6?

It applies to the following package versions (also to older kernels):

- kernel-2.6.32-504.8.1.el6.x86_64
- lksctp-tools-1.0.10-7.el6.x86_64
------------------------------

Note that the API is used in non-blocking mode.

However, I've now seen the problem also with other distributions and
came to realize that it may not be a CentOS specific problem. Can
someone advice whether this may be an issue in the linux kernel sctp
implementation?

Kind regards,

Danny Smit

Re: Memory issue with kernel sctp_connectx

[Adam Endrodi]

On Fri, Mar 06, 2015 at 11:52:05AM +0100, ext Danny Smit wrote:
> 
> I'm look into a possible memory issue when using sctp_connectx. This
> week I've asked to following on the CentOS mailing list:

I think you'll find this thread relevant:
http://www.spinics.net/lists/linux-sctp/msg03705.html

> ------------------------------
> I'm running into a possible memory issue with the SCTP implementation
> in CentOS 6, using lksctp-tools. So far I've not seen the problem with
> other distributions yet.
> 
> The problem is that whenever a SCTP client connection is initiated to
> a second host and port at which no SCTP server application is
> listening
> 
> The scenario is simple:
> - call socket() to create a socket.
> - call sctp_connectx() to establish a connection.
> 
> The last call is repeated periodically to retry to establish a connection.
> 
> When looking on the wire using wireshark it shows that an SCTP INIT
> packet is sent to the second host, which replies with an SCTP ABORT.
> 
> This is exactly according to the SCTP specification. However it
> appears ABORT isn't propagated into to application layer that calls
> sctp_connectx().
[...] 
> Note that the API is used in non-blocking mode.

This case sctp_connectx() doesn't wait until the connection is established.
You need to use sctp_recvmsg() to get the notification.

> Furthermore, because reconnect attempts are made, a steady memory
> increasement occurs. Looking at /proc/meminfo, the increasement occurs
> in SUnreclaim, the kernel slab cache.

Perhaps these are the event being queued up?

-- 
So if you walk away, who is gonna stay?

Re: Memory issue with kernel sctp_connectx

[Danny Smit]

On Fri, Mar 6, 2015 at 12:13 PM, Adam Endrodi wrote:
> On Fri, Mar 06, 2015 at 11:52:05AM +0100, ext Danny Smit wrote:
>>
>> I'm look into a possible memory issue when using sctp_connectx. This
>> week I've asked to following on the CentOS mailing list:
>
> I think you'll find this thread relevant:
> http://www.spinics.net/lists/linux-sctp/msg03705.html

Thanks for pointing me to this thread, it makes things a bit more clear.

>> ------------------------------
>> I'm running into a possible memory issue with the SCTP implementation
>> in CentOS 6, using lksctp-tools. So far I've not seen the problem with
>> other distributions yet.
>>
>> The problem is that whenever a SCTP client connection is initiated to
>> a second host and port at which no SCTP server application is
>> listening
>>
>> The scenario is simple:
>> - call socket() to create a socket.
>> - call sctp_connectx() to establish a connection.
>>
>> The last call is repeated periodically to retry to establish a connection.
>>
>> When looking on the wire using wireshark it shows that an SCTP INIT
>> packet is sent to the second host, which replies with an SCTP ABORT.
>>
>> This is exactly according to the SCTP specification. However it
>> appears ABORT isn't propagated into to application layer that calls
>> sctp_connectx().
> [...]
>> Note that the API is used in non-blocking mode.
>
> This case sctp_connectx() doesn't wait until the connection is established.
> You need to use sctp_recvmsg() to get the notification.

I tried to apply this as suggested, the scenario now is as follows:

First call sctp_connectx() on a socket in non blocking mode, which
results in the error code EINPROGRESS.

Next call sctp_recvmsg(), which results in the error code ENOTCONN
instead of getting any notifications.

This actually feels right because there is no connection established
yet. It also doesn't seem to make a difference for the memory
consumption. Am I missing something?

>> Furthermore, because reconnect attempts are made, a steady memory
>> increasement occurs. Looking at /proc/meminfo, the increasement occurs
>> in SUnreclaim, the kernel slab cache.
>
> Perhaps these are the event being queued up?

I tried disabling events just to test test this and the memory usage
remains stable. Therefore at first sight it indeed does seem to be
related to queued events.


> --
> So if you walk away, who is gonna stay?

Re: Memory issue with kernel sctp_connectx

[Adam Endrodi]

On Fri, Mar 06, 2015 at 04:42:57PM +0100, ext Danny Smit wrote:
> On Fri, Mar 6, 2015 at 12:13 PM, Adam Endrodi wrote:
> > On Fri, Mar 06, 2015 at 11:52:05AM +0100, ext Danny Smit wrote:
> >>
> >> The scenario is simple:
> >> - call socket() to create a socket.
> >> - call sctp_connectx() to establish a connection.
> >>
> >> The last call is repeated periodically to retry to establish a connection.
> >>
> >> When looking on the wire using wireshark it shows that an SCTP INIT
> >> packet is sent to the second host, which replies with an SCTP ABORT.
> >>
> >> This is exactly according to the SCTP specification. However it
> >> appears ABORT isn't propagated into to application layer that calls
> >> sctp_connectx().
> > [...]
> >> Note that the API is used in non-blocking mode.
> >
> > This case sctp_connectx() doesn't wait until the connection is established.
> > You need to use sctp_recvmsg() to get the notification.
> 
> I tried to apply this as suggested, the scenario now is as follows:
> 
> First call sctp_connectx() on a socket in non blocking mode, which
> results in the error code EINPROGRESS.
> 
> Next call sctp_recvmsg(), which results in the error code ENOTCONN
> instead of getting any notifications.
> 
> This actually feels right because there is no connection established
> yet. It also doesn't seem to make a difference for the memory
> consumption. Am I missing something?

How about something like:

for (;;)
{
	sctp_connectx(sfd);
	poll([sfd], -1);
	sctp_recvmsg(sfd);
}

-- 
adam

Re: Memory issue with kernel sctp_connectx

[Danny Smit]

I changed my code to reflect this, but it doesn't really make a
difference. For what its worth, I already tried something similar with
an higher level API to wait for socket notification, which led to the
same results.

Whats interesting is that the poll() call returns immediately (within
milliseconds) regardless of the timeout value. It does set "revents"
on the struct in [sfd], of which the very first occurrence of this
call sets revents to 73 (0x49), but all subsequent calls to poll
within the loop shown above sets revents to 65 (0x48).

If I'm correct this would mean:

0x49 = POLLIN | POLLERR | POLLRDNORM
0x41 = POLLIN | POLLRDNORM

I'm not sure where the first POLLERR comes from and what to conclude
based on this. However the other flag do seem to indicate that there
is data available to be read.

But every call to sctp_recvmsg() still returns with an error and sets
errno to 107 (ENOTCONN). Therefore sctp_recvmsg somehow still seems
unable to read the events.

On Fri, Mar 6, 2015 at 5:29 PM, Adam Endrodi <adam.endrodi@nokia.com> wrote:
> On Fri, Mar 06, 2015 at 04:42:57PM +0100, ext Danny Smit wrote:
>> On Fri, Mar 6, 2015 at 12:13 PM, Adam Endrodi wrote:
>> > On Fri, Mar 06, 2015 at 11:52:05AM +0100, ext Danny Smit wrote:
>> >>
>> >> The scenario is simple:
>> >> - call socket() to create a socket.
>> >> - call sctp_connectx() to establish a connection.
>> >>
>> >> The last call is repeated periodically to retry to establish a connection.
>> >>
>> >> When looking on the wire using wireshark it shows that an SCTP INIT
>> >> packet is sent to the second host, which replies with an SCTP ABORT.
>> >>
>> >> This is exactly according to the SCTP specification. However it
>> >> appears ABORT isn't propagated into to application layer that calls
>> >> sctp_connectx().
>> > [...]
>> >> Note that the API is used in non-blocking mode.
>> >
>> > This case sctp_connectx() doesn't wait until the connection is established.
>> > You need to use sctp_recvmsg() to get the notification.
>>
>> I tried to apply this as suggested, the scenario now is as follows:
>>
>> First call sctp_connectx() on a socket in non blocking mode, which
>> results in the error code EINPROGRESS.
>>
>> Next call sctp_recvmsg(), which results in the error code ENOTCONN
>> instead of getting any notifications.
>>
>> This actually feels right because there is no connection established
>> yet. It also doesn't seem to make a difference for the memory
>> consumption. Am I missing something?
>
> How about something like:
>
> for (;;)
> {
>         sctp_connectx(sfd);
>         poll([sfd], -1);
>         sctp_recvmsg(sfd);
> }
>
> --
> adam

Re: Memory issue with kernel sctp_connectx

[Adam Endrodi]

On Sat, Mar 07, 2015 at 08:44:24PM +0100, ext Danny Smit wrote:
> I changed my code to reflect this, but it doesn't really make a
> difference. For what its worth, I already tried something similar with
> an higher level API to wait for socket notification, which led to the
> same results.
> 
> Whats interesting is that the poll() call returns immediately (within
> milliseconds) regardless of the timeout value. It does set "revents"
> on the struct in [sfd], of which the very first occurrence of this
> call sets revents to 73 (0x49), but all subsequent calls to poll
> within the loop shown above sets revents to 65 (0x48).

So POLLERR (0x08) is reported in all cases.  Perhaps this is the indication
of ABORT you're looking for?

> But every call to sctp_recvmsg() still returns with an error and sets
> errno to 107 (ENOTCONN). Therefore sctp_recvmsg somehow still seems
> unable to read the events.

I'm out of ideas :( Time to involve real experts I guess.

-- 
What doesn't kill you makes you stronger.

Re: Memory issue with kernel sctp_connectx

[Danny Smit]

On Mon, Mar 9, 2015 at 10:40 AM, Adam Endrodi <adam.endrodi@nokia.com> wrote:
> On Sat, Mar 07, 2015 at 08:44:24PM +0100, ext Danny Smit wrote:
>> Whats interesting is that the poll() call returns immediately (within
>> milliseconds) regardless of the timeout value. It does set "revents"
>> on the struct in [sfd], of which the very first occurrence of this
>> call sets revents to 73 (0x49), but all subsequent calls to poll
>> within the loop shown above sets revents to 65 (0x48).
>
> So POLLERR (0x08) is reported in all cases.  Perhaps this is the indication
> of ABORT you're looking for?

I noticed I made a small mistake in the hexadecimal numbers:

The first occurrence gives: 73 (0x49)
All subsequent occurences give: 65 (0x41)

Therefore only the first occurrence includes POLLERR (0x08).

My apologies for the confusion.


I'm still a bit confused about how this is supposed to work though. Is
the POLLERR really the way to go? It doesn't seem to be described
anywhere, rfc6458 doesn't seem to be conclusive about this particular
case (where an ABORT is returned on the SCTP handshake in NON_BLOCKING
mode).

Furthermore I did find a man page which states: "The sctp_recvmsg()
call is used by one-to-one (SOCK_STREAM) type sockets after a
successful connect() call"
Although this originates from FreeBSD
(http://www.freebsd.org/cgi/man.cgi?query=sctp_recvmsg&manpath=FreeBSD+8.0-RELEASE),
this makes me believe that I'm not supposed to use sctp_recvmsg()
before the connection is successfully established. However in
non-blocking mode, how is an ABORT supposed to be detected without
sctp_recvmsg()?


>> But every call to sctp_recvmsg() still returns with an error and sets
>> errno to 107 (ENOTCONN). Therefore sctp_recvmsg somehow still seems
>> unable to read the events.
>
> I'm out of ideas :( Time to involve real experts I guess.

Thanks anyway for trying to help out, it is very much appreciated! I
hope someone can shed some light on this.

Re: Memory issue with kernel sctp_connectx

[Vlad Yasevich]

On 03/09/2015 09:15 AM, Danny Smit wrote:
> On Mon, Mar 9, 2015 at 10:40 AM, Adam Endrodi <adam.endrodi@nokia.com> wrote:
>> On Sat, Mar 07, 2015 at 08:44:24PM +0100, ext Danny Smit wrote:
>>> Whats interesting is that the poll() call returns immediately (within
>>> milliseconds) regardless of the timeout value. It does set "revents"
>>> on the struct in [sfd], of which the very first occurrence of this
>>> call sets revents to 73 (0x49), but all subsequent calls to poll
>>> within the loop shown above sets revents to 65 (0x48).
>>
>> So POLLERR (0x08) is reported in all cases.  Perhaps this is the indication
>> of ABORT you're looking for?
>
> I noticed I made a small mistake in the hexadecimal numbers:
>
> The first occurrence gives: 73 (0x49)
> All subsequent occurences give: 65 (0x41)
>
> Therefore only the first occurrence includes POLLERR (0x08).
>
> My apologies for the confusion.
>
>
> I'm still a bit confused about how this is supposed to work though. Is
> the POLLERR really the way to go? It doesn't seem to be described
> anywhere, rfc6458 doesn't seem to be conclusive about this particular
> case (where an ABORT is returned on the SCTP handshake in NON_BLOCKING
> mode).
>
> Furthermore I did find a man page which states: "The sctp_recvmsg()
> call is used by one-to-one (SOCK_STREAM) type sockets after a
> successful connect() call"
> Although this originates from FreeBSD
> (http://www.freebsd.org/cgi/man.cgi?query=sctp_recvmsg&manpath=FreeBSD+8.0-RELEASE),
> this makes me believe that I'm not supposed to use sctp_recvmsg()
> before the connection is successfully established. However in
> non-blocking mode, how is an ABORT supposed to be detected without
> sctp_recvmsg()?
>
>

Ouch.  Looking at the sctp_recvmsg() code, one of the first things
we do is return error if the socket is not connected.  I agree that this makes
it absolutely impossible to receive any kind of notification data about
connection termination in the case of a 1-to-1 socket.

I think that check really needs to take into consideration the state
of the socket receive queue.

Can you try this patch.

diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index 2625ecc..14fc29d 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -2077,7 +2077,8 @@ static int sctp_recvmsg(struct kiocb *iocb, struct sock *sk,

 	lock_sock(sk);

-	if (sctp_style(sk, TCP) && !sctp_sstate(sk, ESTABLISHED)) {
+	if (sctp_style(sk, TCP) && !sctp_sstate(sk, ESTABLISHED) &&
+	    skb_queue_empty(&sk->sk_receive_queue)) {
 		err = -ENOTCONN;
 		goto out;
 	}

Thanks
-vlad

Re: Memory issue with kernel sctp_connectx

[Danny Smit]

On Tue, Mar 10, 2015 at 8:28 PM, Vlad Yasevich <vyasevich@gmail.com> wrote:
>
> Can you try this patch.
>
> diff --git a/net/sctp/socket.c b/net/sctp/socket.c
> index 2625ecc..14fc29d 100644
> --- a/net/sctp/socket.c
> +++ b/net/sctp/socket.c
> @@ -2077,7 +2077,8 @@ static int sctp_recvmsg(struct kiocb *iocb, struct sock *sk,
>
>       lock_sock(sk);
>
> -     if (sctp_style(sk, TCP) && !sctp_sstate(sk, ESTABLISHED)) {
> +     if (sctp_style(sk, TCP) && !sctp_sstate(sk, ESTABLISHED) &&
> +         skb_queue_empty(&sk->sk_receive_queue)) {
>                 err = -ENOTCONN;
>                 goto out;
>       }


Thanks a lot, this patch seems to work! With this patch sctp_recvmsg()
results in a notification of type SCTP_CANT_STR_ASSOC and the memory
consumption does not increase anymore. Exactly as expected.

Just to be sure, in my previous implementation I didn't even call
sctp_recvmsg(), but repeatedly called sctp_connectx() until it
returned EISCON. We can now conclude that it is just incorrect usage
of the API? And therefore it is correct that with this specific
scenario the sk_recieve_queue keeps piling up (and consuming memory),
even though initialization of the connection fails immediately?