"Operation not permitted" from nf_conntrack under high UDP load

"Operation not permitted" from nf_conntrack under high UDP load

[Sebastian Damm]

Hi,

we have a SIP signalling server, which communicates with hundreds of
thousands of clients all over the world. There are basically just two
services on this host - SIP and STUN. Both services are UDP only.

This server is running with a permanent bandwidth of 20-40 Mbit of UDP
traffic. We have a stateful firewall on it with quite a few rules, the
conntrack table is filled with more than 300k entries, max is at 800k.

In the SIP server log we see about 10k times per day the following messages:

Mar  3 06:25:31 dalke /usr/sbin/kamailio[30372]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7f28fe924000,459,0,89.204.130.xxx:34246,16): Operation
not permitted(1)
Mar  3 06:25:56 dalke /usr/sbin/kamailio[30369]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7f28fe933218,411,0,213.47.42.xxx:5060,16): Operation not
permitted(1)
Mar  3 06:25:57 dalke /usr/sbin/kamailio[30588]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7f82abe4ef28,420,0,217.116.117.xxx:5060,16): Operation
not permitted(1)
Mar  3 06:26:07 dalke /usr/sbin/kamailio[30330]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7f28fe996f70,373,0,178.21.0.xxx:5060,16): Operation not
permitted(1)
Mar  3 06:26:37 dalke /usr/sbin/kamailio[30357]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7f28fe997178,422,0,37.47.161.xxx:5669,16): Operation not
permitted(1)
Mar  3 06:26:56 dalke /usr/sbin/kamailio[31025]: ERROR: <core>
[udp_server.c:550]: udp_send():
sendto(sock,0x7fbee39627c8,477,0,178.27.35.xxx:57454,16): Operation
not permitted(1)

Since every hint in the internet points to iptables for this error, we
cleared all rules. But the messages don't stop.

So we rebooted the machine to have a clear state without firewall.
After that the problem was gone.

Now we just executed "iptables -nvL" and then inserted the modules
nf_conntrack and nf_conntrack_ipv4. After that those messages appeared
again. We can reproduce it by removing and adding those two modules,
causing the errors to disappear and reappear.

We are using Debian Wheezy, and tried it with the standard 3.2.0
kernel and 3.16.0 backports kernel, both show the same behaviour.

Has anybody seen this before? And is there any solution (besides using
a non-stateful firewall)? We tried tweaking the hashsize option when
loading the module, raised the conntrack_max and conntrack_expect_max
values, without any change. Are there other options which might help
in our scenario?

Thanks,
Sebastian

Re: "Operation not permitted" from nf_conntrack under high UDP load

[Kevin Holly]

Hello,

you might want to check your system logs for messages similar to "ip_conntrack: table full, dropping packet." because for me this sounds like a problem with the size of the connection tracking table not being large enough for the number of connections you're dealing there.

This usually indicates that your connection tracking table is filed and you will need to higher the size of the connection tracking table.
To increase the table size, which can be done very easily, I suggest reading out the current value using sysctl net.nf_conntrack_max and setting a higher value, I usually take the existing value and multiply it by 2, or, if the machines have enough RAM available and need to handle really much connections, set it to 1048576 (sysctl net.nf_conntrack_max=1048576) which seems to work out pretty well for us.

Best regards

Kevin Holly - root@hallowe.lt - http://hallowe.lt/

On 03/03/2016 03:31 PM, Sebastian Damm wrote:
> Hi,
> 
> we have a SIP signalling server, which communicates with hundreds of
> thousands of clients all over the world. There are basically just two
> services on this host - SIP and STUN. Both services are UDP only.
> 
> This server is running with a permanent bandwidth of 20-40 Mbit of UDP
> traffic. We have a stateful firewall on it with quite a few rules, the
> conntrack table is filled with more than 300k entries, max is at 800k.
> 
> In the SIP server log we see about 10k times per day the following messages:
> 
> Mar  3 06:25:31 dalke /usr/sbin/kamailio[30372]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7f28fe924000,459,0,89.204.130.xxx:34246,16): Operation
> not permitted(1)
> Mar  3 06:25:56 dalke /usr/sbin/kamailio[30369]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7f28fe933218,411,0,213.47.42.xxx:5060,16): Operation not
> permitted(1)
> Mar  3 06:25:57 dalke /usr/sbin/kamailio[30588]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7f82abe4ef28,420,0,217.116.117.xxx:5060,16): Operation
> not permitted(1)
> Mar  3 06:26:07 dalke /usr/sbin/kamailio[30330]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7f28fe996f70,373,0,178.21.0.xxx:5060,16): Operation not
> permitted(1)
> Mar  3 06:26:37 dalke /usr/sbin/kamailio[30357]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7f28fe997178,422,0,37.47.161.xxx:5669,16): Operation not
> permitted(1)
> Mar  3 06:26:56 dalke /usr/sbin/kamailio[31025]: ERROR: <core>
> [udp_server.c:550]: udp_send():
> sendto(sock,0x7fbee39627c8,477,0,178.27.35.xxx:57454,16): Operation
> not permitted(1)
> 
> Since every hint in the internet points to iptables for this error, we
> cleared all rules. But the messages don't stop.
> 
> So we rebooted the machine to have a clear state without firewall.
> After that the problem was gone.
> 
> Now we just executed "iptables -nvL" and then inserted the modules
> nf_conntrack and nf_conntrack_ipv4. After that those messages appeared
> again. We can reproduce it by removing and adding those two modules,
> causing the errors to disappear and reappear.
> 
> We are using Debian Wheezy, and tried it with the standard 3.2.0
> kernel and 3.16.0 backports kernel, both show the same behaviour.
> 
> Has anybody seen this before? And is there any solution (besides using
> a non-stateful firewall)? We tried tweaking the hashsize option when
> loading the module, raised the conntrack_max and conntrack_expect_max
> values, without any change. Are there other options which might help
> in our scenario?
> 
> Thanks,
> Sebastian
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: "Operation not permitted" from nf_conntrack under high UDP load

[Sebastian Damm]

Hi Kevin,

On Fri, Mar 4, 2016 at 4:59 AM, Kevin Holly <root@hallowe.lt> wrote:
> you might want to check your system logs for messages similar to "ip_conntrack: table full, dropping packet." because for me this sounds like a problem with the size of the connection tracking table not being large enough for the number of connections you're dealing there.

As I wrote, we have a max for the connection tracking table of 800k
set, and it is filled with about 300k entries, so this is definitely
not the problem. We also don't see those log messages you mentioned. I
was aware of those, and I've seen them before on other systems, but on
this machine, this is definitely not the problem.

Any other hints, anyone?

Best Regards,
Sebastian

Re: "Operation not permitted" from nf_conntrack under high UDP load

[Sebastian Damm]

Hi,

just to close this one, after a lot of debugging we found the cause
for this issue. When this error occurs, the SIP server is sending out
two packets to the same destination (IP:port) at roughly the same
time. This causes a race condition in nf_conntrack when inserting a
new tuple into the conntrack table, leading to one of the two packets
getting rejected by the kernel.

Looks like there is no other solution for this problem than building a
stateless firewall and unloading the nf_conntrack module.

Best Regards,
Sebastian