troubles caused by conntrack overlimit in init_netns

troubles caused by conntrack overlimit in init_netns

[Vasily Averin]

Pablo, Florian,

There is an old issue with conntrack limit on multi-netns (read container) nodes.

Any connection to containers hosted on the node creates a conntrack in init_netns.
If the number of conntrack in init_netns reaches the limit, the whole node becomes
unavailable.

To avoid it OpenVz had special patches disabled conntracks on init_ns on openvz nodes, 
but this automatically limits the functionality of host's firewall.

This has been our specific pain for many years, however, containers are now 
being used much more widely than before, and the severity of the described problem
is growing more and more.

Do you know perhaps some alternative solution?

Thank you,
	Vasily Averin

Re: troubles caused by conntrack overlimit in init_netns

[Florian Westphal]

Vasily Averin <vvs@openvz.org> wrote:
> There is an old issue with conntrack limit on multi-netns (read container) nodes.
> 
> Any connection to containers hosted on the node creates a conntrack in init_netns.
> If the number of conntrack in init_netns reaches the limit, the whole node becomes
> unavailable.

Right, from inet_net p.o.v. connections coming from container netns is
no different from different physical host on pyhsical network.

> To avoid it OpenVz had special patches disabled conntracks on init_ns on openvz nodes, 
> but this automatically limits the functionality of host's firewall.
> 
> This has been our specific pain for many years, however, containers are now 
> being used much more widely than before, and the severity of the described problem
> is growing more and more.
> 
> Do you know perhaps some alternative solution?

If you need conntrack in init_net, then no.

If you don't (or only for connections that won't be rerouted to
container netns) you could -j NOTRACK traffic coming from/going to
container.

But, why do you need conntrack in the container netns?
Normally I'd expect that if packet was already handled in init_net,
why re-run skb through conntrack again?

Re: troubles caused by conntrack overlimit in init_netns

[Nikita Yushchenko]

Hi

 > But, why do you need conntrack in the container netns?

Container can hold a default installation of a linux distro, that often enables firewall and adds rules 
that trigger conntrack. But, that is inside container's netns. That is accounted separately and is not 
part of the issue being discussed.

The issue is that traffic directed to container(s) eats out host's conntrack limit, and causes the host 
to be inaccessible via network.

 >> Do you know perhaps some alternative solution?
 >
 > If you need conntrack in init_net, then no.

I suppose this can be fixed by something like:
- adding some "window" on top of host's conntrack limit,
- if out of conntracks, but still within window, conntrack shall be created but marked,
- mark could be removed by a dedicated netfilter rule,
- attaching more than one skb to marked conntrack shall be blocked (i.e. packet dropped instead),
- if skb pointing to a marked conntrack is about to leave the stack (that is, either being delivered 
locally, or queued for transmitting out), it shall be dropped instead, and conntrack removed.

This shall give host's admin a way to explicitly configure a packat path to use as a host management 
interface, that will stay accessible even if containers eat out conntrack limit.

Is that reasonable?  Maybe I can try to implement that...

Nikita

Re: troubles caused by conntrack overlimit in init_netns

[Eric Dumazet]

On 4/2/22 03:33, Vasily Averin wrote:
> Pablo, Florian,
>
> There is an old issue with conntrack limit on multi-netns (read container) nodes.
>
> Any connection to containers hosted on the node creates a conntrack in init_netns.
> If the number of conntrack in init_netns reaches the limit, the whole node becomes
> unavailable.


Can you describe network topology ?


Are you using macvlan, ipvlan, or something else ?


> To avoid it OpenVz had special patches disabled conntracks on init_ns on openvz nodes,
> but this automatically limits the functionality of host's firewall.
>
> This has been our specific pain for many years, however, containers are now
> being used much more widely than before, and the severity of the described problem
> is growing more and more.
>
> Do you know perhaps some alternative solution?
>
> Thank you,
> 	Vasily Averin

Re: troubles caused by conntrack overlimit in init_netns

[Vasily Averin]

On 4/2/22 20:12, Eric Dumazet wrote:
> 
> On 4/2/22 03:33, Vasily Averin wrote:
>> Pablo, Florian,
>>
>> There is an old issue with conntrack limit on multi-netns (read container) nodes.
>>
>> Any connection to containers hosted on the node creates a conntrack in init_netns.
>> If the number of conntrack in init_netns reaches the limit, the whole node becomes
>> unavailable.
> 
> Can you describe network topology ?

              += veth1 <=> veth container1
ethX <=> brX =+= veth2 <=> veth container2
              += vethX <=> veth containerX

> Are you using macvlan, ipvlan, or something else ?

No, we dod not used it earlier, because it was not available in RHEL7, 
but now it looks like good solution for me.

Thank you for the hint,
	Vasily Averin

Re: troubles caused by conntrack overlimit in init_netns

[Eric Dumazet]

On Sat, Apr 2, 2022 at 11:32 AM Vasily Averin <vasily.averin@linux.dev> wrote:
>
> On 4/2/22 20:12, Eric Dumazet wrote:
> >
> > On 4/2/22 03:33, Vasily Averin wrote:
> >> Pablo, Florian,
> >>
> >> There is an old issue with conntrack limit on multi-netns (read container) nodes.
> >>
> >> Any connection to containers hosted on the node creates a conntrack in init_netns.
> >> If the number of conntrack in init_netns reaches the limit, the whole node becomes
> >> unavailable.
> >
> > Can you describe network topology ?
>
>               += veth1 <=> veth container1
> ethX <=> brX =+= veth2 <=> veth container2
>               += vethX <=> veth containerX
>

Could you simply add an iptables rule in init_net to bypass conntrack
for idev=veth* ?

iptables -t raw -I PREROUTING -i veth+ -j NOTRACK

(I have not worked with conntrack in recent years, this might be foolish...)

> > Are you using macvlan, ipvlan, or something else ?
>
> No, we dod not used it earlier, because it was not available in RHEL7,
> but now it looks like good solution for me.
>
> Thank you for the hint,
>         Vasily Averin

Re: troubles caused by conntrack overlimit in init_netns

[Vasily Averin]

On 4/2/22 21:50, Eric Dumazet wrote:
> On Sat, Apr 2, 2022 at 11:32 AM Vasily Averin <vasily.averin@linux.dev> wrote:
>>
>> On 4/2/22 20:12, Eric Dumazet wrote:
>>>
>>> On 4/2/22 03:33, Vasily Averin wrote:
>>>> Pablo, Florian,
>>>>
>>>> There is an old issue with conntrack limit on multi-netns (read container) nodes.
>>>>
>>>> Any connection to containers hosted on the node creates a conntrack in init_netns.
>>>> If the number of conntrack in init_netns reaches the limit, the whole node becomes
>>>> unavailable.
>>>
>>> Can you describe network topology ?
>>
>>               += veth1 <=> veth container1
>> ethX <=> brX =+= veth2 <=> veth container2
>>               += vethX <=> veth containerX
>>
> 
> Could you simply add an iptables rule in init_net to bypass conntrack
> for idev=veth* ?
> 
> iptables -t raw -I PREROUTING -i veth+ -j NOTRACK
> 
> (I have not worked with conntrack in recent years, this might be foolish...)

Great and simple idea.
Thank you very much, we'll investigate it.

	Vasily Averin

Re: troubles caused by conntrack overlimit in init_netns

[Vasily Averin]

On 4/2/22 14:11, Florian Westphal wrote:
> But, why do you need conntrack in the container netns?
> Normally I'd expect that if packet was already handled in init_net,
> why re-run skb through conntrack again?

OpenVz and LXC containers are used for hosting:
so init_netns is controlled by Hoster admins for system-wide purposes,
the container is under the control of the end user, who can configure
any rules for the internal firewall.

Thank you,
	Vasily Averin