allocate specific port range for container?

allocate specific port range for container?

[Hongbing Wang]

Hello LXC experts:

Is it possible to allocate specific transport port range for the socket based applications inside one container? 

Say I have two containers: LXC_a and LXC_b, and each has some socket based applications I cannot modify or have no source code. If I need port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any way to achieve this?

The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust the ephemeral port range, and we do not have the per LXC based setting on the local_port_range. Is this due to that the LXC network namespace isolation is at the L3 level? How could I achieve this per port range LXC?

Regards,
-HB

Re: allocate specific port range for container?

[Serge Hallyn]

Quoting Hongbing Wang (hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org):
> Hello LXC experts:
> 
> Is it possible to allocate specific transport port range for the socket based applications inside one container? 

How do you mean?  You want ports 50000-51000 of the host to be forwarded
to container 1, and 60000-61000 to container 2?

> Say I have two containers: LXC_a and LXC_b, and each has some socket based applications I cannot modify or have no source code. If I need port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any way to achieve this?
> 
> The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust the ephemeral port range, and we do not have the per LXC based setting on the local_port_range. Is this due to that the LXC network namespace isolation is at the L3 level? How could I achieve this per port range LXC?

The network namespaces are actually at L2, not L3.  Each container has
its own routing table.

I think you can get what you want by simply giving each container a veth
nic and using iptables on the host to forward the ports you want to the
appropriate container.  That's how I co-locate web, mail, and other
server containers on the same host.

Re: allocate specific port range for container?

[Hongbing Wang]

________________________________
 From: Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>
To: Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> 
Cc: "containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" <containers-cunTk1MwBs9QetFLy7KEm77oxUfYCNXn@public.gmane.orgion.org> 
Sent: Tuesday, November 5, 2013 1:53 PM
Subject: Re: allocate specific port range for container?
 

Quoting Hongbing Wang (hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org):
> Hello LXC experts:
> 
> Is it possible to allocate specific transport port range for the socket based applications inside one container? 

How do you mean?  You want ports 50000-51000 of the host to be forwarded
to container 1, and 60000-61000 to container 2?

[HB] We want the apps inside LXC_a to use the port b/w 50k - 51k and LXC_b to be allocated the port b/w 60k - 61k.

> Say I have two containers: LXC_a and LXC_b, and each has some socket based applications I cannot modify or have no source code. If I need port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any way to achieve this?
> 
> The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust the ephemeral port range, and we do not have the per LXC based setting on the local_port_range. Is this due to that the LXC network namespace isolation is at the L3 level? How could I achieve this per port range LXC?

The network namespaces are actually at L2, not L3.  Each container has
its own routing table.

I think you can get what you want by simply giving each container a veth
nic and using iptables on the host to forward the ports you want to the
appropriate container.  That's how I co-locate web, mail, and other
server containers on the same host.

[HB] for my specific case we do not use iptables. We can route the packets to the LXC_a or LXC_b, but we want the ports to be within specific ranges.

Regards,
-HB

Re: allocate specific port range for container?

[Zhu Yanhai]

2013/11/6 Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org>:
>
>
>
>
> ________________________________
>  From: Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>
> To: Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org>
> Cc: "containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" <containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>
> Sent: Tuesday, November 5, 2013 1:53 PM
> Subject: Re: allocate specific port range for container?
>
>
> Quoting Hongbing Wang (hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org):
>> Hello LXC experts:
>>
>> Is it possible to allocate specific transport port range for the socket based applications inside one container?
>
> How do you mean?  You want ports 50000-51000 of the host to be forwarded
> to container 1, and 60000-61000 to container 2?
>
> [HB] We want the apps inside LXC_a to use the port b/w 50k - 51k and LXC_b to be allocated the port b/w 60k - 61k.
>
>> Say I have two containers: LXC_a and LXC_b, and each has some socket based applications I cannot modify or have no source code. If I need port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any way to achieve this?
>>
>> The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust the ephemeral port range, and we do not have the per LXC based setting on the local_port_range. Is this due to that the LXC network namespace isolation is at the L3 level? How could I achieve this per port range LXC?
>
> The network namespaces are actually at L2, not L3.  Each container has
> its own routing table.
>
> I think you can get what you want by simply giving each container a veth
> nic and using iptables on the host to forward the ports you want to the
> appropriate container.  That's how I co-locate web, mail, and other
> server containers on the same host.
>
> [HB] for my specific case we do not use iptables. We can route the packets to the LXC_a or LXC_b, but we want the ports to be within specific ranges.

Then how about SElinux ? You can specify the ports rules against each
container. Serge Hallyn has a cookbook about this
(http://www.ibm.com/developerworks/library/l-lxc-security/)
I'm using a SElinux policy based on Serge Hallyn's original edition in
the production system, since I have so many containers on the physical
box  that I can't give them each a individual veth and IP, also NAT is
forbidden by the network topology (it's a long story). In one words, I
don't create separate network namespace for them at all, but let them
use one single network namespace and leverage the SElinux policy for
security.

BTW, there is one thing strange for me about the SElinux ports policy,
it does not prevent the container from name binding the ports in
/proc/sys/net/ipv4/ip_local_port_range, although the ports in
/proc/sys/net/ipv4/ip_local_port_range surly doesn't have the
container's labels. So actually I added a in-house patch as below to
make it work as expected,

--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -3935,24 +3935,18 @@ static int selinux_socket_bind(struct socket
*sock, struct sockaddr *address, in
                }

                if (snum) {
-                       int low, high;
-
-                       inet_get_local_port_range(&low, &high);
-
-                       if (snum < max(PROT_SOCK, low) || snum > high) {
-                               err = sel_netport_sid(sk->sk_protocol,
-                                                     snum, &sid);
-                               if (err)
-                                       goto out;
-                               COMMON_AUDIT_DATA_INIT(&ad, NET);
-                               ad.u.net.sport = htons(snum);
-                               ad.u.net.family = family;
-                               err = avc_has_perm(isec->sid, sid,
-                                                  isec->sclass,
-                                                  SOCKET__NAME_BIND, &ad);
-                               if (err)
-                                       goto out;
-                       }
+                       err = sel_netport_sid(sk->sk_protocol,
+                                       snum, &sid);
+                       if (err)
+                               goto out;
+                       COMMON_AUDIT_DATA_INIT(&ad, NET);
+                       ad.u.net.sport = htons(snum);
+                       ad.u.net.family = family;
+                       err = avc_has_perm(isec->sid, sid,
+                                       isec->sclass,
+                                       SOCKET__NAME_BIND, &ad);
+                       if (err)
+                               goto out;
                }

                switch (isec->sclass) {
-- 
1.8.0.1

--
Thanks,
Zhu Yanhai

>
> Regards,
> -HB
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: allocate specific port range for container?

[Eric W. Biederman]

Hongbing Wang <hongbingwang@sbcglobal.net> writes:

> ________________________________
>  From: Serge Hallyn <serge.hallyn@ubuntu.com>
> To: Hongbing Wang <hongbingwang@sbcglobal.net> 
> Cc: "containers@lists.linux-foundation.org" <containers@lists.linux-foundation.org> 
> Sent: Tuesday, November 5, 2013 1:53 PM
> Subject: Re: allocate specific port range for container?
>  
>
> Quoting Hongbing Wang (hongbingwang@sbcglobal.net):
>> Hello LXC experts:
>> 
>> Is it possible to allocate specific transport port range for the socket based applications inside one container? 
>
> How do you mean?  You want ports 50000-51000 of the host to be forwarded
> to container 1, and 60000-61000 to container 2?
>
> [HB] We want the apps inside LXC_a to use the port b/w 50k - 51k and LXC_b to be allocated the port b/w 60k - 61k.
>
>> Say I have two containers: LXC_a and LXC_b, and each has some socket
>> based applications I cannot modify or have no source code. If I need
>> port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any
>> way to achieve this?
>> 
>> The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust
>> the ephemeral port range, and we do not have the per LXC based
>> setting on the local_port_range. Is this due to that the LXC network
>> namespace isolation is at the L3 level? How could I achieve this per
>> port range LXC?

> The network namespaces are actually at L2, not L3.  Each container has
> its own routing table.
>
> I think you can get what you want by simply giving each container a veth
> nic and using iptables on the host to forward the ports you want to the
> appropriate container.  That's how I co-locate web, mail, and other
> server containers on the same host.
>
> [HB] for my specific case we do not use iptables. We can route the
> packets to the LXC_a or LXC_b, but we want the ports to be within
> specific ranges.

I have been working on this recently.

I have recently made ip_local_port range per netns to help support this.

Beyond that what I am doing is using ingress traffic control to redirect
packets in the appopriate port range into the appropriate network
namespace.

Compared to nat with iptables and some of the other solutions it comes
out fairly clean and it at the very beginning stages at the packet
processing path so unnecessary work can be avoided.

There are a couple of cases you have to watch for icmp, arp and looping
packets back to other namespaces on the same box but it is pretty
straight forward to setup.  The trickiest bit is that you have to read
the tc source to find the tc commands that make the tc classifier easy
to use.

The primary restricition I currently have is that the port ranges need
to be a power of 2 in size, to make the u32 classifier easy to use.

Eric



_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: allocate specific port range for container?

[Hongbing Wang]

________________________________
 From: Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>
To: Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> 
Cc: Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>; "containers-cunTk1MwBs9QetFLy7KEm77oxUfYCNXn@public.gmane.orgion.org" <containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>; Zhu Yanhai <zhu.yanhai@gmail.com> 
Sent: Friday, November 8, 2013 3:40 PM
Subject: Re: allocate specific port range for container?
 

Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> writes:

> ________________________________
>  From: Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>
> To: Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> 
> Cc: "containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" <containers-cunTk1MwBs9QetFLy7KEm2CgeU+1il2p@public.gmane.orgation.org> 
> Sent: Tuesday, November 5, 2013 1:53 PM
> Subject: Re: allocate specific port range for container?
>  
>
> Quoting Hongbing Wang (hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org):
>> Hello LXC experts:
>> 
>> Is it possible to allocate specific transport port range for the socket based applications inside one container? 
>
> How do you mean?  You want ports 50000-51000 of the host to be forwarded
> to container 1, and 60000-61000 to container 2?
>
> [HB] We want the apps inside LXC_a to use the port b/w 50k - 51k and LXC_b to be allocated the port b/w 60k - 61k.
>
>> Say I have two containers: LXC_a and LXC_b, and each has some socket
>> based applications I cannot modify or have no source code. If I need
>> port range 50000 - 51000 for LXC_a and 60000 - 61000 for LXC_b. Any
>> way to achieve this?
>> 
>> The /proc/sys/net/ipv4/ip_local_port_range is for the host to adjust
>> the ephemeral port range, and we do not have the per LXC based
>> setting on the local_port_range. Is this due to that the LXC network
>> namespace isolation is at the L3 level? How could I achieve this per
>> port range LXC?

> The network namespaces are actually at L2, not L3.  Each container has
> its own routing table.
>
> I think you can get what you want by simply giving each container a veth
> nic and using iptables on the host to forward the ports you want to the
> appropriate container.  That's how I co-locate web, mail, and other
> server containers on the same host.
>
> [HB] for my specific case we do not use iptables. We can route the
> packets to the LXC_a or LXC_b, but we want the ports to be within
> specific ranges.

I have been working on this recently.

I have recently made ip_local_port range per netns to help support this.

[HB] That is exactly what I am looking for! Do u have any plan to open-source your work?

Yanhai posted that he is using SELinux, we use different distro with lots of other limitations which require to support the existing features.

Thanks,
-HB

Beyond that what I am doing is using ingress traffic control to redirect
packets in the appopriate port range into the appropriate network
namespace.

Compared to nat with iptables and some of the other solutions it comes
out fairly clean and it at the very beginning stages at the packet
processing path so unnecessary work can be avoided.

There are a couple of cases you have to watch for icmp, arp and looping
packets back to other namespaces on the same box but it is pretty
straight forward to setup.  The trickiest bit is that you have to read
the tc source to find the tc commands that make the tc classifier easy
to use.

The primary restricition I currently have is that the port ranges need
to be a power of 2 in size, to make the u32 classifier easy to use.

Eric

Re: allocate specific port range for container?

[Eric W. Biederman]

Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> writes:

Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> writes:
> > I have been working on this recently.
> >
> > I have recently made ip_local_port range per netns to help support
> > this.
>
> [HB] That is exactly what I am looking for! Do u have any plan to
> open-source your work?

Baring typos this is it.  In due course the code should will also appear
in Mesos.

Before you run this make certain to look and discover the environment
variables you need to set.

linkaddr=$(ip -o link show dev eth0 | sed -e 's|^.*link/ether \(..:..:..:..:..:..\).*$|\1|' )

ip netns add container

ip link add veth0 type veth peer name eth0 netns container address $linkaddr

tc qdisc add dev eth0 ingress

tc filter add dev eth0 ffff: protocol u32 \
   match ether dst $linkaddr \
   match ip dst $ip \
   match ip dport $port_start $port_mask \
   action mirred egress redirect dev $netns-device

tc filter add dev eth0 parent ffff: protocol ip u32 \
   match ip dst $ip \
   match ip protocol 1 0xff \
   action mirred egress mirror dev veth0

tc filter add dev eth0 parent ffff: protocol arp basic \
   action mirred egress mirror dev veth0

tc filter add dev lo parent ffff: protocol u32 \
   match ip dst $ip \
   match ip dport $port_start $port_mask \
   action mirred egress redirect dev veth0


tc qdisc add dev veth0 ingress

tc filter add veth0 parent ffff: protocol arp basic
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip protocol 1 0xff \
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip sport $port_start $port_mask \
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip dst $ip \
   match ip sport $port_start $port_mask \
   action mirred egress redirect dev lo

ip netns exec container ip link set lo up
ip netns exec container tc qdisc add dev lo ingress
ip netns exec container ip link set eth0 up
ip netns exec container ip addr add ${ip}/24 dev eth0
ip netns exec container sysctl net.ipv4.ip_local_port_range="$port_start $port_end"

ip netns exec container tc filter add dev lo parent ffff: protocol ip u32 \
   match ip dst $ip
   action mirred egress redirect dev eth0

ip netns exec container tc filter add dev lo parent ffff: protocol ip u32 \
   match ip dst $ip
   match ip dport $port_start $port_mask

Eric

Re: allocate specific port range for container?

[Hongbing Wang]

Thanks, Eric. I will look into your patch.
-HB


________________________________
 From: Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>
To: Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> 
Cc: Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>; "containers-cunTk1MwBs9QetFLy7KEm77oxUfYCNXn@public.gmane.orgion.org" <containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>; Zhu Yanhai <zhu.yanhai@gmail.com> 
Sent: Friday, November 8, 2013 5:41 PM
Subject: Re: allocate specific port range for container?
 

Hongbing Wang <hongbingwang-rphTv4pjVZMJGwgDXS7ZQA@public.gmane.org> writes:

Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> writes:
> > I have been working on this recently.
> >
> > I have recently made ip_local_port range per netns to help support
> > this.
>
> [HB] That is exactly what I am looking for! Do u have any plan to
> open-source your work?

Baring typos this is it.  In due course the code should will also appear
in Mesos.

Before you run this make certain to look and discover the environment
variables you need to set.

linkaddr=$(ip -o link show dev eth0 | sed -e 's|^.*link/ether \(..:..:..:..:..:..\).*$|\1|' )

ip netns add container

ip link add veth0 type veth peer name eth0 netns container address $linkaddr

tc qdisc add dev eth0 ingress

tc filter add dev eth0 ffff: protocol u32 \
   match ether dst $linkaddr \
   match ip dst $ip \
   match ip dport $port_start $port_mask \
   action mirred egress redirect dev $netns-device

tc filter add dev eth0 parent ffff: protocol ip u32 \
   match ip dst $ip \
   match ip protocol 1 0xff \
   action mirred egress mirror dev veth0

tc filter add dev eth0 parent ffff: protocol arp basic \
   action mirred egress mirror dev veth0

tc filter add dev lo parent ffff: protocol u32 \
   match ip dst $ip \
   match ip dport $port_start $port_mask \
   action mirred egress redirect dev veth0


tc qdisc add dev veth0 ingress

tc filter add veth0 parent ffff: protocol arp basic
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip protocol 1 0xff \
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip sport $port_start $port_mask \
   action mirred egress redirect dev eth0

tc filter add dev veth0 parent ffff: protocol ip u32 \
   match ip dst $ip \
   match ip sport $port_start $port_mask \
   action mirred egress redirect dev lo

ip netns exec container ip link set lo up
ip netns exec container tc qdisc add dev lo ingress
ip netns exec container ip link set eth0 up
ip netns exec container ip addr add ${ip}/24 dev eth0
ip netns exec container sysctl net.ipv4.ip_local_port_range="$port_start $port_end"

ip netns exec container tc filter add dev lo parent ffff: protocol ip u32 \
   match ip dst $ip
   action mirred egress redirect dev eth0

ip netns exec container tc filter add dev lo parent ffff: protocol ip u32 \
   match ip dst $ip
   match ip dport $port_start $port_mask

Eric