How do containers tie to multiple IP's on a NIC?

How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

Hi,

In the containerless world, I often have multiple IPs assigned to a NIC. The
scant documentation I can find on running containers only ever speaks of
single IP assignment schemes. Can I have for example a box with a single NIC
with 8 IPs assigned to it, where the host gets one IP, or perhaps
alternately can see all 8 to run iptables across, but each of the containers
can see only whichever IP or IPs are assigned to it?

If it can work this way, I'd appreciate any hints on the correct way to
implement it. If it can't, I apologize for barking up the wrong tree, and
will need to look at full virtualization methods which can.

Thanks,
Whit

Re: How do containers tie to multiple IP's on a NIC?

[Daniel Lezcano]

On 07/04/2010 05:40 AM, Whit Blauvelt wrote:
> Hi,
>
> In the containerless world, I often have multiple IPs assigned to a NIC. The
> scant documentation I can find on running containers only ever speaks of
> single IP assignment schemes. Can I have for example a box with a single NIC
> with 8 IPs assigned to it, where the host gets one IP, or perhaps
> alternately can see all 8 to run iptables across, but each of the containers
> can see only whichever IP or IPs are assigned to it?
>    

What container userspace command are you using ? libvirt ? liblxc ? 
unshare --net ?

Thanks
   -- Daniel

Re: How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

On Sun, Jul 04, 2010 at 06:51:34PM +0200, Daniel Lezcano wrote:

> What container userspace command are you using ? libvirt ? liblxc ?
> unshare --net ?

Which one do you recommend, considering what I'm trying to do with multiple
IPs on a NIC? I haven't committed to one yet. Which utility do you expect
future development will favor most? I'll be happy to use any tool which gets
the job done, preferably one that has a future.

Thanks,
Whit

Re: How do containers tie to multiple IP's on a NIC?

[Daniel Lezcano]

On 07/04/2010 09:18 PM, Whit Blauvelt wrote:
> On Sun, Jul 04, 2010 at 06:51:34PM +0200, Daniel Lezcano wrote:
>
>    
>> What container userspace command are you using ? libvirt ? liblxc ?
>> unshare --net ?
>>      
> Which one do you recommend, considering what I'm trying to do with multiple
> IPs on a NIC? I haven't committed to one yet. Which utility do you expect
> future development will favor most? I'll be happy to use any tool which gets
> the job done, preferably one that has a future.
>    

Well  ... please don't consider what I will suggest as "preaching for 
its parish" ;)
(not sure it is a correct expression. It is a direct translation from 
French)

I would recommend to use the lxc tools, preferably the 0.7.1 version. 
These tools allow to do what you are expecting that is assign several Ip 
addresses to the same virtual nic.

They are available at:

http://lxc.sourceforge.net/download/lxc/lxc-0.7.1.tar.gz

an older version is certainly available on your distro.

As a quick start:

write a configuration file (eg. lxc.conf)

lxc.network.type=macvlan
lxc.network.link=eth0
lxc.network.flags=up
lxc.network.ipv4=1.2.3.4/24
lxc.network.ipv4=192.168.1.123/24
lxc.network.ipv4=10.0.0.23
lxc.network.ipv4=172.2.1.3

And then lxc-execute -n foo -f lxc.conf /bin/bash

In your shell you should have a new network with one interface and 
several IP addresses.

You can create much more complex configuration but I let you check if 
these tools fit your needs.

Thanks
   -- Daniel

Re: How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

On Sun, Jul 04, 2010 at 09:49:31PM +0200, Daniel Lezcano wrote:

> Well  ... please don't consider what I will suggest as "preaching
> for its parish" ;)

In English, "Preaching to the choir."

> I would recommend to use the lxc tools, preferably the 0.7.1
> version. 

Will do.

> These tools allow to do what you are expecting that is assign several Ip
> addresses to the same virtual nic.

Ah, then what I need to understand is the relationship of the virtual NIC to
the real NIC. That is, some of what I set up is multi-purpose boxes, where
the single machine functions as an iptables firewall, perhaps multi-homed to
two ISPs, with 3 real NICs, one for the IP block assigned by each ISP, and
one for the LAN - which might also have more than on IP on it. But these
aren't just firewalls. They tend to serve a website or two, perhaps ftp,
smtp, dns - spread over serveral of the IPs. They're also doing SNAT and
DNAT for systems behind them.

It would make all sorts of sense to be adding containers to these systems,
in terms of security, isolation, and the flexibility to easily migrate
services to other servers. But unlike the more usual virtualization
instance, where someone has a dozen different boxes and wants to consolidate
them, I'm already fully consolidated. What I need to do is split things
apart more, so they can go into containers, but still consolidated on boxes
which continue to be multi-purpose, and where each single NIC may have over
a dozen IPs assigned to it, but as a rule from within a single block per
NIC.

I've seen discussions elsewhere (using Google to try to find hints for this)
where people have given a machine two IPs on the same LAN by actually using
two physical NICs (and then need to play STP tricks). My attitude is "Why
use two pieces of hardware where one can do the job?"

Time for some trial-and-error with lxc tools.

Regards,
Whit

Re: How do containers tie to multiple IP's on a NIC?

[Pavel Labushev]

05.07.2010 07:08, Whit Blauvelt пишет:

> Ah, then what I need to understand is the relationship of the virtual NIC to
> the real NIC. That is, some of what I set up is multi-purpose boxes, where

What exactly are you trying to achieve? A transparent packet forwarding
between containers and external networks?
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

On Mon, Jul 05, 2010 at 05:50:38PM +0800, Pavel Labushev wrote:

> What exactly are you trying to achieve? A transparent packet forwarding
> between containers and external networks?

I'm trying to get the overview of what can be achieved, and how. Unless I've
missed it, there's not much documentation on even moderately complex use of
containers. Since the capabilities are rapidly advancing, maybe I'm just
asking the question a few months too early? From the outside, as someone new
to containers, it looks like a maze where there are a number of entrances,
each of which may lead approximately to the goal, but some of which may be
dead ends. 

The examples I have found are along the lines of: Here's how to start a
container, bridge it to a NIC which has a single IP assigned, ssh to it ...
and the examples stop there. What I'd like to achieve is a setup where, say,
a box with 6 IPs on an external network - on eth0 before bridging (or its
alternatives) - can have 5 of those IPs each dedicated to different single
container.

I'm not committed to a particular way of achieving that yet, just looking at
the maze entrances wondering which to choose. Standard packet forwarding,
routing and firewalling in Linux isn't what I'm asking about, I use that
stuff frequently in complex ways, and I already run some things in simple
chroots. I'm trying to learn how, on a test basis, to set up something like
a production environment with lxc, where it involves multiple IPs, WAN or
LAN, on each single NIC, behind some of which should be containers which
effectively own individual IPs, publicly available.

It's probably looking harder to me than it is, because I haven't found a
clear description of it yet. 

Thanks,
Whit

Re: How do containers tie to multiple IP's on a NIC?

[Daniel Lezcano]

On 07/05/2010 04:07 PM, Whit Blauvelt wrote:
> On Mon, Jul 05, 2010 at 05:50:38PM +0800, Pavel Labushev wrote:
>
>    
>> What exactly are you trying to achieve? A transparent packet forwarding
>> between containers and external networks?
>>      
> I'm trying to get the overview of what can be achieved, and how. Unless I've
> missed it, there's not much documentation on even moderately complex use of
> containers. Since the capabilities are rapidly advancing, maybe I'm just
> asking the question a few months too early? From the outside, as someone new
> to containers, it looks like a maze where there are a number of entrances,
> each of which may lead approximately to the goal, but some of which may be
> dead ends.
>    
Hi Whit,

may be this documents can help you:

http://lxc.sourceforge.net/doc/sigops/appcr.pdf

Re: How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

On Mon, Jul 05, 2010 at 11:13:34PM +0200, Daniel Lezcano wrote:
> may be this documents can help you:
> 
> http://lxc.sourceforge.net/doc/sigops/appcr.pdf

Thanks much Daniel. A clarifying and enjoyable read. 

Whit

Re: How do containers tie to multiple IP's on a NIC?

[Pavel Labushev]

05.07.2010 22:07, Whit Blauvelt пишет:

> The examples I have found are along the lines of: Here's how to start a
> container, bridge it to a NIC which has a single IP assigned, ssh to it ...
> and the examples stop there. What I'd like to achieve is a setup where, say,
> a box with 6 IPs on an external network - on eth0 before bridging (or its
> alternatives) - can have 5 of those IPs each dedicated to different single
> container.

I have a setup similar to that you describe. It's a bit hackish, but I
like it's transparency and isolation capabilities (so I can leave
CAP_NET_RAW and CAP_NET_ADMIN for containers). It looks like that:

For the host:

host # ip addr add 1.1.128.2/20 dev eth0
host # route add -net default gw 1.1.128.1


For lxc1:

host # cat /etc/lxc/lxc1/config | grep net
lxc.network.type = veth
lxc.network.veth.pair = lxc1_veth0
lxc.network.flags = up

host # arp -Ds 1.1.128.3 eth0 pub
host # arp -Ds 1.1.128.4 eth0 pub
host # arp -Ds 1.1.128.5 eth0 pub
host # arp -Ds 1.1.128.6 eth0 pub

host # sysctl -w net.ipv4.conf.lxc1_veth0.proxy_arp = 1

host # ip addr add 10.0.1.1/24 dev lxc1_veth0

host # route add -host 1.1.128.3 gw 10.0.1.2
host # route add -host 1.1.128.4 gw 10.0.1.2
host # route add -host 1.1.128.5 gw 10.0.1.2
host # route add -host 1.1.128.6 gw 10.0.1.2

lxc1 # ip addr add 10.0.1.2/24 dev lxc1_veth0

lxc1 # ip addr add 1.1.128.3/20 dev eth0
lxc1 # ip addr add 1.1.128.4/20 dev eth0
lxc1 # ip addr add 1.1.128.5/20 dev eth0
lxc1 # ip addr add 1.1.128.6/20 dev eth0

lxc1 # route add -net default gw 1.1.128.1


For lxc2:

host # cat /etc/lxc/lxc2/config | grep net
lxc.network.type = veth
lxc.network.veth.pair = lxc2_veth0
lxc.network.flags = up

host # arp -Ds 1.1.128.7 eth0 pub
host # sysctl -w net.ipv4.conf.lxc2_veth0.proxy_arp = 1
host # ip addr add 10.0.2.1/24 dev lxc2_veth0
host # route add -host 1.1.128.7 gw 10.0.2.2

lxc2 # ip addr add 10.0.2.2/24 dev eth0
lxc2 # ip addr add 1.1.128.7/20 dev eth0
lxc2 # route add -net default gw 1.1.128.1


Hope it helps. Btw, I use custom udev scripts to do the host part, and
stock init scripts inside the containers to do the lxc part.
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: How do containers tie to multiple IP's on a NIC?

[Pavel Labushev]

06.07.2010 23:00, Pavel Labushev пишет:

> lxc1 # ip addr add 10.0.1.2/24 dev lxc1_veth0

Uh, it's eth0 instead of lxc1_veth0.
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: How do containers tie to multiple IP's on a NIC?

[Eric W. Biederman]

Whit Blauvelt <whit-M6G8SDWvnhfby3iVrkZq2A@public.gmane.org> writes:

> On Mon, Jul 05, 2010 at 05:50:38PM +0800, Pavel Labushev wrote:
>
>> What exactly are you trying to achieve? A transparent packet forwarding
>> between containers and external networks?
>
> I'm trying to get the overview of what can be achieved, and how. Unless I've
> missed it, there's not much documentation on even moderately complex use of
> containers. Since the capabilities are rapidly advancing, maybe I'm just
> asking the question a few months too early? From the outside, as someone new
> to containers, it looks like a maze where there are a number of entrances,
> each of which may lead approximately to the goal, but some of which may be
> dead ends. 
>
> The examples I have found are along the lines of: Here's how to start a
> container, bridge it to a NIC which has a single IP assigned, ssh to it ...
> and the examples stop there. What I'd like to achieve is a setup where, say,
> a box with 6 IPs on an external network - on eth0 before bridging (or its
> alternatives) - can have 5 of those IPs each dedicated to different single
> container.
>
> I'm not committed to a particular way of achieving that yet, just looking at
> the maze entrances wondering which to choose. Standard packet forwarding,
> routing and firewalling in Linux isn't what I'm asking about, I use that
> stuff frequently in complex ways, and I already run some things in simple
> chroots. I'm trying to learn how, on a test basis, to set up something like
> a production environment with lxc, where it involves multiple IPs, WAN or
> LAN, on each single NIC, behind some of which should be containers which
> effectively own individual IPs, publicly available.
>
> It's probably looking harder to me than it is, because I haven't found a
> clear description of it yet. 

The paper has probably already answered this but the sound bite answer is:

Each network namespace appears to userspace as separate instance of
the network stack.  Separate network device, separate forwarding
tables, separate iptables rules etc.  Network devices can be moved
between network namespaces.

paired veth devices are interesting because you can put one end of a
logical tunnel in each network namespace.

macvlan devices are interesting because you can create assign multiple
mac addresses to a nic and have a different network device for each
mac address.

You can use special tools like lxc to set these things up, but you can also
just run commands inside the network namespace and setup the environment
like you would normally.

Eric

Re: How do containers tie to multiple IP's on a NIC?

[Whit Blauvelt]

On Wed, Jul 07, 2010 at 05:55:22AM -0700, Eric W. Biederman wrote:

> paired veth devices are interesting because you can put one end of a
> logical tunnel in each network namespace.
> 
> macvlan devices are interesting because you can create assign multiple
> mac addresses to a nic and have a different network device for each
> mac address.

These two statements could be section headers in a valuable article - or
book chapter.

The problem with the standard VM concep: it takes the metaphor of separate
computers too literally. The potential for lessening the materials expense
and environmental cost of producing so much hardware and the electricity to
power it - is huge. Conversion to containers instead of VMs can be a major
economic win.

But only if the knowledge of how to do it becomes widespread. 

Whit