Simple way of putting a VM on a LAN

Simple way of putting a VM on a LAN

[Bill Davidsen]

I seem to be missing the "right way" to put a VM on the LAN such that it 
appears to really be a machine on the LAN. This is a requirement for 
being able to deploy internal servers as needed.

For instance, consider a DNS server or outbound mailer at IP 10.x.y.z 
and how to do that with a VM. Current I have not found a way other than 
putting the IP on as an alias, like eth0:1, setting up a tunnel using a 
bunch of stuff from Debian, plugging in a set of moderately complex 
iptables rules, starting the VM using a bunch of obscure -nic options 
unreleated to the actual IP to be provided, and generally a ton of 
complexity to install and remove.

I often want to provide momentary service not requiring preserving a lot 
of state, like mailers, DNS, etc. Surely there's a better way?

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Simple way of putting a VM on a LAN

[Mike Snitzer]

On Thu, Jun 26, 2008 at 9:54 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> I seem to be missing the "right way" to put a VM on the LAN such that it
> appears to really be a machine on the LAN. This is a requirement for being
> able to deploy internal servers as needed.
>
> For instance, consider a DNS server or outbound mailer at IP 10.x.y.z and
> how to do that with a VM. Current I have not found a way other than putting
> the IP on as an alias, like eth0:1, setting up a tunnel using a bunch of
> stuff from Debian, plugging in a set of moderately complex iptables rules,
> starting the VM using a bunch of obscure -nic options unreleated to the
> actual IP to be provided, and generally a ton of complexity to install and
> remove.
>
> I often want to provide momentary service not requiring preserving a lot of
> state, like mailers, DNS, etc. Surely there's a better way?

I've taken to using a bridge (or in virt-manager speak "shared
physical device").  The 'network-bridge' script (and supporting
xen-network-common.sh and xen-script-common.sh) that are provided with
xen rpms (e.g. xen-3.1.0-13.fc8.x86_64.rpm) make this relatively
painless.

The overall solution is not what I'd call "simple" but once I've
started the bridge I just defer to libvirtd to abstract away the
complexity associated with exposing each kvm guest to the physical
network.

Re: Simple way of putting a VM on a LAN

[Chris Lalancette]

Mike Snitzer wrote:
> I've taken to using a bridge (or in virt-manager speak "shared
> physical device").  The 'network-bridge' script (and supporting
> xen-network-common.sh and xen-script-common.sh) that are provided with
> xen rpms (e.g. xen-3.1.0-13.fc8.x86_64.rpm) make this relatively
> painless.
> 
> The overall solution is not what I'd call "simple" but once I've
> started the bridge I just defer to libvirtd to abstract away the
> complexity associated with exposing each kvm guest to the physical
> network.

Yep, exactly.  Actually, generally your distribution of choice provides nice
startup scripts to such things; in Fedora, you create an
/etc/sysconfig/network-scripts/ifcfg-eth0 that has a BRIDGE=br0, and an
/etc/sysconfig/network-scripts/ifcfg-br0 that defines the actual bridge with
TYPE=Bridge, and the system will bring up the bridge at bootup and plug your
eth0 into it.  I'm sure the other distributions have similar mechanisms.

Chris Lalancette

Re: Simple way of putting a VM on a LAN

[Freddie Cash]

On Thu, Jun 26, 2008 at 6:54 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> I seem to be missing the "right way" to put a VM on the LAN such that it
> appears to really be a machine on the LAN. This is a requirement for being
> able to deploy internal servers as needed.
>
> For instance, consider a DNS server or outbound mailer at IP 10.x.y.z and
> how to do that with a VM. Current I have not found a way other than putting
> the IP on as an alias, like eth0:1, setting up a tunnel using a bunch of
> stuff from Debian, plugging in a set of moderately complex iptables rules,
> starting the VM using a bunch of obscure -nic options unreleated to the
> actual IP to be provided, and generally a ton of complexity to install and
> remove.

Do some reading on bridged networking, and for configuring bridged
interfaces in Debian.

You'll need to install the tun-utils and bridge-utils packages, and
then configure /etc/network/interfaces to create the bridge device at
boot.

Basically, just move the "address", "netmask", and "gateway" lines
from eth0 to kvmbr0.

As an example:
# eth0 is the primary NIC
# This just needs to be "up" in order for the bridge to use it
auto eth0

# kvmbr0 is the bridge device used for KVM virtual machines
auto  kvmbr0
iface kvmbr0 inet static
        bridge-ports eth0
        bridge-maxwait 5
        address 10.1.x.x
        netmask 255.255.255.0
        gateway 10.1.x.1

Then configure /etc/kvm/kvm-ifup to use kvmbr0 as the bridge.

Then start your VMs.  So long as you are using -nic tap, they should
automatically be added to the kvmbr0 interface, and appear on the
network the same as any other server.

There's plenty of how-tos for this on the kvm wiki:
http://kvm.qumranet.com/kvmwiki

-- 
Freddie Cash
fjwcash@gmail.com

Re: Simple way of putting a VM on a LAN

[Bill Davidsen]

Chris Lalancette wrote:
> Mike Snitzer wrote:
>> I've taken to using a bridge (or in virt-manager speak "shared
>> physical device").  The 'network-bridge' script (and supporting
>> xen-network-common.sh and xen-script-common.sh) that are provided with
>> xen rpms (e.g. xen-3.1.0-13.fc8.x86_64.rpm) make this relatively
>> painless.
>>
>> The overall solution is not what I'd call "simple" but once I've
>> started the bridge I just defer to libvirtd to abstract away the
>> complexity associated with exposing each kvm guest to the physical
>> network.
> 
> Yep, exactly.  Actually, generally your distribution of choice provides nice
> startup scripts to such things; in Fedora, you create an
> /etc/sysconfig/network-scripts/ifcfg-eth0 that has a BRIDGE=br0, and an
> /etc/sysconfig/network-scripts/ifcfg-br0 that defines the actual bridge with
> TYPE=Bridge, and the system will bring up the bridge at bootup and plug your
> eth0 into it.  I'm sure the other distributions have similar mechanisms.
> 
A bit of the original problem seems to have been clipped before you read 
it, or I stated it poorly.
- the problems are immediate, not permanent hosting. So startup 
anythings are out, I'd have to put in scripts for every machine I might 
ever want to host on every machine capable of hosting.
- I need to bind an IP, unless you can point me to a different bridge 
package. If eth0 is x.y.z.10 and I put x.y.z.20 on eth0:1
     ifconfig eth0:1 x.y.z.20 up
adding eth0:1 moves the whole NIC to the bridge, and the normal 
functions of the machine come to a halt. I'm probably doing something 
wrong, currently I'm getting this done by ugly iptables abuse.

I am missing some piece on doing this quickly and selectively, for the 
case of "dns02 just dropped a cooling fan, we need another server, run 
it on your {whatever} machine for a few minutes." These are immediate 
and short term, but often done on machines in burn-in state, someone's 
desktop, etc. Little load, but the service must be running.

Does that clarify?

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Simple way of putting a VM on a LAN

[Javier Guerra]

On Wed, Jul 9, 2008 at 11:28 AM, Bill Davidsen <davidsen@tmr.com> wrote:
> A bit of the original problem seems to have been clipped before you read it,
> or I stated it poorly.

i think you're very confused.  maybe you got it working the hard way,
but it's really simple to do the easy way.

first, you have to do some small preparations on the host machine, but
nothing difficult.  this is what i do on my workstation (kubuntu) so
that i can fire up a test VM at any moment's whim:

- setup a bridge, and use it as main interface
- add a /etc/qemu-ifup script
- kvm kernel module
- make sure /dev/kvm and /dev/net/tun have the correct privilege access.

for the first one, in debian-like systems just use the following in
/dev/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto br0
iface br0 inet static
        address 172.18.13.66
        netmask 255.255.0.0
        network 172.18.0.0
        broadcast 172.18.255.255
        gateway 172.18.0.1
        bridge_ports eth0

that makes br0 my main interface, and adds eth0 to it. when i'm not
running any VM, it doesn't interfere in any way, except for any
utilities that default to eth0... if that were a problem, i could
simply rename eth0=>peth0 and br0=>eth0 (i think the Xen scripts do
similar tricks)

when that's set, /etc/qemu-ifup just have to setup the tun/tap
interface and add to the bridge:
#!/bin/sh
ifconfig $1 0.0.0.0 promisc up
brctl addif br0 $1

and that's it!  no need to meddle with iptables.  note that i don't
even set the IP, the VM is connected to the LAN, and it setups it's
own IP "from inside"

hope that helps

-- 
Javier

Re: Simple way of putting a VM on a LAN

[Bill Davidsen]

Javier Guerra wrote:
> On Wed, Jul 9, 2008 at 11:28 AM, Bill Davidsen <davidsen@tmr.com> wrote:
>   
>> A bit of the original problem seems to have been clipped before you read it,
>> or I stated it poorly.
>>     
>
> i think you're very confused.  maybe you got it working the hard way,
> but it's really simple to do the easy way.
>   

Your easy way seems to mean using Debian, other distributions don't have 
some of the scripts, or they are in different places or do different 
things. Other thoughts below.
> first, you have to do some small preparations on the host machine, but
> nothing difficult.  this is what i do on my workstation (kubuntu) so
> that i can fire up a test VM at any moment's whim:
>
> - setup a bridge, and use it as main interface
> - add a /etc/qemu-ifup script
> - kvm kernel module
> - make sure /dev/kvm and /dev/net/tun have the correct privilege access.
>
> for the first one, in debian-like systems just use the following in
> /dev/network/interfaces:
>   

That's Debian thing.
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> # The primary network interface
> auto br0
> iface br0 inet static
>         address 172.18.13.66
>         netmask 255.255.0.0
>         network 172.18.0.0
>         broadcast 172.18.255.255
>         gateway 172.18.0.1
>         bridge_ports eth0
>
> that makes br0 my main interface, and adds eth0 to it. when i'm not
> running any VM, it doesn't interfere in any way, except for any
> utilities that default to eth0... if that were a problem, i could
> simply rename eth0=>peth0 and br0=>eth0 (i think the Xen scripts do
> similar tricks)
>   

That's an interesting trick. I don't know of any problems I'm having 
which require it, but a useful thing to consider.
> when that's set, /etc/qemu-ifup just have to setup the tun/tap
> interface and add to the bridge:
> #!/bin/sh
> ifconfig $1 0.0.0.0 promisc up
> brctl addif br0 $1
>
> and that's it!  no need to meddle with iptables.  note that i don't
> even set the IP, the VM is connected to the LAN, and it setups it's
> own IP "from inside"
>   

Not being a trusting person I find that a bridge is an ineffective 
firewall, but with a bit of trickery that could live on the VM, to the 
extent it's needed. Now the "sets up its own IP" is a mystery, since 
there's no place I have told it what the IP of the machine it replaces 
might be. I did take the obvious step of setting the macadrs of the tap 
to that of the NIC in the original machine, but here I find a problem, 
at boot DHCP is not being used, or perhaps the issue is that some 
internal kvm DHCP service is being used instead of sending the requests 
out and letting my server provide the IP (and gateway, and nameservice, 
and etc). Setting up the IP and routing by hand does result in a working 
configuration, however, so other than the lack of control from using 
iptables to forward packets, it works well.

If the DHCP worked as expected it would really be easy.
> hope that helps
>   

I thank you for sharing your info, it was a good starting point even 
though some of the steps were not portable.

Well, it provides an easier way to get things working, there's one case 
where the iptables is still desirable, but that is a corner case for 
sure. Modulo the DHCP issue it works well, so I can say it did help, 
although not in the way you expected, I suspect.

I'm going to write it up while I can remember what I did and understand 
my notes. I have a bunch of tcpdump files from the physical NIC (eth0) 
and the bridge (br0), and wireshark, so I may get some idea of why DHCP 
isn't working, that would finish the job in most cases. Even if I have 
to do a bit of manual setup, it's faster than setting up iptables, and 
acceptably secure as long as the kvm host is at least as secure as the 
original.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Simple way of putting a VM on a LAN

[Javier Guerra]

On Wed, Jul 23, 2008 at 11:15 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> Your easy way seems to mean using Debian, other distributions don't have
> some of the scripts, or they are in different places or do different things.
> Other thoughts below.

yep, on Gentoo and SuSE i didn't find the included scripts flexible
enough, so i did the same 'by hand'.  that was a few years ago, it
might be better now; but it's not hard to do anyway.


> Not being a trusting person I find that a bridge is an ineffective firewall,

a bridge isn't a firewall.  it's the software equivalent of plugging
both your host and guest to an ethernet switch.  in most ways, your
host 'steps out of the way'.

> but with a bit of trickery that could live on the VM, to the extent it's
> needed. Now the "sets up its own IP" is a mystery, since there's no place I
> have told it what the IP of the machine it replaces might be. I did take the

as said before, it's as if your VM is directly plugged to the LAN.
you just configure its network 'from inside'.  the host doesn't care
what IP numbers it uses.  in fact, it could be using totally different
protocols, just as long as they go over ethernet.

> hand does result in a working configuration, however, so other than the lack
> of control from using iptables to forward packets, it works well.

you can use iptables.  maybe you have to setup ebtables, but in the
end, just put rules in the FORWARD chains.  google for 'transparent
firewall', or 'bridge iptables'

> of manual setup, it's faster than setting up iptables, and acceptably secure
> as long as the kvm host is at least as secure as the original.

just do with your VM as you do with a 'real' box.  after that, you can
use the fact that every packet to the VM has to pass through your eth0
device; even if they don't appear on your INPUT chains.

-- 
Javier

Re: Simple way of putting a VM on a LAN

[Bill Davidsen]

Javier Guerra wrote:
> On Wed, Jul 23, 2008 at 11:15 PM, Bill Davidsen <davidsen@tmr.com> wrote:
>> Your easy way seems to mean using Debian, other distributions don't have
>> some of the scripts, or they are in different places or do different things.
>> Other thoughts below.
> 
> yep, on Gentoo and SuSE i didn't find the included scripts flexible
> enough, so i did the same 'by hand'.  that was a few years ago, it
> might be better now; but it's not hard to do anyway.
> 
> 
>> Not being a trusting person I find that a bridge is an ineffective firewall,
> 
> a bridge isn't a firewall.  it's the software equivalent of plugging
> both your host and guest to an ethernet switch.  in most ways, your
> host 'steps out of the way'.

Maybe I didn't have my tongue far enough in my cheek... I do know what a 
bridge is, etc, I was referring to the desirability of using iptables 
for the forwarding. I must have looked at ebtables at one time, the 
package is loaded, but I don't remember having any instant "this is 
great" moments with it, so I'll have to reread the docs if I need more 
than the bridge.
> 
>> but with a bit of trickery that could live on the VM, to the extent it's
>> needed. Now the "sets up its own IP" is a mystery, since there's no place I
>> have told it what the IP of the machine it replaces might be. I did take the
> 
> as said before, it's as if your VM is directly plugged to the LAN.
> you just configure its network 'from inside'.  the host doesn't care
> what IP numbers it uses.  in fact, it could be using totally different
> protocols, just as long as they go over ethernet.

But when the host is really on the network, it uses DHCP to set the IP, 
while in a VM it never sends any DHCP packets, the setting of the IP 
times out, and I wind up with no IP until I set it. I have checked with 
tcpdump, the DHCP requests for IP appear on the bridge, but not on the 
eth0 NIC, and so are never seen by the DHCP server.

Do you see this problem, or have any information about it? Obviously 
suggestions on fixing this are needed, since the dhcp server is a 
candidate for virtualization in the future.
> 
>> hand does result in a working configuration, however, so other than the lack
>> of control from using iptables to forward packets, it works well.
> 
> you can use iptables.  maybe you have to setup ebtables, but in the
> end, just put rules in the FORWARD chains.  google for 'transparent
> firewall', or 'bridge iptables'
> 
>> of manual setup, it's faster than setting up iptables, and acceptably secure
>> as long as the kvm host is at least as secure as the original.
> 
> just do with your VM as you do with a 'real' box.  after that, you can
> use the fact that every packet to the VM has to pass through your eth0
> device; even if they don't appear on your INPUT chains.
> 


-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Simple way of putting a VM on a LAN

[Stuart Jansen]

On Fri, 2008-07-25 at 12:44 -0400, Bill Davidsen wrote:
> But when the host is really on the network, it uses DHCP to set the IP, 
> while in a VM it never sends any DHCP packets, the setting of the IP 
> times out, and I wind up with no IP until I set it. I have checked with 
> tcpdump, the DHCP requests for IP appear on the bridge, but not on the 
> eth0 NIC, and so are never seen by the DHCP server.
> 
> Do you see this problem, or have any information about it? Obviously 
> suggestions on fixing this are needed, since the dhcp server is a 
> candidate for virtualization in the future.

Just to be certain of the obvious, you added eth0 to the bridge, right?

brctl addif br0 eth0

And the tap device is active, right?

ifconfig tap0 up

Assuming it isn't something so obvious, I'm suspecting spanning tree.

brctl stp br0 off

Re: Simple way of putting a VM on a LAN

[Bill Davidsen]

Stuart Jansen wrote:
> On Fri, 2008-07-25 at 12:44 -0400, Bill Davidsen wrote:
>> But when the host is really on the network, it uses DHCP to set the IP, 
>> while in a VM it never sends any DHCP packets, the setting of the IP 
>> times out, and I wind up with no IP until I set it. I have checked with 
>> tcpdump, the DHCP requests for IP appear on the bridge, but not on the 
>> eth0 NIC, and so are never seen by the DHCP server.
>>
>> Do you see this problem, or have any information about it? Obviously 
>> suggestions on fixing this are needed, since the dhcp server is a 
>> candidate for virtualization in the future.
> 
> Just to be certain of the obvious, you added eth0 to the bridge, right?
> 
> brctl addif br0 eth0

Yes, everything works except the DHCP discovery. Once I bring up the VM 
NIC by hand and set the default route everything works really well with 
TCP, UDP, and ICMP, as well as the usual ARP packets, etc.
> 
> And the tap device is active, right?
> 
> ifconfig tap0 up

UP and based on something I saw in another script is tried adding 
promiscuous, which really didn't change anything.
> 
> Assuming it isn't something so obvious, I'm suspecting spanning tree.
> 
> brctl stp br0 off

It was never on (unless it was turned on by something more automated 
than my fingers, but it's definitely off now, and make no difference.

For a test I modified the network setup to a static IP and routing. That 
did work, although it is undesirable, since it invites having the DNS 
wrong. I moved to DHCP to be sure that the IPs are always right, a 
master list gets turned into entries in both dhchd.conf and the 
appropriate DNS files (forward and reverse lookeps are always right, 
too). Shot myself in that foot way back in ARPAnet days :-(

Thanks for the ideas, I have one more, but I have to do a little 
research before I can ask an intelligent question.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot