Re: Networkconfiguration with KVM

[Dan Johansson <kvm@dmj.nu>]

On Monday 05 April 2010 13.34:26 mathias dufresne wrote:
> Hi,
> 
> For networks 1 and 2 (respectively 192.168.1.0/24 and 192.168.2.0/24, I'll
> acll also network 3 for the last one) you'll have to use bridges.
> For network 3 you can use vde which is quite simply to use and gives
> possibility to add new virtual networks without the need of physical
> interface.
> 
> For bridges configuration:
> brctl addbr ${brname}
> brctl setfd ${brname} 0
> brctl sethello ${brname} 1
> brctl stp ${brname} off
> brct addif ${brname} ${physical_if}
> 
> Here you can add an IP to you're bridge interface if needed for the host
>  has access to this network. Host's IP must by set up on bridge interface
>  with configuration, if IP is set up on physical interface it is not
>  working. I'm wondering why, my network knowledge is not much :/
> 
> Then when you'll run qemu to set up a new VM you'll have to use a tap
>  device for the VM
> 
> One important thing when use several networks in the same host is then vlan
> ID you'll use for networks. These vlans aren't (from my understanding : )
> 802.1Q vlans but pure qemu/kvm concepts. If you don't set up each VM's
> interface on the right kvm's vlan they will all be set up on vlan number 0,
> which means packets from all networks will appear everywhere (tcpdump on
> 192.168.1.0/24 card will see traffic for others networks).
> 
> Here is the command I use to launch a VM with two bridged interfaces and
>  one other on some vde network:
> screen -d -m qemu-system-x86_64 -m 64 \
> -net tap,vlan=0,ifname=tap0,script=/path/to/scripts/kvm-ifup.br0 \
> -net nic,vlan=0,model=e1000,macaddr=52:54:00:01:ab:cd \
> -net tap,vlan=3,ifname=tap1,script=/path/to/scripts/kvm-ifup.br1 \
> -net nic,vlan=3,model=e1000,macaddr=52:54:00:03:ab:cd \
> -net vde,vlan=2,sock=/var/vde/switches/vdelan0 \
> -net nic,vlan=2,model=e1000,macaddr=52:54:00:02:ab:cd \
> -hda /path/to/disks/vm.vmdk \
> -nographic
> 
> Note the "vlan=X" option set on both -net arguments for each interface.
> 
> For VDE switchs:
> /usr/bin/vde_switch -m 770 -g qemu -daemon -n 8 -t dummy10 -s
> /var/vde/switches/vdelan0 -M /var/vde/management/vdelan0
> 
> I use dummy interface on host to set up vde networks because the host
> mustn't have access on these networks.
> -s directory where VDE's unix sockets will be store for this network
> -M management socket (I haven't yet understood how to use them :p)
> -n number of port on the switch
> other options are quite easy to understand.
> 
> Finally my scripts to set up bridged interface are the following:
> ------------------
> #!/bin/sh
> #set -x
> switch=`echo $0 | awk -F\. '{ print $NF}'`
> if [ -n "$1" ];then
>         #/usr/bin/tunctl -u `whoami` -t $1
>         /sbin/ip link set $1 up
>         sleep 0.5s
>         /sbin/brctl addif $switch $1
>         exit 0
> else
>         echo "Error: no interface specified"
>         exit 1
> fi
> ------------------
> I use one script per bridge named kvm-ifup.${brname} because the script's
> name define the bridge on which the interface will be attached
>  (kvm-ifup.br0 -> br0 and kvm-ifup.br1 -> br1).
> 
> I use VDE switch for networks on which host isn't plugged because they are
> simpler to use for me but it is not the only way to proceed.
> 
> 
> 
> A last thing, I have a physical card on which is plugged my modem. As I
> don't want to expose the host on Internet I use a VM to act as a firewall.
> As I just can't give this VM direct access on the physical card dedicated
>  to Internet I use a bridge:
> 
> modem ------- physical card ----- bridge ----- tap ---- VM's internal card
> 
> The modem knows only one MAC address: the one from physical interface but
> this physical interface as no IP. The bridge has same MAC address as
> physical interface but has no more IP. The card with an IP is the VM's
> internal card so for packets pass through all these interface without IP I
> use ebtables to "nat" MAC address between physical interface and VM's
> internal card:
> ebtables -t nat -A PREROUTING -i $PHYS_DEV_NAME -d $PHYS_DEV_MAC -j dnat
> --to-destination $VIRT_DEV_MAC
> ebtables -t nat -A POSTROUTING -o $VIRT_DEV_NAME -s $VIRT_DEV_MAC -j snat
> --to-source $PHYS_DEV_MAC
> 
> Finally there are some interesting links:
> forum's thread related to vlan issues
> http://serverfault.com/questions/101477/issue-with-multiple-bridging-for-kv
> m-hosts
> 
> something about bridges:
> http://www.savelono.com/linux/using-multiple-interfaces-with-kvm-and-xen.ht
> ml
> 
> different method to build networks using qemu/kvm:
> http://people.gnome.org/~markmc/qemu-networking.html<http://people.gnome.or
> g/%7Emarkmc/qemu-networking.html>
> 
> Hoping this helps.
> 
> Kindly regards,
> 
> mathias
> 
> 2010/4/5 Dan Johansson <kvm@dmj.nu>
> 
> > On Sunday 04 April 2010 22.23:28 Held Bernhard wrote:
> > > Am 04.04.2010 20:02, schrieb Dan Johansson:
> > > > On Sunday 04 April 2010 15.00:26 sudhir kumar wrote:
> > > >> On Sun, Apr 4, 2010 at 5:47 PM, Dan Johansson <kvm@dmj.nu> wrote:
> > > >>> Hi,
> > > >>>
> > > >>> I am new to this list and to KVM (and qemu) so please be gentle
> > > >>> with me. Up until now I have been running my virtualizing  using
> > > >>> VMWare-Server. Now I want to try KVM due to some issues with the
> > > >>> VMWare-Server and I am having some troubles with the networking
> > > >>> part
> >
> > of
> >
> > > >>> KVM.
> > > >>>
> > > >>> This is a small example of what I want (best viewed in a fix-font):
> > > >>>
> > > >>>  +-----------------------------------+
> > > >>>
> > > >>>  | Host                              |
> > > >>>  |  +----------+                eth0 |---- 192.168.1.0/24
> > > >>>  |
> > > >>>  |  |      eth0|-- +                 |
> > > >>>  |  | VM1  eth1|---(---+------- eth1 |---- 192.168.2.0/24
> > > >>>  |  |      eth2|---(---(---+         |
> > > >>>  |
> > > >>>  |  +----------+   |   |   |         |
> > > >>>  |
> > > >>>  |  +----------+   +---(---(--- eth2 |---- 192.168.1.0/24
> > > >>>  |
> > > >>>  |  |      eth0|---+   |   |         |
> > > >>>  |  | VM2  eth1|-------+   +--- eth3 |---- 192.168.3.0/24
> > > >>>  |  |      eth2|-----------+         |
> > > >>>  |
> > > >>>  |  +----------+                     |
> > > >>>
> > > >>>  +-----------------------------------+
> > > >>>
> > > >>> Host-eth0 is only for the Host (no VM)
> > > >>> Host-eth1 is shared between the Host and the VM's (VM?-eth1)
> > > >>> Host-eth2 and Host-eth3 are only for the VMs (eth0 and eth2)
> > > >>>
> > > >>> The Host and the VMs all have fixed IPs (no dhcp or likewise).
> > > >>> In this example th IPs could be:
> > > >>> Host-eth0:      192.168.1.1
> > > >>> Host-eth1:      192.168.2.1
> > > >>> Host-eth2:      -
> > > >>> Host-eth3:      -
> > > >>> VM1-eth0:               192.168.1.11
> > > >>> VM1-eth1:               192.168.2.11
> > > >>> VM1-eth2:               192.168.3.11
> > > >>> VM2-eth0:               192.168.1.22
> > > >>> VM2-eth1:               192.168.2.22
> > > >>> VM3-eth2:               192.168.3.22
> > > >>>
> > > >>> And, yes, Host-eth0 and Host-eth2 are in the same subnet, with eth0
> > > >>> dedicated to the Host and eth2 dedicated to the VMs.
> > > >>>
> > > >>> In VMWare this was quite easy to setup (three bridged networks).
> > > >>
> > > >> Its easy with KVM too. You want 3 NICs per VM, so you need to pass
> > > >> the corresponding parameters(including qemu-ifup script) for 3 NICs
> > > >> to each VM.
> > > >> In the host you need to create 2 bridges: say br-eth1 and br-eth2.
> > > >> Make them as the interface on the host in place of the corresponding
> > > >> eth interfaces.(brct addbr br-eth1; ifcfg eth1 0.0.0.0 up; brctl
> > > >> addif br-eth eth1; assign eth1's ip and routes to breth1; same for
> > > >> eth2). In the corresponding qemu-ifup scripts of each interface use
> > > >> bridge=br-ethN (This basicaly translates to brctl addif br-ethN $1,
> > > >> where $ is the tap device created)
> > > >> This should work perfectly fine with your existing NW setup.
> > > >> For a quick reference use: http://www.linux-kvm.org/page/Networking
> > > >
> > > > Thanks for your help, but... I am still not able to get it to work
> > > > the way I want.
> > > > This is what I have don so far:
> > > > brctl addbr br-eth1
> > > > brctl addbr br-eth3
> > > >
> > > > ip link set eth1 up
> > > > ip link set eth3 up
> > > >
> > > > brctl addif br-eth1 eth1
> > > > brctl addif br-eth3 eth3
> > > >
> > > > tunctl -b -t qtap1
> > > > tunctl -b -t qtap3
> > > >
> > > > brctl addif br-eth1 qtap1
> > > > brctl addif br-eth3 qtap3
> > > >
> > > > ifconfig qtap1 up 0.0.0.0 promisc
> > > > ifconfig qtap3 up 0.0.0.0 promisc
> > > >
> > > > # ifconfig
> > > > eth0      Link encap:Ethernet  HWaddr 00:0d:88:52:51:24
> > > >           inet addr:192.168.1.3  Bcast:192.168.1.255
> >
> >  Mask:255.255.255.0
> >
> > > >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:443638 errors:0 dropped:0 overruns:0 frame:0
> > > >           TX packets:758540 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:1000
> > > >           RX bytes:47041686 (44.8 MiB)  TX bytes:990115354 (944.2
> > > > MiB) Interrupt:19 Base address:0xec00
> > > >
> > > > eth1      Link encap:Ethernet  HWaddr 00:0d:88:52:51:25
> > > >           inet addr:192.168.4.1  Bcast:192.168.4.255
> >
> >  Mask:255.255.255.0
> >
> > > >           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> > > >           TX packets:6 errors:0 dropped:0 overruns:0 carrier:6
> > > >           collisions:0 txqueuelen:1000
> > > >           RX bytes:0 (0.0 B)  TX bytes:360 (360.0 B)
> > > >           Interrupt:18 Base address:0xe880
> > > >
> > > > eth3      Link encap:Ethernet  HWaddr 00:0d:88:52:51:27
> > > >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> > > >           TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:1000
> > > >           RX bytes:0 (0.0 B)  TX bytes:240 (240.0 B)
> > > >           Interrupt:16 Base address:0xe480
> > > >
> > > > qtap1     Link encap:Ethernet  HWaddr 26:c0:de:df:c5:e4
> > > >           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:351 errors:0 dropped:0 overruns:0 frame:0
> > > >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:500
> > > >           RX bytes:14742 (14.3 KiB)  TX bytes:0 (0.0 B)
> > > >
> > > > qtap3     Link encap:Ethernet  HWaddr 26:3e:ba:2d:97:bc
> > > >           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:6 errors:0 dropped:0 overruns:0 frame:0
> > > >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:500
> > > >           RX bytes:252 (252.0 B)  TX bytes:0 (0.0 B)
> > > >
> > > >
> > > > # brctl show
> > > > bridge name     bridge id               STP enabled     interfaces
> > > > br-eth1         8000.000d88525125       no              eth1
> > > >                                                         qtap1
> > > > br-eth3         8000.000d88525127       no              eth3
> > > >                                                         qtap3
> > > >
> > > >
> > > > This is the way I start the guest:
> > > > kvm -net nic,vlan=1,model=rtl8139,macaddr=52:54:00:12:34:56 -net
> > > > tap,vlan=1,ifname=qtap1,script=no,downscript=no -net
> > > > nic,vlan=3,model=rtl8139,macaddr=52:54:00:12:34:58 -net
> > > > tap,vlan=3,ifname=qtap3,script=no,downscript=no Robbie.img -m 1024
> > > >
> > > >
> > > > The eth3/br-eth3/qtap3 looks OK (I can ping the "default-GW" on that
> > > > network from the guest) but the connection to the "shared" interface
> > > > (eth1/br- eth1/qtap1) does not work, I can not ping or ssh to/from
> > > > the guest from/to the host. Do not ask me if I can ping any other
> > > > host on that network - there are no other host on the network yet,
> > > > just the
> >
> > Host
> >
> > > > and the guest.
> > > >
> > > > Any suggestions?
> > >
> > > eth1 should not have an IP address:
> > > # ifconfig eth1 0.0.0.0
> > >
> > > br-eth1 is not activated (it's missing in `ifconfig`), and it needs an
> > > IP address:
> > > # ifconfig br-eth1 192.168.2.1/24
> > >
> > > Even if it works I would explicitly activate br-eth3 too:
> > > # ifconfig br-eth3 0.0.0.0 up
> > >
> > > Looking at the output of `ifconfig` shows that the IP-address of eth0
> > > (192.168.1.3) doesn't match 192.168.1.1 from your address list, and
> > > eth1 (192.168.4.1) is in a different network than the specified
> >
> > 192.168.2.1/24.
> >
> > OK, the 192.168.1.3 - 192.168.1.1 is my typo, 192.168.1.3 is the correct
> > IP for the Host on this interface.
> > I was first planning to use the 192.168.2.0/24 network here, but I had
> > forgotten that that was already used somewhere else in my network, so I
> > just
> > picked the next "free" network address. Sorry for the confusion.
> >
> > I have tried you suggestions above but I still have no network
> > connection.
> >
> > # ifconfig eth1
> > eth1      Link encap:Ethernet  HWaddr 00:0d:88:52:51:25
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:3 errors:0 dropped:0 overruns:0 carrier:3
> >          collisions:0 txqueuelen:1000
> >          RX bytes:0 (0.0 B)  TX bytes:180 (180.0 B)
> >           Interrupt:18 Base address:0xe880
> >
> >
> > # ifconfig br-eth1
> > br-eth1   Link encap:Ethernet  HWaddr 00:0d:88:52:51:25
> >           inet addr:192.168.4.1  Bcast:192.168.4.255  Mask:255.255.255.0
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX packets:65 errors:0 dropped:0 overruns:0 frame:0
> >          TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
> >          collisions:0 txqueuelen:0
> >          RX bytes:3518 (3.4 KiB)  TX bytes:4750 (4.6 KiB)
> >
> > # ifconfig br-eth3
> > br-eth3   Link encap:Ethernet  HWaddr 00:0d:88:52:51:27
> >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:0
> >          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> >
> > Regards,
OK, lets simplify it a bit. This is now the layout that I am playing with and 
trying to get KVM to work .

 +-----------------------------------+
  | Host                              |
  |  +----------+                eth0 |---- 192.168.1.0/24
  |  | VM   eth1|------qtap1          |
  |  |          |        |            |
  |  | eth0     |      br-eth1---eth1 |---- 192.168.4.0/24
  |  +----------+                     |
  |                                   |
  +-----------------------------------+

Host-eth0 is only for the Host (no VM I/O)
Host-eth1 is shared between the Host and the VM (Host <--> VM, Host<--
>"external", VM <--> "external")
VM-eth0 is unused at the moment.

This is the commands I have been using to setup the Networking:
brctl addbr br-eth1
brctl setfd br-eth1 0
brctl sethello br-eth1 1
brctl stp br-eth1 off
brctl addif br-eth1 eth1
tunctl -b -t qtap1
brctl addif br-eth1 qtap1
ifconfig qtap1 up 0.0.0.0 promisc
ifconfig br-eth1 192.168.4.1/24 up
ifconfig eth1 0.0.0.0 up

And this is the way I start the VM:
kvm -net nic,vlan=1,model=rtl8139,macaddr=52:54:00:12:34:56 -net 
tap,vlan=1,ifname=qtap1,script=no,downscript=no Robbie.img -m 1024 -k de-ch

On the Host I have the following:
# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:0d:88:52:51:25
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:18
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:1080 (1.0 KiB)
          Interrupt:18 Base address:0xe880

# ifconfig br-eth1
br-eth1   Link encap:Ethernet  HWaddr 00:0d:88:52:51:25
          inet addr:192.168.4.1  Bcast:192.168.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:501 errors:0 dropped:0 overruns:0 frame:0
          TX packets:479 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:26378 (25.7 KiB)  TX bytes:35080 (34.2 KiB)

# ifconfig qtap1
qtap1     Link encap:Ethernet  HWaddr da:77:3a:a9:40:23
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:503 errors:0 dropped:0 overruns:0 frame:0
          TX packets:481 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:33524 (32.7 KiB)  TX bytes:35240 (34.4 KiB)

# brctl show
bridge name     bridge id               STP enabled     interfaces
br-eth1         8000.000d88525125       no              eth1
                                                        qtap1
# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.4.0     0.0.0.0         255.255.255.0   U         0 0          0 br-
eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
239.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0


And in the VM I have:
# ifconfig
eth1      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
          inet addr:192.168.4.4  Bcast:192.168.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1040 (1.0 KiB)  TX bytes:1018 (1018.0 B)
          Interrupt:11 Base address:0xc000

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.4.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         192.168.4.1     0.0.0.0         UG        0 0          0 eth1

But Still I get the following:
>From the Host to the VM:
$ ssh root@192.168.4.4
ssh: connect to host 192.168.4.4 port 22: Connection refused

And from the VM to the Host:
# ssh root@192.168.4.1
ssh: connect to host 192.168.4.1 port 22: Connection refused

Any more suggestions?
-- 
Dan Johansson, <http://www.dmj.nu>
***************************************************
This message is printed on 100% recycled electrons!
***************************************************