Regarding routed networking with KVM

Regarding routed networking with KVM

[Rajiv Rajaian]

Hi all
I have installed KVM with fedora13 operating system and I have
successfully created guest operating system with virt-install.
Also I have configured routed networking in host machine. Here's my
routed networking configuration

[root@kvmcluster ~]# virsh net-dumpxml routed
<network>
  <name>routed</name>
  <uuid>78b64a7e-48ed-4296-8794-a8450798d283</uuid>
  <forward dev='eth0' mode='route'/>
  <bridge name='virbr1' stp='on' delay='0' />
  <ip address='144.68.100.100' netmask='255.255.255.0'>
    <dhcp>
      <range start='144.68.100.1' end='144.68.100.254' />
    </dhcp>
  </ip>
</network>

Now I can reach the guest VM which is having static ip address
144.68.100.1 from the host machine(kvmcluster)

[root@kvmcluster ~]# ping 144.68.100.1
PING 144.68.100.1 (144.68.100.1) 56(84) bytes of data.
64 bytes from 144.68.100.1: icmp_seq=1 ttl=64 time=0.910 ms


The routing table entry of the host machine looks like this

[root@kvmcluster ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
144.68.100.0    0.0.0.0         255.255.255.0   U     0      0        0 virbr1
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
0.0.0.0         10.2.0.100      0.0.0.0         UG    0      0        0 eth0

>From the Default gateway machine (10.2.0.100) which is in the same
subnet as that of host machine  I have added a routing table entry as
[root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
gw 10.2.0.100

Now I can't reach the guest VM with this routing entry.

But when I have added the routing table entry with 10.2.0.20(host
machine's ip) as gateway I can reach the guest VM
[root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
gw 10.2.0.20

[root@careserver ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
144.68.100.0    10.2.0.20       255.255.255.0   UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         10.2.0.1        0.0.0.0         UG    0      0        0 eth0

Why the default gateway machine can't reach the guest VM with
10.2.0.100 as gateway. For each VM whether have to add separate
routing table entry?
Is there any other way to have a common routing table entry. Please
help me to solve this issue


Regards.
Rajiv. R
Project Associate,
CARE , CT Dept,
Anna University Chennai

Re: Regarding routed networking with KVM

[Daniel P. Berrange]

On Tue, Sep 14, 2010 at 03:28:56PM +0530, Rajiv Rajaian wrote:
> Hi all
> I have installed KVM with fedora13 operating system and I have
> successfully created guest operating system with virt-install.
> Also I have configured routed networking in host machine. Here's my
> routed networking configuration
> 
> [root@kvmcluster ~]# virsh net-dumpxml routed
> <network>
>   <name>routed</name>
>   <uuid>78b64a7e-48ed-4296-8794-a8450798d283</uuid>
>   <forward dev='eth0' mode='route'/>
>   <bridge name='virbr1' stp='on' delay='0' />
>   <ip address='144.68.100.100' netmask='255.255.255.0'>
>     <dhcp>
>       <range start='144.68.100.1' end='144.68.100.254' />
>     </dhcp>
>   </ip>
> </network>
> 
> Now I can reach the guest VM which is having static ip address
> 144.68.100.1 from the host machine(kvmcluster)
> 
> [root@kvmcluster ~]# ping 144.68.100.1
> PING 144.68.100.1 (144.68.100.1) 56(84) bytes of data.
> 64 bytes from 144.68.100.1: icmp_seq=1 ttl=64 time=0.910 ms
> 
> 
> The routing table entry of the host machine looks like this
> 
> [root@kvmcluster ~]# route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
> 10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
> 192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
> 144.68.100.0    0.0.0.0         255.255.255.0   U     0      0        0 virbr1
> 169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
> 0.0.0.0         10.2.0.100      0.0.0.0         UG    0      0        0 eth0
> 
> From the Default gateway machine (10.2.0.100) which is in the same
> subnet as that of host machine  I have added a routing table entry as
> [root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
> gw 10.2.0.100
> 
> Now I can't reach the guest VM with this routing entry.

This is because '10.2.0.100' is not the gateway for the subnet
you created. When you create a routed network in libvirt like
this, the gateway for that network is the host on which it is
running. This routing entry you added on the gateway machine
is just pointing the gateway back to itself - it still has
no clue which host has the 144.68.100.0 subnet. Your gateway
needs to know 'where do I send packets for 144.68.100.0?'.
The answer is the host 10.2.0.20...


Hence this second routing entry you show is the correct one:

> But when I have added the routing table entry with 10.2.0.20(host
> machine's ip) as gateway I can reach the guest VM
> [root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
> gw 10.2.0.20

> 
> [root@careserver ~]# route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
> 10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
> 192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
> 144.68.100.0    10.2.0.20       255.255.255.0   UG    0      0        0 eth0
> 169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
> 0.0.0.0         10.2.0.1        0.0.0.0         UG    0      0        0 eth0
> 
> Why the default gateway machine can't reach the guest VM with
> 10.2.0.100 as gateway. For each VM whether have to add separate
> routing table entry?

You aren't adding a routing table entry per VM here. You added
an entry for the entire subnet 144.68.100.0/255.255.255.0, which
you said has one VM on 144.68.100.1. This subnet can have many 
more VMs running on it without adding more routing table entries.


Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: Regarding routed networking with KVM

[Rajiv Rajaian]

Thanks for your reply Daniel.
Suppose I want to create on more VM with static ip address
144.68.100.0 on host having ip address 10.2.0.25 then do i need to add
one more routing table entry by specifying 10.2.0.25 as gateway or
10.2.0.20 as previous entry is enough to access that VM?
Is there any provision to specify the gateway machine for (virbr1)
while creating the routed networking with libvirt?
Here the thing is I want to reach VMs running on different host
machines having same subnet and same default gateway from another
machine which is running on the LAN. ie VM1 running at Host1 and VM2
running at Host2  should be accessed from Host3 which is in LAN.  Is
there any other steps I have to do to reach the VMs residing on
different host machines??
Thanks in Advance

On Tue, Sep 14, 2010 at 3:45 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Tue, Sep 14, 2010 at 03:28:56PM +0530, Rajiv Rajaian wrote:
>> Hi all
>> I have installed KVM with fedora13 operating system and I have
>> successfully created guest operating system with virt-install.
>> Also I have configured routed networking in host machine. Here's my
>> routed networking configuration
>>
>> [root@kvmcluster ~]# virsh net-dumpxml routed
>> <network>
>>   <name>routed</name>
>>   <uuid>78b64a7e-48ed-4296-8794-a8450798d283</uuid>
>>   <forward dev='eth0' mode='route'/>
>>   <bridge name='virbr1' stp='on' delay='0' />
>>   <ip address='144.68.100.100' netmask='255.255.255.0'>
>>     <dhcp>
>>       <range start='144.68.100.1' end='144.68.100.254' />
>>     </dhcp>
>>   </ip>
>> </network>
>>
>> Now I can reach the guest VM which is having static ip address
>> 144.68.100.1 from the host machine(kvmcluster)
>>
>> [root@kvmcluster ~]# ping 144.68.100.1
>> PING 144.68.100.1 (144.68.100.1) 56(84) bytes of data.
>> 64 bytes from 144.68.100.1: icmp_seq=1 ttl=64 time=0.910 ms
>>
>>
>> The routing table entry of the host machine looks like this
>>
>> [root@kvmcluster ~]# route -n
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>> 10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
>> 192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
>> 144.68.100.0    0.0.0.0         255.255.255.0   U     0      0        0 virbr1
>> 169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
>> 0.0.0.0         10.2.0.100      0.0.0.0         UG    0      0        0 eth0
>>
>> From the Default gateway machine (10.2.0.100) which is in the same
>> subnet as that of host machine  I have added a routing table entry as
>> [root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
>> gw 10.2.0.100
>>
>> Now I can't reach the guest VM with this routing entry.
>
> This is because '10.2.0.100' is not the gateway for the subnet
> you created. When you create a routed network in libvirt like
> this, the gateway for that network is the host on which it is
> running. This routing entry you added on the gateway machine
> is just pointing the gateway back to itself - it still has
> no clue which host has the 144.68.100.0 subnet. Your gateway
> needs to know 'where do I send packets for 144.68.100.0?'.
> The answer is the host 10.2.0.20...
>
>
> Hence this second routing entry you show is the correct one:
>
>> But when I have added the routing table entry with 10.2.0.20(host
>> machine's ip) as gateway I can reach the guest VM
>> [root@careserver ~]# route add -net 144.68.100.0 netmask 255.255.255.0
>> gw 10.2.0.20
>
>>
>> [root@careserver ~]# route -n
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>> 10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
>> 192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
>> 144.68.100.0    10.2.0.20       255.255.255.0   UG    0      0        0 eth0
>> 169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
>> 0.0.0.0         10.2.0.1        0.0.0.0         UG    0      0        0 eth0
>>
>> Why the default gateway machine can't reach the guest VM with
>> 10.2.0.100 as gateway. For each VM whether have to add separate
>> routing table entry?
>
> You aren't adding a routing table entry per VM here. You added
> an entry for the entire subnet 144.68.100.0/255.255.255.0, which
> you said has one VM on 144.68.100.1. This subnet can have many
> more VMs running on it without adding more routing table entries.
>
>
> Daniel
> --
> |: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
> |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
> |: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
> |: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
>

Re: Regarding routed networking with KVM

[Daniel P. Berrange]

On Tue, Sep 14, 2010 at 04:13:09PM +0530, Rajiv Rajaian wrote:
> Thanks for your reply Daniel.
> Suppose I want to create on more VM with static ip address
> 144.68.100.0 on host having ip address 10.2.0.25 then do i need to add
> one more routing table entry by specifying 10.2.0.25 as gateway or
> 10.2.0.20 as previous entry is enough to access that VM?

Each host must have a separate subnet for VMs. So if you have 3 
hosts, you'd want to repeat your libvirt network configure using
something like these in the XML

 Host A (10.2.0.20): 144.68.100.0/255.255.255.0
 Host B (10.2.0.25): 144.68.101.0/255.255.255.0
 Host C (10.2.0.30): 144.68.102.0/255.255.255.0

And then on your LAN gateway setup a route for each host's
subnet

 # route add -net 144.68.100.0 netmask 255.255.255.0 gw 10.2.0.20
 # route add -net 144.68.101.0 netmask 255.255.255.0 gw 10.2.0.25
 # route add -net 144.68.102.0 netmask 255.255.255.0 gw 10.2.0.30


> Is there any provision to specify the gateway machine for (virbr1)
> while creating the routed networking with libvirt?

Not with the current design. We require the main LAN gateway to have
static routes added. In the future we might provide an impl based
on Proxy-ARP or IP-subnetting which would simplify this.

> Here the thing is I want to reach VMs running on different host
> machines having same subnet and same default gateway from another
> machine which is running on the LAN. ie VM1 running at Host1 and VM2
> running at Host2  should be accessed from Host3 which is in LAN.  Is
> there any other steps I have to do to reach the VMs residing on
> different host machines??

Each host that is running VMs would need its own routed network
with dedicated subnet defined. Once the LAN gateway has static 
routes for each subnet, any host on your LAN should be able to 
access any VM on any of your hosts, and vica-verca.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: Regarding routed networking with KVM

[Rajiv Rajaian]

Thanks for your kind information Daniel.
Consider this scenario
VM1(144.68.100.1) and VM2(144.68.100.2) running on Host1(10.2.0.20)
and Host2(10.2.0.30) respectively. Is it possible to access the VM1
and VM2 from Host3(10.2.0.100). How to add a static route for this
scenario?? Here I don't need separate subnets for VMs.
All VMs should be in same subnet ie 144.68.100.0/255.255.255.0
Is there any way to configure this ??

On Tue, Sep 14, 2010 at 4:21 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Tue, Sep 14, 2010 at 04:13:09PM +0530, Rajiv Rajaian wrote:
>> Thanks for your reply Daniel.
>> Suppose I want to create on more VM with static ip address
>> 144.68.100.0 on host having ip address 10.2.0.25 then do i need to add
>> one more routing table entry by specifying 10.2.0.25 as gateway or
>> 10.2.0.20 as previous entry is enough to access that VM?
>
> Each host must have a separate subnet for VMs. So if you have 3
> hosts, you'd want to repeat your libvirt network configure using
> something like these in the XML
>
>  Host A (10.2.0.20): 144.68.100.0/255.255.255.0
>  Host B (10.2.0.25): 144.68.101.0/255.255.255.0
>  Host C (10.2.0.30): 144.68.102.0/255.255.255.0
>
> And then on your LAN gateway setup a route for each host's
> subnet
>
>  # route add -net 144.68.100.0 netmask 255.255.255.0 gw 10.2.0.20
>  # route add -net 144.68.101.0 netmask 255.255.255.0 gw 10.2.0.25
>  # route add -net 144.68.102.0 netmask 255.255.255.0 gw 10.2.0.30
>
>
>> Is there any provision to specify the gateway machine for (virbr1)
>> while creating the routed networking with libvirt?
>
> Not with the current design. We require the main LAN gateway to have
> static routes added. In the future we might provide an impl based
> on Proxy-ARP or IP-subnetting which would simplify this.
>
>> Here the thing is I want to reach VMs running on different host
>> machines having same subnet and same default gateway from another
>> machine which is running on the LAN. ie VM1 running at Host1 and VM2
>> running at Host2  should be accessed from Host3 which is in LAN.  Is
>> there any other steps I have to do to reach the VMs residing on
>> different host machines??
>
> Each host that is running VMs would need its own routed network
> with dedicated subnet defined. Once the LAN gateway has static
> routes for each subnet, any host on your LAN should be able to
> access any VM on any of your hosts, and vica-verca.
>
> Daniel
> --
> |: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
> |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
> |: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
> |: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
>

Re: Regarding routed networking with KVM

[Daniel P. Berrange]

On Tue, Sep 14, 2010 at 04:58:46PM +0530, Rajiv Rajaian wrote:
> Thanks for your kind information Daniel.
> Consider this scenario
> VM1(144.68.100.1) and VM2(144.68.100.2) running on Host1(10.2.0.20)
> and Host2(10.2.0.30) respectively. Is it possible to access the VM1
> and VM2 from Host3(10.2.0.100). How to add a static route for this
> scenario?? Here I don't need separate subnets for VMs.
> All VMs should be in same subnet ie 144.68.100.0/255.255.255.0
> Is there any way to configure this ??

No, in the setup libvirt does, each host must have a separate subnet.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: Regarding routed networking with KVM

[Rajiv Rajaian]

Thanks for your information Daniel

On Tue, Sep 14, 2010 at 5:46 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Tue, Sep 14, 2010 at 04:58:46PM +0530, Rajiv Rajaian wrote:
>> Thanks for your kind information Daniel.
>> Consider this scenario
>> VM1(144.68.100.1) and VM2(144.68.100.2) running on Host1(10.2.0.20)
>> and Host2(10.2.0.30) respectively. Is it possible to access the VM1
>> and VM2 from Host3(10.2.0.100). How to add a static route for this
>> scenario?? Here I don't need separate subnets for VMs.
>> All VMs should be in same subnet ie 144.68.100.0/255.255.255.0
>> Is there any way to configure this ??
>
> No, in the setup libvirt does, each host must have a separate subnet.
>
> Regards,
> Daniel
> --
> |: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
> |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
> |: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
> |: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
>