Weird networking problem

Weird networking problem

[Lorenzo Milesi]

Hi.
I'm experiencing weird network problems on a KVM installation.
OS is Ubuntu 12.04, qemu 1.0+noroms-0ubuntu14.3, kernel 3.2.0-34-generic.

eth0 is attached to LAN -> br0
eth2 is attached to WAN -> br1

Debian config follows:

auto eth0
iface eth0 inet manual
auto br0
iface br0 inet static
        address 192.168.1.47
        netmask 255.255.255.0
        gateway 192.168.1.1
        dns-nameservers 192.168.1.1 8.8.8.8
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

auto eth2
iface eth2 inet manual
auto br1
iface br1 inet manual
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0



I've configured a single guest to work a firewall (pfsense). Using version 2.1 beta which supports virtualized drivers.
XML config [1].

Problem: I've configured a VPN to another network (network B).
>From Network B, I can ping & ssh to 192.168.1.49 (another physical host on the lan), but I can only ping my kvm physical host, all TCP connection (ssh) gets lost. 
I have similar problem with port forward, while I can ssh to .49 I cannot to .47.
I managed to catch a tcpdump while trying to ssh to .47:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 96 bytes
12:18:21.720364 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2912170 ecr 0,nop,wscale 7], length 0
12:18:21.720760 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127905521 ecr 2912170,nop,wscale 7], length 0
12:18:22.718447 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2912420 ecr 0,nop,wscale 7], length 0
12:18:22.718814 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127905770 ecr 2912170,nop,wscale 7], length 0
12:18:22.923054 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127905821 ecr 2912170,nop,wscale 7], length 0
12:18:24.723703 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2912921 ecr 0,nop,wscale 7], length 0
12:18:24.724103 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127906272 ecr 2912170,nop,wscale 7], length 0
12:18:24.935085 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127906325 ecr 2912170,nop,wscale 7], length 0
12:18:28.734360 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2913924 ecr 0,nop,wscale 7], length 0
12:18:28.734737 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127907274 ecr 2912170,nop,wscale 7], length 0
12:18:28.947166 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127907328 ecr 2912170,nop,wscale 7], length 0
12:18:36.751056 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2915928 ecr 0,nop,wscale 7], length 0
12:18:36.751477 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127909279 ecr 2912170,nop,wscale 7], length 0
12:18:36.975114 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127909335 ecr 2912170,nop,wscale 7], length 0





I know it's not an issue with the firewall, because I've tried another distro and I had other kind of issue, always network related.
Any idea?
thanks!


P.S. please reply all as I'm not subscribed

[1]
<domain type='qemu' id='5'>
  <name>pfsense</name>
  <uuid>36d77162-3e9c-5317-d011-9b61a9bfb887</uuid>
  <memory>1548288</memory>
  <currentMemory>1548288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/depsrv01lv/pfsense'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:7e:03:aa'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:08:e5:84'/>
      <source bridge='br1'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-36d77162-3e9c-5317-d011-9b61a9bfb887</label>
    <imagelabel>libvirt-36d77162-3e9c-5317-d011-9b61a9bfb887</imagelabel>
  </seclabel>
</domain>


-- 
Lorenzo Milesi - lorenzo.milesi@yetopen.it

GPG/PGP Key-Id: 0xE704E230 - http://keyserver.linux.it

Re: Weird networking problem

[Stefan Hajnoczi]

On Tue, Dec 18, 2012 at 05:45:05PM +0100, Lorenzo Milesi wrote:
> From Network B, I can ping & ssh to 192.168.1.49 (another physical host on the lan), but I can only ping my kvm physical host, all TCP connection (ssh) gets lost. 
> I have similar problem with port forward, while I can ssh to .49 I cannot to .47.
> I managed to catch a tcpdump while trying to ssh to .47:
> 
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on vtnet0, link-type EN10MB (Ethernet), capture size 96 bytes
> 12:18:21.720364 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2912170 ecr 0,nop,wscale 7], length 0
> 12:18:21.720760 IP 192.168.1.47.ssh > my.host.com.34242: Flags [S.], seq 1214622068, ack 2689263165, win 14480, options [mss 1460,sackOK,TS val 127905521 ecr 2912170,nop,wscale 7], length 0
> 12:18:22.718447 IP my.host.com.34242 > 192.168.1.47.ssh: Flags [S], seq 2689263164, win 14600, options [mss 1412,sackOK,TS val 2912420 ecr 0,nop,wscale 7], length 0

This looks like a generic networking problem with your bridges, VPN,
etc.  I don't see anything that points to KVM itself being involved.

my.host.com sends SYN to .47
.47 replies with SYN+ACK to my.host.com
...1 second of silence...
my.host.com tries sending SYN to .47 again

Did you run tcpdump on my.host.com?

If you did not run it on my.host.com then this output suggests that
my.host.com isn't receiving the SYN+ACK reply.  Therefore it doesn't
complete the 3-way handshake with an ACK reply.

Stefan