nfs, tap & vlan issues

nfs, tap & vlan issues

[Fabio Coatti]

Hi all,
I'm facing a nasty issue trying to access NFS volume from a kvm machine.
scenario:
host machine is a dual quadcore amd with kvm-75 release (tried even kvm-73, no 
difference).
Linux 2.6.26.5 vanilla on top of a gentoo distro. (host: x86_64, guest: i386)
on host machine we have two nics, eth0/1.
eth0 is used by host machine and not for guest traffic, while eth1 is enslaved 
in a bridge and then connected to tap0 device, for virtual machine use.

eth1 is connected to network with several vlans (cisco trunk) and the 
configuration (host side) is more or less the following:

4: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
qlen 1000
    link/ether 00:1f:29:69:2a:c4 brd ff:ff:ff:ff:ff:ff
5: kvmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 00:1f:29:69:2a:c4 brd ff:ff:ff:ff:ff:ff
18: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 500
    link/ether 00:ff:e6:89:f6:a7 brd ff:ff:ff:ff:ff:ff

the command line used on kvm is the following:


kvm -m 1G -drive file=/dev/kvm-pool/disk0,if=virtio,boot=on -localtime -net 
nic,macaddr=DE:AD:BE:EF:15:5,model=virtio -net tap,ifname=tap0 

(we tried also using raw and qcow2 images)

The network on guest machine is set up like this:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
3: vlan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.5/24 brd 192.168.61.255 scope global vlan3
4: vlan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.33/24 brd 10.0.0.255 scope global vlan4


The vlan3 is used to reach an nfs server, while vlan4 for all other services. 
network seems to work just fine, as expected: ssh, http, etc..
The problem shows up using nfs; basically we can mount the remote directory 
and issue a df -k (jus an example) but at the first slightly more complex 
operation (a even a simple ls, that produces a bit more of output) the nfs 
locks up and there is no way to get out. wild guess: small packets are ok, 
but when data size becomes similar to mtu size, problems arises.
mount options:

rw,intr,nfsvers=3,rsize=8192,wsize=8192,nolock,timeo=4,retrans=9,bg,tcp,addr=192.168.0.9

in fact, tcpdumping the traffic on guest machine, something interesting 
appears:
16:23:41.287515 IP 192.168.0.5.570425344 > 192.168.0.9.2049: 0 proc-774977080
16:23:41.303954 IP truncated-ip - 4 bytes missing! 192.168.0.9.2049 > 
192.168.0.5.1796115507: reply ok 1448 readdirplus [|nfs]
16:23:42.086867 IP 192.168.0.5.1796115507 > 192.168.0.9.2049: 180 
proc-774977080
16:23:42.103119 IP truncated-ip - 4 bytes missing! 192.168.0.9.2049 > 
192.168.0.5.1796115507: reply ok 14

This "4 bytes truncated" makes me think that the issue is mtu related. I don't 
know if this can be a bridge culprit or a tap one, but I've tried to mount 
the same nfs server on host machine, using the same bridge used by tap0, with 
a vlan device, and no problems shows up. So if something fails in handling 
the packet, it doesn't seems to be the bridge, but tap or kvm itself. I've 
tried to modify MTU on all the devices involved (pyhs, bridge, vlans, tap on 
guest and host) but without success.
I'm unable to discover if this is a kvm strictly related issue, but I'm able 
to reproduce it only usign kvm and not in other ways.
tried with 2.6.26.3 and .5 kernels (both guest and host), kvm 73 and 75; net 
driver: virtio. I've tried to search for similar problems (and solutions), 
but archives and google didn't helped me.
Of course I'm available for any other information and willing to try any 
suggestion that will arrive.

many thanks for any help.

-- 
Fabio "Cova" Coatti    http://members.ferrara.linux.it/cova     
Ferrara Linux Users Group           http://ferrara.linux.it
GnuPG fp:9765 A5B6 6843 17BC A646  BE8C FA56 373A 5374 C703
Old SysOps never die... they simply forget their password.

Re: nfs, tap & vlan issues

[Javier Guerra]

On Thu, Sep 18, 2008 at 11:47 AM, Fabio Coatti <cova@ferrara.linux.it> wrote:
> The network on guest machine is set up like this:
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
>    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
> 3: vlan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
>    inet 192.168.0.5/24 brd 192.168.61.255 scope global vlan3
> 4: vlan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
>    inet 10.0.0.33/24 brd 10.0.0.255 scope global vlan4


there's your problem:

your vlan interfaces (vlan3@eth0, vlan4@eth0) have an MTU of 1500. to
encapsulate that in eth0, it has to add 4 bytes of tagging, therefore
eth0 should have an MTU of 1504.  also, the bridge and eth1 on Dom0
must have MTUs of 1504.

i don't know if the bridge can support 1504, if not, you would have to
set eth0 at 1500, and the vlan interfaces at 1496


-- 
Javier

Re: nfs, tap & vlan issues

[Fabio Coatti]

giovedì 18 settembre 2008, Javier Guerra ha scritto:
> On Thu, Sep 18, 2008 at 11:47 AM, Fabio Coatti <cova@ferrara.linux.it> 
wrote:
> > The network on guest machine is set up like this:
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
> >    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
> > 1000 link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
> > 3: vlan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> >    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
> >    inet 192.168.0.5/24 brd 192.168.61.255 scope global vlan3
> > 4: vlan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> >    link/ether de:ad:be:ef:15:05 brd ff:ff:ff:ff:ff:ff
> >    inet 10.0.0.33/24 brd 10.0.0.255 scope global vlan4
>
> there's your problem:
>
> your vlan interfaces (vlan3@eth0, vlan4@eth0) have an MTU of 1500. to
> encapsulate that in eth0, it has to add 4 bytes of tagging, therefore
> eth0 should have an MTU of 1504.  also, the bridge and eth1 on Dom0
> must have MTUs of 1504.
>
> i don't know if the bridge can support 1504, if not, you would have to
> set eth0 at 1500, and the vlan interfaces at 1496

I see your point; I've tried right now... and it worked! Thanks for your help, 
really greatly appreciated.
Anyway, a small detail:

At first, I've tried to bring host interfaces (eth1,kvmbr0 and tap0) up to 
1504, no problem. Then I've started guest machine and tried to raise eth0 to 
1504, but it seems that virtio driver refuses mtu bigger than 1500.

So the only difference with your recipe is that I've left all host interfaces 
(physical, tap and bridge) to an MTU of 1500 and set, on guest, eth0 to 1500, 
vlans to 1496.

So it was eth0 on guest to complain and not bridge, but now it's working like 
a charm.
(I was sure to have tried even the combination you suggested, in all my tries, 
but obvously I was wrong)


-- 
Fabio "Cova" Coatti    http://members.ferrara.linux.it/cova     
Ferrara Linux Users Group           http://ferrara.linux.it
GnuPG fp:9765 A5B6 6843 17BC A646  BE8C FA56 373A 5374 C703
Old SysOps never die... they simply forget their password.