UDP problem with virtio...

UDP problem with virtio...

[Erik Brakkee]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Hi,


I am running a guest that exposes a NFS share to be used by a TViX box. The TViX box is not that advanced and uses NFS over UDP instead of TCP.
What I am seeing is that when I use the virtio network driver, the TViX box cannot mount the filesystem. However, when I use rtl8139 instead, it can.

The problem is easily reproduced:
1. Create a guest that uses a virtio network interface (bridging setup)
2. Expose an NFS share on the guest
3. mount the NFS share using 'mount -o udp host:/share /localdir'
4. verify that the NFS share cannot be mounted.
5. Modify the guest to use rtl8139 device emulation and stop the guest and start it
6. mount the NFS share using 'mount -o udp host:/share /localdir'
7. Verify tha the NFS share can be mounted

Now the bug reporting guidelines say that I should always use the latest KVM version but compiling KVM from source and using that is not really an option. The server is important enough not to mess around with. Therefore, my question is whether this problem has been seen before or not.

My configuration is as follows:

*Host*

centos 6.2
uname -a gives "Linux falcon.fritz.box 2.6.32-220.17.1.el6.x86_64 #1 SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux"

KVM software: qemu-kvm-0.12.1.2-2.209.el6_2.5.x86_64

*Guest*

OpenSuSE 11.3
uname -a gives "Linux shikra 2.6.34.7-0.7-default #1 SMP 2010-12-13 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux"

I also see in the instructions I should try running qemu with some alternative switches, but I am not running qemu directly. I am using virt-manager and a domain.xml to configure how qemu runs. How should I modify the domain.xml to help in troubleshooting this issue?

Cheers
   Erik

PS. the domain.xml is attached.






[-- Attachment #2: shikra.xml --]
[-- Type: text/xml, Size: 3675 bytes --]

<domain type='kvm' id='5'>
  <name>shikra</name>
  <uuid>3a524d9f-a9df-4280-d0d7-e7562a860147</uuid>
  <memory>6291456</memory>
  <currentMemory>6291456</currentMemory>
  <vcpu>8</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/bootdisks/shikra'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/shikraroot/disk'/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/shikradata/disk'/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/shikravcr/disk'/>
      <target dev='vdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:01:01:01'/>
      <source bridge='br0'/>
      <target dev='vnet4'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:01:01:02'/>
      <source network='default'/>
      <target dev='vnet5'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5902' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x08' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x09' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Re: UDP problem with virtio...

[Erik Brakkee]

Hi all,


I have done some more experiments and it does work when using a Centos 
6.2 guest. Therefore, it is most likely a compatibility issues between 
the guest and host virtio implementation.
I read somewhere there is some sort of compatibility mode for virtio but 
could not find how to activate it. Is there a way to make these 
different virtio host/guest implementations talk to each other?

Cheers
   Erik


Erik Brakkee wrote:
> Hi,
>
>
> I am running a guest that exposes a NFS share to be used by a TViX 
> box. The TViX box is not that advanced and uses NFS over UDP instead 
> of TCP.
> What I am seeing is that when I use the virtio network driver, the 
> TViX box cannot mount the filesystem. However, when I use rtl8139 
> instead, it can.
>
> The problem is easily reproduced:
> 1. Create a guest that uses a virtio network interface (bridging setup)
> 2. Expose an NFS share on the guest
> 3. mount the NFS share using 'mount -o udp host:/share /localdir'
> 4. verify that the NFS share cannot be mounted.
> 5. Modify the guest to use rtl8139 device emulation and stop the guest 
> and start it
> 6. mount the NFS share using 'mount -o udp host:/share /localdir'
> 7. Verify tha the NFS share can be mounted
>
> Now the bug reporting guidelines say that I should always use the 
> latest KVM version but compiling KVM from source and using that is not 
> really an option. The server is important enough not to mess around 
> with. Therefore, my question is whether this problem has been seen 
> before or not.
>
> My configuration is as follows:
>
> *Host*
>
> centos 6.2
> uname -a gives "Linux falcon.fritz.box 2.6.32-220.17.1.el6.x86_64 #1 
> SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux"
>
> KVM software: qemu-kvm-0.12.1.2-2.209.el6_2.5.x86_64
>
> *Guest*
>
> OpenSuSE 11.3
> uname -a gives "Linux shikra 2.6.34.7-0.7-default #1 SMP 2010-12-13 
> 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux"
>
> I also see in the instructions I should try running qemu with some 
> alternative switches, but I am not running qemu directly. I am using 
> virt-manager and a domain.xml to configure how qemu runs. How should I 
> modify the domain.xml to help in troubleshooting this issue?
>
> Cheers
>   Erik
>
> PS. the domain.xml is attached.
>
>
>
>
>


-- 

Nonsense and other useful things: http://brakkee.org
MountainHoppers: http://mountainhoppers.nl
Track Detective: http://trackdetective.com
Twitter: http://twitter.com/ErikBrakkee

Re: UDP problem with virtio...

[Stefan Hajnoczi]

On Sat, May 26, 2012 at 11:23 AM, Erik Brakkee <erik@brakkee.org> wrote:
> I have done some more experiments and it does work when using a Centos 6.2
> guest. Therefore, it is most likely a compatibility issues between the guest
> and host virtio implementation.
> I read somewhere there is some sort of compatibility mode for virtio but
> could not find how to activate it. Is there a way to make these different
> virtio host/guest implementations talk to each other?

They should be compatible.

I noticed in your domain XML the VM has two virtio-net interfaces.
Could that be affecting your mount success?

Did you collect a packet capture inside the guest?  (tcpdump -nli eth0
-s0 -w /tmp/dump.pcap)

It would be interesting to understand what the guest sees during the
failed and successful UDP NFS mounts.

Can you be more specific about the mount failure.  Is there an error
message?  Anything interesting in the guest's dmesg?

Did you collect a packet capture on the host?

Stefan

Re: UDP problem with virtio...

[Michael S. Tsirkin]

On Fri, May 25, 2012 at 10:55:37PM +0200, Erik Brakkee wrote:
> I also see in the instructions I should try running qemu with some
> alternative switches, but I am not running qemu directly. I am using
> virt-manager and a domain.xml to configure how qemu runs. How should I
> modify the domain.xml to help in troubleshooting this issue?

See
http://libvirt.org/drvqemu.html#qemucommand

Re: UDP problem with virtio...

[Erik Brakkee]

Stefan Hajnoczi wrote:
> On Sat, May 26, 2012 at 11:23 AM, Erik Brakkee<erik@brakkee.org>  wrote:
>> I have done some more experiments and it does work when using a Centos 6.2
>> guest. Therefore, it is most likely a compatibility issues between the guest
>> and host virtio implementation.
>> I read somewhere there is some sort of compatibility mode for virtio but
>> could not find how to activate it. Is there a way to make these different
>> virtio host/guest implementations talk to each other?
> They should be compatible.
>
> I noticed in your domain XML the VM has two virtio-net interfaces.
> Could that be affecting your mount success?
>
> Did you collect a packet capture inside the guest?  (tcpdump -nli eth0
> -s0 -w /tmp/dump.pcap)
>
> It would be interesting to understand what the guest sees during the
> failed and successful UDP NFS mounts.
>
> Can you be more specific about the mount failure.  Is there an error
> message?  Anything interesting in the guest's dmesg?
>
> Did you collect a packet capture on the host?
>
> Stefan

Hi all,

I sent a reply earlier but apparently not to this mailing list.
The problem went away after installing the latest software updates on 
the openSuSE 11.3 guest. I tried reproducing it from a backup of the VM 
running the older kernel but was unable to do this on a openSuSE 12.1 
host. To reproduce it I would have had to use the centos 6.2 host but 
that would be quite a lot more work to do. So I am leaving the issue now 
as resolved. It was probably some earlier bug that was fixed.

Cheers
   Erik


-- 

Nonsense and other useful things: http://brakkee.org
MountainHoppers: http://mountainhoppers.nl
Track Detective: http://trackdetective.com
Twitter: http://twitter.com/ErikBrakkee