Network poblems using virtio and tap.

Network poblems using virtio and tap.

[Fabian Deutsch]

Hey.

I'm observing some network problems when using virtio and a tap if.
The trouble is a drop of network troughput from abot 90Mbit/s to about 90kbit/s,
after about 1.7 TB of transferred data.

Setup
Host: 
- Fedora 8, nearly all updates
- Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
- 8GB RAM
- 2 Guests

Guest:
- Fedora 9 all updates
- virtio nic using tap
- samba sharing a mountpoint using an iscsi target

Steps to reproduce:
1. Setup a guets using virtio and tap
2. Mount an iscsi target and wokr on the fs or  create some traffic on the nic.
2.a I used:
# while true ; do sync ; \
    time dd if=/dev/zero of=test.bin bs=1M count=1024 ; sync ; 
    time md5sum test.bin ; done
3. Wait a couple of days or about 1.7 TB of data on a 100Mbit/s.

Expected results:
Everything runs fine, even after 2.5 TB of transferred data or more.

Actual results:
Everything runs fine up to some point. 
Then: Troughput drops a bit (from 11 MB/s to about 6MB/s). 
At some point later the troughput drops to about 90kBit/s.
No messages on guest and host. No errors on ifconfig, no dmesg errors.
Ping latency increases over time. Logged here: 
http://www.meinsandkasten.de/log-virtio.pdf


Did someone observ something similar or so?

Greetings
fabian
-- 
Fabian Deutsch <fabian.deutsch@gmx.de>

Re: Network poblems using virtio and tap.

[Fabian Deutsch]

Hello.

It seems to be a problem of virtio. Using e1000 again, I don't get into
trouble anymore.

- fabian

Am Montag, den 01.09.2008, 14:42 +0200 schrieb Fabian Deutsch:
> Hey.
> 
> I'm observing some network problems when using virtio and a tap if.
> The trouble is a drop of network troughput from abot 90Mbit/s to about 90kbit/s,
> after about 1.7 TB of transferred data.
> 
> Setup
> Host: 
> - Fedora 8, nearly all updates
> - Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
> - 8GB RAM
> - 2 Guests
> 
> Guest:
> - Fedora 9 all updates
> - virtio nic using tap
> - samba sharing a mountpoint using an iscsi target
> 
> Steps to reproduce:
> 1. Setup a guets using virtio and tap
> 2. Mount an iscsi target and wokr on the fs or  create some traffic on the nic.
> 2.a I used:
> # while true ; do sync ; \
>     time dd if=/dev/zero of=test.bin bs=1M count=1024 ; sync ; 
>     time md5sum test.bin ; done
> 3. Wait a couple of days or about 1.7 TB of data on a 100Mbit/s.
> 
> Expected results:
> Everything runs fine, even after 2.5 TB of transferred data or more.
> 
> Actual results:
> Everything runs fine up to some point. 
> Then: Troughput drops a bit (from 11 MB/s to about 6MB/s). 
> At some point later the troughput drops to about 90kBit/s.
> No messages on guest and host. No errors on ifconfig, no dmesg errors.
> Ping latency increases over time. Logged here: 
> http://www.meinsandkasten.de/log-virtio.pdf
> 
> 
> Did someone observ something similar or so?
> 
> Greetings
> fabian

Re: Network poblems using virtio and tap.

[Christian Borntraeger]

Am Montag, 1. September 2008 schrieb Fabian Deutsch:
> Hey.
> 
> I'm observing some network problems when using virtio and a tap if.
> The trouble is a drop of network troughput from abot 90Mbit/s to about 
90kbit/s,
> after about 1.7 TB of transferred data.
> 
> Setup
> Host: 
> - Fedora 8, nearly all updates
> - Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
> - 8GB RAM
> - 2 Guests
> 
> Guest:
> - Fedora 9 all updates
> - virtio nic using tap
> - samba sharing a mountpoint using an iscsi target

Can you tell us what host kernel and what guest kernel you are using? The were 
some changes to tun/tap for virtio lately which could make a difference.

Christian

Re: Network poblems using virtio and tap.

[Fabian Deutsch]

Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
> Am Montag, 1. September 2008 schrieb Fabian Deutsch:
> > Hey.
> > 
> > I'm observing some network problems when using virtio and a tap if.
> > The trouble is a drop of network troughput from abot 90Mbit/s to
about 
> 90kbit/s,
> > after about 1.7 TB of transferred data.
> > 
> > Setup
> > Host: 
> > - Fedora 8, nearly all updates
> > - Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
> > - 8GB RAM
> > - 2 Guests
> > 
> > Guest:
> > - Fedora 9 all updates
> > - virtio nic using tap
> > - samba sharing a mountpoint using an iscsi target
> 
> Can you tell us what host kernel and what guest kernel you are using?
The were 
> some changes to tun/tap for virtio lately which could make a
difference.

Host:
Fedora 8 x86_64
2.6.25.14-69.fc8 SMP
kvm-72

Guest:
Fedora 9 i386
2.6.25.11-97.fc9.i686


Greetings
- fabian

Re: Network poblems using virtio and tap.

[richard lucassen]

On Sat, 06 Sep 2008 14:04:30 +0200
Fabian Deutsch <fabian.deutsch@gmx.de> wrote:

> Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
> > Am Montag, 1. September 2008 schrieb Fabian Deutsch:
> > > Hey.
> > > 
> > > I'm observing some network problems when using virtio and a tap
> > > if. The trouble is a drop of network troughput from abot 90Mbit/s
> > > to
> about 
> > 90kbit/s,
> > > after about 1.7 TB of transferred data.
> > > 
> > > Setup
> > > Host: 
> > > - Fedora 8, nearly all updates
> > > - Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
> > > - 8GB RAM
> > > - 2 Guests
> > > 
> > > Guest:
> > > - Fedora 9 all updates
> > > - virtio nic using tap
> > > - samba sharing a mountpoint using an iscsi target
> > 
> > Can you tell us what host kernel and what guest kernel you are
> > using?
> The were 
> > some changes to tun/tap for virtio lately which could make a
> difference.
> 
> Host:
> Fedora 8 x86_64
> 2.6.25.14-69.fc8 SMP
> kvm-72
> 
> Guest:
> Fedora 9 i386
> 2.6.25.11-97.fc9.i686

I noticed the same behaviour using a rtl8139 and a tap device. After
some MB's (!) I had to invoke a in the guest:

ifdown eth0; ifup eth0

to get the network back again. I replaced the rtl-8139 by a virtio
device and the problem seems to have gone, but I haven't tried yet with
a large amount of data.

kvm-73 (kvm74 gives compile errors)
host: 64bit ubuntu-8.04-LTS with vanilla kernel 2.6.26.3 SMP
guest: ubuntu-8.04-LTS with ubuntu kernel 32-bit

On the same host runs a 32bit w2k3 (virtio NIC) and I haven't noticed
such problems, even when this host ran with an rtl-8139 device.

Sorry if I missed some postings concerning the issue, I just
(re)subscribed to the list.

R.

-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+

Re: Network poblems using virtio and tap.

[Fabian Deutsch]

Am Samstag, den 06.09.2008, 14:26 +0200 schrieb richard lucassen:
> On Sat, 06 Sep 2008 14:04:30 +0200
> Fabian Deutsch <fabian.deutsch@gmx.de> wrote:
> 
> > Am Samstag, den 06.09.2008, 09:40 +0200 schrieb Christian Borntraeger:
> > > Am Montag, 1. September 2008 schrieb Fabian Deutsch:
> > > > Hey.
> > > > 
> > > > I'm observing some network problems when using virtio and a tap
> > > > if. The trouble is a drop of network troughput from abot 90Mbit/s
> > > > to
> > about 
> > > 90kbit/s,
> > > > after about 1.7 TB of transferred data.
> > > > 
> > > > Setup
> > > > Host: 
> > > > - Fedora 8, nearly all updates
> > > > - Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
> > > > - 8GB RAM
> > > > - 2 Guests
> > > > 
> > > > Guest:
> > > > - Fedora 9 all updates
> > > > - virtio nic using tap
> > > > - samba sharing a mountpoint using an iscsi target
> > > 
> > > Can you tell us what host kernel and what guest kernel you are
> > > using?
> > The were 
> > > some changes to tun/tap for virtio lately which could make a
> > difference.
> > 
> > Host:
> > Fedora 8 x86_64
> > 2.6.25.14-69.fc8 SMP
> > kvm-72
> > 
> > Guest:
> > Fedora 9 i386
> > 2.6.25.11-97.fc9.i686
> 
> I noticed the same behaviour using a rtl8139 and a tap device. After
> some MB's (!) I had to invoke a in the guest:
> 
> ifdown eth0; ifup eth0

ifdown ; ifup doesn't help in this case.
Even a guest-restart "init 6" doesn't help.
Just shutting it completely down "init 0" and powering it on again
helps.

> 
> to get the network back again. I replaced the rtl-8139 by a virtio
> device and the problem seems to have gone, but I haven't tried yet with
> a large amount of data.

Virtio net works fast and withouth trouble - up to some point, then
packet loss occurs.

> 
> kvm-73 (kvm74 gives compile errors)
> host: 64bit ubuntu-8.04-LTS with vanilla kernel 2.6.26.3 SMP
> guest: ubuntu-8.04-LTS with ubuntu kernel 32-bit
> 
> On the same host runs a 32bit w2k3 (virtio NIC) and I haven't noticed
> such problems, even when this host ran with an rtl-8139 device.
> 
> Sorry if I missed some postings concerning the issue, I just
> (re)subscribed to the list.
> 
> R.
> 

Re: Network poblems using virtio and tap.

[richard lucassen]

On Sat, 06 Sep 2008 15:47:23 +0200
Fabian Deutsch <fabian.deutsch@gmx.de> wrote:

> > I noticed the same behaviour using a rtl8139 and a tap device. After
> > some MB's (!) I had to invoke a in the guest:
> > 
> > ifdown eth0; ifup eth0
> 
> ifdown ; ifup doesn't help in this case.
> Even a guest-restart "init 6" doesn't help.
> Just shutting it completely down "init 0" and powering it on again
> helps.

No, for me no need for shutdown.

> > to get the network back again. I replaced the rtl-8139 by a virtio
> > device and the problem seems to have gone, but I haven't tried yet
> > with a large amount of data.
> 
> Virtio net works fast and withouth trouble - up to some point, then
> packet loss occurs.

I'll try to transfer /dev/urandom through ssh to another host (vdisk
too small)

But not now, I'm too busy at the moment :-(

-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+

Re: Network poblems using virtio and tap.

[richard lucassen]

On Sat, 06 Sep 2008 15:47:23 +0200
Fabian Deutsch <fabian.deutsch@gmx.de> wrote:

> Virtio net works fast and withouth trouble - up to some point, then
> packet loss occurs.

FYI, after some time I got this:

Received disconnect from 10.1.255.2: 2: Bad packet length 1367736372.

I was still able to relogon using ssh.

-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+