Loss of network connectivity with high load

Loss of network connectivity with high load

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

Hi all!

I'm using Linux 2.6.31.13 compiled with the kernel.org source code on
KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian
repositories. I'm using qemu-kvm 0.12.3.

These are the parameters I'm using to start the virtual machine:


escher:~# ps ax|grep kvm
 6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
-drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
-k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
telnet:localhost:4042,server,nowait


When I try to copy directories of several gigabytes in this VM using
rsync to other hosts, this virtual machine loses connectivity with the
rest of the network. Checking, using a serial console connection,
/var/log/syslog and /var/log/messages in search of the problem, I don't
see information that might give a clue to the problem.

This problem is not with the network driver by default (rtl8139), so I
think that should be something related to Virtio.

Thanks in advance for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Loss of network connectivity with high load

[Brian Jackson]

On Tuesday 15 June 2010 16:00:38 Daniel Bareiro wrote:
> Hi all!
> 
> I'm using Linux 2.6.31.13 compiled with the kernel.org source code on
> KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
> GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian
> repositories. I'm using qemu-kvm 0.12.3.
> 
> These are the parameters I'm using to start the virtual machine:
> 
> 
> escher:~# ps ax|grep kvm
>  6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
> -drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
> file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
> file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
> nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
> -k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
> telnet:localhost:4042,server,nowait
> 
> 
> When I try to copy directories of several gigabytes in this VM using
> rsync to other hosts, this virtual machine loses connectivity with the
> rest of the network. Checking, using a serial console connection,
> /var/log/syslog and /var/log/messages in search of the problem, I don't
> see information that might give a clue to the problem.
> 
> This problem is not with the network driver by default (rtl8139), so I
> think that should be something related to Virtio.


There have been a few bugs similar to this reported that should be pretty easy 
to find. Basically, I'd say try 0.12.4, it should have a few fixes in this 
area over 0.12.3.


> 
> Thanks in advance for your reply.
> 
> Regards,
> Daniel

Re: Loss of network connectivity with high load

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]

Hi, Brian.

On Tuesday, 15 June 2010 16:13:03 -0500,
Brian Jackson wrote:

> > I'm using Linux 2.6.31.13 compiled with the kernel.org source code
> > on KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
> > GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from
> > Debian repositories. I'm using qemu-kvm 0.12.3.
> > 
> > These are the parameters I'm using to start the virtual machine:
> > 
> > 
> > escher:~# ps ax|grep kvm
> >  6299 ?        Rl   113:40 /usr/local/qemu-kvm/bin/qemu-system-x86_64
> > -drive file=/dev/vm/belvedere-raiz,cache=none,if=virtio,boot=on -drive
> > file=/dev/vm/belvedere-u01,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u02,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u03,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u04,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u05,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u06,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u07,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u08,cache=none,if=virtio -drive
> > file=/dev/vm/belvedere-u09,cache=none,if=virtio -m 3072 -smp 2 -net
> > nic,model=virtio,macaddr=00:16:3e:00:00:56 -net tap -daemonize -vnc :2
> > -k es -localtime -monitor telnet:localhost:4002,server,nowait -serial
> > telnet:localhost:4042,server,nowait
> > 
> > 
> > When I try to copy directories of several gigabytes in this VM using
> > rsync to other hosts, this virtual machine loses connectivity with
> > the rest of the network. Checking, using a serial console
> > connection, /var/log/syslog and /var/log/messages in search of the
> > problem, I don't see information that might give a clue to the
> > problem.
> > 
> > This problem is not with the network driver by default (rtl8139), so
> > I think that should be something related to Virtio.

> There have been a few bugs similar to this reported that should be
> pretty easy to find. Basically, I'd say try 0.12.4, it should have a
> few fixes in this area over 0.12.3.

I was testing with qemu-kvm 0.12.4 but I'm having the same problem. I
re-read the announce about qemu-kvm 0.12.4 sent by Avi, but I didn't
find in it some reference about fixes related to Virtio.

If there's anything else I can try, please tell me.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Loss of network connectivity with high load

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Wednesday, 16 June 2010 00:36:43 -0300,
Daniel Bareiro wrote:

> > There have been a few bugs similar to this reported that should be
> > pretty easy to find. Basically, I'd say try 0.12.4, it should have a
> > few fixes in this area over 0.12.3.

> I was testing with qemu-kvm 0.12.4 but I'm having the same problem. I
> re-read the announce about qemu-kvm 0.12.4 sent by Avi, but I didn't
> find in it some reference about fixes related to Virtio.
> 
> If there's anything else I can try, please tell me.

I found some references to similar problems:

http://forum.proxmox.com/threads/3117-virtio-net-crashing-(stop-sending-traffic)
http://www.mail-archive.com/kvm@vger.kernel.org/msg26033.html
http://marc.info/?l=kvm&m=126564542625725&w=2


In particular, I note the patches provided by Tom Lendacky and Herbert
Xu. Are these patches applied to qemu-kvm 0.12.3 or 0.12.4? If so, it
could be a regression?


Thanks in advance for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Loss of network connectivity with high load

[Avi Kivity]

On 06/16/2010 12:00 AM, Daniel Bareiro wrote:
> Hi all!
>
> I'm using Linux 2.6.31.13 compiled with the kernel.org source code on
> KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
> GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from Debian
> repositories. I'm using qemu-kvm 0.12.3.
>
>    

2.6.26 is really old.  Can you try a new kernel in the guest?  Recent 
kernels had many virtio fixes.

-- 
error compiling committee.c: too many arguments to function

Re: Loss of network connectivity with high load

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

Hi, Avi.

On Wednesday, 16 June 2010 17:14:54 +0300,
Avi Kivity wrote:

>> I'm using Linux 2.6.31.13 compiled with the kernel.org source code on
>> KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
>> GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from
>> Debian repositories. I'm using qemu-kvm 0.12.3.

> 2.6.26 is really old.  Can you try a new kernel in the guest?  Recent
> kernels had many virtio fixes.

I tested with Linux 2.6.32-bpo.5-amd64 from backports repositories and
so far I didn't have a problem. If the problem persists, I will comment
to the list.

Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Loss of network connectivity with high load

[Thomas Besser]

Hi all,

Daniel Bareiro wrote:
> On Wednesday, 16 June 2010 17:14:54 +0300,
> Avi Kivity wrote:
>>> I'm using Linux 2.6.31.13 compiled with the kernel.org source code on
>>> KVM host with Debian GNU/Linux Lenny amd64. Also I'm using Debian
>>> GNU/Linux Lenny amd64 virtual machine with kernel 2.6.26-2 from
>>> Debian repositories. I'm using qemu-kvm 0.12.3.
> 
>> 2.6.26 is really old.  Can you try a new kernel in the guest?  Recent
>> kernels had many virtio fixes.
> 
> I tested with Linux 2.6.32-bpo.5-amd64 from backports repositories and
> so far I didn't have a problem. If the problem persists, I will comment
> to the list.

I think we have the same problem here: after about 3 GB of transfered data 
with rsync, network (virtio) in the guest stops working (restart of network 
solves the network stop). 

Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4. 
Will try 2.6.34 in guest and report, if the problem still exists. 

Guest starting options:

/usr/bin/kvm -name sysimager -vnc :35 -usbdevice tablet -net 
nic,vlan=0,model=virtio,macaddr=00:08:15:00:09:95 -net 
tap,vlan=0,ifname=tap35,script=no -m 512 -monitor 
unix:/var/run/kvm/sysimager,server,nowait -drive 
index=0,media=disk,file=/vm/sysimager/sysimager.raw,if=virtio,boot=on -drive 
index=1,media=disk,file=/vm/sysimager/daten.raw,if=virtio -daemonize -vga 
std -k de -pidfile /var/run/kvm/sysimager.pid -tdf

No error messages in /var/log/syslog, but via vnc on console. Any hint how 
to copy and paste via vncviewer the message?

Regards
Thomas

Re: Loss of network connectivity with high load

[Thomas Besser]

Thomas Besser wrote:
>> On Wednesday, 16 June 2010 17:14:54 +0300,
>> Avi Kivity wrote:
>>> 2.6.26 is really old.  Can you try a new kernel in the guest?  Recent
>>> kernels had many virtio fixes.
> I think we have the same problem here: after about 3 GB of transfered data
> with rsync, network (virtio) in the guest stops working (restart of
> network solves the network stop).
> 
> Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4.
> Will try 2.6.34 in guest and report, if the problem still exists.

With a 2.6.34 guest kernel the problem seems to be gone (after about 6 GB 
transferred data).

Regards
Thomas

Re: Loss of network connectivity with high load

[RW]

I've had the same issues too. I'm using Gentoo.
Kernel packages < 2.6.32-r7 (Release 7 not RC-7)
have had this problem. Haven't tested -r8 to -r10.
Kernel 2.6.32-r11 is currently working fine. 2.6.34
was working too but it crashed after two days with a
clock syncing issue. I'm hopping that 2.6.32-r11 doesn't
have this issue. It's currently running for 24 hours
now. KVM is 0.12.4.

- Robert


On 06/22/10 14:32, Thomas Besser wrote:
> Thomas Besser wrote:
>>> On Wednesday, 16 June 2010 17:14:54 +0300,
>>> Avi Kivity wrote:
>>>> 2.6.26 is really old.  Can you try a new kernel in the guest?  Recent
>>>> kernels had many virtio fixes.
>> I think we have the same problem here: after about 3 GB of transfered data
>> with rsync, network (virtio) in the guest stops working (restart of
>> network solves the network stop).
>>
>> Both host and guest Debian Lenny with 2.6.32. qemu-kvm is version 0.12.4.
>> Will try 2.6.34 in guest and report, if the problem still exists.
> 
> With a 2.6.34 guest kernel the problem seems to be gone (after about 6 GB 
> transferred data).
> 
> Regards
> Thomas
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html