Networking Problems with Debian Stretch + KVM + old Linux Guest

Networking Problems with Debian Stretch + KVM + old Linux Guest

[Hermann Himmelbauer]

Hi,
I today upgraded my KVM host from Debian 8 to the latest Debian 9
(Stretch). This worked perfectly, however, 2 old guest systems (SuSE
9.1, kernel 2.6.7 / 2.6.5) have no network access.

All other machines running on this host are Linux Debian machines and
use the "virtio" networking drivere whereas those two old machines use
RTL8139 (or e1000, makes no difference).

On the guest side, the networking interface (eth0 / rtl8139) is up, it
states "Link Up / 100MBit" in the log file, everything looks fine, but I
can't get out, no ping, empty arp table etc.

Basically, I use bridging for the virtual hosts, this looks like this:

br0             8000.0026186273f4       no              eth0
                                                        vnet0
                                                        vnet1

or like so:

port no mac addr                is local?       ageing timer
  1     00:00:24:cc:c7:85       no                 0.42
  1     00:19:66:b3:cb:34       no                 3.97
  1     00:22:b0:cf:04:b2       no                 0.03


What is interesting is that I cannot find the MAC Address of the 2
machines in the above table, which is probably not good.

Forwarding is enabled for all bridges and there is no packet filter /
firewall.

I have no clue how to solve this problem - do you have any idea?

Kernel version (uname -rv):
Linux version 4.9.0-5-amd64 (debian-kernel@lists.debian.org) (gcc
version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.65-3+deb9u2
(2018-01-04)

kvm --version:
QEMU emulator version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3)


Best Regards,
Hermann

-- 
hermann@qwer.tk
PGP/GPG: 299893C7 (on keyservers)

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Liran Alon]

----- hermann@qwer.tk wrote:

> Hi,
> I today upgraded my KVM host from Debian 8 to the latest Debian 9
> (Stretch). This worked perfectly, however, 2 old guest systems (SuSE
> 9.1, kernel 2.6.7 / 2.6.5) have no network access.
> 
> All other machines running on this host are Linux Debian machines and
> use the "virtio" networking drivere whereas those two old machines
> use
> RTL8139 (or e1000, makes no difference).
> 
> On the guest side, the networking interface (eth0 / rtl8139) is up,
> it
> states "Link Up / 100MBit" in the log file, everything looks fine, but
> I
> can't get out, no ping, empty arp table etc.
> 
> Basically, I use bridging for the virtual hosts, this looks like
> this:
> 
> br0             8000.0026186273f4       no              eth0
>                                                         vnet0
>                                                         vnet1
> 
> or like so:
> 
> port no mac addr                is local?       ageing timer
>   1     00:00:24:cc:c7:85       no                 0.42
>   1     00:19:66:b3:cb:34       no                 3.97
>   1     00:22:b0:cf:04:b2       no                 0.03
> 
> 
> What is interesting is that I cannot find the MAC Address of the 2
> machines in the above table, which is probably not good.
> 
> Forwarding is enabled for all bridges and there is no packet filter /
> firewall.
> 
> I have no clue how to solve this problem - do you have any idea?
> 
> Kernel version (uname -rv):
> Linux version 4.9.0-5-amd64 (debian-kernel@lists.debian.org) (gcc
> version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian
> 4.9.65-3+deb9u2
> (2018-01-04)
> 
> kvm --version:
> QEMU emulator version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3)
> 
> 
> Best Regards,
> Hermann
> 
> -- 
> hermann@qwer.tk
> PGP/GPG: 299893C7 (on keyservers)

1. What do you see when sniffing (with tcpdump) the QEMU's tap devices which represent the guest's NICs? Do you see any traffic there?

2. Examining the KVM events trace may also be helpful. It's very easy to extract:
# echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
# cat /sys/kernel/debug/tracing/trace_pipe > /tmp/trace
# echo 0 >/sys/kernel/debug/tracing/events/kvm/enable

Regards,
-Liran

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Binarus]

On 20.01.2018 17:04, Hermann Himmelbauer wrote:
> Hi,
> I today upgraded my KVM host from Debian 8 to the latest Debian 9
> (Stretch). This worked perfectly, however, 2 old guest systems (SuSE
> 9.1, kernel 2.6.7 / 2.6.5) have no network access.
> 
> All other machines running on this host are Linux Debian machines and
> use the "virtio" networking drivere whereas those two old machines use
> RTL8139 (or e1000, makes no difference).
> 
> On the guest side, the networking interface (eth0 / rtl8139) is up, it
> states "Link Up / 100MBit" in the log file, everything looks fine, but I
> can't get out, no ping, empty arp table etc.
> 
> Basically, I use bridging for the virtual hosts, this looks like this:
> 
> br0             8000.0026186273f4       no              eth0
>                                                         vnet0
>                                                         vnet1
> 
> or like so:
> 
> port no mac addr                is local?       ageing timer
>   1     00:00:24:cc:c7:85       no                 0.42
>   1     00:19:66:b3:cb:34       no                 3.97
>   1     00:22:b0:cf:04:b2       no                 0.03
> 
> 
> What is interesting is that I cannot find the MAC Address of the 2
> machines in the above table, which is probably not good.

I had a similar problem some years ago. In my case, the network in the
guests generally worked, but was stuttering such extremely that it
actually could not be used.

After some research, it turned out that I had to assign a MAC address to
the guests' (virtual) NICs explicitly.

You haven't told us how you configure / start your VMs. I personally
don't use libvirt and colleagues, but I am starting the VMs from the
command line. Consequently, I don't know how to assign MAC addresses
using the high-level configuration tools. What I am doing on the command
line (as far as it concerns the network) is something like that (please
imagine the following code to be on one line):

/usr/bin/qemu-system-x86_64
[...]
-device virtio-net-pci,vlan=0,mac=02:01:01:01:02:01
-net
tap,vlan=0,name=dax,ifname=dax0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
[...]

Please note how the MAC address is explicitly assigned. script and
downscript are just two simple wrappers which add the virtual NIC to an
existing bridge which connects them with the host and the other VMs.

The disadvantage of this method is that you have to manage the MAC
addresses yourself (you ultimately must make sure that every VM gets its
own and that no other device in the world (in practice in most cases: in
your network) has the same as one of your virtual NICs). The MAC address
shown above is from the semi-official private MAC address space, so I am
hopefully out of trouble in this respect.

Please let us know if this solves your problem.

Regards,

Binarus

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Hermann Himmelbauer]

Am 21.01.2018 um 10:53 schrieb Binarus:
> On 20.01.2018 17:04, Hermann Himmelbauer wrote:
>> Hi,
>> I today upgraded my KVM host from Debian 8 to the latest Debian 9
>> (Stretch). This worked perfectly, however, 2 old guest systems (SuSE
>> 9.1, kernel 2.6.7 / 2.6.5) have no network access.
>>
>> All other machines running on this host are Linux Debian machines and
>> use the "virtio" networking drivere whereas those two old machines use
>> RTL8139 (or e1000, makes no difference).
>>
>> On the guest side, the networking interface (eth0 / rtl8139) is up, it
>> states "Link Up / 100MBit" in the log file, everything looks fine, but I
>> can't get out, no ping, empty arp table etc.
>>
>> Basically, I use bridging for the virtual hosts, this looks like this:
>>
>> br0             8000.0026186273f4       no              eth0
>>                                                         vnet0
>>                                                         vnet1
>>
>> or like so:
>>
>> port no mac addr                is local?       ageing timer
>>   1     00:00:24:cc:c7:85       no                 0.42
>>   1     00:19:66:b3:cb:34       no                 3.97
>>   1     00:22:b0:cf:04:b2       no                 0.03
>>
>>
>> What is interesting is that I cannot find the MAC Address of the 2
>> machines in the above table, which is probably not good.
> 
> I had a similar problem some years ago. In my case, the network in the
> guests generally worked, but was stuttering such extremely that it
> actually could not be used.
> 
> After some research, it turned out that I had to assign a MAC address to
> the guests' (virtual) NICs explicitly.
> 
> You haven't told us how you configure / start your VMs. I personally
> don't use libvirt and colleagues, but I am starting the VMs from the
> command line. Consequently, I don't know how to assign MAC addresses
> using the high-level configuration tools. What I am doing on the command
> line (as far as it concerns the network) is something like that (please
> imagine the following code to be on one line):
> 
> /usr/bin/qemu-system-x86_64
> [...]
> -device virtio-net-pci,vlan=0,mac=02:01:01:01:02:01
> -net
> tap,vlan=0,name=dax,ifname=dax0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> [...]
> 
> Please note how the MAC address is explicitly assigned. script and
> downscript are just two simple wrappers which add the virtual NIC to an
> existing bridge which connects them with the host and the other VMs.
> 
> The disadvantage of this method is that you have to manage the MAC
> addresses yourself (you ultimately must make sure that every VM gets its
> own and that no other device in the world (in practice in most cases: in
> your network) has the same as one of your virtual NICs). The MAC address
> shown above is from the semi-official private MAC address space, so I am
> hopefully out of trouble in this respect.
> 
> Please let us know if this solves your problem.

Thank you for your quick reply, I did not (yet) solve the problem, but I
unfortunately don't fully understand this "tap" device.

I start my virtual machines via libvirt (virt-manage / virsh). However,
looking at the command line, the virtual machine is started like this:
qemu-system-x86_64 -enable-kvm -name guest=horn,debug-threads=on -S
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-11-horn/master-key.aes
-machine pc-1.1,accel=kvm,usb=off,dump-guest-core=off
-cpu qemu32 -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1
-uuid af3996ca-e192-65f0-4318-bd5054311d42 -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-11-horn/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/dev/gaia_data1/horn,format=raw,if=none,id=drive-ide0-0-0
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-netdev tap,fd=34,id=hostnet0
-device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:9a:db:c6,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0
-vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

So, the MAC-Address is assigend fixed. However, doing a "ip tuntap"
shows the following:

vnet8: tap
vnet6: tap vnet_hdr
vnet4: tap vnet_hdr
vnet2: tap
vnet0: tap vnet_hdr
vnet9: tap vnet_hdr
vnet7: tap vnet_hdr
vnet5: tap vnet_hdr
vnet3: tap vnet_hdr
vnet1: tap vnet_hdr

vnet2 / vnet8 are exactly the machines that have no network access.

So, I wonder if my problem has something to do with VLANs? I read
somewhere that only virtio support VLANs, the rtl8139 does not. I do not
use VLANs in any way, but perhaps there's something misconfigured?

What I do not know is how these "vnetX" - devices are created - you
mentioned two scripts - what do they do? Are the MAC addresses somehow
assigned to the "ventX" - devices, and if yes, how?

Best Regards,
Hermann

-- 
hermann@qwer.tk
PGP/GPG: 299893C7 (on keyservers)

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Hermann Himmelbauer]

Am 20.01.2018 um 21:57 schrieb Liran Alon:
> 
> ----- hermann@qwer.tk wrote:
> 
>> Hi,
>> I today upgraded my KVM host from Debian 8 to the latest Debian 9
>> (Stretch). This worked perfectly, however, 2 old guest systems (SuSE
>> 9.1, kernel 2.6.7 / 2.6.5) have no network access.
>>
>> All other machines running on this host are Linux Debian machines and
>> use the "virtio" networking drivere whereas those two old machines
>> use
>> RTL8139 (or e1000, makes no difference).
>>
>> On the guest side, the networking interface (eth0 / rtl8139) is up,
>> it
>> states "Link Up / 100MBit" in the log file, everything looks fine, but
>> I
>> can't get out, no ping, empty arp table etc.
>>
>> Basically, I use bridging for the virtual hosts, this looks like
>> this:
>>
>> br0             8000.0026186273f4       no              eth0
>>                                                         vnet0
>>                                                         vnet1
>>
>> or like so:
>>
>> port no mac addr                is local?       ageing timer
>>   1     00:00:24:cc:c7:85       no                 0.42
>>   1     00:19:66:b3:cb:34       no                 3.97
>>   1     00:22:b0:cf:04:b2       no                 0.03
>>
>>
>> What is interesting is that I cannot find the MAC Address of the 2
>> machines in the above table, which is probably not good.
>>
>> Forwarding is enabled for all bridges and there is no packet filter /
>> firewall.
>>
>> I have no clue how to solve this problem - do you have any idea?
>>
>> Kernel version (uname -rv):
>> Linux version 4.9.0-5-amd64 (debian-kernel@lists.debian.org) (gcc
>> version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian
>> 4.9.65-3+deb9u2
>> (2018-01-04)
>>
>> kvm --version:
>> QEMU emulator version 2.8.1(Debian 1:2.8+dfsg-6+deb9u3)
>>
>>
>> Best Regards,
>> Hermann
>>
>> -- 
>> hermann@qwer.tk
>> PGP/GPG: 299893C7 (on keyservers)
> 
> 1. What do you see when sniffing (with tcpdump) the QEMU's tap devices which represent the guest's NICs? Do you see any traffic there?
> 
> 2. Examining the KVM events trace may also be helpful. It's very easy to extract:
> # echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
> # cat /sys/kernel/debug/tracing/trace_pipe > /tmp/trace
> # echo 0 >/sys/kernel/debug/tracing/events/kvm/enable

Thanks for your quick reply.
For tcpdump, I see on the guest side ARP-requests. On the host side on
the tap device I do see traffic but from other machines, none of the
specific guest.

For the KVM events trace - any hint about how to interprete the output?
There is massive amounts of data, which look mostly like this:

CPU 0/KVM-18088 [001] .... 161817.943123: kvm_pio: pio_read at 0x40 size
1 count 1 val 0x4
       CPU 0/KVM-18088 [001] d... 161817.943124: kvm_entry: vcpu 0
       CPU 0/KVM-18088 [001] .... 161817.943125: kvm_exit: reason
IO_INSTRUCTION rip 0xc010feb1 info 200040 0
       CPU 0/KVM-2451  [003] .... 161817.943125: kvm_exit: reason
APIC_ACCESS rip 0xffffffff84a4f8d2 info 1380 0

Is there anything I should look for? Moreover, I don't know which
virtual machine is e.g. "KVM-18088"?

Best Regards,
Hermann


-- 
hermann@qwer.tk
PGP/GPG: 299893C7 (on keyservers)

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Hermann Himmelbauer]

Am 21.01.2018 um 10:53 schrieb Binarus:
> On 20.01.2018 17:04, Hermann Himmelbauer wrote:
>> Hi,
>> I today upgraded my KVM host from Debian 8 to the latest Debian 9
>> (Stretch). This worked perfectly, however, 2 old guest systems (SuSE
>> 9.1, kernel 2.6.7 / 2.6.5) have no network access.
>>
>> All other machines running on this host are Linux Debian machines and
>> use the "virtio" networking drivere whereas those two old machines use
>> RTL8139 (or e1000, makes no difference).
>>
>> On the guest side, the networking interface (eth0 / rtl8139) is up, it
>> states "Link Up / 100MBit" in the log file, everything looks fine, but I
>> can't get out, no ping, empty arp table etc.
>>
>> Basically, I use bridging for the virtual hosts, this looks like this:
>>
>> br0             8000.0026186273f4       no              eth0
>>                                                         vnet0
>>                                                         vnet1
>>
>> or like so:
>>
>> port no mac addr                is local?       ageing timer
>>   1     00:00:24:cc:c7:85       no                 0.42
>>   1     00:19:66:b3:cb:34       no                 3.97
>>   1     00:22:b0:cf:04:b2       no                 0.03
>>
>>
>> What is interesting is that I cannot find the MAC Address of the 2
>> machines in the above table, which is probably not good.
> 
> I had a similar problem some years ago. In my case, the network in the
> guests generally worked, but was stuttering such extremely that it
> actually could not be used.
> 
> After some research, it turned out that I had to assign a MAC address to
> the guests' (virtual) NICs explicitly.
> 
> You haven't told us how you configure / start your VMs. I personally
> don't use libvirt and colleagues, but I am starting the VMs from the
> command line. Consequently, I don't know how to assign MAC addresses
> using the high-level configuration tools. What I am doing on the command
> line (as far as it concerns the network) is something like that (please
> imagine the following code to be on one line):
> 
> /usr/bin/qemu-system-x86_64
> [...]
> -device virtio-net-pci,vlan=0,mac=02:01:01:01:02:01
> -net
> tap,vlan=0,name=dax,ifname=dax0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> [...]
> 
> Please note how the MAC address is explicitly assigned. script and
> downscript are just two simple wrappers which add the virtual NIC to an
> existing bridge which connects them with the host and the other VMs.
> 
> The disadvantage of this method is that you have to manage the MAC
> addresses yourself (you ultimately must make sure that every VM gets its
> own and that no other device in the world (in practice in most cases: in
> your network) has the same as one of your virtual NICs). The MAC address
> shown above is from the semi-official private MAC address space, so I am
> hopefully out of trouble in this respect.
> 
> Please let us know if this solves your problem.

O.k., just got it running - I upgraded the machine from kernel 2.6.7 to
2.6.27 and now it works - also with the rtl8139 driver.

So it seems that networking of QEMU/KVM has some problems with old Linux
kernels - I did not investigate this further, however.

Best Regards and thanks for help,
Hermann


-- 
hermann@qwer.tk
PGP/GPG: 299893C7 (on keyservers)

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Binarus]

On 21.01.2018 13:30, Hermann Himmelbauer wrote:

> Thank you for your quick reply, I did not (yet) solve the problem, but I
> unfortunately don't fully understand this "tap" device.

You could imagine a tap device to be a NIC at the host side which is
completely realized in software (as opposed to the many NIC card types
which are passed to the VM as hardware device (which is emulated or
para-virtualized, but that doesn't matter in this context)).

> I start my virtual machines via libvirt (virt-manage / virsh). However,
> looking at the command line, the virtual machine is started like this:
> qemu-system-x86_64 -enable-kvm -name guest=horn,debug-threads=on -S
> -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-11-horn/master-key.aes
> -machine pc-1.1,accel=kvm,usb=off,dump-guest-core=off
> -cpu qemu32 -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1
> -uuid af3996ca-e192-65f0-4318-bd5054311d42 -no-user-config -nodefaults
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-11-horn/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot strict=on
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -drive file=/dev/gaia_data1/horn,format=raw,if=none,id=drive-ide0-0-0
> -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -netdev tap,fd=34,id=hostnet0
> -device
> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:9a:db:c6,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0
> -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4
> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
> 
> So, the MAC-Address is assigend fixed. However, doing a "ip tuntap"
> shows the following:
> 
> vnet8: tap
> vnet6: tap vnet_hdr
> vnet4: tap vnet_hdr
> vnet2: tap
> vnet0: tap vnet_hdr
> vnet9: tap vnet_hdr
> vnet7: tap vnet_hdr
> vnet5: tap vnet_hdr
> vnet3: tap vnet_hdr
> vnet1: tap vnet_hdr
> 
> vnet2 / vnet8 are exactly the machines that have no network access.

This is a whole lot of options, among them many I have never used.

Nevertheless, as you already have seen yourself, the MAC address is
assigned explicitly. So the next thing I would do is checking if both of
the two VMs in question are being assigned the *same* MAC address by
accident (I don't know how libvirt and friends calculate / select MAC
addresses, but I could imagine that such accidents happen from time to
time). If that doesn't solve the problem, you may try the suggestion below.

> So, I wonder if my problem has something to do with VLANs? I read
> somewhere that only virtio support VLANs, the rtl8139 does not. I do not
> use VLANs in any way, but perhaps there's something misconfigured?

First, the rtl8139 device does support the vlan parameter, at least on
my systems:

root@cerberus:~/qemu-kvm# qemu-system-x86_64 -device rtl8139,help
rtl8139.rombar=uint32
rtl8139.x-pcie-lnksta-dllla=bool (on/off)
rtl8139.bootindex=int32
rtl8139.multifunction=bool (on/off)
rtl8139.romfile=str
rtl8139.vlan=int32 (Integer VLAN id to connect to)
rtl8139.command_serr_enable=bool (on/off)
rtl8139.addr=int32 (Slot and optional function number, example: 06.0 or 06)
rtl8139.mac=str (Ethernet 6-byte MAC Address, example: 52:54:00:12:34:56)
rtl8139.netdev=str (ID of a netdev to use as a backend)

Please note the sixth line of the output.

Regarding the VLANs in general:

You usually have one NIC which belongs to the VM side. This NIC is
hardware from the VM's point of view (the fact that it is emulated or
para-virtualized doesn't matter here). Secondly, you have another NIC
(the TAP device) which belongs to the host and which is realized in
software.

Now you need a way to connect those two; otherwise, you can't pass
network traffic between them. One possible method to connect them are
VLANs. By passing the parameter ",vlan=n" to both NICs, you are
connecting them (of course, n is the same for both NICs).

Don't be worried when you see examples where this is not done
explicitly. There are default values for the "vlan=n" parameter for
network devices. Please refer to your qemu man page if you'd like to
know more about that subject.

Another method to connect the host NIC with the VM NIC is the method
your command line shows: The "netdev=" option of the rtl8139 determines
which host network device should be used as the "backend" for the
emulated hardware.

Since the latter method does not seem to work in your case, you could do
the following for testing purposes:

Copy the command line you have shown into a file, make it executable and
make the following changes:

Replace -netdev tap[...] by

-net
tap,vlan=29,ifname=spock0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifup

(you probably already have this script, but maybe at another location,
and see the comments at the end)
(I have chosen a random weird VLAN number to make sure it doesn't crash
with a VLAN from you other VMs)
(Instead of spock0, you can choose another arbitrary name for the interface)

Then replace -device rtl8139[...] by

-device rtl8139,vlan=29,macaddr=<YOUR_MAC_HERE>,bus=pci.0,addr=0x3

Then shut down the VM in question and try to start it from the command
line by executing the file you have just created.

Please note that I can't test this at the moment, so it might refuse to
start and emit error messages. If this is the case, please report.

> What I do not know is how these "vnetX" - devices are created - you
> mentioned two scripts - what do they do? Are the MAC addresses somehow
> assigned to the "ventX" - devices, and if yes, how?

The vnetX devices seem to be the tap devices in your case. The qemu
manual tells us that the O/S chooses a name for the tap device if none
is given explicitly. These devices are created automatically (by qemu)
when starting the respective VM (provided you have defined a tap device
for the VM).

According to qemu's manual, there is no option for a tap device to
assign a MAC, and I never have seen such an example. I haven't read the
source code, and thus I don't know if a tap device uses an own MAC
address internally (I don't think that, though), but in every case this
should not matter here. What does matter is the MAC address you have
assigned to the emulated / para-virtualized NIC which is passed to the VM.

Regarding the scripts: After having connected the VM's NIC and the
host's (tap) NIC, you are not done yet. What you have so far is a street
with two ends (the host and the VM), but no additional crossroads which
would enable you to discover the world.

To do that, you usually connect the tap device to a bridge which is also
connected to the real hardware NIC of your host (if desired), to the
other tap devices from the other VMs (if desired) and so on.

Please note that there are other methods, too (for example, you could
define a virtual network switch, and there are even other methods), but
I have found bridging to be easy to understand, efficient and scalable.
Of course, you can define as many bridges a you want and separate your
VMs that way.

In my case, as I wrote in my previous message, the script just adds the
tap device to a bridge (I have only one since my setup is simple: each
of my VMs should see the full network traffic). It is not worth showing
the contents of that script because it is the default one delivered with
qemu, so you should have it already.

As a final hint, you probably don't need that script:

a) There is an option ",br=bridge" to -net tap. I suppose that this
connects the tap device to the respective bridge immediately when it is
created, and thus I didn't need the script if I would use this option.
But please note that I did not try this yet, so I might be wrong.

b) There is a parameter "-net
bridge[,vlan=n][,name=name][,br=bridge][...]" which seems to meant for
connecting a vlan to a bridge directly; this would make using the script
unnecessary as well. Please note that I didn't try that either, so I
might be wrong again.

In general, to solve your problem, I would recommend you to play around
with the command line. To get the right feeling, you might want to read
qemu's man page; this will lead to a deeper understanding of what
libvirt and friends are doing behind the scenes.

By the way, I never managed to damage a VM by giving the wrong
parameters to qemu. But if you are concerned about this, just make a
backup copy of the image the respective VM boots from. Then, nothing
will stop you from playing around :-)

Regards,

Binarus

Re: Networking Problems with Debian Stretch + KVM + old Linux Guest

[Hermann Himmelbauer]

Dear Binaurus,
Many thanks for your in-depth reply, it helped me to understand things a
lot better.

I partially solved the problem by upgrading the Linux kernels of the
guest machines from 2.6.5/2.6.7 to 2.6.23. (2.6.22 did not work). After
that upgrade, the machines had network access again.

(Duplicate MAC Addresses seem not to be the case as everything is
assigned properly)

However, when transferring huge amounts of data the network was gone
again. Rebooting the machine did not help, but stopping / starting (and
thus recreating the tap-interface did help.

I now changed the driver in the guest kernel from RTL8193 to Intel E1000
(module 8239too -> e1000) - and it seems that the network access is stable.

So, to me this really smells like some kind of kernel bug (hardware
incompatibility?) - possibly at the host side, no clue, however. I found
some links to people who had the same problem, unfortunately without
solutions:

https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1694625
https://www.centos.org/forums/viewtopic.php?f=47&t=59614&sid=7b5cd81b80a6806cb4ae322715bb7766&start=10
https://bugs.centos.org/view.php?id=5526

Anyway, thanks for help, I keep my fingers crossed that the problem is gone.

Best Regards,
Hermann



Am 22.01.2018 um 09:54 schrieb Binarus:
> On 21.01.2018 13:30, Hermann Himmelbauer wrote:
> 
>> Thank you for your quick reply, I did not (yet) solve the problem, but I
>> unfortunately don't fully understand this "tap" device.
> 
> You could imagine a tap device to be a NIC at the host side which is
> completely realized in software (as opposed to the many NIC card types
> which are passed to the VM as hardware device (which is emulated or
> para-virtualized, but that doesn't matter in this context)).
> 
>> I start my virtual machines via libvirt (virt-manage / virsh). However,
>> looking at the command line, the virtual machine is started like this:
>> qemu-system-x86_64 -enable-kvm -name guest=horn,debug-threads=on -S
>> -object
>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-11-horn/master-key.aes
>> -machine pc-1.1,accel=kvm,usb=off,dump-guest-core=off
>> -cpu qemu32 -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1
>> -uuid af3996ca-e192-65f0-4318-bd5054311d42 -no-user-config -nodefaults
>> -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-11-horn/monitor.sock,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> -no-shutdown -boot strict=on
>> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> -drive file=/dev/gaia_data1/horn,format=raw,if=none,id=drive-ide0-0-0
>> -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>> -netdev tap,fd=34,id=hostnet0
>> -device
>> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:9a:db:c6,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0
>> -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
>> -device intel-hda,id=sound0,bus=pci.0,addr=0x4
>> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
>>
>> So, the MAC-Address is assigend fixed. However, doing a "ip tuntap"
>> shows the following:
>>
>> vnet8: tap
>> vnet6: tap vnet_hdr
>> vnet4: tap vnet_hdr
>> vnet2: tap
>> vnet0: tap vnet_hdr
>> vnet9: tap vnet_hdr
>> vnet7: tap vnet_hdr
>> vnet5: tap vnet_hdr
>> vnet3: tap vnet_hdr
>> vnet1: tap vnet_hdr
>>
>> vnet2 / vnet8 are exactly the machines that have no network access.
> 
> This is a whole lot of options, among them many I have never used.
> 
> Nevertheless, as you already have seen yourself, the MAC address is
> assigned explicitly. So the next thing I would do is checking if both of
> the two VMs in question are being assigned the *same* MAC address by
> accident (I don't know how libvirt and friends calculate / select MAC
> addresses, but I could imagine that such accidents happen from time to
> time). If that doesn't solve the problem, you may try the suggestion below.
> 
>> So, I wonder if my problem has something to do with VLANs? I read
>> somewhere that only virtio support VLANs, the rtl8139 does not. I do not
>> use VLANs in any way, but perhaps there's something misconfigured?
> 
> First, the rtl8139 device does support the vlan parameter, at least on
> my systems:
> 
> root@cerberus:~/qemu-kvm# qemu-system-x86_64 -device rtl8139,help
> rtl8139.rombar=uint32
> rtl8139.x-pcie-lnksta-dllla=bool (on/off)
> rtl8139.bootindex=int32
> rtl8139.multifunction=bool (on/off)
> rtl8139.romfile=str
> rtl8139.vlan=int32 (Integer VLAN id to connect to)
> rtl8139.command_serr_enable=bool (on/off)
> rtl8139.addr=int32 (Slot and optional function number, example: 06.0 or 06)
> rtl8139.mac=str (Ethernet 6-byte MAC Address, example: 52:54:00:12:34:56)
> rtl8139.netdev=str (ID of a netdev to use as a backend)
> 
> Please note the sixth line of the output.
> 
> Regarding the VLANs in general:
> 
> You usually have one NIC which belongs to the VM side. This NIC is
> hardware from the VM's point of view (the fact that it is emulated or
> para-virtualized doesn't matter here). Secondly, you have another NIC
> (the TAP device) which belongs to the host and which is realized in
> software.
> 
> Now you need a way to connect those two; otherwise, you can't pass
> network traffic between them. One possible method to connect them are
> VLANs. By passing the parameter ",vlan=n" to both NICs, you are
> connecting them (of course, n is the same for both NICs).
> 
> Don't be worried when you see examples where this is not done
> explicitly. There are default values for the "vlan=n" parameter for
> network devices. Please refer to your qemu man page if you'd like to
> know more about that subject.
> 
> Another method to connect the host NIC with the VM NIC is the method
> your command line shows: The "netdev=" option of the rtl8139 determines
> which host network device should be used as the "backend" for the
> emulated hardware.
> 
> Since the latter method does not seem to work in your case, you could do
> the following for testing purposes:
> 
> Copy the command line you have shown into a file, make it executable and
> make the following changes:
> 
> Replace -netdev tap[...] by
> 
> -net
> tap,vlan=29,ifname=spock0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifup
> 
> (you probably already have this script, but maybe at another location,
> and see the comments at the end)
> (I have chosen a random weird VLAN number to make sure it doesn't crash
> with a VLAN from you other VMs)
> (Instead of spock0, you can choose another arbitrary name for the interface)
> 
> Then replace -device rtl8139[...] by
> 
> -device rtl8139,vlan=29,macaddr=<YOUR_MAC_HERE>,bus=pci.0,addr=0x3
> 
> Then shut down the VM in question and try to start it from the command
> line by executing the file you have just created.
> 
> Please note that I can't test this at the moment, so it might refuse to
> start and emit error messages. If this is the case, please report.
> 
>> What I do not know is how these "vnetX" - devices are created - you
>> mentioned two scripts - what do they do? Are the MAC addresses somehow
>> assigned to the "ventX" - devices, and if yes, how?
> 
> The vnetX devices seem to be the tap devices in your case. The qemu
> manual tells us that the O/S chooses a name for the tap device if none
> is given explicitly. These devices are created automatically (by qemu)
> when starting the respective VM (provided you have defined a tap device
> for the VM).
> 
> According to qemu's manual, there is no option for a tap device to
> assign a MAC, and I never have seen such an example. I haven't read the
> source code, and thus I don't know if a tap device uses an own MAC
> address internally (I don't think that, though), but in every case this
> should not matter here. What does matter is the MAC address you have
> assigned to the emulated / para-virtualized NIC which is passed to the VM.
> 
> Regarding the scripts: After having connected the VM's NIC and the
> host's (tap) NIC, you are not done yet. What you have so far is a street
> with two ends (the host and the VM), but no additional crossroads which
> would enable you to discover the world.
> 
> To do that, you usually connect the tap device to a bridge which is also
> connected to the real hardware NIC of your host (if desired), to the
> other tap devices from the other VMs (if desired) and so on.
> 
> Please note that there are other methods, too (for example, you could
> define a virtual network switch, and there are even other methods), but
> I have found bridging to be easy to understand, efficient and scalable.
> Of course, you can define as many bridges a you want and separate your
> VMs that way.
> 
> In my case, as I wrote in my previous message, the script just adds the
> tap device to a bridge (I have only one since my setup is simple: each
> of my VMs should see the full network traffic). It is not worth showing
> the contents of that script because it is the default one delivered with
> qemu, so you should have it already.
> 
> As a final hint, you probably don't need that script:
> 
> a) There is an option ",br=bridge" to -net tap. I suppose that this
> connects the tap device to the respective bridge immediately when it is
> created, and thus I didn't need the script if I would use this option.
> But please note that I did not try this yet, so I might be wrong.
> 
> b) There is a parameter "-net
> bridge[,vlan=n][,name=name][,br=bridge][...]" which seems to meant for
> connecting a vlan to a bridge directly; this would make using the script
> unnecessary as well. Please note that I didn't try that either, so I
> might be wrong again.
> 
> In general, to solve your problem, I would recommend you to play around
> with the command line. To get the right feeling, you might want to read
> qemu's man page; this will lead to a deeper understanding of what
> libvirt and friends are doing behind the scenes.
> 
> By the way, I never managed to damage a VM by giving the wrong
> parameters to qemu. But if you are concerned about this, just make a
> backup copy of the image the respective VM boots from. Then, nothing
> will stop you from playing around :-)
> 
> Regards,
> 
> Binarus
> 

-- 
hermann@qwer.tk
PGP/GPG: 299893C7 (on keyservers)