[REGRESSION] Virtio networking issues on v6.17-rc1

[REGRESSION] Virtio networking issues on v6.17-rc1

[Cristian Marussi]

Hi,

Not sure if it has been already reported but in a kvmtool/guest setup moving
the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
functionality....in a very funny way, though, I'd say...

In fact NO error is apparently reported in the guest kernel log and the
interfaces seems perfectly up an running both sides, but looking at the
host/guest interfaces you can see that ALL received frames are indeed dropped:


enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
....
        ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
        RX packets 125  bytes 17948 (17.5 KiB)
        RX errors 0  dropped 125 overruns 0  frame 0
        TX packets 1207  bytes 51182 (49.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


...on the host same..(taken later on...)

tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
        ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
        RX packets 804  bytes 43904 (42.8 KiB)
        RX errors 0  dropped 804  overruns 0  frame 0
        TX packets 101  bytes 14408 (14.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

....and for a good reason, apparently, since sniffing around on the Host TAP
interface I can see a never ending stream of:

$ sudo tcpdump -i tap0
listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
	0x0020:  0000 0000 c0a8 2101                      ......!.

... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
SRC mac address 0215 1515 1515  :P

...or on a different instance:

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:57:57.127171 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0x3333), length 82: 
	0x0000:  0000 0002 0215 1515 1515 86dd 6000 0000  ............`...
	0x0010:  0010 3aff fe80 0000 0000 0000 0015 15ff  ..:.............
	0x0020:  fe15 1515 ff02 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0002 8500 26b0 0000 0000 0101 0215  ......&.........
	0x0040:  1515 1515                                ....

...so it seems the frame is 'shifted' by 10 bytes or more and random
garbage pollutes the Ether frame...

I am using a vanilla kvmtool at:

	ba6830e vfio: include libgen.h (for musl compatibility)

and run with:

	lkvm run -c 4 -m 4G -k $IMAGE_DEF --network virtio -d $ROOTFS_DEF -p "earlycon loglevel=8"

and I could reproduce this on a RaspberryPI_5 with a Host Kernel v6.6 and on an APPLE M1
with Host Kernel v6.12.

Has anybody seen something similar ? Did I miss something obvious ?

...never saw this happening before on any previous kernel version.

Thanks,
Cristian

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Cristian Marussi]

On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
> Hi,
> 

Hi,

> Not sure if it has been already reported but in a kvmtool/guest setup moving
> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
> functionality....in a very funny way, though, I'd say...
> 
> In fact NO error is apparently reported in the guest kernel log and the
> interfaces seems perfectly up an running both sides, but looking at the
> host/guest interfaces you can see that ALL received frames are indeed dropped:
> 
> 
> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> ....
>         ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
>         RX packets 125  bytes 17948 (17.5 KiB)
>         RX errors 0  dropped 125 overruns 0  frame 0
>         TX packets 1207  bytes 51182 (49.9 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> 
> ...on the host same..(taken later on...)
> 
> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
>         ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
>         RX packets 804  bytes 43904 (42.8 KiB)
>         RX errors 0  dropped 804  overruns 0  frame 0
>         TX packets 101  bytes 14408 (14.0 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> ....and for a good reason, apparently, since sniffing around on the Host TAP
> interface I can see a never ending stream of:
> 
> $ sudo tcpdump -i tap0
> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
> 	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
> 	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
> 	0x0020:  0000 0000 c0a8 2101                      ......!.
> 
> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
> SRC mac address 0215 1515 1515  :P
>

I bisected this regression to:

56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit

commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
Author: Paolo Abeni <pabeni@redhat.com>
Date:   Tue Jul 8 17:55:02 2025 +0200

    virtio_net: enable gso over UDP tunnel support.
    
    If the related virtio feature is set, enable transmission and reception
    of gso over UDP tunnel packets.
    
    Most of the work is done by the previously introduced helper, just need
    to determine the UDP tunnel features inside the virtio_net_hdr and
    update accordingly the virtio net hdr size.
    
    Acked-by: Jason Wang <jasowang@redhat.com>
    Signed-off-by: Paolo Abeni <pabeni@redhat.com>

 drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------

Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.

Thanks,
Cristian

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Paolo Abeni]

On 8/13/25 8:09 PM, Cristian Marussi wrote:
> On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
>> Not sure if it has been already reported but in a kvmtool/guest setup moving
>> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
>> functionality....in a very funny way, though, I'd say...
>>
>> In fact NO error is apparently reported in the guest kernel log and the
>> interfaces seems perfectly up an running both sides, but looking at the
>> host/guest interfaces you can see that ALL received frames are indeed dropped:
>>
>>
>> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>> ....
>>         ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
>>         RX packets 125  bytes 17948 (17.5 KiB)
>>         RX errors 0  dropped 125 overruns 0  frame 0
>>         TX packets 1207  bytes 51182 (49.9 KiB)
>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>>
>> ...on the host same..(taken later on...)
>>
>> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>         inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
>>         ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
>>         RX packets 804  bytes 43904 (42.8 KiB)
>>         RX errors 0  dropped 804  overruns 0  frame 0
>>         TX packets 101  bytes 14408 (14.0 KiB)
>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> ....and for a good reason, apparently, since sniffing around on the Host TAP
>> interface I can see a never ending stream of:
>>
>> $ sudo tcpdump -i tap0
>> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
>> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
>> 	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
>> 	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
>> 	0x0020:  0000 0000 c0a8 2101                      ......!.
>>
>> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
>> SRC mac address 0215 1515 1515  :P
>>
> 
> I bisected this regression to:
> 
> 56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit
> 
> commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
> Author: Paolo Abeni <pabeni@redhat.com>
> Date:   Tue Jul 8 17:55:02 2025 +0200
> 
>     virtio_net: enable gso over UDP tunnel support.
>     
>     If the related virtio feature is set, enable transmission and reception
>     of gso over UDP tunnel packets.
>     
>     Most of the work is done by the previously introduced helper, just need
>     to determine the UDP tunnel features inside the virtio_net_hdr and
>     update accordingly the virtio net hdr size.
>     
>     Acked-by: Jason Wang <jasowang@redhat.com>
>     Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> 
>  drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------
> 
> Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.

Thanks for reporting, I was not aware yet of this regression.

Apparently there is mismatch between the negotiated features (that
determinate the virtio net header size) and the actually virtio header
used by both the guest and the host, which is totally unexpected.

Could you please share the host kernel version, the arch and the kvmtool
version? Also I understand you only upgraded the guest kernel, without
any other setup changes, am I correct?

Thanks,

Paolo

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Cristian Marussi]

On Wed, Aug 13, 2025 at 09:51:30PM +0200, Paolo Abeni wrote:
> On 8/13/25 8:09 PM, Cristian Marussi wrote:
> > On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
> >> Not sure if it has been already reported but in a kvmtool/guest setup moving
> >> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
> >> functionality....in a very funny way, though, I'd say...
> >>
> >> In fact NO error is apparently reported in the guest kernel log and the
> >> interfaces seems perfectly up an running both sides, but looking at the
> >> host/guest interfaces you can see that ALL received frames are indeed dropped:
> >>
> >>
> >> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >> ....
> >>         ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
> >>         RX packets 125  bytes 17948 (17.5 KiB)
> >>         RX errors 0  dropped 125 overruns 0  frame 0
> >>         TX packets 1207  bytes 51182 (49.9 KiB)
> >>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >>
> >>
> >> ...on the host same..(taken later on...)
> >>
> >> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >>         inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
> >>         ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
> >>         RX packets 804  bytes 43904 (42.8 KiB)
> >>         RX errors 0  dropped 804  overruns 0  frame 0
> >>         TX packets 101  bytes 14408 (14.0 KiB)
> >>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >>
> >> ....and for a good reason, apparently, since sniffing around on the Host TAP
> >> interface I can see a never ending stream of:
> >>
> >> $ sudo tcpdump -i tap0
> >> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
> >> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
> >> 	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
> >> 	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
> >> 	0x0020:  0000 0000 c0a8 2101                      ......!.
> >>
> >> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
> >> SRC mac address 0215 1515 1515  :P
> >>
> > 
> > I bisected this regression to:
> > 
> > 56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit
> > 
> > commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
> > Author: Paolo Abeni <pabeni@redhat.com>
> > Date:   Tue Jul 8 17:55:02 2025 +0200
> > 
> >     virtio_net: enable gso over UDP tunnel support.
> >     
> >     If the related virtio feature is set, enable transmission and reception
> >     of gso over UDP tunnel packets.
> >     
> >     Most of the work is done by the previously introduced helper, just need
> >     to determine the UDP tunnel features inside the virtio_net_hdr and
> >     update accordingly the virtio net hdr size.
> >     
> >     Acked-by: Jason Wang <jasowang@redhat.com>
> >     Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> > 
> >  drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------
> > 
> > Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.
> 
> Thanks for reporting, I was not aware yet of this regression.
> 

Hi Paolo,

I spotted this issue on my setup this morning.

> Apparently there is mismatch between the negotiated features (that
> determinate the virtio net header size) and the actually virtio header
> used by both the guest and the host, which is totally unexpected.
> 

Have the UAPI headers been updated ? ... I could not see any change...
(not that I am very familiar with virtio core :D)

> Could you please share the host kernel version, the arch and the kvmtool
> version? Also I understand you only upgraded the guest kernel, without
> any other setup changes, am I correct?
> 

The arch is arm64, I could reproduce on a Raspberry Pi5 with host v6.6 and on
an Apple M1 (running Linux natively) with a v6.12.

Both host kernels are from the original distros unchanged (Debian and Fedora)

In both cases I compiled from scratch (make clean && make) kvmtool on the
respective arm64 Host using the current top of tree from origin/master at:

https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/log/

	commit 	ba6830eec0f5 ("vfio: include libgen.h (for musl compatibility)")

Run kvmtoool baer simple with
	
	lkvm run -c 4 -m 4G -k $IMAGE_DEF --network virtio -d $ROOTFS_DEF -p "earlycon loglevel=8"

All was fine till v6.16, and it is my usual setup and update procedure that never failed really
as long as I update kvmtool in sync.
Not sure if something is still missing from kvmtool master repo for v6.17 but it seems reasonably
updated (July25).

I wonder if no one else had this issue on arm64/kvmtool....that would made my 'regression'
suspiciously tied tomy setup..

Thanks for having a look,
Cristian

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Paolo Abeni]

On 8/13/25 10:31 PM, Cristian Marussi wrote:
> On Wed, Aug 13, 2025 at 09:51:30PM +0200, Paolo Abeni wrote:
>> On 8/13/25 8:09 PM, Cristian Marussi wrote:
>>> On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
>>>> Not sure if it has been already reported but in a kvmtool/guest setup moving
>>>> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
>>>> functionality....in a very funny way, though, I'd say...
>>>>
>>>> In fact NO error is apparently reported in the guest kernel log and the
>>>> interfaces seems perfectly up an running both sides, but looking at the
>>>> host/guest interfaces you can see that ALL received frames are indeed dropped:
>>>>
>>>>
>>>> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>>> ....
>>>>         ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
>>>>         RX packets 125  bytes 17948 (17.5 KiB)
>>>>         RX errors 0  dropped 125 overruns 0  frame 0
>>>>         TX packets 1207  bytes 51182 (49.9 KiB)
>>>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>>>
>>>>
>>>> ...on the host same..(taken later on...)
>>>>
>>>> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>>>         inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
>>>>         ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
>>>>         RX packets 804  bytes 43904 (42.8 KiB)
>>>>         RX errors 0  dropped 804  overruns 0  frame 0
>>>>         TX packets 101  bytes 14408 (14.0 KiB)
>>>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>>>
>>>> ....and for a good reason, apparently, since sniffing around on the Host TAP
>>>> interface I can see a never ending stream of:
>>>>
>>>> $ sudo tcpdump -i tap0
>>>> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
>>>> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
>>>> 	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
>>>> 	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
>>>> 	0x0020:  0000 0000 c0a8 2101                      ......!.
>>>>
>>>> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
>>>> SRC mac address 0215 1515 1515  :P
>>>>
>>>
>>> I bisected this regression to:
>>>
>>> 56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit
>>>
>>> commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
>>> Author: Paolo Abeni <pabeni@redhat.com>
>>> Date:   Tue Jul 8 17:55:02 2025 +0200
>>>
>>>     virtio_net: enable gso over UDP tunnel support.
>>>     
>>>     If the related virtio feature is set, enable transmission and reception
>>>     of gso over UDP tunnel packets.
>>>     
>>>     Most of the work is done by the previously introduced helper, just need
>>>     to determine the UDP tunnel features inside the virtio_net_hdr and
>>>     update accordingly the virtio net hdr size.
>>>     
>>>     Acked-by: Jason Wang <jasowang@redhat.com>
>>>     Signed-off-by: Paolo Abeni <pabeni@redhat.com>
>>>
>>>  drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------
>>>
>>> Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.
>>
>> Thanks for reporting, I was not aware yet of this regression.
>>
> 
> Hi Paolo,
> 
> I spotted this issue on my setup this morning.
> 
>> Apparently there is mismatch between the negotiated features (that
>> determinate the virtio net header size) and the actually virtio header
>> used by both the guest and the host, which is totally unexpected.
>>
> 
> Have the UAPI headers been updated ? ... I could not see any change...
> (not that I am very familiar with virtio core :D)

The virtio feature space has been extended from 64 bits to 128 and the
blamed commit is the first using the 'extended' virtio features
negotiation.

But, modulo bugs, the extended space should be used only if the
hypervisors exposes the relevant capabilities. kvmtool does not do that.

>> Could you please share the host kernel version, the arch and the kvmtool
>> version? Also I understand you only upgraded the guest kernel, without
>> any other setup changes, am I correct?
>>
> 
> The arch is arm64, I could reproduce on a Raspberry Pi5 with host v6.6 and on
> an Apple M1 (running Linux natively) with a v6.12.
> 
> Both host kernels are from the original distros unchanged (Debian and Fedora)
> 
> In both cases I compiled from scratch (make clean && make) kvmtool on the
> respective arm64 Host using the current top of tree from origin/master at:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/log/
> 
> 	commit 	ba6830eec0f5 ("vfio: include libgen.h (for musl compatibility)")
> 
> Run kvmtoool baer simple with
> 	
> 	lkvm run -c 4 -m 4G -k $IMAGE_DEF --network virtio -d $ROOTFS_DEF -p "earlycon loglevel=8"
> 
> All was fine till v6.16, and it is my usual setup and update procedure that never failed really
> as long as I update kvmtool in sync.
> Not sure if something is still missing from kvmtool master repo for v6.17 but it seems reasonably
> updated (July25).
> 
> I wonder if no one else had this issue on arm64/kvmtool....that would made my 'regression'
> suspiciously tied tomy setup..

Thanks for the detailed reproduction steps.

I agree it should be some bad iteration with the arm arch and/or
kvmtool. I hope an arm system is not required to reproduce the issue,
because I don't have any of them handy.

I should be able to dig into this tomorrow afternoon my time.

Cheers,

Paolo

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Cristian Marussi]

On Wed, Aug 13, 2025 at 10:41:28PM +0200, Paolo Abeni wrote:
> On 8/13/25 10:31 PM, Cristian Marussi wrote:
> > On Wed, Aug 13, 2025 at 09:51:30PM +0200, Paolo Abeni wrote:
> >> On 8/13/25 8:09 PM, Cristian Marussi wrote:
> >>> On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
> >>>> Not sure if it has been already reported but in a kvmtool/guest setup moving
> >>>> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
> >>>> functionality....in a very funny way, though, I'd say...
> >>>>
> >>>> In fact NO error is apparently reported in the guest kernel log and the
> >>>> interfaces seems perfectly up an running both sides, but looking at the
> >>>> host/guest interfaces you can see that ALL received frames are indeed dropped:
> >>>>
> >>>>
> >>>> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >>>> ....
> >>>>         ether 02:15:15:15:15:15  txqueuelen 1000  (Ethernet)           <<<<<<<<<<<<<<<<
> >>>>         RX packets 125  bytes 17948 (17.5 KiB)
> >>>>         RX errors 0  dropped 125 overruns 0  frame 0
> >>>>         TX packets 1207  bytes 51182 (49.9 KiB)
> >>>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >>>>
> >>>>
> >>>> ...on the host same..(taken later on...)
> >>>>
> >>>> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >>>>         inet 192.168.33.1  netmask 255.255.255.0  broadcast 192.168.33.255
> >>>>         ether 8a:10:f6:df:a1:70  txqueuelen 1000  (Ethernet)
> >>>>         RX packets 804  bytes 43904 (42.8 KiB)
> >>>>         RX errors 0  dropped 804  overruns 0  frame 0
> >>>>         TX packets 101  bytes 14408 (14.0 KiB)
> >>>>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >>>>
> >>>> ....and for a good reason, apparently, since sniffing around on the Host TAP
> >>>> interface I can see a never ending stream of:
> >>>>
> >>>> $ sudo tcpdump -i tap0
> >>>> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
> >>>> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
> >>>> 	0x0000:  ffff ffff 0215 1515 1515 0806 0001 0800  ................       <<<<<<<<<<<<<
> >>>> 	0x0010:  0604 0001 0215 1515 1515 c0a8 2102 0000  ............!...
> >>>> 	0x0020:  0000 0000 c0a8 2101                      ......!.
> >>>>
> >>>> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
> >>>> SRC mac address 0215 1515 1515  :P
> >>>>
> >>>
> >>> I bisected this regression to:
> >>>
> >>> 56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit
> >>>
> >>> commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
> >>> Author: Paolo Abeni <pabeni@redhat.com>
> >>> Date:   Tue Jul 8 17:55:02 2025 +0200
> >>>
> >>>     virtio_net: enable gso over UDP tunnel support.
> >>>     
> >>>     If the related virtio feature is set, enable transmission and reception
> >>>     of gso over UDP tunnel packets.
> >>>     
> >>>     Most of the work is done by the previously introduced helper, just need
> >>>     to determine the UDP tunnel features inside the virtio_net_hdr and
> >>>     update accordingly the virtio net hdr size.
> >>>     
> >>>     Acked-by: Jason Wang <jasowang@redhat.com>
> >>>     Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> >>>
> >>>  drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------
> >>>
> >>> Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.
> >>
> >> Thanks for reporting, I was not aware yet of this regression.
> >>
> > 
> > Hi Paolo,
> > 
> > I spotted this issue on my setup this morning.
> > 
> >> Apparently there is mismatch between the negotiated features (that
> >> determinate the virtio net header size) and the actually virtio header
> >> used by both the guest and the host, which is totally unexpected.
> >>
> > 
> > Have the UAPI headers been updated ? ... I could not see any change...
> > (not that I am very familiar with virtio core :D)
> 
> The virtio feature space has been extended from 64 bits to 128 and the
> blamed commit is the first using the 'extended' virtio features
> negotiation.

Understood, thanks for the explanation.

> 
> But, modulo bugs, the extended space should be used only if the
> hypervisors exposes the relevant capabilities. kvmtool does not do that.
> 

Ah, ok...I was hoping it was just a matter of kvmtool still to have to
update headers but maybe there's something more.

> >> Could you please share the host kernel version, the arch and the kvmtool
> >> version? Also I understand you only upgraded the guest kernel, without
> >> any other setup changes, am I correct?
> >>
> > 
> > The arch is arm64, I could reproduce on a Raspberry Pi5 with host v6.6 and on
> > an Apple M1 (running Linux natively) with a v6.12.
> > 
> > Both host kernels are from the original distros unchanged (Debian and Fedora)
> > 
> > In both cases I compiled from scratch (make clean && make) kvmtool on the
> > respective arm64 Host using the current top of tree from origin/master at:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/log/
> > 
> > 	commit 	ba6830eec0f5 ("vfio: include libgen.h (for musl compatibility)")
> > 
> > Run kvmtoool baer simple with
> > 	
> > 	lkvm run -c 4 -m 4G -k $IMAGE_DEF --network virtio -d $ROOTFS_DEF -p "earlycon loglevel=8"
> > 
> > All was fine till v6.16, and it is my usual setup and update procedure that never failed really
> > as long as I update kvmtool in sync.
> > Not sure if something is still missing from kvmtool master repo for v6.17 but it seems reasonably
> > updated (July25).
> > 
> > I wonder if no one else had this issue on arm64/kvmtool....that would made my 'regression'
> > suspiciously tied tomy setup..
> 
> Thanks for the detailed reproduction steps.
> 
> I agree it should be some bad iteration with the arm arch and/or
> kvmtool. I hope an arm system is not required to reproduce the issue,
> because I don't have any of them handy.

Thanks for having a look, hope not to have made noise for nothing and
maybe it is just a matter of a pending udpate from kvmtool guys...indeed
looking at their mailing list there is a pending patch to update headers
to v6.16 (for unrelated reasons)...

https://lore.kernel.org/kvm/20250729095745.3148294-2-andre.przywara@arm.com/T/#u

....but I suppose it would not be enough to catch with v6.17 changes
anyway.

Anyway if you need to test any fix on arm64 just let me know and I can do it
(...modulo my incoming holiday end-of-next-week :P)

> 
> I should be able to dig into this tomorrow afternoon my time.
>

No worries...no rush...I have a good workaround for now...I just wanted to
raise a flag to make sure I am not hallucinating on my own :D 

Thanks for you help

Cheers,
Cristian

Re: [REGRESSION] Virtio networking issues on v6.17-rc1

[Paolo Abeni]

On 8/13/25 10:41 PM, Paolo Abeni wrote:
> Thanks for the detailed reproduction steps.

They definitely helped to reproduce the issue here. It took a little
time because I'm a complete noob WRT kvmtool.

Apparently it was indeed a kvmtool issue.

I just posted a patch that addresses the issue for me:

https://lore.kernel.org/kvm/ed62443b8fd3fef87bd313a54f821cf363f647a5.1755185758.git.pabeni@redhat.com/T/#u

Cheers,

Paolo