EVL: Kernel WARNING

EVL: Kernel WARNING

[Hannes Diethelm]

Hi

After updating to the lastest v6.12.y-cip-evl-rebase, I have a kernel WARNING in the igb driver if I use my oob-net-icmp.c (Patch submitted)

Exact commits good...bad:
linux-evl efed612...55591d4
libevl 3e5577d..f7b7df4

Regards
Hannes

[17022.558481] ------------[ cut here ]------------
[17022.558482] WARNING: CPU: 3 PID: 10425 at drivers/net/ethernet/intel/igb/igb_main.c:9513 igb_alloc_rx_buffers+0xd3/0x230 [igb]
[17022.558497] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nf_tables libcrc32c intel_rapl_msr binfmt_misc intel_rapl_common snd_soc_avs x86_pkg_temp_thermal intel_powerclamp coretemp snd_soc_hda_codec crct10dif_pclmul ghash_clmulni_intel snd_hda_ext_core nls_ascii sha512_ssse3 nls_cp437 vfat sha256_ssse3 snd_soc_core sha1_ssse3 fat snd_hda_codec_realtek mei_pxp snd_hda_codec_generic eeepc_wmi snd_compress snd_pcm_dmaengine snd_hda_scodec_component snd_hda_codec_hdmi mei_hdcp aesni_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi gf128mul snd_hda_codec crypto_simd asus_wmi cryptd sparse_keymap rapl intel_cstate platform_profile snd_hda_core ee1004 battery intel_uncore snd_hwdep snd_pcm wmi_bmof rfkill mei_me mei snd_timer snd soundcore intel_pmc_core intel_vsec pmt_telemetry pmt_class acpi_pad joydev evdev sg parport_pc ppdev lp parport efi_pstore configfs nfnetlink efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache jbd2 mlx4_ib ib_uverbs ib_core mlx4_en hid_multitouch uas
[17022.558533]  usb_storage amdgpu hid_generic sr_mod sd_mod cdrom usbhid amdxcp drm_exec hid gpu_sched drm_buddy drm_suballoc_helper drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper ahci libahci drm libata mlx4_core iTCO_wdt intel_pmc_bxt iTCO_vendor_support xhci_pci xhci_hcd watchdog mxm_wmi usbcore igb scsi_mod i2c_i801 crc32_pclmul r8168(OE) crc32c_intel i2c_smbus i2c_algo_bit fan dca scsi_common crc16 usb_common video wmi button
[17022.558554] CPU: 3 UID: 0 PID: 10425 Comm: enp2s0f0.rx Tainted: G           OE      6.12.90-cip24-xenomai4 #1
[17022.558557] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[17022.558558] Hardware name: System manufacturer System Product Name/H110M-A, BIOS 3805 05/07/2018
[17022.558559] IRQ stage: EVL
[17022.558560] RIP: 0010:igb_alloc_rx_buffers+0xd3/0x230 [igb]
[17022.558569] Code: 73 10 48 8b 03 49 8b 7e 18 48 01 c6 f6 87 fc 02 00 00 20 75 be b9 02 00 00 00 4c 89 ea e8 55 37 ff c9 8b 73 10 48 03 33 eb a9 <0f> 0b 49 83 86 d0 00 00 00 01 66 41 03 ae 80 00 00 00 66 41 39 ae
[17022.558570] RSP: 0018:ffffc90d09adfcb0 EFLAGS: 00010246
[17022.558572] RAX: 0000000000000000 RBX: ffffc90d00fe5768 RCX: ffff88ae2ede2b00
[17022.558573] RDX: 0000000000000000 RSI: 0000000000002820 RDI: ffff88ad035e5110
[17022.558574] RBP: 000000000000ff4f R08: 0000000000000000 R09: 0000000000000000
[17022.558575] R10: 0000000000000000 R11: ffff88ad144fe050 R12: ffff88ad1b1184f0
[17022.558576] R13: 0000000000000600 R14: ffff88ad144fe240 R15: 0000000000000001
[17022.558577] FS:  0000000000000000(0000) GS:ffff88ae2ed80000(0000) knlGS:0000000000000000
[17022.558579] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17022.558580] CR2: 0000000000000000 CR3: 000000017c6ec001 CR4: 00000000003706f0
[17022.558581] Call Trace:
[17022.558583]  <TASK>
[17022.558585]  igb_poll+0x1096/0x1630 [igb]
[17022.558595]  ? __switch_to_asm+0x37/0x70
[17022.558598]  ? __switch_to+0x113/0x4b0
[17022.558602]  __napi_poll+0x28/0x160
[17022.558605]  napi_poll+0x122/0x140
[17022.558608]  evl_net_do_rx+0x2dc/0x540
[17022.558613]  ? __pfx_kthread_trampoline+0x10/0x10
[17022.558615]  kthread_trampoline+0x107/0x210
[17022.558618]  kthread+0xcf/0x100
[17022.558620]  ? __pfx_kthread+0x10/0x10
[17022.558623]  ret_from_fork+0x31/0x50
[17022.558625]  ? __pfx_kthread+0x10/0x10
[17022.558626]  ret_from_fork_asm+0x1a/0x30
[17022.558631]  </TASK>
[17022.558631] ---[ end trace 0000000000000000 ]---

Re: EVL: Kernel WARNING

[Hannes Diethelm]

Am 04.07.26 um 20:05 schrieb Hannes Diethelm:
> Hi
> 
> After updating to the lastest v6.12.y-cip-evl-rebase, I have a kernel 
> WARNING in the igb driver if I use my oob-net-icmp.c (Patch submitted)
> 
> Exact commits good...bad:
> linux-evl efed612...55591d4
> libevl 3e5577d..f7b7df4
> 
> Regards
> Hannes
> ...

Update:

- It fails after ~240...253 packages. After that, no ping response is received any more
- I use "sudo ./oob-net-ping -i enp7s0 -a 192.168.122.1 -w 1000" so it fails faster
- Also without -w 1000, it fails, you just have to wait longer
- With an I350 card, I get the kernel WARNING
- In a VM with an e1000e, I don't get a warning but still after ~250 packages, no response is
   received any more
- After evl net -di / evl net -ei, it works for another ~250 packages until it fails again
- With the good version, I tested ~1M packages, all fine, max 60 us response time on the real PC

Re: EVL: Kernel WARNING

[Philippe Gerum]

Hannes Diethelm <hannes.diethelm@gmail.com> writes:

> Am 04.07.26 um 20:05 schrieb Hannes Diethelm:
>> Hi
>> After updating to the lastest v6.12.y-cip-evl-rebase, I have a
>> kernel WARNING in the igb driver if I use my oob-net-icmp.c (Patch
>> submitted)
>> Exact commits good...bad:
>> linux-evl efed612...55591d4
>> libevl 3e5577d..f7b7df4
>> Regards
>> Hannes
>> ...
>
> Update:
>
> - It fails after ~240...253 packages. After that, no ping response is received any more
> - I use "sudo ./oob-net-ping -i enp7s0 -a 192.168.122.1 -w 1000" so it fails faster
> - Also without -w 1000, it fails, you just have to wait longer
> - With an I350 card, I get the kernel WARNING
> - In a VM with an e1000e, I don't get a warning but still after ~250 packages, no response is
>   received any more
> - After evl net -di / evl net -ei, it works for another ~250 packages until it fails again
> - With the good version, I tested ~1M packages, all fine, max 60 us response time on the real PC

Ok, I can reproduce this easily with oob-net-ping as you described. This
also happens with an (arm-based) FEC driver, which confirms that this is
a generic issue - maybe in the packet protocol module? - not a driver
specific bug. Working on it.

-- 
Philippe.

Re: EVL: Kernel WARNING

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>
>> Am 04.07.26 um 20:05 schrieb Hannes Diethelm:
>>> Hi
>>> After updating to the lastest v6.12.y-cip-evl-rebase, I have a
>>> kernel WARNING in the igb driver if I use my oob-net-icmp.c (Patch
>>> submitted)
>>> Exact commits good...bad:
>>> linux-evl efed612...55591d4
>>> libevl 3e5577d..f7b7df4
>>> Regards
>>> Hannes
>>> ...
>>
>> Update:
>>
>> - It fails after ~240...253 packages. After that, no ping response is received any more
>> - I use "sudo ./oob-net-ping -i enp7s0 -a 192.168.122.1 -w 1000" so it fails faster
>> - Also without -w 1000, it fails, you just have to wait longer
>> - With an I350 card, I get the kernel WARNING
>> - In a VM with an e1000e, I don't get a warning but still after ~250 packages, no response is
>>   received any more
>> - After evl net -di / evl net -ei, it works for another ~250 packages until it fails again
>> - With the good version, I tested ~1M packages, all fine, max 60 us response time on the real PC
>
> Ok, I can reproduce this easily with oob-net-ping as you described. This
> also happens with an (arm-based) FEC driver, which confirms that this is
> a generic issue - maybe in the packet protocol module? - not a driver
> specific bug. Working on it.

I forgot to mention that this issue boils down to a shortage of buffers
to convey ingress packets, which explains the behavior once
triggered. Some memory leakage in the packet layer seems to be at work,
because it does not trigger when spraying the receiving target with
ICMP/UDP requests.

-- 
Philippe.

Re: EVL: Kernel WARNING

[Philippe Gerum]

Hannes Diethelm <hannes.diethelm@gmail.com> writes:

> - With the good version, I tested ~1M packages, all fine, max 60 us
> response time on the real PC

You mean 60us for the overall round-trip time between two evl-enabled
targets in full/end-to-end oob mode?

-- 
Philippe.

Re: EVL: Kernel WARNING

[Hannes Diethelm]

Am 05.07.26 um 17:06 schrieb Philippe Gerum:
> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
> 
>> - With the good version, I tested ~1M packages, all fine, max 60 us
>> response time on the real PC
> 
> You mean 60us for the overall round-trip time between two evl-enabled
> targets in full/end-to-end oob mode?
> 

That's between an evl-enabled host and an FPGA board as a target[1]
directly connected with 100M Ethernet and an Intel I350 after some
heavy tuning. Most tuning was done for PREEMT_RT. I have to test what
is really needed for Xenomai4 and what can be dropped.

I don't have two PC's to test (yet).

https://store.mesanet.com/index.php?route=product/product&product_id=374

Re: EVL: Kernel WARNING

[Philippe Gerum]

Hannes Diethelm <hannes.diethelm@gmail.com> writes:

> Am 05.07.26 um 17:06 schrieb Philippe Gerum:
>> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>> 
>>> - With the good version, I tested ~1M packages, all fine, max 60 us
>>> response time on the real PC
>> You mean 60us for the overall round-trip time between two
>> evl-enabled
>> targets in full/end-to-end oob mode?
>> 
>
> That's between an evl-enabled host and an FPGA board as a target[1]
> directly connected with 100M Ethernet and an Intel I350 after some
> heavy tuning. Most tuning was done for PREEMT_RT. I have to test what
> is really needed for Xenomai4 and what can be dropped.
>
> I don't have two PC's to test (yet).
>
> https://store.mesanet.com/index.php?route=product/product&product_id=374

Ok, thanks for the information. Meanwhile, I found the culprit. Now I
have to figure out why I did that. Oh, well..

commit a793acf3cab9a608d70a39d740f04c6a6a1ec77c
Author: Philippe Gerum <rpm@xenomai.org>
Date:   Tue May 26 10:16:44 2026 +0200

    evl/net: packet: forward status of ETH_P_ALL delivery

-- 
Philippe.

Re: EVL: Kernel WARNING

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>
>> Am 05.07.26 um 17:06 schrieb Philippe Gerum:
>>> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>>> 
>>>> - With the good version, I tested ~1M packages, all fine, max 60 us
>>>> response time on the real PC
>>> You mean 60us for the overall round-trip time between two
>>> evl-enabled
>>> targets in full/end-to-end oob mode?
>>> 
>>
>> That's between an evl-enabled host and an FPGA board as a target[1]
>> directly connected with 100M Ethernet and an Intel I350 after some
>> heavy tuning. Most tuning was done for PREEMT_RT. I have to test what
>> is really needed for Xenomai4 and what can be dropped.
>>
>> I don't have two PC's to test (yet).
>>
>> https://store.mesanet.com/index.php?route=product/product&product_id=374
>
> Ok, thanks for the information. Meanwhile, I found the culprit. Now I
> have to figure out why I did that. Oh, well..
>
> commit a793acf3cab9a608d70a39d740f04c6a6a1ec77c
> Author: Philippe Gerum <rpm@xenomai.org>
> Date:   Tue May 26 10:16:44 2026 +0200
>
>     evl/net: packet: forward status of ETH_P_ALL delivery

Fixed by #bb06b3a8f100.

-- 
Philippe.

Re: EVL: Kernel WARNING

[Hannes Diethelm]

Am 06.07.26 um 07:56 schrieb Philippe Gerum:
> Philippe Gerum <rpm@xenomai.org> writes:
> 
>> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>>
>>> Am 05.07.26 um 17:06 schrieb Philippe Gerum:
>>>> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>>>>
>>>>> - With the good version, I tested ~1M packages, all fine, max 60 us
>>>>> response time on the real PC
>>>> You mean 60us for the overall round-trip time between two
>>>> evl-enabled
>>>> targets in full/end-to-end oob mode?
>>>>
>>>
>>> That's between an evl-enabled host and an FPGA board as a target[1]
>>> directly connected with 100M Ethernet and an Intel I350 after some
>>> heavy tuning. Most tuning was done for PREEMT_RT. I have to test what
>>> is really needed for Xenomai4 and what can be dropped.
>>>
>>> I don't have two PC's to test (yet).
>>>
>>> https://store.mesanet.com/index.php?route=product/product&product_id=374
>>
>> Ok, thanks for the information. Meanwhile, I found the culprit. Now I
>> have to figure out why I did that. Oh, well..
>>
>> commit a793acf3cab9a608d70a39d740f04c6a6a1ec77c
>> Author: Philippe Gerum <rpm@xenomai.org>
>> Date:   Tue May 26 10:16:44 2026 +0200
>>
>>      evl/net: packet: forward status of ETH_P_ALL delivery
> 
> Fixed by #bb06b3a8f100.
> 

Thanks, tested and it works.