[REGRESSION] in 6.17, failing __dev_change_net_namespace+0xb89/0xc30

[REGRESSION] in 6.17, failing __dev_change_net_namespace+0xb89/0xc30

[Cynthia]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

When I updated my machine to the newest kernel, a bug started to appear. 
The system does not panic, but an error kept happening in dmesg.

The bug happens with LXC/Incus when it tries to start a new container. 
(but probably other things are affected too)


Steps to Reproduce: the bug can be reproduced in a libvirt VM, no need 
for a specific system. Also I suspect the bug is also 
architecture-independent, but I cannot verify that.
1) Install ArchLinux (all dependencies are available). I was testing 
with vanilla kernel, so any linux distro should be affected.
https://aur.archlinux.org/packages/linux-mainline can be installed, this 
is the vanilla kernel with a generally good kernel config for most PCs.
2) Install LXC/Incus (pacman -S incus)
3) configure incus and start a container:
usermod -v 1000000-1000999999 -w 1000000-1000999999 root &&
incus admin init &&
incus launch images:debian/12 first # start a container
4) Previous step should trigger incus to do namespaces. I'm not sure 
what syscall is causing the bug, I do not have a mini C program. These 
steps should be enough to see the log in the dmesg.

I also did a git bisect, the first commit to have this bug is this:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c17270f9b920e4e1777488f1911bbfdaf2af3be

I initially reported this bug on Bugzilla, but after seeing 6 year old 
bugs there, I'm not sure if that platform is still in use.
https://bugzilla.kernel.org/show_bug.cgi?id=220649

Since my initial report, 6.17.2 was released, the bug is still happening.

I'm attaching 2 files:
dmesg_slice: the slice of dmesg containing the problematic frame (on a 
bare-metal linux with AMD srso mitigation disabled)
bisect_log: a log of the git bisect process

[-- Attachment #2: bisect_log --]
[-- Type: text/plain, Size: 2884 bytes --]

git bisect start
# status: waiting for both good and bad commits
# good: [8d05316d79d8afd20ba767efea8706d8238a9d46] dm-raid: do not include dm-core.h
git bisect good 8d05316d79d8afd20ba767efea8706d8238a9d46
# status: waiting for bad commit, 1 good commit known
# bad: [8f5ae30d69d7543eee0d70083daf4de8fe15d585] Linux 6.17-rc1
git bisect bad 8f5ae30d69d7543eee0d70083daf4de8fe15d585
# bad: [8be4d31cb8aaeea27bde4b7ddb26e28a89062ebf] Merge tag 'net-next-6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect bad 8be4d31cb8aaeea27bde4b7ddb26e28a89062ebf
# good: [0d5ec7919f3747193f051036b2301734a4b5e1d6] Merge tag 'char-misc-6.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
git bisect good 0d5ec7919f3747193f051036b2301734a4b5e1d6
# good: [beb6c8326eb4e7006c4aa16b0fee3e303d42e685] Merge tag 'uml-for-linux-6.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/uml/linux
git bisect good beb6c8326eb4e7006c4aa16b0fee3e303d42e685
# bad: [0a49abff43782de9546497392caab0f8bc507545] Merge branch 'net-migrate-remaining-drivers-to-dedicated-_rxfh_context-ops'
git bisect bad 0a49abff43782de9546497392caab0f8bc507545
# bad: [bbb7d478d91ac4d5c288e226cc8744daf3820798] net: phy: Add interface types for 50G and 100G
git bisect bad bbb7d478d91ac4d5c288e226cc8744daf3820798
# bad: [b11344f63fdd9e8c5121148a6965b41079071dd2] gve: Return error for unknown admin queue command
git bisect bad b11344f63fdd9e8c5121148a6965b41079071dd2
# good: [c969149bafbeb8ae113747e00ae3ef97461f2cd4] net: amt: convert to use secs_to_jiffies
git bisect good c969149bafbeb8ae113747e00ae3ef97461f2cd4
# bad: [3b5b1c428260152e47c9584bc176f358b87ca82d] eth: gianfar: migrate to new RXFH callbacks
git bisect bad 3b5b1c428260152e47c9584bc176f358b87ca82d
# bad: [0550890b2f80697e7ebf1e4916373e884a921492] Merge branch 'gve-add-rx-hw-timestamping-support'
git bisect bad 0550890b2f80697e7ebf1e4916373e884a921492
# bad: [ccc7edf0ada83d395b634506eff9616360a99b5a] netpoll: move netpoll_print_options to netconsole
git bisect bad ccc7edf0ada83d395b634506eff9616360a99b5a
# bad: [a1acde1e1bcf18efc0549d7d317daed54138014e] net: phy: make phy_package a separate module
git bisect bad a1acde1e1bcf18efc0549d7d317daed54138014e
# bad: [8909f5f4ecd551c2299b28e05254b77424c8c7dc] net: stmmac: qcom-ethqos: add ethqos_pcs_set_inband()
git bisect bad 8909f5f4ecd551c2299b28e05254b77424c8c7dc
# bad: [0c17270f9b920e4e1777488f1911bbfdaf2af3be] net: sysfs: Implement is_visible for phys_(port_id, port_name, switch_id)
git bisect bad 0c17270f9b920e4e1777488f1911bbfdaf2af3be
# good: [ffe8a49091767f71802a3c601c121aa0ab84ac5f] net: ti: icssg-prueth: Read firmware-names from device tree
git bisect good ffe8a49091767f71802a3c601c121aa0ab84ac5f
# first bad commit: [0c17270f9b920e4e1777488f1911bbfdaf2af3be] net: sysfs: Implement is_visible for phys_(port_id, port_name, switch_id)

[-- Attachment #3: dmesg_slice --]
[-- Type: text/plain, Size: 6832 bytes --]

[Oct 8 18:41] NET: Registered PF_VSOCK protocol family
[  +0.897142] lbr: port 1(veth07d00b7e) entered blocking state
[  +0.000008] lbr: port 1(veth07d00b7e) entered disabled state
[  +0.000005] veth07d00b7e: entered allmulticast mode
[  +0.000021] veth07d00b7e: entered promiscuous mode
[  +0.139625] ------------[ cut here ]------------
[  +0.000004] WARNING: CPU: 6 PID: 2938 at net/core/dev.c:12410 __dev_change_net_namespace+0xb89/0xc30
[  +0.000006] Modules linked in: veth vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock snd_seq_dummy snd_hrtimer snd_seq rfcomm bridge stp llc nft_masq nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 r8153_ecm cdc_ether usbnet nf_tables r8152 mii libphy mdio_bus cmac algif_hash algif_skcipher af_alg bnep vfat fat snd_sof_amd_acp70 snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_pci_ps snd_soc_acpi_amd_match snd_amd_sdw_acpi soundwire_amd soundwire_generic_allocation soundwire_bus snd_hda_codec_alc269 snd_soc_sdca snd_hda_scodec_component mt7921e snd_hda_codec_realtek_lib snd_soc_core hid_sensor_als mt7921_common snd_hda_codec_generic hid_sensor_trigger snd_compress snd_hda_codec_atihdmi mt792x_lib industrialio_triggered_buffer ac97_bus snd_hda_codec_hdmi kfifo_buf
[  +0.000034]  snd_pcm_dmaengine mt76_connac_lib intel_rapl_msr amd_atl hid_sensor_iio_common snd_rpl_pci_acp6x intel_rapl_common mt76 industrialio leds_cros_ec snd_hda_intel cros_usbpd_charger snd_acp_pci snd_amd_acpi_mach mac80211 cros_usbpd_notify btusb snd_usb_audio led_class_multicolor cros_charge_control cros_ec_chardev cros_ec_debugfs snd_acp_legacy_common cros_ec_hwmon cros_usbpd_logger cros_ec_sysfs cros_kbd_led_backlight gpio_cros_ec snd_hda_codec btrtl spd5118 snd_pci_acp6x snd_usbmidi_lib btintel kvm_amd snd_hda_core libarc4 snd_pci_acp5x ucsi_acpi snd_ump cros_ec_dev sp5100_tco snd_intel_dspcfg typec_ucsi snd_rn_pci_acp3x btbcm cfg80211 snd_rawmidi snd_intel_sdw_acpi kvm snd_acp_config cros_ec_lpcs typec btmtk snd_hwdep snd_seq_device cros_ec snd_soc_acpi roles i2c_piix4 hid_multitouch hid_sensor_hub cros_ec_proto bluetooth snd_pcm snd_pci_acp3x snd_timer irqbypass wmi_bmof rapl amdxdna pcspkr amd_pmf k10temp i2c_smbus mousedev rfkill thunderbolt joydev amdtee snd i2c_hid_acpi amd_sfh soundcore
[  +0.000053]  platform_profile i2c_hid amd_pmc mc mac_hid usbip_host usbip_core i2c_dev crypto_user loop nfnetlink zram 842_decompress 842_compress lz4hc_compress lz4_compress dm_crypt encrypted_keys trusted asn1_encoder tee amdgpu dm_mod amdxcp i2c_algo_bit drm_ttm_helper ttm drm_exec gpu_sched drm_suballoc_helper nvme drm_panel_backlight_quirks drm_buddy nvme_core polyval_clmulni ghash_clmulni_intel drm_display_helper nvme_keyring aesni_intel ccp nvme_auth cec video wmi serio_raw
[  +0.000020] CPU: 6 UID: 0 PID: 2938 Comm: incusd Not tainted 6.17.1-1-mainline #1 PREEMPT(full)  4b783b4a638669fb644857f484487d17cb45ed1f
[  +0.000002] Hardware name: Framework Laptop 13 (AMD Ryzen 7040Series)/FRANMDCP07, BIOS 03.07 02/19/2025
[  +0.000001] RIP: 0010:__dev_change_net_namespace+0xb89/0xc30
[  +0.000003] Code: 00 00 0f 85 74 f7 ff ff 48 8b 7c 24 20 e8 6f c5 2a 00 e9 65 f7 ff ff 31 f6 48 89 ef e8 d0 9d fe ff 89 44 24 18 e9 f9 f6 ff ff <0f> 0b e9 f0 f5 ff ff 0f 0b e9 82 fb ff ff 48 c7 c7 00 60 c5 bd e8
[  +0.000001] RSP: 0018:ffffd2cf4a997488 EFLAGS: 00010282
[  +0.000001] RAX: 00000000fffffffe RBX: ffff8a75cb245000 RCX: 0000000000000001
[  +0.000001] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8a75cb2455e8
[  +0.000001] RBP: ffff8a757f6f5d00 R08: ffff8a7541468140 R09: ffff8a7541468320
[  +0.000000] R10: 00000000ffffffff R11: ffffffffbcdf231b R12: ffff8a75cb2451a0
[  +0.000001] R13: ffff8a75cb245190 R14: ffff8a75cb2455b0 R15: ffff8a75cb2451b8
[  +0.000001] FS:  00007f1ad8ff96c0(0000) GS:ffff8a82e6689000(0000) knlGS:0000000000000000
[  +0.000001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000000] CR2: 000024dbeebc9000 CR3: 00000003ce468000 CR4: 0000000000f50ef0
[  +0.000001] PKRU: 55555554
[  +0.000001] Call Trace:
[  +0.000002]  <TASK>
[  +0.000004]  ? if6_seq_show+0x30/0x50
[  +0.000006]  do_setlink.isra.0+0xc7/0x1270
[  +0.000004]  ? __nla_validate_parse+0x5c/0xcc0
[  +0.000003]  ? security_capable+0x94/0x1a0
[  +0.000003]  rtnl_newlink+0x858/0xc20
[  +0.000002]  ? update_curr+0x8e/0x1c0
[  +0.000004]  ? update_entity_lag+0x71/0x80
[  +0.000002]  ? sched_balance_newidle+0x358/0x450
[  +0.000001]  ? psi_task_switch+0x113/0x2a0
[  +0.000002]  ? __pfx_rtnl_newlink+0x10/0x10
[  +0.000003]  rtnetlink_rcv_msg+0x346/0x3e0
[  +0.000002]  ? sched_clock+0x10/0x30
[  +0.000003]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
[  +0.000002]  netlink_rcv_skb+0x59/0x110
[  +0.000004]  netlink_unicast+0x285/0x3c0
[  +0.000002]  ? __alloc_skb+0xdb/0x1a0
[  +0.000003]  netlink_sendmsg+0x20d/0x430
[  +0.000002]  ____sys_sendmsg+0x39f/0x3d0
[  +0.000003]  ? import_iovec+0x2f/0x40
[  +0.000003]  ___sys_sendmsg+0x99/0xe0
[  +0.000003]  __sys_sendmsg+0x8a/0xf0
[  +0.000003]  do_syscall_64+0x81/0x970
[  +0.000003]  ? __sys_bind+0xe3/0x110
[  +0.000002]  ? syscall_exit_work+0x143/0x1b0
[  +0.000004]  ? do_syscall_64+0x244/0x970
[  +0.000003]  ? sock_alloc_file+0x63/0xc0
[  +0.000002]  ? syscall_exit_work+0x143/0x1b0
[  +0.000002]  ? do_syscall_64+0x244/0x970
[  +0.000002]  ? alloc_fd+0x12e/0x190
[  +0.000002]  ? put_unused_fd+0x2a/0x70
[  +0.000002]  ? do_sys_openat2+0xa2/0xe0
[  +0.000003]  ? syscall_exit_work+0x143/0x1b0
[  +0.000002]  ? do_syscall_64+0x244/0x970
[  +0.000002]  ? exc_page_fault+0x7e/0x1a0
[  +0.000002]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  +0.000003] RIP: 0033:0x7f1c2a89f042
[  +0.000030] Code: 08 0f 85 d1 40 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66
[  +0.000001] RSP: 002b:00007f1ad8fe77d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[  +0.000002] RAX: ffffffffffffffda RBX: 00007f1ad8fe7900 RCX: 00007f1c2a89f042
[  +0.000000] RDX: 0000000000004000 RSI: 00007f1ad8fe7870 RDI: 000000000000000c
[  +0.000001] RBP: 00007f1ad8fe7800 R08: 0000000000000000 R09: 0000000000000000
[  +0.000000] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1abc016570
[  +0.000001] R13: 00007f1abc006045 R14: 0000000000000000 R15: 0000000000000006
[  +0.000001]  </TASK>
[  +0.000001] ---[ end trace 0000000000000000 ]---
[  +0.000151] phys5V1v52: renamed from veth35b149fb
[  +0.002227] eth0: renamed from phys5V1v52
[  +0.000172] lbr: port 1(veth07d00b7e) entered blocking state
[  +0.000005] lbr: port 1(veth07d00b7e) entered forwarding state

Re: [REGRESSION] in 6.17, failing __dev_change_net_namespace+0xb89/0xc30

[Jakub Kicinski]

On Tue, 14 Oct 2025 10:04:43 +0000 Cynthia wrote:
> When I updated my machine to the newest kernel, a bug started to appear. 
> The system does not panic, but an error kept happening in dmesg.
> 
> The bug happens with LXC/Incus when it tries to start a new container. 
> (but probably other things are affected too)
> 
> 
> Steps to Reproduce: the bug can be reproduced in a libvirt VM, no need 
> for a specific system. Also I suspect the bug is also 
> architecture-independent, but I cannot verify that.
> 1) Install ArchLinux (all dependencies are available). I was testing 
> with vanilla kernel, so any linux distro should be affected.
> https://aur.archlinux.org/packages/linux-mainline can be installed, this 
> is the vanilla kernel with a generally good kernel config for most PCs.
> 2) Install LXC/Incus (pacman -S incus)
> 3) configure incus and start a container:
> usermod -v 1000000-1000999999 -w 1000000-1000999999 root &&
> incus admin init &&
> incus launch images:debian/12 first # start a container
> 4) Previous step should trigger incus to do namespaces. I'm not sure 
> what syscall is causing the bug, I do not have a mini C program. These 
> steps should be enough to see the log in the dmesg.
> 
> I also did a git bisect, the first commit to have this bug is this:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c17270f9b920e4e1777488f1911bbfdaf2af3be
> 
> I initially reported this bug on Bugzilla, but after seeing 6 year old 
> bugs there, I'm not sure if that platform is still in use.
> https://bugzilla.kernel.org/show_bug.cgi?id=220649
> 
> Since my initial report, 6.17.2 was released, the bug is still happening.
> 
> I'm attaching 2 files:
> dmesg_slice: the slice of dmesg containing the problematic frame (on a 
> bare-metal linux with AMD srso mitigation disabled)
> bisect_log: a log of the git bisect process

Thanks a lot for bisecting! Looking at the code my guess is that sysfs
gives us ENOENT when we try to change owner of a file that isn't
visible. Adding sysfs maintainers - should sysfs_group_attrs_change_owner() 
call is_visible before trying to touch the attr?

Re: [REGRESSION] in 6.17, failing __dev_change_net_namespace+0xb89/0xc30

[Greg Kroah-Hartman]

On Wed, Oct 15, 2025 at 01:31:20PM -0700, Jakub Kicinski wrote:
> On Tue, 14 Oct 2025 10:04:43 +0000 Cynthia wrote:
> > When I updated my machine to the newest kernel, a bug started to appear. 
> > The system does not panic, but an error kept happening in dmesg.
> > 
> > The bug happens with LXC/Incus when it tries to start a new container. 
> > (but probably other things are affected too)
> > 
> > 
> > Steps to Reproduce: the bug can be reproduced in a libvirt VM, no need 
> > for a specific system. Also I suspect the bug is also 
> > architecture-independent, but I cannot verify that.
> > 1) Install ArchLinux (all dependencies are available). I was testing 
> > with vanilla kernel, so any linux distro should be affected.
> > https://aur.archlinux.org/packages/linux-mainline can be installed, this 
> > is the vanilla kernel with a generally good kernel config for most PCs.
> > 2) Install LXC/Incus (pacman -S incus)
> > 3) configure incus and start a container:
> > usermod -v 1000000-1000999999 -w 1000000-1000999999 root &&
> > incus admin init &&
> > incus launch images:debian/12 first # start a container
> > 4) Previous step should trigger incus to do namespaces. I'm not sure 
> > what syscall is causing the bug, I do not have a mini C program. These 
> > steps should be enough to see the log in the dmesg.
> > 
> > I also did a git bisect, the first commit to have this bug is this:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c17270f9b920e4e1777488f1911bbfdaf2af3be
> > 
> > I initially reported this bug on Bugzilla, but after seeing 6 year old 
> > bugs there, I'm not sure if that platform is still in use.
> > https://bugzilla.kernel.org/show_bug.cgi?id=220649
> > 
> > Since my initial report, 6.17.2 was released, the bug is still happening.
> > 
> > I'm attaching 2 files:
> > dmesg_slice: the slice of dmesg containing the problematic frame (on a 
> > bare-metal linux with AMD srso mitigation disabled)
> > bisect_log: a log of the git bisect process
> 
> Thanks a lot for bisecting! Looking at the code my guess is that sysfs
> gives us ENOENT when we try to change owner of a file that isn't
> visible. Adding sysfs maintainers - should sysfs_group_attrs_change_owner() 
> call is_visible before trying to touch the attr?

Oh, I never considered that call-path, and given that I haven't seen a
bug report about this yet, it's pretty rare :)

So yes, that should be checked.  Can anyone knock up a patch for this?
I'm busy all today with other stuff, sorry.

greg k-h

Re: [REGRESSION] in 6.17, failing __dev_change_net_namespace+0xb89/0xc30

[Fernando Fernandez Mancera]

On 10/16/25 8:25 AM, Greg Kroah-Hartman wrote:
> On Wed, Oct 15, 2025 at 01:31:20PM -0700, Jakub Kicinski wrote:
>> On Tue, 14 Oct 2025 10:04:43 +0000 Cynthia wrote:
>>> When I updated my machine to the newest kernel, a bug started to appear.
>>> The system does not panic, but an error kept happening in dmesg.
>>>
>>> The bug happens with LXC/Incus when it tries to start a new container.
>>> (but probably other things are affected too)
>>>
>>>
>>> Steps to Reproduce: the bug can be reproduced in a libvirt VM, no need
>>> for a specific system. Also I suspect the bug is also
>>> architecture-independent, but I cannot verify that.
>>> 1) Install ArchLinux (all dependencies are available). I was testing
>>> with vanilla kernel, so any linux distro should be affected.
>>> https://aur.archlinux.org/packages/linux-mainline can be installed, this
>>> is the vanilla kernel with a generally good kernel config for most PCs.
>>> 2) Install LXC/Incus (pacman -S incus)
>>> 3) configure incus and start a container:
>>> usermod -v 1000000-1000999999 -w 1000000-1000999999 root &&
>>> incus admin init &&
>>> incus launch images:debian/12 first # start a container
>>> 4) Previous step should trigger incus to do namespaces. I'm not sure
>>> what syscall is causing the bug, I do not have a mini C program. These
>>> steps should be enough to see the log in the dmesg.
>>>
>>> I also did a git bisect, the first commit to have this bug is this:
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c17270f9b920e4e1777488f1911bbfdaf2af3be
>>>
>>> I initially reported this bug on Bugzilla, but after seeing 6 year old
>>> bugs there, I'm not sure if that platform is still in use.
>>> https://bugzilla.kernel.org/show_bug.cgi?id=220649
>>>
>>> Since my initial report, 6.17.2 was released, the bug is still happening.
>>>
>>> I'm attaching 2 files:
>>> dmesg_slice: the slice of dmesg containing the problematic frame (on a
>>> bare-metal linux with AMD srso mitigation disabled)
>>> bisect_log: a log of the git bisect process
>>
>> Thanks a lot for bisecting! Looking at the code my guess is that sysfs
>> gives us ENOENT when we try to change owner of a file that isn't
>> visible. Adding sysfs maintainers - should sysfs_group_attrs_change_owner()
>> call is_visible before trying to touch the attr?
> 
> Oh, I never considered that call-path, and given that I haven't seen a
> bug report about this yet, it's pretty rare :)
> 
> So yes, that should be checked.  Can anyone knock up a patch for this?
> I'm busy all today with other stuff, sorry.
> 
> greg k-h
> 

I can send a patch, I hit this yesterday too.

Thanks,
Fernando.