[REGRESSION] Apparmor deadlock in 6.12.101 in complain mode

[REGRESSION] Apparmor deadlock in 6.12.101 in complain mode

[Mattias Åsander]

Hello

With the 6.12.101 kernel on debian13 we suddenly got frequent deadlocks in apparmor for our Apache webservers.

It appears to be caused by the commit "apparmor: grab ns lock and refresh when looking up changehat child profiles"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=32e92764d6f8d251c1bca62be33793287b453a81

As of that commit the function change_hat() now takes the ns->lock mutex, and later with the lock held it calls a function that also takes the same namespace lock, causing a deadlock.

So, security/apparmor/domain.c:change_hat() calls security/apparmor/domain.c:build_change_hat(), and in complain-mode that calls security/apparmor/policy.c:aa_new_learning_profile(),
which also tries to takes the profile->ns->lock mutex.

###########
INFO: task /usr/sbin/apach:125024 blocked for more than 120 seconds.
      Not tainted 6.12.101+deb13-amd64 #1 Debian 6.12.101-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:/usr/sbin/apach state:D stack:0     pid:125024 tgid:125024 ppid:2366   flags:0x00004006
Call Trace:
 <TASK>
 __schedule+0x505/0xc00
 schedule+0x27/0xc0
 schedule_preempt_disabled+0x15/0x30
 __mutex_lock.constprop.0+0x41b/0x720
 ? srso_alias_return_thunk+0x5/0xfbef5
 aa_new_learning_profile+0x149/0x1e0
 build_change_hat+0x28b/0x3a0
 change_hat.isra.0+0x60e/0xd90
 aa_change_hat+0x301/0x620
 aa_setprocattr_changehat+0xf8/0x230
 do_setattr+0x2d1/0x380
 proc_pid_attr_write+0xe0/0x150
 vfs_write+0xf5/0x440
 ? __pte_offset_map+0x1b/0x180
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? mutex_lock+0x12/0x30
 ? srso_alias_return_thunk+0x5/0xfbef5
 ksys_write+0x6d/0xf0
 do_syscall_64+0x87/0x1b0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? __count_memcg_events+0x53/0xf0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? count_memcg_events.constprop.0+0x1a/0x30
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? handle_mm_fault+0x1bb/0x2c0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? do_user_addr_fault+0x36c/0x620
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? arch_exit_to_user_mode_prepare.isra.0+0x16/0xa0
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f4719c279ee
RSP: 002b:00007ffdb983ec48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00007f4719b15c00 RCX: 00007f4719c279ee
RDX: 00000000000000d3 RSI: 00005610b4669ce0 RDI: 000000000000008c
RBP: 00005610b4669ce0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000000d3
R13: 00005610b4669ce0 R14: 00007f4719a8c660 R15: 00007f4719acd054
 </TASK>
###########


Some more background on how the issue gets triggered for us:
We run apache with MPM-prefork and apparmor has it in complain-mode, and a bunch of different vhosts configured with their own profiles in /etc/apparmor.d/apache2.d/,
everything is fine as long as you call a vhost with a matching apparmor profile, but calling it with a hostname that does not have a explicit profile (ex the ip-address) causes an immediate deadlock.
So with ex a vhost as "SiteName example.org":
curl http://example.org <- works, no issues
curl http://127.0.0.1 <- deadlock

A quick workaround for our issue in Apache is to define AADefaultHatName globally so that it avoids the aa_new_learning_profile code-path,
we have it set to "AADefaultHatName DEFAULT_URI" as that hat/profile was available by default in the shipped /etc/apparmor.d/usr.sbin.apache2 ruleset in debian13.

Best Regards
//Mattias