[REGRESSION] Apparmor deadlock in 6.12.101 in complain mode

[REGRESSION] Apparmor deadlock in 6.12.101 in complain mode

[Mattias Åsander]

Hello

With the 6.12.101 kernel on debian13 we suddenly got frequent deadlocks in apparmor for our Apache webservers.

It appears to be caused by the commit "apparmor: grab ns lock and refresh when looking up changehat child profiles"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=32e92764d6f8d251c1bca62be33793287b453a81

As of that commit the function change_hat() now takes the ns->lock mutex, and later with the lock held it calls a function that also takes the same namespace lock, causing a deadlock.

So, security/apparmor/domain.c:change_hat() calls security/apparmor/domain.c:build_change_hat(), and in complain-mode that calls security/apparmor/policy.c:aa_new_learning_profile(),
which also tries to takes the profile->ns->lock mutex.

###########
INFO: task /usr/sbin/apach:125024 blocked for more than 120 seconds.
      Not tainted 6.12.101+deb13-amd64 #1 Debian 6.12.101-1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:/usr/sbin/apach state:D stack:0     pid:125024 tgid:125024 ppid:2366   flags:0x00004006
Call Trace:
 <TASK>
 __schedule+0x505/0xc00
 schedule+0x27/0xc0
 schedule_preempt_disabled+0x15/0x30
 __mutex_lock.constprop.0+0x41b/0x720
 ? srso_alias_return_thunk+0x5/0xfbef5
 aa_new_learning_profile+0x149/0x1e0
 build_change_hat+0x28b/0x3a0
 change_hat.isra.0+0x60e/0xd90
 aa_change_hat+0x301/0x620
 aa_setprocattr_changehat+0xf8/0x230
 do_setattr+0x2d1/0x380
 proc_pid_attr_write+0xe0/0x150
 vfs_write+0xf5/0x440
 ? __pte_offset_map+0x1b/0x180
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? mutex_lock+0x12/0x30
 ? srso_alias_return_thunk+0x5/0xfbef5
 ksys_write+0x6d/0xf0
 do_syscall_64+0x87/0x1b0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? __count_memcg_events+0x53/0xf0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? count_memcg_events.constprop.0+0x1a/0x30
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? handle_mm_fault+0x1bb/0x2c0
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? do_user_addr_fault+0x36c/0x620
 ? srso_alias_return_thunk+0x5/0xfbef5
 ? arch_exit_to_user_mode_prepare.isra.0+0x16/0xa0
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f4719c279ee
RSP: 002b:00007ffdb983ec48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00007f4719b15c00 RCX: 00007f4719c279ee
RDX: 00000000000000d3 RSI: 00005610b4669ce0 RDI: 000000000000008c
RBP: 00005610b4669ce0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000000d3
R13: 00005610b4669ce0 R14: 00007f4719a8c660 R15: 00007f4719acd054
 </TASK>
###########


Some more background on how the issue gets triggered for us:
We run apache with MPM-prefork and apparmor has it in complain-mode, and a bunch of different vhosts configured with their own profiles in /etc/apparmor.d/apache2.d/,
everything is fine as long as you call a vhost with a matching apparmor profile, but calling it with a hostname that does not have a explicit profile (ex the ip-address) causes an immediate deadlock.
So with ex a vhost as "SiteName example.org":
curl http://example.org <- works, no issues
curl http://127.0.0.1 <- deadlock

A quick workaround for our issue in Apache is to define AADefaultHatName globally so that it avoids the aa_new_learning_profile code-path,
we have it set to "AADefaultHatName DEFAULT_URI" as that hat/profile was available by default in the shipped /etc/apparmor.d/usr.sbin.apache2 ruleset in debian13.

Best Regards
//Mattias

Re: [REGRESSION] Apparmor deadlock in 6.12.101 in complain mode

[John Johansen]

thanks for the report

This should be fixed by
   commit 4ec11f14d1d6f apparmor: fix deadlock in complain-mode change_hat

in apparmor-next, linux-next. It went into testing early last week, and got a small revision a couple of days last, and got pushed into linux-next late last week



On 8/12/26 01:03, Mattias Åsander wrote:
> Hello
> 
> With the 6.12.101 kernel on debian13 we suddenly got frequent deadlocks in apparmor for our Apache webservers.
> 
> It appears to be caused by the commit "apparmor: grab ns lock and refresh when looking up changehat child profiles"
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=32e92764d6f8d251c1bca62be33793287b453a81
> 
> As of that commit the function change_hat() now takes the ns->lock mutex, and later with the lock held it calls a function that also takes the same namespace lock, causing a deadlock.
> 
> So, security/apparmor/domain.c:change_hat() calls security/apparmor/domain.c:build_change_hat(), and in complain-mode that calls security/apparmor/policy.c:aa_new_learning_profile(),
> which also tries to takes the profile->ns->lock mutex.
> 
> ###########
> INFO: task /usr/sbin/apach:125024 blocked for more than 120 seconds.
>        Not tainted 6.12.101+deb13-amd64 #1 Debian 6.12.101-1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:/usr/sbin/apach state:D stack:0     pid:125024 tgid:125024 ppid:2366   flags:0x00004006
> Call Trace:
>   <TASK>
>   __schedule+0x505/0xc00
>   schedule+0x27/0xc0
>   schedule_preempt_disabled+0x15/0x30
>   __mutex_lock.constprop.0+0x41b/0x720
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   aa_new_learning_profile+0x149/0x1e0
>   build_change_hat+0x28b/0x3a0
>   change_hat.isra.0+0x60e/0xd90
>   aa_change_hat+0x301/0x620
>   aa_setprocattr_changehat+0xf8/0x230
>   do_setattr+0x2d1/0x380
>   proc_pid_attr_write+0xe0/0x150
>   vfs_write+0xf5/0x440
>   ? __pte_offset_map+0x1b/0x180
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? mutex_lock+0x12/0x30
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ksys_write+0x6d/0xf0
>   do_syscall_64+0x87/0x1b0
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? __count_memcg_events+0x53/0xf0
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? count_memcg_events.constprop.0+0x1a/0x30
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? handle_mm_fault+0x1bb/0x2c0
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? do_user_addr_fault+0x36c/0x620
>   ? srso_alias_return_thunk+0x5/0xfbef5
>   ? arch_exit_to_user_mode_prepare.isra.0+0x16/0xa0
>   entry_SYSCALL_64_after_hwframe+0x76/0x7e
> RIP: 0033:0x7f4719c279ee
> RSP: 002b:00007ffdb983ec48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 00007f4719b15c00 RCX: 00007f4719c279ee
> RDX: 00000000000000d3 RSI: 00005610b4669ce0 RDI: 000000000000008c
> RBP: 00005610b4669ce0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000000d3
> R13: 00005610b4669ce0 R14: 00007f4719a8c660 R15: 00007f4719acd054
>   </TASK>
> ###########
> 
> 
> Some more background on how the issue gets triggered for us:
> We run apache with MPM-prefork and apparmor has it in complain-mode, and a bunch of different vhosts configured with their own profiles in /etc/apparmor.d/apache2.d/,
> everything is fine as long as you call a vhost with a matching apparmor profile, but calling it with a hostname that does not have a explicit profile (ex the ip-address) causes an immediate deadlock.
> So with ex a vhost as "SiteName example.org":
> curl http://example.org <- works, no issues
> curl http://127.0.0.1 <- deadlock
> 
> A quick workaround for our issue in Apache is to define AADefaultHatName globally so that it avoids the aa_new_learning_profile code-path,
> we have it set to "AADefaultHatName DEFAULT_URI" as that hat/profile was available by default in the shipped /etc/apparmor.d/usr.sbin.apache2 ruleset in debian13.
> 
> Best Regards
> //Mattias