[REGRESSION] apparmor: AF_UNIX datagram send slowdown after 6456ccbd2ff7

[REGRESSION] apparmor: AF_UNIX datagram send slowdown after 6456ccbd2ff7

[Chengfeng Lin]

Hi John and Georgia,

I found a repeatable AF_UNIX datagram send slowdown caused by the exact source
delta of 6456ccbd2ff7 ("apparmor: fix regression in fs based unix sockets when
using old abi"). In a bare-metal parent/child/parent test, a standalone
sendmmsg() workload was 15.30% slower after the change. An independent larger
runner reproduced the slowdown with both sendmmsg() and io_uring SEND, so this
is not specific to io_uring.

#regzbot introduced: 6456ccbd2ff7
#regzbot title: apparmor AF_UNIX datagram send slowdown

The standalone workload uses socketpair(AF_UNIX, SOCK_DGRAM). Each sendmmsg()
call sends 32 messages of 128 bytes. Peer draining and payload validation are
outside the timed region. Each point had 3 warm-up rounds followed by 15
measured rounds of 65,536 messages.

Because 6456ccbd2ff7 was part of an AppArmor series, I isolated its source
delta on one controlled baseline:

  parent 0bfa1c2da7a8: AppArmor topic prefix through 50d56a1a366a
  child  30cb02a874b4: parent plus only the exact two-file 6456ccbd2ff7 diff

The child tree matches the same baseline with the AppArmor prefix advanced
through 6456ccbd2ff7. I used a fresh boot for each point:

  point       mean ns/message
  parent A          343.382
  child             395.831
  parent B          343.259

The child was 15.295% slower than the parent midpoint. Dropping the first
measured round gave 15.289%; parent drift was -0.036%, and the maximum CV was
0.137%. All 45 measured rows passed the payload, length, count, and empty-queue
checks.

The machine was an Intel Core i7-12700KF system with 32 GiB RAM. The workload
was pinned to P-core CPU 2, with governor and EPP set to performance and Turbo
disabled. All three runs used actual preempt=none and reported
/proc/self/attr/current as unconfined.

As a separate check, the larger original runner used matched actual
preempt=full. Its direct sendmmsg() profile was 14.841% slower, and its
io_uring SEND profile was 17.953% slower. I am keeping these results separate
from the standalone percentage.

A separate v6.16.12 -> v6.17.13 perf comparison was directionally consistent.
The children overhead reported for security_unix_may_send was 0.27% in both
v6.16.12 controls and 6.77% at v6.17.13. The latter included the path through
apparmor_unix_may_send(), aa_unix_peer_perm(), and unix_peer_perm(). This was
a release-endpoint comparison rather than the exact source pair above, so I
use it only as supporting path evidence, not as an internal decomposition of
the 15.295% slowdown.

I understand that 6456ccbd2ff7 fixes a real old AppArmor policy ABI issue,
so I am not suggesting a revert. Would it be possible to retain that fix while
avoiding the added cost on this unconfined AF_UNIX datagram send path?

Evidence bundle:

  https://github.com/lcf0399/linux-regression-evidence/tree/7b9862c7ac2d5f9622725912d7f0df95011bcbfd/apparmor-af-unix-send-old-abi-6456cc

Thanks,
Chengfeng

Re: [REGRESSION] apparmor: AF_UNIX datagram send slowdown after 6456ccbd2ff7

[John Johansen]

On 8/5/26 10:33, Chengfeng Lin wrote:
> Hi John and Georgia,
> 
> I found a repeatable AF_UNIX datagram send slowdown caused by the exact source
> delta of 6456ccbd2ff7 ("apparmor: fix regression in fs based unix sockets when
> using old abi"). In a bare-metal parent/child/parent test, a standalone
> sendmmsg() workload was 15.30% slower after the change. An independent larger
> runner reproduced the slowdown with both sendmmsg() and io_uring SEND, so this
> is not specific to io_uring.
> 
> #regzbot introduced: 6456ccbd2ff7
> #regzbot title: apparmor AF_UNIX datagram send slowdown
> 
> The standalone workload uses socketpair(AF_UNIX, SOCK_DGRAM). Each sendmmsg()
> call sends 32 messages of 128 bytes. Peer draining and payload validation are
> outside the timed region. Each point had 3 warm-up rounds followed by 15
> measured rounds of 65,536 messages.
> 
> Because 6456ccbd2ff7 was part of an AppArmor series, I isolated its source
> delta on one controlled baseline:
> 
>    parent 0bfa1c2da7a8: AppArmor topic prefix through 50d56a1a366a
>    child  30cb02a874b4: parent plus only the exact two-file 6456ccbd2ff7 diff
> 
> The child tree matches the same baseline with the AppArmor prefix advanced
> through 6456ccbd2ff7. I used a fresh boot for each point:
> 
>    point       mean ns/message
>    parent A          343.382
>    child             395.831
>    parent B          343.259
> 
> The child was 15.295% slower than the parent midpoint. Dropping the first
> measured round gave 15.289%; parent drift was -0.036%, and the maximum CV was
> 0.137%. All 45 measured rows passed the payload, length, count, and empty-queue
> checks.
> 
> The machine was an Intel Core i7-12700KF system with 32 GiB RAM. The workload
> was pinned to P-core CPU 2, with governor and EPP set to performance and Turbo
> disabled. All three runs used actual preempt=none and reported
> /proc/self/attr/current as unconfined.
> 
> As a separate check, the larger original runner used matched actual
> preempt=full. Its direct sendmmsg() profile was 14.841% slower, and its
> io_uring SEND profile was 17.953% slower. I am keeping these results separate
> from the standalone percentage.
> 
> A separate v6.16.12 -> v6.17.13 perf comparison was directionally consistent.
> The children overhead reported for security_unix_may_send was 0.27% in both
> v6.16.12 controls and 6.77% at v6.17.13. The latter included the path through
> apparmor_unix_may_send(), aa_unix_peer_perm(), and unix_peer_perm(). This was
> a release-endpoint comparison rather than the exact source pair above, so I
> use it only as supporting path evidence, not as an internal decomposition of
> the 15.295% slowdown.
> 
> I understand that 6456ccbd2ff7 fixes a real old AppArmor policy ABI issue,
> so I am not suggesting a revert. Would it be possible to retain that fix while
> avoiding the added cost on this unconfined AF_UNIX datagram send path?
> 

thanks for the report, we will have to investigate and get back to you on this,
generally speaking we try to keep the mediation overhead for unconfined tasks
to a minimum using early bailout and some caching. The referenced commit does
look to be doing early bailout, so we are going to have to do some analysis, on
where exactly the overhead is coming from.

I can't promise we can get it back to what it was before but we should be able
to do better than the current regression.




> Evidence bundle:
> 
>    https://github.com/lcf0399/linux-regression-evidence/tree/7b9862c7ac2d5f9622725912d7f0df95011bcbfd/apparmor-af-unix-send-old-abi-6456cc
> 
> Thanks,
> Chengfeng