PR_SET_PTRACER can also EINVAL when Yama is not configured

PR_SET_PTRACER can also EINVAL when Yama is not configured

[Cesar Eduardo Barros]

The man page for PR_SET_PTRACER mentions three cases in which 
prctl(PR_SET_PTRACER, ...) can fail with EINVAL, but there's a fourth 
case: when CONFIG_SECURITY_YAMA=n

The code which deals with the three currently documented cases is at 
security/yama/yama_lsm.c which is only compiled when "Yama support" is 
configured. When it isn't, it falls through to the default case at 
kernel/sys.c which returns EINVAL.

(I was alerted to this at 
https://github.com/cesarb/count_instructions/pull/1 where I was told 
that it fails with EINVAL when the Yama LSM is not installed. I had 
written the code based on the man page, so I hadn't considered this 
fourth case.)

-- 
Cesar Eduardo Barros
cesarb@cesarb.eti.br

Re: PR_SET_PTRACER can also EINVAL when Yama is not configured

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

Hi Cesar,

> Date: 2026-11-07 07:26:32-0300
> From: Cesar Eduardo Barros <cesarb@cesarb.eti.br>
>
> The man page for PR_SET_PTRACER mentions three cases in which
> prctl(PR_SET_PTRACER, ...) can fail with EINVAL, but there's a fourth case:
> when CONFIG_SECURITY_YAMA=n
> 
> The code which deals with the three currently documented cases is at
> security/yama/yama_lsm.c which is only compiled when "Yama support" is
> configured. When it isn't, it falls through to the default case at
> kernel/sys.c which returns EINVAL.
> 
> (I was alerted to this at
> https://github.com/cesarb/count_instructions/pull/1 where I was told that it
> fails with EINVAL when the Yama LSM is not installed. I had written the code
> based on the man page, so I hadn't considered this fourth case.)

Thanks for the report!  Would you mind sending a patch?


Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]