[STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

Hello stable team,

Please consider backporting the following two commits to the stable trees:

  - b0c857049153680e1b6ccb59abf359a4ca523561 ("ftrace: deprecate
disabling via ftrace_enabled sysctl")
  - 88b3e7fedc07d940c32eb5c0733a780e944b6b47 ("selftests/livepatch:
update test-ftrace.sh for deprecated ftrace_enabled")

Both commits were merged into mainline in v7.3 via the 'ftrace-v7.3' tag.

This change addresses a long-standing issue: setting
kernel.ftrace_enabled=0 silently disables BPF trampolines
(fentry/fexit) and ftrace-based kprobes/kretprobes. The write succeeds
and the hooks stop firing without error, while re-enabling silently
restores them. Livepatch resolved this via FTRACE_OPS_FL_PERMANENT,
which blocks disabling ftrace while a permanent ops is registered and
blocks registration while ftrace is disabled. Trampolines had this
same protection from 2019 to 2022 while sharing a global direct_ops
marked permanent, but lost it during the 2022 per-trampoline-ops
split. ftrace based Kprobes never had this protection.

The breakage was introduced in commit 00963a2e75a8 ("bpf: Support
bpf_trampoline on functions with IPMODIFY (e.g. livepatch)").

With these commits, writing 0 is refused with -EOPNOTSUPP and a
one-time warning is logged; reads and writing 1 are unaffected. Steven
Rostedt requested and reviewed this change on the mailing list,
including the follow-up selftest update.

We would like to backport this to LTS branches starting with 6.1.y,
where the breaking commit already exists.

Mailing list threads for reference:

  - RFC v1: https://lore.kernel.org/bpf/20260730163544.2042327-1-andrey.grodzovsky@crowdstrike.com/
  - RFC v3 (final):
https://lore.kernel.org/bpf/20260806153000.4184871-1-andrey.grodzovsky@crowdstrike.com/

I can provide rebased backports for any trees where these commits do
not apply cleanly.

Thanks,
Andrey Grodzovsky

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Steven Rostedt]

On Thu, 20 Aug 2026 17:30:40 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> Hello stable team,
> 
> Please consider backporting the following two commits to the stable trees:
> 
>   - b0c857049153680e1b6ccb59abf359a4ca523561 ("ftrace: deprecate
> disabling via ftrace_enabled sysctl")
>   - 88b3e7fedc07d940c32eb5c0733a780e944b6b47 ("selftests/livepatch:
> update test-ftrace.sh for deprecated ftrace_enabled")
> 
> Both commits were merged into mainline in v7.3 via the 'ftrace-v7.3' tag.
> 
> This change addresses a long-standing issue: setting
> kernel.ftrace_enabled=0 silently disables BPF trampolines
> (fentry/fexit) and ftrace-based kprobes/kretprobes. The write succeeds
> and the hooks stop firing without error, while re-enabling silently
> restores them. Livepatch resolved this via FTRACE_OPS_FL_PERMANENT,
> which blocks disabling ftrace while a permanent ops is registered and
> blocks registration while ftrace is disabled. Trampolines had this
> same protection from 2019 to 2022 while sharing a global direct_ops
> marked permanent, but lost it during the 2022 per-trampoline-ops
> split. ftrace based Kprobes never had this protection.
> 
> The breakage was introduced in commit 00963a2e75a8 ("bpf: Support
> bpf_trampoline on functions with IPMODIFY (e.g. livepatch)").
> 
> With these commits, writing 0 is refused with -EOPNOTSUPP and a
> one-time warning is logged; reads and writing 1 are unaffected. Steven
> Rostedt requested and reviewed this change on the mailing list,
> including the follow-up selftest update.
> 
> We would like to backport this to LTS branches starting with 6.1.y,
> where the breaking commit already exists.
> 
> Mailing list threads for reference:
> 
>   - RFC v1: https://lore.kernel.org/bpf/20260730163544.2042327-1-andrey.grodzovsky@crowdstrike.com/
>   - RFC v3 (final):
> https://lore.kernel.org/bpf/20260806153000.4184871-1-andrey.grodzovsky@crowdstrike.com/
> 
> I can provide rebased backports for any trees where these commits do
> not apply cleanly.

I'm really nervous about backporting patches that may be reverted
upstream. These change user space API and if anything depends on the
old behavior, it will be reverted. That would also mean stable would
need to revert it too.

Instead, please post the first two patches of your original series:

  https://lore.kernel.org/all/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com/

I rather push these upstream even if they are not really needed. But
those are much safer to backport because it "fixes" the problem even if
the main fix gets reverted.

-- Steve

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

On Thu, Aug 20, 2026 at 6:20 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 20 Aug 2026 17:30:40 -0400
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > Hello stable team,
> >
> > Please consider backporting the following two commits to the stable trees:
> >
> >   - b0c857049153680e1b6ccb59abf359a4ca523561 ("ftrace: deprecate
> > disabling via ftrace_enabled sysctl")
> >   - 88b3e7fedc07d940c32eb5c0733a780e944b6b47 ("selftests/livepatch:
> > update https://urldefense.com/v3/__http://test-ftrace.sh__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOzKIOK6Jg$  for deprecated ftrace_enabled")
> >
> > Both commits were merged into mainline in v7.3 via the 'ftrace-v7.3' tag.
> >
> > This change addresses a long-standing issue: setting
> > kernel.ftrace_enabled=0 silently disables BPF trampolines
> > (fentry/fexit) and ftrace-based kprobes/kretprobes. The write succeeds
> > and the hooks stop firing without error, while re-enabling silently
> > restores them. Livepatch resolved this via FTRACE_OPS_FL_PERMANENT,
> > which blocks disabling ftrace while a permanent ops is registered and
> > blocks registration while ftrace is disabled. Trampolines had this
> > same protection from 2019 to 2022 while sharing a global direct_ops
> > marked permanent, but lost it during the 2022 per-trampoline-ops
> > split. ftrace based Kprobes never had this protection.
> >
> > The breakage was introduced in commit 00963a2e75a8 ("bpf: Support
> > bpf_trampoline on functions with IPMODIFY (e.g. livepatch)").
> >
> > With these commits, writing 0 is refused with -EOPNOTSUPP and a
> > one-time warning is logged; reads and writing 1 are unaffected. Steven
> > Rostedt requested and reviewed this change on the mailing list,
> > including the follow-up selftest update.
> >
> > We would like to backport this to LTS branches starting with 6.1.y,
> > where the breaking commit already exists.
> >
> > Mailing list threads for reference:
> >
> >   - RFC v1: https://urldefense.com/v3/__https://lore.kernel.org/bpf/20260730163544.2042327-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOzV4jPY7w$
> >   - RFC v3 (final):
> > https://urldefense.com/v3/__https://lore.kernel.org/bpf/20260806153000.4184871-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOxUSL-qOg$
> >
> > I can provide rebased backports for any trees where these commits do
> > not apply cleanly.
>
> I'm really nervous about backporting patches that may be reverted
> upstream. These change user space API and if anything depends on the
> old behavior, it will be reverted. That would also mean stable would
> need to revert it too.
>
> Instead, please post the first two patches of your original series:
>
>   https://urldefense.com/v3/__https://lore.kernel.org/all/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOyr231hEA$
>
> I rather push these upstream even if they are not really needed. But
> those are much safer to backport because it "fixes" the problem even if
> the main fix gets reverted.
>
> -- Steve

I see, ok, I will add this to my TODO list.

Thanks,
Andrey

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

On Thu, Aug 20, 2026 at 6:20 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 20 Aug 2026 17:30:40 -0400
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > Hello stable team,
> >
> > Please consider backporting the following two commits to the stable trees:
> >
> >   - b0c857049153680e1b6ccb59abf359a4ca523561 ("ftrace: deprecate
> > disabling via ftrace_enabled sysctl")
> >   - 88b3e7fedc07d940c32eb5c0733a780e944b6b47 ("selftests/livepatch:
> > update https://urldefense.com/v3/__http://test-ftrace.sh__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOzKIOK6Jg$  for deprecated ftrace_enabled")
> >
> > Both commits were merged into mainline in v7.3 via the 'ftrace-v7.3' tag.
> >
> > This change addresses a long-standing issue: setting
> > kernel.ftrace_enabled=0 silently disables BPF trampolines
> > (fentry/fexit) and ftrace-based kprobes/kretprobes. The write succeeds
> > and the hooks stop firing without error, while re-enabling silently
> > restores them. Livepatch resolved this via FTRACE_OPS_FL_PERMANENT,
> > which blocks disabling ftrace while a permanent ops is registered and
> > blocks registration while ftrace is disabled. Trampolines had this
> > same protection from 2019 to 2022 while sharing a global direct_ops
> > marked permanent, but lost it during the 2022 per-trampoline-ops
> > split. ftrace based Kprobes never had this protection.
> >
> > The breakage was introduced in commit 00963a2e75a8 ("bpf: Support
> > bpf_trampoline on functions with IPMODIFY (e.g. livepatch)").
> >
> > With these commits, writing 0 is refused with -EOPNOTSUPP and a
> > one-time warning is logged; reads and writing 1 are unaffected. Steven
> > Rostedt requested and reviewed this change on the mailing list,
> > including the follow-up selftest update.
> >
> > We would like to backport this to LTS branches starting with 6.1.y,
> > where the breaking commit already exists.
> >
> > Mailing list threads for reference:
> >
> >   - RFC v1: https://urldefense.com/v3/__https://lore.kernel.org/bpf/20260730163544.2042327-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOzV4jPY7w$
> >   - RFC v3 (final):
> > https://urldefense.com/v3/__https://lore.kernel.org/bpf/20260806153000.4184871-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOxUSL-qOg$
> >
> > I can provide rebased backports for any trees where these commits do
> > not apply cleanly.
>
> I'm really nervous about backporting patches that may be reverted
> upstream. These change user space API and if anything depends on the
> old behavior, it will be reverted. That would also mean stable would
> need to revert it too.
>
> Instead, please post the first two patches of your original series:
>
>   https://urldefense.com/v3/__https://lore.kernel.org/all/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOyr231hEA$
>
> I rather push these upstream even if they are not really needed. But
> those are much safer to backport because it "fixes" the problem even if
> the main fix gets reverted.
>
> -- Steve

Hi Steve, coming back to this topic (sorry it took me a while - other
work related stuff kept me busy). Are you still willing to accept  [1]
 upstream ? If yes, can you also please clarify why only the first 2
patches and not the third, selftest patch?

Andrey

[1] - https://urldefense.com/v3/__https://lore.kernel.org/all/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com/__;!!BmdzS3_lV9HdKG8!1ja9D51Ex2VTTMdOEl4Eb-3ORL8CXBX1biYMPMLmT1iP3r8I8tfDE41DOth6sc54E3ODLfRqYSaIdTR60lcLWOyr231hEA$

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Steven Rostedt]

On Tue, 3 Nov 2026 21:51:42 -0500
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> Hi Steve, coming back to this topic (sorry it took me a while - other
> work related stuff kept me busy). Are you still willing to accept  [1]
>  upstream ? If yes, can you also please clarify why only the first 2
> patches and not the third, selftest patch?

I was thinking for backporting purposes. The selftest patch is fine, but
not something that needs to be backported.

-- Steve

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

On Wed, Nov 4, 2026 at 8:39 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 3 Nov 2026 21:51:42 -0500
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > Hi Steve, coming back to this topic (sorry it took me a while - other
> > work related stuff kept me busy). Are you still willing to accept  [1]
> >  upstream ? If yes, can you also please clarify why only the first 2
> > patches and not the third, selftest patch?
>
> I was thinking for backporting purposes. The selftest patch is fine, but
> not something that needs to be backported.
>
> -- Steve

Ok, so just to clear my own confusion: do we want to directly present
this to stable as a backport fixing a bug, or go through merge to
mainline first and then only backport? If we do want to take it into
mainline then I think the selftest patch is beneficial especially
because Jiri found some case that breaks there that needs furhter
invistigation [1].

Andrey

[1] - https://lore.kernel.org/all/amn-vbX6AitcIcC0@krava/

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Greg KH]

On Wed, Nov 04, 2026 at 10:43:37AM -0500, Andrey Grodzovsky wrote:
> On Wed, Nov 4, 2026 at 8:39 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Tue, 3 Nov 2026 21:51:42 -0500
> > Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> >
> > > Hi Steve, coming back to this topic (sorry it took me a while - other
> > > work related stuff kept me busy). Are you still willing to accept  [1]
> > >  upstream ? If yes, can you also please clarify why only the first 2
> > > patches and not the third, selftest patch?
> >
> > I was thinking for backporting purposes. The selftest patch is fine, but
> > not something that needs to be backported.
> >
> > -- Steve
> 
> Ok, so just to clear my own confusion: do we want to directly present
> this to stable as a backport fixing a bug, or go through merge to
> mainline first and then only backport? If we do want to take it into
> mainline then I think the selftest patch is beneficial especially
> because Jiri found some case that breaks there that needs furhter
> invistigation [1].

We can not take patches in stable that are not upstream first unless
there is a VERY good reason why that can not happen.

thanks,

greg k-h

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Steven Rostedt]

On Wed, 4 Nov 2026 10:43:37 -0500
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> Ok, so just to clear my own confusion: do we want to directly present
> this to stable as a backport fixing a bug, or go through merge to
> mainline first and then only backport? If we do want to take it into
> mainline then I think the selftest patch is beneficial especially
> because Jiri found some case that breaks there that needs furhter
> invistigation [1].

It has to go to mainline before going to stable. I'm fine either way for
the selftest as that's really up to the bpf maintainers.

-- Steve

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

On Wed, Nov 4, 2026 at 12:30 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 4 Nov 2026 10:43:37 -0500
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > Ok, so just to clear my own confusion: do we want to directly present
> > this to stable as a backport fixing a bug, or go through merge to
> > mainline first and then only backport? If we do want to take it into
> > mainline then I think the selftest patch is beneficial especially
> > because Jiri found some case that breaks there that needs furhter
> > invistigation [1].
>
> It has to go to mainline before going to stable. I'm fine either way for
> the selftest as that's really up to the bpf maintainers.
>
> -- Steve

I will work on the failure Jiri reported and then will submit a V2 of
the original.

Thanks,
Andrey

Re: [STABLE REQUEST] ftrace: deprecate the ftrace_enabled disable switch

[Andrey Grodzovsky]

On Wed, Nov 4, 2026 at 12:32 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
>
> On Wed, Nov 4, 2026 at 12:30 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Wed, 4 Nov 2026 10:43:37 -0500
> > Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> >
> > > Ok, so just to clear my own confusion: do we want to directly present
> > > this to stable as a backport fixing a bug, or go through merge to
> > > mainline first and then only backport? If we do want to take it into
> > > mainline then I think the selftest patch is beneficial especially
> > > because Jiri found some case that breaks there that needs furhter
> > > invistigation [1].
> >
> > It has to go to mainline before going to stable. I'm fine either way for
> > the selftest as that's really up to the bpf maintainers.
> >
> > -- Steve
>
> I will work on the failure Jiri reported and then will submit a V2 of
> the original.
>
> Thanks,
> Andrey

I reproduced and that a simple test corner case bug but it's all
irrelevant anyway. The tests basically attach fentry, fexit, kprobe,
and kretprobe programs in turn and confirm kernel.ftrace_enabled=0 is
refused with EBUSY and vice versa. Since mainline has [1], we cannot
test this for mainline kernel anyway and so the first 2 patches is
exactly what's needed both for mainline and for backport to stable.

Andrey

[1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b0c857049153680e1b6ccb59abf359a4ca523561