KFENCE: included in x86 defconfig?

KFENCE: included in x86 defconfig?

[Matthieu Baerts]

Hi Alexander, Marco, Dmitry,

I hope you are well!

First, thank you for your nice work with KFENCE!

When talking to Jakub about the kernel config used by the new CI for the
net tree [1], Jakub suggested [2] to check if KFENCE could not be
enabled by default for x86 architecture.

As KFENCE maintainers, what do you think about that? Do you see some
blocking points? Do you plan to add it in x86_64_defconfig?

[1] https://netdev.bots.linux.dev/status.html
[2] https://lore.kernel.org/netdev/20240207072159.33198b36@kernel.org/

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: KFENCE: included in x86 defconfig?

[Marco Elver]

[Cc'ing a bunch more people to get input]

Hi Matt,

On Wed, 7 Feb 2024 at 17:16, Matthieu Baerts <matttbe@kernel.org> wrote:
[...]
> When talking to Jakub about the kernel config used by the new CI for the
> net tree [1], Jakub suggested [2] to check if KFENCE could not be
> enabled by default for x86 architecture.
>
> As KFENCE maintainers, what do you think about that? Do you see some
> blocking points? Do you plan to add it in x86_64_defconfig?

We have no concrete plans to add it to x86 defconfig. I don't think
there'd be anything wrong with that from a technical point of view,
but I think defconfig should remain relatively minimal.

I guess different groups of people will disagree here: as kernel
maintainers, it'd be a good thing because we get more coverage and
higher probability of catching memory-safety bugs; as a user, I think
having defconfig enable KFENCE seems unintuitive.

I think this would belong into some "hardening" config - while KFENCE
is not a mitigation (due to sampling) it has the performance
characteristics of unintrusive hardening techniques, so I think it
would be a good fit. I think that'd be
"kernel/configs/hardening.config".

Preferences?

Thanks,
-- Marco

Re: KFENCE: included in x86 defconfig?

[Borislav Petkov]

On Wed, Feb 07, 2024 at 07:05:31PM +0100, Marco Elver wrote:
> I think this would belong into some "hardening" config - while KFENCE
> is not a mitigation (due to sampling) it has the performance
> characteristics of unintrusive hardening techniques, so I think it
> would be a good fit. I think that'd be
> "kernel/configs/hardening.config".

Instead of doing a special config for all the parties out there, why
don't parties simply automate their testing efforts by merging config
snippets into the default configs using

scripts/kconfig/merge_config.sh

before they run their specialized tests?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: KFENCE: included in x86 defconfig?

[Matthieu Baerts]

Hi Marco,

Thank you for your reply!

On 07/02/2024 19:05, Marco Elver wrote:
> [Cc'ing a bunch more people to get input]
> 
> Hi Matt,
> 
> On Wed, 7 Feb 2024 at 17:16, Matthieu Baerts <matttbe@kernel.org> wrote:
> [...]
>> When talking to Jakub about the kernel config used by the new CI for the
>> net tree [1], Jakub suggested [2] to check if KFENCE could not be
>> enabled by default for x86 architecture.
>>
>> As KFENCE maintainers, what do you think about that? Do you see some
>> blocking points? Do you plan to add it in x86_64_defconfig?
> 
> We have no concrete plans to add it to x86 defconfig. I don't think
> there'd be anything wrong with that from a technical point of view,
> but I think defconfig should remain relatively minimal.
> 
> I guess different groups of people will disagree here: as kernel
> maintainers, it'd be a good thing because we get more coverage and
> higher probability of catching memory-safety bugs; as a user, I think
> having defconfig enable KFENCE seems unintuitive.

Thank you for having shared your point of view. I agree with you, the
x86_64_defconfig is probably not the right place.

> I think this would belong into some "hardening" config - while KFENCE
> is not a mitigation (due to sampling) it has the performance
> characteristics of unintrusive hardening techniques, so I think it
> would be a good fit. I think that'd be
> "kernel/configs/hardening.config".
> 
> Preferences?
I didn't think about the hardening kconfig. It seems to make sense!

I will wait for people from the Linux Hardening ML to comment if that's
OK :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: KFENCE: included in x86 defconfig?

[Matthieu Baerts]

Hi Boris,

Thank you for your reply.

On 07/02/2024 19:16, Borislav Petkov wrote:
> On Wed, Feb 07, 2024 at 07:05:31PM +0100, Marco Elver wrote:
>> I think this would belong into some "hardening" config - while KFENCE
>> is not a mitigation (due to sampling) it has the performance
>> characteristics of unintrusive hardening techniques, so I think it
>> would be a good fit. I think that'd be
>> "kernel/configs/hardening.config".
> 
> Instead of doing a special config for all the parties out there, why
> don't parties simply automate their testing efforts by merging config
> snippets into the default configs using
> 
> scripts/kconfig/merge_config.sh
> 
> before they run their specialized tests?

Sorry, I'm sure I understand your suggestion: do you mean not including
KFENCE in hardening.config either, but in another one?

For the networking tests, we are already merging .config files, e.g. the
debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
can be elsewhere, and we don't mind merging other .config files if they
are maintained.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: KFENCE: included in x86 defconfig?

[Borislav Petkov]

On Wed, Feb 07, 2024 at 07:35:53PM +0100, Matthieu Baerts wrote:
> Sorry, I'm sure I understand your suggestion: do you mean not including
> KFENCE in hardening.config either, but in another one?
> 
> For the networking tests, we are already merging .config files, e.g. the
> debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
> can be elsewhere, and we don't mind merging other .config files if they
> are maintained.

Well, depends on where should KFENCE be enabled? Do you want people to
run their tests with it too, or only the networking tests? If so, then
hardening.config probably makes sense. 

Judging by what Documentation/dev-tools/kfence.rst says:

"KFENCE is designed to be enabled in production kernels, and has near zero
performance overhead."

this reads like it should be enabled *everywhere* - not only in some
hardening config.

But then again I've never played with it so I don't really know.

If only the networking tests should enable it, then it should be a local
.config snippet which is not part of the kernel.

Makes more sense?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: KFENCE: included in x86 defconfig?

[Matthieu Baerts]

On 07/02/2024 20:04, Borislav Petkov wrote:
> On Wed, Feb 07, 2024 at 07:35:53PM +0100, Matthieu Baerts wrote:
>> Sorry, I'm sure I understand your suggestion: do you mean not including
>> KFENCE in hardening.config either, but in another one?
>>
>> For the networking tests, we are already merging .config files, e.g. the
>> debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
>> can be elsewhere, and we don't mind merging other .config files if they
>> are maintained.
> 
> Well, depends on where should KFENCE be enabled? Do you want people to
> run their tests with it too, or only the networking tests? If so, then
> hardening.config probably makes sense. 
> 
> Judging by what Documentation/dev-tools/kfence.rst says:
> 
> "KFENCE is designed to be enabled in production kernels, and has near zero
> performance overhead."
> 
> this reads like it should be enabled *everywhere* - not only in some
> hardening config.
> 
> But then again I've never played with it so I don't really know.
> 
> If only the networking tests should enable it, then it should be a local
> .config snippet which is not part of the kernel.
> 
> Makes more sense?

Yes, thank you!

On my side, KFENCE is currently in local .config snippet, not part of
the kernel. If it has near zero performance overhead and can be used in
productions kernel, maybe it can be set elsewhere to be used by more
people? But not everywhere, according to Marco.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: KFENCE: included in x86 defconfig?

[Marco Elver]

On Wed, 7 Feb 2024 at 23:12, Matthieu Baerts <matttbe@kernel.org> wrote:
>
> On 07/02/2024 20:04, Borislav Petkov wrote:
> > On Wed, Feb 07, 2024 at 07:35:53PM +0100, Matthieu Baerts wrote:
> >> Sorry, I'm sure I understand your suggestion: do you mean not including
> >> KFENCE in hardening.config either, but in another one?
> >>
> >> For the networking tests, we are already merging .config files, e.g. the
> >> debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
> >> can be elsewhere, and we don't mind merging other .config files if they
> >> are maintained.
> >
> > Well, depends on where should KFENCE be enabled? Do you want people to
> > run their tests with it too, or only the networking tests? If so, then
> > hardening.config probably makes sense.
> >
> > Judging by what Documentation/dev-tools/kfence.rst says:
> >
> > "KFENCE is designed to be enabled in production kernels, and has near zero
> > performance overhead."
> >
> > this reads like it should be enabled *everywhere* - not only in some
> > hardening config.
> >
> > But then again I've never played with it so I don't really know.
> >
> > If only the networking tests should enable it, then it should be a local
> > .config snippet which is not part of the kernel.
> >
> > Makes more sense?
>
> Yes, thank you!
>
> On my side, KFENCE is currently in local .config snippet, not part of
> the kernel. If it has near zero performance overhead and can be used in
> productions kernel, maybe it can be set elsewhere to be used by more
> people? But not everywhere, according to Marco.

At the moment we still think this decision is to be made by the
distribution, system administrator, or whoever decides on kernel
config. I'm aware that several major Linux distributions enable KFENCE
in their kernels. The tool was designed for in-production use - we use
it in production [1] - but I'm not sure we can and should make this
decision for _every_ production kernel. The hardening config seems
like a good place, and I've put that on the TODO list.

Thanks,
-- Marco

[1] https://arxiv.org/abs/2311.09394 (see Linux section)

Re: KFENCE: included in x86 defconfig?

[Jakub Kicinski]

On Wed, 7 Feb 2024 20:04:44 +0100 Borislav Petkov wrote:
> On Wed, Feb 07, 2024 at 07:35:53PM +0100, Matthieu Baerts wrote:
> > Sorry, I'm sure I understand your suggestion: do you mean not including
> > KFENCE in hardening.config either, but in another one?
> > 
> > For the networking tests, we are already merging .config files, e.g. the
> > debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
> > can be elsewhere, and we don't mind merging other .config files if they
> > are maintained.  
> 
> Well, depends on where should KFENCE be enabled? Do you want people to
> run their tests with it too, or only the networking tests? If so, then
> hardening.config probably makes sense. 
> 
> Judging by what Documentation/dev-tools/kfence.rst says:
> 
> "KFENCE is designed to be enabled in production kernels, and has near zero
> performance overhead."
> 
> this reads like it should be enabled *everywhere* - not only in some
> hardening config.

Right, a lot of distros enable it and so do hyperscalers (Fedora, Meta
and Google at least, AFAIK). Linus is pretty clear on the policy that
"feature" type Kconfig options should default to disabled. But for
something like KFENCE we were wondering what the cut-over point is
for making it enabled by default.

Re: KFENCE: included in x86 defconfig?

[Marco Elver]

On Thu, 8 Feb 2024 at 00:33, Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Wed, 7 Feb 2024 20:04:44 +0100 Borislav Petkov wrote:
> > On Wed, Feb 07, 2024 at 07:35:53PM +0100, Matthieu Baerts wrote:
> > > Sorry, I'm sure I understand your suggestion: do you mean not including
> > > KFENCE in hardening.config either, but in another one?
> > >
> > > For the networking tests, we are already merging .config files, e.g. the
> > > debug.config one. We are not pushing to have KFENCE in x86 defconfig, it
> > > can be elsewhere, and we don't mind merging other .config files if they
> > > are maintained.
> >
> > Well, depends on where should KFENCE be enabled? Do you want people to
> > run their tests with it too, or only the networking tests? If so, then
> > hardening.config probably makes sense.
> >
> > Judging by what Documentation/dev-tools/kfence.rst says:
> >
> > "KFENCE is designed to be enabled in production kernels, and has near zero
> > performance overhead."
> >
> > this reads like it should be enabled *everywhere* - not only in some
> > hardening config.
>
> Right, a lot of distros enable it and so do hyperscalers (Fedora, Meta
> and Google at least, AFAIK). Linus is pretty clear on the policy that
> "feature" type Kconfig options should default to disabled. But for
> something like KFENCE we were wondering what the cut-over point is
> for making it enabled by default.

That's a good question, and I don't have the answer to that - maybe we
need to ask Linus then.

We could argue that to improve memory safety of the Linux kernel more
rapidly, enablement of KFENCE by default (on the "big" architectures
like x86) might actually be a net benefit at ~zero performance
overhead and the cost of 2 MiB of RAM (default config). One big
assumption is that CI systems or whoever will look at their kernel
logs and report the warnings (a quick web search does confirm that
KFENCE reports are reported by random users as well and not just devs
or CI systems).

Thanks,
-- Marco

Re: KFENCE: included in x86 defconfig?

[Borislav Petkov]

On Thu, Feb 08, 2024 at 08:47:37AM +0100, Marco Elver wrote:
> That's a good question, and I don't have the answer to that - maybe we
> need to ask Linus then.

Right, before that, lemme put my user hat on.

> We could argue that to improve memory safety of the Linux kernel more
> rapidly, enablement of KFENCE by default (on the "big" architectures
> like x86) might actually be a net benefit at ~zero performance
> overhead and the cost of 2 MiB of RAM (default config).

What about its benefit?

I haven't seen a bug fix saying "found by KFENCE" or so but that doesn't
mean a whole lot.

The more important question is would I, as a user, have a way of
reporting such issues, would those issues be taken seriously and so on.

We have a whole manual about it:

Documentation/admin-guide/reporting-issues.rst

maybe the kfence splat would have a pointer to that? Perhaps...

Personally, I don't mind running it if it really is a ~zero overhead
KASAN replacement. Maybe as a preliminary step we should enable it on
devs machines who know how to report such things.

/me goes and enables it in a guest...

[    0.074294] kfence: initialized - using 2097152 bytes for 255 objects at 0xffff88807d600000-0xffff88807d800000

Guest looks ok to me, no reports.

What now? :-)

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: KFENCE: included in x86 defconfig?

[Linus Torvalds]

On Thu, 8 Feb 2024 at 10:55, Borislav Petkov <bp@alien8.de> wrote:
>
> What about its benefit?
>
> I haven't seen a bug fix saying "found by KFENCE" or so but that doesn't
> mean a whole lot.

It does find some things. You can search for "BUG: KFENCE" on lore,
and there are real bug reports.

That said, there are real downsides too. Yes, you potentially find
bugs, but the act of finding the bugs might also cause issues. And
that means that anybody who enables KFENCE then needs to be willing to
deal with said issues and have the infrastructure to debug and report
them upstream.

I think that's the *real* cost there - KFENCE is likely a good idea,
but I'm not convinced it should be a defconfig thing, it should be a
conscious decision.

           Linus

Re: KFENCE: included in x86 defconfig?

[Marco Elver]

On Thu, 8 Feb 2024 at 11:55, Borislav Petkov <bp@alien8.de> wrote:
>
> On Thu, Feb 08, 2024 at 08:47:37AM +0100, Marco Elver wrote:
> > That's a good question, and I don't have the answer to that - maybe we
> > need to ask Linus then.
>
> Right, before that, lemme put my user hat on.
>
> > We could argue that to improve memory safety of the Linux kernel more
> > rapidly, enablement of KFENCE by default (on the "big" architectures
> > like x86) might actually be a net benefit at ~zero performance
> > overhead and the cost of 2 MiB of RAM (default config).
>
> What about its benefit?
>
> I haven't seen a bug fix saying "found by KFENCE" or so but that doesn't
> mean a whole lot.

git log --grep 'BUG: KFENCE: '

There are more I'm aware of - also plenty I know of in downstream
kernels (https://arxiv.org/pdf/2311.09394.pdf - Section 5.7).

> The more important question is would I, as a user, have a way of
> reporting such issues, would those issues be taken seriously and so on.

This is a problem shared by all other diagnostic and error reports the
kernel produces.

> We have a whole manual about it:
>
> Documentation/admin-guide/reporting-issues.rst
>
> maybe the kfence splat would have a pointer to that? Perhaps...
>
> Personally, I don't mind running it if it really is a ~zero overhead
> KASAN replacement. Maybe as a preliminary step we should enable it on
> devs machines who know how to report such things.

It's not a KASAN replacement, since it's sampling based. From the
Documentation: "KFENCE is designed to be enabled in production
kernels, and has near zero performance overhead. Compared to KASAN,
KFENCE trades performance for precision. The main motivation behind
KFENCE's design, is that with enough total uptime KFENCE will detect
bugs in code paths not typically exercised by non-production test
workloads. One way to quickly achieve a large enough total uptime is
when the tool is deployed across a large fleet of machines."

Enabling it in as many kernels as possible will help towards the
"deployed across a large fleet of machines".  That being said, KFENCE
is already deployed across O(millions) of devices where the reporting
story is also taken care of. Enabling it in even more systems where
the reporting story is not as clear may or may not be helpful - it'd
be an experiment.

> /me goes and enables it in a guest...
>
> [    0.074294] kfence: initialized - using 2097152 bytes for 255 objects at 0xffff88807d600000-0xffff88807d800000
>
> Guest looks ok to me, no reports.
>
> What now? :-)

No reports are good. Doesn't mean absence of bugs though. :-)

Thanks,
-- Marco

Re: KFENCE: included in x86 defconfig?

[Borislav Petkov]

On Thu, Feb 08, 2024 at 12:12:19PM +0100, Marco Elver wrote:
> git log --grep 'BUG: KFENCE: '
> 
> There are more I'm aware of - also plenty I know of in downstream
> kernels (https://arxiv.org/pdf/2311.09394.pdf - Section 5.7).

Good.

> This is a problem shared by all other diagnostic and error reports the
> kernel produces.

Yes, and it becomes a problem if you expose it to the wider audience.

And yes, nothing new here - it is the same ol' question of getting good
bug reports.

> It's not a KASAN replacement, since it's sampling based.

I meant this: "Compared to KASAN, KFENCE trades performance for
precision."

And yeah, I did read what you pasted.

> From the Documentation: "KFENCE is designed to be enabled in
> production kernels, and has near zero performance overhead. Compared
> to KASAN, KFENCE trades performance for precision. The main motivation
> behind KFENCE's design, is that with enough total uptime KFENCE will
> detect bugs in code paths not typically exercised by non-production
> test workloads.

What is that double negation supposed to mean?

That it'll detect bugs in code paths that are typically exercised by
production test workloads?

> One way to quickly achieve a large enough total uptime is
> when the tool is deployed across a large fleet of machines."

In any case, I'll enable it on my test machines and see what happens.

> No reports are good. Doesn't mean absence of bugs though. :-)

As long as I don't know about them, I'm good. :-P

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: KFENCE: included in x86 defconfig?

[Kees Cook]

On February 7, 2024 10:05:31 AM PST, Marco Elver <elver@google.com> wrote:
>On Wed, 7 Feb 2024 at 17:16, Matthieu Baerts <matttbe@kernel.org> wrote:
>[...]
>> When talking to Jakub about the kernel config used by the new CI for the
>> net tree [1], Jakub suggested [2] to check if KFENCE could not be
>> enabled by default for x86 architecture.
>
>I think this would belong into some "hardening" config - while KFENCE
>is not a mitigation (due to sampling) it has the performance
>characteristics of unintrusive hardening techniques, so I think it
>would be a good fit. I think that'd be
>"kernel/configs/hardening.config".

I would be happy to see it added to the hardening fragment! Send me a patch and I'll put it in my tree. :)

-Kees

-- 
Kees Cook