[bug report]warning about entry_64.S from objtool in v5.4 LTS

[bug report]warning about entry_64.S from objtool in v5.4 LTS

[Xinghui Li]

Hi all
We found a warning from objtool:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1d1:
unsupported intra-function call

and if we enable retpoline in config:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1c1:
unsupported intra-function call
arch/x86/entry/entry_64.o: warning: objtool: If this is a retpoline,
please patch it in with alternatives and annotate it with
ANNOTATE_NOSPEC_ALTERNATIVE.

I found this issue has been introduced since “x86/speculation: Change
FILL_RETURN_BUFFER to work with objtool( commit 8afd1c7da2)”backported
in v5.4.217.
Comparing with the upstream version(commit 089dd8e53):
There is no “ANNOTATE_INTRA_FUNCTION_CALL” in v5.4 for missing
dependency patch. When the “ANNOTATE_NOSPEC_ALTERNATIVE” is removed,
this issue just occurs.

I tried to backport “ANNOTATE_INTRA_FUNCTION_CALL”and its dependency
patchs in v5.4, but I met the CFA miss match issue from objtool.
So, please help check this issue in v5.4 LTS version.

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Greg KH]

On Mon, Feb 06, 2023 at 11:09:48AM +0800, Xinghui Li wrote:
> Hi all
> We found a warning from objtool:
> arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1d1:
> unsupported intra-function call
> 
> and if we enable retpoline in config:
> arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1c1:
> unsupported intra-function call
> arch/x86/entry/entry_64.o: warning: objtool: If this is a retpoline,
> please patch it in with alternatives and annotate it with
> ANNOTATE_NOSPEC_ALTERNATIVE.
> 
> I found this issue has been introduced since “x86/speculation: Change
> FILL_RETURN_BUFFER to work with objtool( commit 8afd1c7da2)”backported
> in v5.4.217.
> Comparing with the upstream version(commit 089dd8e53):
> There is no “ANNOTATE_INTRA_FUNCTION_CALL” in v5.4 for missing
> dependency patch. When the “ANNOTATE_NOSPEC_ALTERNATIVE” is removed,
> this issue just occurs.
> 
> I tried to backport “ANNOTATE_INTRA_FUNCTION_CALL”and its dependency
> patchs in v5.4, but I met the CFA miss match issue from objtool.
> So, please help check this issue in v5.4 LTS version.

If you rely on the 5.4.y kernel tree, and you need this speculation
fixes and feel this is a real problem, please provide some backported
patches to resolve the problem.

It's been reported many times in the past, but no one seems to actually
want to fix this bad enough to send in a patch :(

Usually people just move to a newer kernel, what is preventing you from
doing that right now?

thanks,

greg k-h

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Xinghui Li]

Greg KH <gregkh@linuxfoundation.org> 于2023年2月6日周一 13:40写道：

> If you rely on the 5.4.y kernel tree, and you need this speculation
> fixes and feel this is a real problem, please provide some backported
> patches to resolve the problem.
>
> It's been reported many times in the past, but no one seems to actually
> want to fix this bad enough to send in a patch :(
>
> Usually people just move to a newer kernel, what is preventing you from
> doing that right now?
>
> thanks,
>
> greg k-h
Thanks for your reply~ I am sorry about reporting the known Issue.
I am trying to fix this issue right now. And I met the CFA issue, so I
just want to discuss it with the community.

We keep staying in v5.4 because we developed the product base on v5.4
3 years ago.
So it is unstable and difficult to update the kernel version.

I will try to find out a way to fix this issue.

Thanks a lot.

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Greg KH]

On Mon, Feb 06, 2023 at 05:22:54PM +0800, Xinghui Li wrote:
> Greg KH <gregkh@linuxfoundation.org> 于2023年2月6日周一 13:40写道：
> 
> > If you rely on the 5.4.y kernel tree, and you need this speculation
> > fixes and feel this is a real problem, please provide some backported
> > patches to resolve the problem.
> >
> > It's been reported many times in the past, but no one seems to actually
> > want to fix this bad enough to send in a patch :(
> >
> > Usually people just move to a newer kernel, what is preventing you from
> > doing that right now?
> >
> > thanks,
> >
> > greg k-h
> Thanks for your reply~ I am sorry about reporting the known Issue.
> I am trying to fix this issue right now. And I met the CFA issue, so I
> just want to discuss it with the community.

Is this an actual real problem that you can detect with testing?  Or is
it just a warning message in the build?

> We keep staying in v5.4 because we developed the product base on v5.4
> 3 years ago.
> So it is unstable and difficult to update the kernel version.

That is very odd, why is it hard to update to a new kernel?  What
happens if 5.4 stops being maintained tomorrow, what are your plans to
move to a more modern kernel?  Being stuck with an old kernel version
with no plans to move does not seem like a wise business decision:(

> I will try to find out a way to fix this issue.

Great, and we will be glad to review patches.

thanks,

greg k-h

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Kuniyuki Iwashima]

Hi,

From:   Greg KH <gregkh@linuxfoundation.org>
Date:   Mon, 6 Feb 2023 10:39:45 +0100
> On Mon, Feb 06, 2023 at 05:22:54PM +0800, Xinghui Li wrote:
> > Greg KH <gregkh@linuxfoundation.org> 于2023年2月6日周一 13:40写道：
> > 
> > > If you rely on the 5.4.y kernel tree, and you need this speculation
> > > fixes and feel this is a real problem, please provide some backported
> > > patches to resolve the problem.
> > >
> > > It's been reported many times in the past, but no one seems to actually
> > > want to fix this bad enough to send in a patch :(
> > >
> > > Usually people just move to a newer kernel, what is preventing you from
> > > doing that right now?
> > >
> > > thanks,
> > >
> > > greg k-h
> > Thanks for your reply~ I am sorry about reporting the known Issue.
> > I am trying to fix this issue right now. And I met the CFA issue, so I
> > just want to discuss it with the community.
> 
> Is this an actual real problem that you can detect with testing?  Or is
> it just a warning message in the build?

I've just received a related report from a customer and I found the
same commit was the first bad one while investigating.  I've attached
a simple repro below.

After 8afd1c7da2b0 ("x86/speculation: Change FILL_RETURN_BUFFER to
work with objtool"), calling stack_trace_save_tsk_reliable() from
the kernel module below fails with -EINVAL if CONFIG_UNWINDER_ORC=y
and CONFIG_RETPOLINE=y.

I confirmed that this issue does not happen on 5.10 and 5.15 at
least (and told the customer to use either).

Interestingly, after the commit, stack_trace_save_tsk_reliable()
fails but seems to fill the buffer with the correct stack trace.

---8<---
#include <linux/module.h>
#include <linux/kallsyms.h>

typedef int (*func_t)(struct task_struct *tsk,
		      unsigned long *store, unsigned int size);

static __init int buggy_orc_init(void)
{
	unsigned long store[32] = {0};
	func_t func;
	int ret, i;

	func = (func_t)kallsyms_lookup_name("stack_trace_save_tsk_reliable");
	if (!func)
		return -ENOENT;

	ret = func(current, store, ARRAY_SIZE(store));

	for (i = 0; i < ARRAY_SIZE(store); i++)
		printk(KERN_ERR "%px: %pS\n", (void *)store[i], (void *)store[i]);

	return ret > 0 ? 0 : ret;
}

module_init(buggy_orc_init);

MODULE_LICENSE("GPL");
---8<---

---8<---
# insmod buggy_orc.ko
[    8.576683] buggy_orc: loading out-of-tree module taints kernel.
[    8.578414] ffffffff810d1538: stack_trace_save_tsk_reliable+0x78/0xc0
[    8.578414] ffffffffc000405c: buggy_orc_init+0x5c/0x1000 [buggy_orc]
[    8.579066] ffffffff81000b56: do_one_initcall+0x46/0x1f0
[    8.579066] ffffffff810f005c: do_init_module+0x4c/0x240
[    8.579066] ffffffff810f2cbf: __do_sys_finit_module+0xbf/0xe0
[    8.580379] ffffffff81002198: do_syscall_64+0x48/0x110
[    8.580379] ffffffff81c00094: entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[    8.581328] 0000000000000000: 0x0
[    8.581328] 0000000000000000: 0x0
[    8.581328] 0000000000000000: 0x0
[    8.582145] 0000000000000000: 0x0
[    8.582145] 0000000000000000: 0x0
[    8.582145] 0000000000000000: 0x0
[    8.582145] 0000000000000000: 0x0
[    8.583229] 0000000000000000: 0x0
[    8.583229] 0000000000000000: 0x0
[    8.583229] 0000000000000000: 0x0
[    8.584046] 0000000000000000: 0x0
[    8.584046] 0000000000000000: 0x0
[    8.584046] 0000000000000000: 0x0
[    8.584046] 0000000000000000: 0x0
[    8.585130] 0000000000000000: 0x0
[    8.585130] 0000000000000000: 0x0
[    8.585130] 0000000000000000: 0x0
[    8.585130] 0000000000000000: 0x0
[    8.586221] 0000000000000000: 0x0
[    8.586221] 0000000000000000: 0x0
[    8.586221] 0000000000000000: 0x0
[    8.587038] 0000000000000000: 0x0
[    8.587038] 0000000000000000: 0x0
[    8.587038] 0000000000000000: 0x0
[    8.587038] 0000000000000000: 0x0
insmod: ERROR: could not insert module buggy_orc.ko: Invalid parameters
---8<---

Thank you,
Kuniyuki

> 
> > We keep staying in v5.4 because we developed the product base on v5.4
> > 3 years ago.
> > So it is unstable and difficult to update the kernel version.
> 
> That is very odd, why is it hard to update to a new kernel?  What
> happens if 5.4 stops being maintained tomorrow, what are your plans to
> move to a more modern kernel?  Being stuck with an old kernel version
> with no plans to move does not seem like a wise business decision:(
> 
> > I will try to find out a way to fix this issue.
> 
> Great, and we will be glad to review patches.
> 
> thanks,
> 
> greg k-h

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Xinghui Li]

Greg KH <gregkh@linuxfoundation.org> 于2023年2月6日周一 17:39写道：
>
> Is this an actual real problem that you can detect with testing?  Or is
> it just a warning message in the build?
>
So far, I have not met the actual error in running time.
But according to Kuniyuki's report, it seems like this commit will
cause an actual running time error.
I will continue to analyze these error reports.
> That is very odd, why is it hard to update to a new kernel?  What
> happens if 5.4 stops being maintained tomorrow, what are your plans to
> move to a more modern kernel?  Being stuck with an old kernel version
> with no plans to move does not seem like a wise business decision:(
>
The product base on v5.4 is the LTS version just like the
stable-kernel in the upstream community.
> Great, and we will be glad to review patches.
>
> thanks,
>
> greg k-h
Anyway, I will try to figure it out :-)

Thanks

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Greg KH]

On Tue, Feb 07, 2023 at 02:43:17PM +0800, Xinghui Li wrote:
> Greg KH <gregkh@linuxfoundation.org> 于2023年2月6日周一 17:39写道：
> >
> > Is this an actual real problem that you can detect with testing?  Or is
> > it just a warning message in the build?
> >
> So far, I have not met the actual error in running time.
> But according to Kuniyuki's report, it seems like this commit will
> cause an actual running time error.

I do not understand the actual runtime error, sorry.  Stack traces don't
matter at runtime, do they?

> > That is very odd, why is it hard to update to a new kernel?  What
> > happens if 5.4 stops being maintained tomorrow, what are your plans to
> > move to a more modern kernel?  Being stuck with an old kernel version
> > with no plans to move does not seem like a wise business decision:(
> >
> The product base on v5.4 is the LTS version just like the
> stable-kernel in the upstream community.

That was not the question I asked :(

thanks,

greg k-h

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Xinghui Li]

Greg KH <gregkh@linuxfoundation.org> 于2023年2月7日周二 14:52写道：
> I do not understand the actual runtime error, sorry.  Stack traces don't
> matter at runtime, do they?
>
Sorry for the inaccurate description~ I mean it might not just a
compiling warning.
> > > That is very odd, why is it hard to update to a new kernel?  What
Some features we developed based on v5.4's API. Update the kernel
verison could cause the KABI issue.
> > > happens if 5.4 stops being maintained tomorrow, what are your plans to
We will align with the LTS's lifecycle on our product
lifecycle(actually shorter).
If v5.4 do stop being maintained(I know it is not real), It looks like
we can only maintain it all by ourselves :-(.
> > > move to a more modern kernel?  Being stuck with an old kernel version
> > > with no plans to move does not seem like a wise business decision:(
> > >
> > The product base on v5.4 is the LTS version just like the
> > stable-kernel in the upstream community.
>
> That was not the question I asked :(
>
Will the above reply answer your question?
Besides, we do base the new kernel to develop the new version product
such as v5.18 and 6.x.

Thanks~

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Greg KH]

On Tue, Feb 07, 2023 at 03:19:06PM +0800, Xinghui Li wrote:
> Greg KH <gregkh@linuxfoundation.org> 于2023年2月7日周二 14:52写道：
> > I do not understand the actual runtime error, sorry.  Stack traces don't
> > matter at runtime, do they?
> >
> Sorry for the inaccurate description~ I mean it might not just a
> compiling warning.
> > > > That is very odd, why is it hard to update to a new kernel?  What
> Some features we developed based on v5.4's API. Update the kernel
> verison could cause the KABI issue.

What abi issue are you referring to?  The user/kernel api?  Or
out-of-tree-code that is in your kernel tree?

If out-of-tree stuff, just forward port it, like you are going to have
to do anyway.  You have to always have a plan to move forward, otherwise
you are guaranteed to have an insecure kernel someday.

Moving to a new kernel version should always be planned, and possible.
If not, something is very wrong with your management process :(

> > > > happens if 5.4 stops being maintained tomorrow, what are your plans to
> We will align with the LTS's lifecycle on our product
> lifecycle(actually shorter).
> If v5.4 do stop being maintained(I know it is not real), It looks like
> we can only maintain it all by ourselves :-(.

Again, why?  What is so magical about this release?

thanks,

greg k-h

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Xinghui Li]

Greg KH <gregkh@linuxfoundation.org> 于2023年2月7日周二 15:30写道：
>
> On Tue, Feb 07, 2023 at 03:19:06PM +0800, Xinghui Li wrote:
> > Greg KH <gregkh@linuxfoundation.org> 于2023年2月7日周二 14:52写道：
> > > I do not understand the actual runtime error, sorry.  Stack traces don't
> > > matter at runtime, do they?
> > >
> > Sorry for the inaccurate description~ I mean it might not just a
> > compiling warning.
> > > > > That is very odd, why is it hard to update to a new kernel?  What
> > Some features we developed based on v5.4's API. Update the kernel
> > verison could cause the KABI issue.
>
> What abi issue are you referring to?  The user/kernel api?  Or
> out-of-tree-code that is in your kernel tree?
>
> If out-of-tree stuff, just forward port it, like you are going to have
> to do anyway.  You have to always have a plan to move forward, otherwise
> you are guaranteed to have an insecure kernel someday.
>
> Moving to a new kernel version should always be planned, and possible.
> If not, something is very wrong with your management process :(
>
> > > > > happens if 5.4 stops being maintained tomorrow, what are your plans to
> > We will align with the LTS's lifecycle on our product
> > lifecycle(actually shorter).
> > If v5.4 do stop being maintained(I know it is not real), It looks like
> > we can only maintain it all by ourselves :-(.
>
> Again, why?  What is so magical about this release?
>
> thanks,
>
> greg k-h

Hi Greg, first of all, I have to say I totally agree with your opinion
about keeping updating the product's kernel.
But maybe because I'm not a native English speaker, it seems I didn't
convey my meaning clearly or didn't get you well.

We do have plans to update the kernel of the product and we keep doing that.

However, for conservative reasons, parts of our customers are opposed
to making significant upgrades to the products.
Meanwhile, for commercial purposes, we are committed to maintaining
the products we sell over a period of time.
For the above reasons, we also maintain older versions of the product,
so we will fix/report the issues in older kernels just like this time.

As for your question about why we chose v5.4, most likely because it
was the latest LTS release at the time.

Thanks~

Re: [bug report]warning about entry_64.S from objtool in v5.4 LTS

[Greg KH]

On Mon, Feb 13, 2023 at 08:20:03PM +0800, Xinghui Li wrote:
> However, for conservative reasons, parts of our customers are opposed
> to making significant upgrades to the products.

Your customers are not concervative at all if they are consuming the
RETBLEED and other Spectre-like fixes into your kernels.  Those are
major changes as you can see (affecting every single file you build), so
the "do not change anything" argument does not make any sense here,
right?

> Meanwhile, for commercial purposes, we are committed to maintaining
> the products we sell over a period of time.
> For the above reasons, we also maintain older versions of the product,
> so we will fix/report the issues in older kernels just like this time.

That's great, but some help in fixing this reported issue would also be
most appreciated, as you are the best one to test and verify the fix.

thanks,

greg k-h