Unknown ORC entry type 5 on RHEL 9 kernel

Unknown ORC entry type 5 on RHEL 9 kernel

[Stephen Brennan]

Hi Philipp,

I wanted to let you know about an issue with drgn's stack unwinding on
the RHEL 9 kernel. It looks like that kernel backports some ORC changes,
without including a change Omar made (including an .orc_header section)
to help identify the ORC format version. I've included more details on
this drgn issue:

https://github.com/osandov/drgn/issues/578

The result is that drgn can't unwind stacks on the RHEL 9 kernel. I
think the best path forward would be to include the .orc_header patch:
b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version identifier")

As it is, the RHEL 9 drgn can't unwind stacks on the RHEL 9 kernel,
which seems less than ideal.

Thanks,
Stephen

Re: Unknown ORC entry type 5 on RHEL 9 kernel

[Philipp Rudo]

Hi Stephen,

thanks for letting me know. And yes that is something we need to fix in
the kernel rather than drgn.

I took a quick look and found that the change was included when
backporting fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two")
for 9.6. Thing is that I'll be heading to LPC tomorrow and won't have
time to work on it properly before that...

One more hint. When you want to get the history of RHEL, it's better to
check the centos-stream repo then the changelog. There you have the
full git history (modulo some embargoed CVE fixes).

https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9

Thanks
Philipp


On Thu, 04 Dec 2025 12:06:54 -0800
Stephen Brennan <stephen.s.brennan@oracle.com> wrote:

> Hi Philipp,
> 
> I wanted to let you know about an issue with drgn's stack unwinding on
> the RHEL 9 kernel. It looks like that kernel backports some ORC changes,
> without including a change Omar made (including an .orc_header section)
> to help identify the ORC format version. I've included more details on
> this drgn issue:
> 
> https://github.com/osandov/drgn/issues/578
> 
> The result is that drgn can't unwind stacks on the RHEL 9 kernel. I
> think the best path forward would be to include the .orc_header patch:
> b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version identifier")
> 
> As it is, the RHEL 9 drgn can't unwind stacks on the RHEL 9 kernel,
> which seems less than ideal.
> 
> Thanks,
> Stephen
> 

Re: Unknown ORC entry type 5 on RHEL 9 kernel

[Stephen Brennan]

Philipp Rudo <prudo@redhat.com> writes:
> Hi Stephen,
>
> thanks for letting me know. And yes that is something we need to fix in
> the kernel rather than drgn.
>
> I took a quick look and found that the change was included when
> backporting fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two")
> for 9.6. Thing is that I'll be heading to LPC tomorrow and won't have
> time to work on it properly before that...

Thanks for taking a deeper look! No worries on timing, it's not urgent.
If you think the most likely fix will be to backport commit
b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version
identifier"), then I believe the drgn workaround I proposed in the
Github issue [1] should work fine: once the .orc_header section is
found, drgn will detect it and use the correct hash. The only difference
being that the "true" first broken version was 5.14.0-517.el9, which
might be worth encoding in the code in case somebody ever runs drgn
against a weird unreleased version from that development period.

[1]: https://github.com/osandov/drgn/issues/578#issuecomment-3614826789

> One more hint. When you want to get the history of RHEL, it's better to
> check the centos-stream repo then the changelog. There you have the
> full git history (modulo some embargoed CVE fixes).
>
> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9

Thanks, that helps a lot!
Stephen

> Thanks
> Philipp
>
>
> On Thu, 04 Dec 2025 12:06:54 -0800
> Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
>
>> Hi Philipp,
>> 
>> I wanted to let you know about an issue with drgn's stack unwinding on
>> the RHEL 9 kernel. It looks like that kernel backports some ORC changes,
>> without including a change Omar made (including an .orc_header section)
>> to help identify the ORC format version. I've included more details on
>> this drgn issue:
>> 
>> https://github.com/osandov/drgn/issues/578
>> 
>> The result is that drgn can't unwind stacks on the RHEL 9 kernel. I
>> think the best path forward would be to include the .orc_header patch:
>> b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version identifier")
>> 
>> As it is, the RHEL 9 drgn can't unwind stacks on the RHEL 9 kernel,
>> which seems less than ideal.
>> 
>> Thanks,
>> Stephen
>> 

Re: Unknown ORC entry type 5 on RHEL 9 kernel

[Philipp Rudo]

Hi Stephen,
Hi Omar,

On Fri, 05 Dec 2025 10:38:05 -0800
Stephen Brennan <stephen.s.brennan@oracle.com> wrote:

> Philipp Rudo <prudo@redhat.com> writes:
> > Hi Stephen,
> >
> > thanks for letting me know. And yes that is something we need to fix in
> > the kernel rather than drgn.
> >
> > I took a quick look and found that the change was included when
> > backporting fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two")
> > for 9.6. Thing is that I'll be heading to LPC tomorrow and won't have
> > time to work on it properly before that...  
> 
> Thanks for taking a deeper look! No worries on timing, it's not urgent.
> If you think the most likely fix will be to backport commit
> b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version
> identifier"), then I believe the drgn workaround I proposed in the
> Github issue [1] should work fine: once the .orc_header section is
> found, drgn will detect it and use the correct hash. The only difference
> being that the "true" first broken version was 5.14.0-517.el9, which
> might be worth encoding in the code in case somebody ever runs drgn
> against a weird unreleased version from that development period.
> 
> [1]: https://github.com/osandov/drgn/issues/578#issuecomment-3614826789

I had a look today. Unfortunately backporting b9f174c811e3
("x86/unwind/orc: Add ELF section with ORC version identifier") isn't as
straight forward as I hoped. The problem is that the RHEL9 kernel is
missing commit 321648455061 ("kbuild: use obj-y instead extra-y for
objects placed at the head"). Without that commit Omar's fix won't
work as intended and in fact causes problems...

I found an internal discussion about backporting 321648455061 ("kbuild:
use obj-y instead extra-y for objects placed at the head"). But back then
(~2 years ago) it was deemed to be too invasive to backport for RHEL9. I
don't think this will change now even later in the RHEL9 life cycle.

All in all I'm afraid we are stuck with Stephen's work around.

Thanks
Philipp

> 
> > One more hint. When you want to get the history of RHEL, it's better to
> > check the centos-stream repo then the changelog. There you have the
> > full git history (modulo some embargoed CVE fixes).
> >
> > https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9  
> 
> Thanks, that helps a lot!
> Stephen
> 
> > Thanks
> > Philipp
> >
> >
> > On Thu, 04 Dec 2025 12:06:54 -0800
> > Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
> >  
> >> Hi Philipp,
> >> 
> >> I wanted to let you know about an issue with drgn's stack unwinding on
> >> the RHEL 9 kernel. It looks like that kernel backports some ORC changes,
> >> without including a change Omar made (including an .orc_header section)
> >> to help identify the ORC format version. I've included more details on
> >> this drgn issue:
> >> 
> >> https://github.com/osandov/drgn/issues/578
> >> 
> >> The result is that drgn can't unwind stacks on the RHEL 9 kernel. I
> >> think the best path forward would be to include the .orc_header patch:
> >> b9f174c811e3a ("x86/unwind/orc: Add ELF section with ORC version identifier")
> >> 
> >> As it is, the RHEL 9 drgn can't unwind stacks on the RHEL 9 kernel,
> >> which seems less than ideal.
> >> 
> >> Thanks,
> >> Stephen
> >>   
>