[Question] How can I get floating point registers on arm64

[Question] How can I get floating point registers on arm64

[Grant Seltzer Richman]

Hi everyone,

I'm writing a uprobe program that I'm attaching to a function in a go
program on arm64. The function takes a float and as such loads the
parameters via 64-bit floating point registers i.e. `D0`.

However, the struct pt_regs context that uprobe programs have access
to only has a single set of 31 64-bit registers. These appear to be
the regular general purpose integer registers. My question is - how do
I access the second set of registers? If this question doesn't make
sense, am I misunderstanding how arm64 works?

Thanks so much,
Grant

Re: [Question] How can I get floating point registers on arm64

[Andrii Nakryiko]

On Thu, Mar 2, 2023 at 11:06 AM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> Hi everyone,
>
> I'm writing a uprobe program that I'm attaching to a function in a go
> program on arm64. The function takes a float and as such loads the
> parameters via 64-bit floating point registers i.e. `D0`.
>
> However, the struct pt_regs context that uprobe programs have access
> to only has a single set of 31 64-bit registers. These appear to be
> the regular general purpose integer registers. My question is - how do
> I access the second set of registers? If this question doesn't make
> sense, am I misunderstanding how arm64 works?
>

cc'ing Dave, as he was looking at this problem in the past (in the
context of accessing xmm registers, but similar problem).

The conclusion was that we'd need to add a new helper (kfunc nowadays)
that would do it for BPF program. Few things to consider:

  - designing generic enough interface to allow reading various
families of registers (FPU, XMM, etc) in some generic way
  - consider whether do platform-specific or platform-agnostic
interface (both possible)
  - and most annoyingly, we'd need to handle kernel potentially
modifying FPU state without (yet) restoring it. Dave investigated
this, and in some recent kernels it seems like kernel code doesn't
necessarily restore FPU state right after it's done with it, and
rather sets some special flag to restore FPU state as kernel exits to
user-space.

Hopefully Dave can correct me and fill in details.


> Thanks so much,
> Grant

Re: [Question] How can I get floating point registers on arm64

[Grant Seltzer Richman]

On Tue, Mar 7, 2023 at 7:28 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Mar 2, 2023 at 11:06 AM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > I'm writing a uprobe program that I'm attaching to a function in a go
> > program on arm64. The function takes a float and as such loads the
> > parameters via 64-bit floating point registers i.e. `D0`.
> >
> > However, the struct pt_regs context that uprobe programs have access
> > to only has a single set of 31 64-bit registers. These appear to be
> > the regular general purpose integer registers. My question is - how do
> > I access the second set of registers? If this question doesn't make
> > sense, am I misunderstanding how arm64 works?
> >
>
> cc'ing Dave, as he was looking at this problem in the past (in the
> context of accessing xmm registers, but similar problem).
>
> The conclusion was that we'd need to add a new helper (kfunc nowadays)
> that would do it for BPF program. Few things to consider:
>
>   - designing generic enough interface to allow reading various
> families of registers (FPU, XMM, etc) in some generic way
>   - consider whether do platform-specific or platform-agnostic
> interface (both possible)
>   - and most annoyingly, we'd need to handle kernel potentially
> modifying FPU state without (yet) restoring it. Dave investigated
> this, and in some recent kernels it seems like kernel code doesn't
> necessarily restore FPU state right after it's done with it, and
> rather sets some special flag to restore FPU state as kernel exits to
> user-space.

Thanks for this info Andrii! I think your first couple points are
manageable but I'm not familiar with FPU context switching. Will read
up on it, and Dave if you're willing to give some guidance I'd happily
put in the work to get this helper introduced!

>
> Hopefully Dave can correct me and fill in details.
>
>
> > Thanks so much,
> > Grant

Re: [Question] How can I get floating point registers on arm64

[Dave Marchevsky]

On 3/8/23 9:20 AM, Grant Seltzer Richman wrote:
> On Tue, Mar 7, 2023 at 7:28 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>>
>> On Thu, Mar 2, 2023 at 11:06 AM Grant Seltzer Richman
>> <grantseltzer@gmail.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> I'm writing a uprobe program that I'm attaching to a function in a go
>>> program on arm64. The function takes a float and as such loads the
>>> parameters via 64-bit floating point registers i.e. `D0`.
>>>
>>> However, the struct pt_regs context that uprobe programs have access
>>> to only has a single set of 31 64-bit registers. These appear to be
>>> the regular general purpose integer registers. My question is - how do
>>> I access the second set of registers? If this question doesn't make
>>> sense, am I misunderstanding how arm64 works?
>>>
>>
>> cc'ing Dave, as he was looking at this problem in the past (in the
>> context of accessing xmm registers, but similar problem).
>>
>> The conclusion was that we'd need to add a new helper (kfunc nowadays)
>> that would do it for BPF program. Few things to consider:
>>
>>   - designing generic enough interface to allow reading various
>> families of registers (FPU, XMM, etc) in some generic way
>>   - consider whether do platform-specific or platform-agnostic
>> interface (both possible)
>>   - and most annoyingly, we'd need to handle kernel potentially
>> modifying FPU state without (yet) restoring it. Dave investigated
>> this, and in some recent kernels it seems like kernel code doesn't
>> necessarily restore FPU state right after it's done with it, and
>> rather sets some special flag to restore FPU state as kernel exits to
>> user-space.
> 
> Thanks for this info Andrii! I think your first couple points are
> manageable but I'm not familiar with FPU context switching. Will read
> up on it, and Dave if you're willing to give some guidance I'd happily
> put in the work to get this helper introduced!
> 

Hi Grant,

I attempted to tackle this in a patchset a while back [0]. Had to abandon it to
focus on other things, please feel free to use it as a starting point.

Happy to elaborate on Andrii's 3rd point above, there's definitely some nuance
there that the series may not explain well. But need a day or so to page it back
in :). Will update this thread with details.

  [0]: https://lore.kernel.org/bpf/20220512074321.2090073-1-davemarchevsky@fb.com/

>>
>> Hopefully Dave can correct me and fill in details.
>>
>>
>>> Thanks so much,
>>> Grant

Re: [Question] How can I get floating point registers on arm64

[Grant Seltzer Richman]

On Wed, Mar 8, 2023 at 10:54 PM Dave Marchevsky <davemarchevsky@meta.com> wrote:
>
> On 3/8/23 9:20 AM, Grant Seltzer Richman wrote:
> > On Tue, Mar 7, 2023 at 7:28 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> >>
> >> On Thu, Mar 2, 2023 at 11:06 AM Grant Seltzer Richman
> >> <grantseltzer@gmail.com> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> I'm writing a uprobe program that I'm attaching to a function in a go
> >>> program on arm64. The function takes a float and as such loads the
> >>> parameters via 64-bit floating point registers i.e. `D0`.
> >>>
> >>> However, the struct pt_regs context that uprobe programs have access
> >>> to only has a single set of 31 64-bit registers. These appear to be
> >>> the regular general purpose integer registers. My question is - how do
> >>> I access the second set of registers? If this question doesn't make
> >>> sense, am I misunderstanding how arm64 works?
> >>>
> >>
> >> cc'ing Dave, as he was looking at this problem in the past (in the
> >> context of accessing xmm registers, but similar problem).
> >>
> >> The conclusion was that we'd need to add a new helper (kfunc nowadays)
> >> that would do it for BPF program. Few things to consider:
> >>
> >>   - designing generic enough interface to allow reading various
> >> families of registers (FPU, XMM, etc) in some generic way
> >>   - consider whether do platform-specific or platform-agnostic
> >> interface (both possible)
> >>   - and most annoyingly, we'd need to handle kernel potentially
> >> modifying FPU state without (yet) restoring it. Dave investigated
> >> this, and in some recent kernels it seems like kernel code doesn't
> >> necessarily restore FPU state right after it's done with it, and
> >> rather sets some special flag to restore FPU state as kernel exits to
> >> user-space.
> >
> > Thanks for this info Andrii! I think your first couple points are
> > manageable but I'm not familiar with FPU context switching. Will read
> > up on it, and Dave if you're willing to give some guidance I'd happily
> > put in the work to get this helper introduced!
> >
>
> Hi Grant,
>
> I attempted to tackle this in a patchset a while back [0]. Had to abandon it to
> focus on other things, please feel free to use it as a starting point.
>
> Happy to elaborate on Andrii's 3rd point above, there's definitely some nuance
> there that the series may not explain well. But need a day or so to page it back
> in :). Will update this thread with details.

Thanks Dave! I'm going to spend time over the new few days
familiarizing myself with this code and will certainly follow up with
questions. I see this patchset seems to specifically tackle x86 for
now, I'm hoping to additionally get arm64 support as that's the
priority for the project I work on.

>   [0]: https://lore.kernel.org/bpf/20220512074321.2090073-1-davemarchevsky@fb.com/
>
> >>
> >> Hopefully Dave can correct me and fill in details.
> >>
> >>
> >>> Thanks so much,
> >>> Grant