Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Stafford Horne]

+CC Lists,

They should have been included for all of these.

On Fri, Dec 13, 2024 at 11:22:32AM -0500, Michael Jeanson wrote:
> On 2024-12-12 07:41, Stafford Horne wrote:
> > On Tue, Dec 10, 2024 at 03:30:05PM -0500, Michael Jeanson wrote:
> >> On 2024-12-10 13:56, Michael Jeanson wrote:
> >>>> I started adding rseq support to OpenRISC, but it seems I need to do a bit more
> >>>> for me than just call rseq_signal_deliver().  OpenRISC does not implement
> >>>> HAVE_REGS_AND_STACK_ACCESS_API yet, so I will need to do that first.  Also I
> >>>> need to think of an instruction to use for RSEQ_SIG, but that should not be too
> >>>> hard.
> >>>
> >>> Do you have a WIP tree somewhere I can have a look at? Assuming you add
> >>> HAVE_REGS_AND_STACK_ACCESS_API, the rest should be pretty simple.
> >>
> >> I had a quick look at the kernel code and it looks pretty straightforward,
> >> I hacked this together just to see if it would build :
> >>
> >>   https://github.com/mjeanson/linux/commits/openrisc-rseq/
> >>
> >> This is thoroughly untested and only cross-compiled.
> > 
> > Thanks, while you were doing this I did something similar but took a much
> > shorter route, I only implemented the APIs used by rseq.
> > 
> > I have pushed branches for linux and glibc here:
> > 
> >  - https://github.com/stffrdhrn/or1k-glibc/commits/or1k-rseq/
> >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> 
> You might also want to add a call to 'rseq_syscall' in arch/openrisc/kernel/entry.S
> on return to userspace when CONFIG_DEBUG_RSEQ is enabled.

Yes, I am aware of that one, but I think I discovered an issue with the return to
userspace code that needs some cleanup before I can add that in.

> > 
> > With these the glibc tests for rseq do work on openrisc, see output below.  Do
> > you think with proper cleanups the glibc tests only would be enough to push this
> > upstream? As I haven't started writing the linux selftests for rseq.
> 
> It would really be better to implement the selftests, the current glibc test only
> do registration of the rseq area, there is no critical section / functional tests
> of the rseq feature.
> 
> This is something me and Mathieu could assist you with, first adding support for
> openrisc in librseq [1] and then we mostly just have to copy the headers to the
> kernel selftests to add support.
> 
> I can do most of the boilerplate of adding the architecture but you would be
> better suited to write the assembly used in the critical sections.

OK, I think I can add the boilerplace too by copying from another architecture.
Thanks for the tip on using librseq. I was looking at this code earlier today.

> Do you have access to hardware you could share temporarily for testing?

I have been doing most of my testing with qemu-system-or1k, built from source.
I run the virt platform which boots with Linux virt_defconfig.  The filesystem I
run is a setup from builtroot [1].

For "real" harware I have run OpenRISC in litex [2] on FPGA boards.  There is no
real hardware that I have access too.  The QEMU system emulator is the fastest
though so I have been using that almost exclusively lately.

-Stafford

[1] https://github.com/stffrdhrn/or1k-utils/tree/master/buildroot
[2] https://github.com/enjoy-digital/litex

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Stafford Horne]

On Sat, Dec 14, 2024 at 11:03:27AM +0000, Stafford Horne wrote:
> +CC Lists,
> 
> They should have been included for all of these.
> 
> On Fri, Dec 13, 2024 at 11:22:32AM -0500, Michael Jeanson wrote:
> > On 2024-12-12 07:41, Stafford Horne wrote:
> > > On Tue, Dec 10, 2024 at 03:30:05PM -0500, Michael Jeanson wrote:
> > >> On 2024-12-10 13:56, Michael Jeanson wrote:
> > >>>> I started adding rseq support to OpenRISC, but it seems I need to do a bit more
> > >>>> for me than just call rseq_signal_deliver().  OpenRISC does not implement
> > >>>> HAVE_REGS_AND_STACK_ACCESS_API yet, so I will need to do that first.  Also I
> > >>>> need to think of an instruction to use for RSEQ_SIG, but that should not be too
> > >>>> hard.
> > >>>
> > >>> Do you have a WIP tree somewhere I can have a look at? Assuming you add
> > >>> HAVE_REGS_AND_STACK_ACCESS_API, the rest should be pretty simple.
> > >>
> > >> I had a quick look at the kernel code and it looks pretty straightforward,
> > >> I hacked this together just to see if it would build :
> > >>
> > >>   https://github.com/mjeanson/linux/commits/openrisc-rseq/
> > >>
> > >> This is thoroughly untested and only cross-compiled.
> > > 
> > > Thanks, while you were doing this I did something similar but took a much
> > > shorter route, I only implemented the APIs used by rseq.
> > > 
> > > I have pushed branches for linux and glibc here:
> > > 
> > >  - https://github.com/stffrdhrn/or1k-glibc/commits/or1k-rseq/
> > >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> > 
> > You might also want to add a call to 'rseq_syscall' in arch/openrisc/kernel/entry.S
> > on return to userspace when CONFIG_DEBUG_RSEQ is enabled.
> 
> Yes, I am aware of that one, but I think I discovered an issue with the return to
> userspace code that needs some cleanup before I can add that in.

I think this ended up being ok.

I have added the call to rseq_syscall and implemented self tests on my branch
now.

 - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
 - commit 1fa73dd6c2d3 ("rseq/selftests: Add support for OpenRISC")

I haven't got the tests to complete fully yet though.  Do you have a recommended
approach for building, testing and debugging them?  I am using my glibc
toolchain, but I assume the original implementations didnt have glibc support
available when they were testing.

My stack now:

  - QEMU virt
  - Linux virt_defconfig (or1k-rseq branch)
    - rseq selftests - built with gcc/glibc toolchain (or1k-rseq branch)
  - rootfs - Buildroot with my glibc (or1k-rseq branch)
    - gdb
    - strace

In general I am using the latest git HEADs for qemu, gcc, binutils etc.

Once, everything is working on QEMU I will test again on the FPGA hardware.

Currently tests are failing with SIGSEGV:

    TAP version 13
    1..10
    # timeout set to 0
    # selftests: rseq: basic_test
    # testing current cpu
    ok 1 selftests: rseq: basic_test
    # timeout set to 0
    # selftests: rseq: basic_percpu_ops_test
    # spinlock
    # ./kselftest/runner.sh: line 37:   772 Segmentation fault      /usr/bin/timeout  --foreground "$kselftest_timeout" /usr/bin/timeout "$kselftest_timeout" $1
    not ok 2 selftests: rseq: basic_percpu_ops_test # exit=139

In gdb it looks to be happening in in an mprotect syscall in glibc and at that
point the stack seems to be corrupt already.  So its taking me a bit of time to
untangle.

-Stafford

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Stafford Horne]

On Tue, Dec 24, 2024 at 08:20:00PM +0000, Stafford Horne wrote:
> On Sat, Dec 14, 2024 at 11:03:27AM +0000, Stafford Horne wrote:
> > +CC Lists,
> > 
> > They should have been included for all of these.
> > 
> > On Fri, Dec 13, 2024 at 11:22:32AM -0500, Michael Jeanson wrote:
> > > On 2024-12-12 07:41, Stafford Horne wrote:
> > > > On Tue, Dec 10, 2024 at 03:30:05PM -0500, Michael Jeanson wrote:
> > > >> On 2024-12-10 13:56, Michael Jeanson wrote:
> > > >>>> I started adding rseq support to OpenRISC, but it seems I need to do a bit more
> > > >>>> for me than just call rseq_signal_deliver().  OpenRISC does not implement
> > > >>>> HAVE_REGS_AND_STACK_ACCESS_API yet, so I will need to do that first.  Also I
> > > >>>> need to think of an instruction to use for RSEQ_SIG, but that should not be too
> > > >>>> hard.
> > > >>>
> > > >>> Do you have a WIP tree somewhere I can have a look at? Assuming you add
> > > >>> HAVE_REGS_AND_STACK_ACCESS_API, the rest should be pretty simple.
> > > >>
> > > >> I had a quick look at the kernel code and it looks pretty straightforward,
> > > >> I hacked this together just to see if it would build :
> > > >>
> > > >>   https://github.com/mjeanson/linux/commits/openrisc-rseq/
> > > >>
> > > >> This is thoroughly untested and only cross-compiled.
> > > > 
> > > > Thanks, while you were doing this I did something similar but took a much
> > > > shorter route, I only implemented the APIs used by rseq.
> > > > 
> > > > I have pushed branches for linux and glibc here:
> > > > 
> > > >  - https://github.com/stffrdhrn/or1k-glibc/commits/or1k-rseq/
> > > >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> > > 
> > > You might also want to add a call to 'rseq_syscall' in arch/openrisc/kernel/entry.S
> > > on return to userspace when CONFIG_DEBUG_RSEQ is enabled.
> > 
> > Yes, I am aware of that one, but I think I discovered an issue with the return to
> > userspace code that needs some cleanup before I can add that in.
> 
> I think this ended up being ok.
> 
> I have added the call to rseq_syscall and implemented self tests on my branch
> now.
> 
>  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
>  - commit 1fa73dd6c2d3 ("rseq/selftests: Add support for OpenRISC")
> 
> I haven't got the tests to complete fully yet though.  Do you have a recommended
> approach for building, testing and debugging them?  I am using my glibc
> toolchain, but I assume the original implementations didnt have glibc support
> available when they were testing.
> 
> My stack now:
> 
>   - QEMU virt
>   - Linux virt_defconfig (or1k-rseq branch)
>     - rseq selftests - built with gcc/glibc toolchain (or1k-rseq branch)
>   - rootfs - Buildroot with my glibc (or1k-rseq branch)
>     - gdb
>     - strace
> 
> In general I am using the latest git HEADs for qemu, gcc, binutils etc.
> 
> Once, everything is working on QEMU I will test again on the FPGA hardware.
> 
> Currently tests are failing with SIGSEGV:
> 
>     TAP version 13
>     1..10
>     # timeout set to 0
>     # selftests: rseq: basic_test
>     # testing current cpu
>     ok 1 selftests: rseq: basic_test
>     # timeout set to 0
>     # selftests: rseq: basic_percpu_ops_test
>     # spinlock
>     # ./kselftest/runner.sh: line 37:   772 Segmentation fault      /usr/bin/timeout  --foreground "$kselftest_timeout" /usr/bin/timeout "$kselftest_timeout" $1
>     not ok 2 selftests: rseq: basic_percpu_ops_test # exit=139
> 
> In gdb it looks to be happening in in an mprotect syscall in glibc and at that
> point the stack seems to be corrupt already.  So its taking me a bit of time to
> untangle.

I was able to get this fixed, the issue was with how I was setting up the
rseq_cs in rseq-or1k.h.  There were some other problems too which I fixed.

Now, some of the tests are passing.

I have been making progress with some kernel printk's in kernel/rseq.c and using
gdb.  I should be able to get the remaining issues fixed up before too long.

-Stafford

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Stafford Horne]

On Thu, Jan 02, 2025 at 01:08:18AM +0000, Stafford Horne wrote:
> On Tue, Dec 24, 2024 at 08:20:00PM +0000, Stafford Horne wrote:
> > On Sat, Dec 14, 2024 at 11:03:27AM +0000, Stafford Horne wrote:
> > > +CC Lists,
> > > 
> > > They should have been included for all of these.
> > > 
> > > On Fri, Dec 13, 2024 at 11:22:32AM -0500, Michael Jeanson wrote:
> > > > On 2024-12-12 07:41, Stafford Horne wrote:
> > > > > On Tue, Dec 10, 2024 at 03:30:05PM -0500, Michael Jeanson wrote:
> > > > >> On 2024-12-10 13:56, Michael Jeanson wrote:
> > > > >>>> I started adding rseq support to OpenRISC, but it seems I need to do a bit more
> > > > >>>> for me than just call rseq_signal_deliver().  OpenRISC does not implement
> > > > >>>> HAVE_REGS_AND_STACK_ACCESS_API yet, so I will need to do that first.  Also I
> > > > >>>> need to think of an instruction to use for RSEQ_SIG, but that should not be too
> > > > >>>> hard.
> > > > >>>
> > > > >>> Do you have a WIP tree somewhere I can have a look at? Assuming you add
> > > > >>> HAVE_REGS_AND_STACK_ACCESS_API, the rest should be pretty simple.
> > > > >>
> > > > >> I had a quick look at the kernel code and it looks pretty straightforward,
> > > > >> I hacked this together just to see if it would build :
> > > > >>
> > > > >>   https://github.com/mjeanson/linux/commits/openrisc-rseq/
> > > > >>
> > > > >> This is thoroughly untested and only cross-compiled.
> > > > > 
> > > > > Thanks, while you were doing this I did something similar but took a much
> > > > > shorter route, I only implemented the APIs used by rseq.
> > > > > 
> > > > > I have pushed branches for linux and glibc here:
> > > > > 
> > > > >  - https://github.com/stffrdhrn/or1k-glibc/commits/or1k-rseq/
> > > > >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> > > > 
> > > > You might also want to add a call to 'rseq_syscall' in arch/openrisc/kernel/entry.S
> > > > on return to userspace when CONFIG_DEBUG_RSEQ is enabled.
> > > 
> > > Yes, I am aware of that one, but I think I discovered an issue with the return to
> > > userspace code that needs some cleanup before I can add that in.
> > 
> > I think this ended up being ok.
> > 
> > I have added the call to rseq_syscall and implemented self tests on my branch
> > now.
> > 
> >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> >  - commit 1fa73dd6c2d3 ("rseq/selftests: Add support for OpenRISC")
> > 
> > I haven't got the tests to complete fully yet though.  Do you have a recommended
> > approach for building, testing and debugging them?  I am using my glibc
> > toolchain, but I assume the original implementations didnt have glibc support
> > available when they were testing.
> > 
> > My stack now:
> > 
> >   - QEMU virt
> >   - Linux virt_defconfig (or1k-rseq branch)
> >     - rseq selftests - built with gcc/glibc toolchain (or1k-rseq branch)
> >   - rootfs - Buildroot with my glibc (or1k-rseq branch)
> >     - gdb
> >     - strace
> > 
> > In general I am using the latest git HEADs for qemu, gcc, binutils etc.
> > 
> > Once, everything is working on QEMU I will test again on the FPGA hardware.
> > 
> > Currently tests are failing with SIGSEGV:
> > 
> >     TAP version 13
> >     1..10
> >     # timeout set to 0
> >     # selftests: rseq: basic_test
> >     # testing current cpu
> >     ok 1 selftests: rseq: basic_test
> >     # timeout set to 0
> >     # selftests: rseq: basic_percpu_ops_test
> >     # spinlock
> >     # ./kselftest/runner.sh: line 37:   772 Segmentation fault      /usr/bin/timeout  --foreground "$kselftest_timeout" /usr/bin/timeout "$kselftest_timeout" $1
> >     not ok 2 selftests: rseq: basic_percpu_ops_test # exit=139
> > 
> > In gdb it looks to be happening in in an mprotect syscall in glibc and at that
> > point the stack seems to be corrupt already.  So its taking me a bit of time to
> > untangle.
> 
> I was able to get this fixed, the issue was with how I was setting up the
> rseq_cs in rseq-or1k.h.  There were some other problems too which I fixed.
> 
> Now, some of the tests are passing.
> 
> I have been making progress with some kernel printk's in kernel/rseq.c and using
> gdb.  I should be able to get the remaining issues fixed up before too long.

Hello,

I have all of the tests passing now under OpenRISC.  I would like to clean up
the commit messages and send the changes for review.  I incorporated your
HAVE_REGS_AND_STACK_ACCESS_API patch with my changes put it on my branch here:

 https://github.com/stffrdhrn/linux/commits/or1k-rseq/

Your commit does not have a Signed-off-by.  I am happy with the change, could you
fixup the commit message and submit it as a patch?  Or is OK if I do that and
add your Signed-off-by?

-Stafford

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Michael Jeanson]

On 2025-01-05 01:48, Stafford Horne wrote: 
> Hello,
> 
> I have all of the tests passing now under OpenRISC.  I would like to clean up
> the commit messages and send the changes for review.  I incorporated your
> HAVE_REGS_AND_STACK_ACCESS_API patch with my changes put it on my branch here:
> 
>  https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> 
> Your commit does not have a Signed-off-by.  I am happy with the change, could you
> fixup the commit message and submit it as a patch?  Or is OK if I do that and
> add your Signed-off-by?
> 
> -Stafford

Hi,

You can add my Signed-off-by and submit at your convenience.

Thanks!

Michael

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Michael Jeanson]

On 2024-12-24 15:20, Stafford Horne wrote: 
> I think this ended up being ok.
> 
> I have added the call to rseq_syscall and implemented self tests on my branch
> now.
> 
>  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
>  - commit 1fa73dd6c2d3 ("rseq/selftests: Add support for OpenRISC")
> 
> I haven't got the tests to complete fully yet though.  Do you have a recommended
> approach for building, testing and debugging them?  I am using my glibc
> toolchain, but I assume the original implementations didnt have glibc support
> available when they were testing.
> 
> My stack now:
> 
>   - QEMU virt
>   - Linux virt_defconfig (or1k-rseq branch)
>     - rseq selftests - built with gcc/glibc toolchain (or1k-rseq branch)
>   - rootfs - Buildroot with my glibc (or1k-rseq branch)
>     - gdb
>     - strace
> 
> In general I am using the latest git HEADs for qemu, gcc, binutils etc.
> 
> Once, everything is working on QEMU I will test again on the FPGA hardware.
> 
> Currently tests are failing with SIGSEGV:
> 
>     TAP version 13
>     1..10
>     # timeout set to 0
>     # selftests: rseq: basic_test
>     # testing current cpu
>     ok 1 selftests: rseq: basic_test
>     # timeout set to 0
>     # selftests: rseq: basic_percpu_ops_test
>     # spinlock
>     # ./kselftest/runner.sh: line 37:   772 Segmentation fault      /usr/bin/timeout  --foreground "$kselftest_timeout" /usr/bin/timeout "$kselftest_timeout" $1
>     not ok 2 selftests: rseq: basic_percpu_ops_test # exit=139
> 
> In gdb it looks to be happening in in an mprotect syscall in glibc and at that
> point the stack seems to be corrupt already.  So its taking me a bit of time to
> untangle.
> 
> -Stafford

Hi,

Sorry for the late reply, I was away for the holidays. From your other message I
think you managed to get through this. Is there anything I can still help you
with?

Michael

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Stafford Horne]

On Mon, Jan 06, 2025 at 01:26:48PM -0500, Michael Jeanson wrote:
> On 2024-12-24 15:20, Stafford Horne wrote: 
> > I think this ended up being ok.
> > 
> > I have added the call to rseq_syscall and implemented self tests on my branch
> > now.
> > 
> >  - https://github.com/stffrdhrn/linux/commits/or1k-rseq/
> >  - commit 1fa73dd6c2d3 ("rseq/selftests: Add support for OpenRISC")
> > 
> > I haven't got the tests to complete fully yet though.  Do you have a recommended
> > approach for building, testing and debugging them?  I am using my glibc
> > toolchain, but I assume the original implementations didnt have glibc support
> > available when they were testing.
> > 
> > My stack now:
> > 
> >   - QEMU virt
> >   - Linux virt_defconfig (or1k-rseq branch)
> >     - rseq selftests - built with gcc/glibc toolchain (or1k-rseq branch)
> >   - rootfs - Buildroot with my glibc (or1k-rseq branch)
> >     - gdb
> >     - strace
> > 
> > In general I am using the latest git HEADs for qemu, gcc, binutils etc.
> > 
> > Once, everything is working on QEMU I will test again on the FPGA hardware.
> > 
> > Currently tests are failing with SIGSEGV:
> > 
> >     TAP version 13
> >     1..10
> >     # timeout set to 0
> >     # selftests: rseq: basic_test
> >     # testing current cpu
> >     ok 1 selftests: rseq: basic_test
> >     # timeout set to 0
> >     # selftests: rseq: basic_percpu_ops_test
> >     # spinlock
> >     # ./kselftest/runner.sh: line 37:   772 Segmentation fault      /usr/bin/timeout  --foreground "$kselftest_timeout" /usr/bin/timeout "$kselftest_timeout" $1
> >     not ok 2 selftests: rseq: basic_percpu_ops_test # exit=139
> > 
> > In gdb it looks to be happening in in an mprotect syscall in glibc and at that
> > point the stack seems to be corrupt already.  So its taking me a bit of time to
> > untangle.
> > 
> > -Stafford
> 
> Hi,
> 
> Sorry for the late reply, I was away for the holidays. From your other message I
> think you managed to get through this. Is there anything I can still help you
> with?

No problem,

I figured you were on vacation.  I just kept sending mails to document my
progress.  The holiday's are when I get the most time to work on this stuff.

As of now there is nothing else needed.

-Stafford

Re: [PATCH] nptl: Add <thread_pointer.h> for or1k

[Michael Jeanson]

On 2025-01-06 15:18, Stafford Horne wrote:
> 
> No problem,
> 
> I figured you were on vacation.  I just kept sending mails to document my
> progress.  The holiday's are when I get the most time to work on this stuff.
> 
> As of now there is nothing else needed.
> 
> -Stafford

Thanks for taking care of this!

Michael