* Re: Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <725895933.460.1555337504175.JavaMail.zimbra@efficios.com>
@ 2019-04-15 14:14 ` Mathieu Desnoyers
2019-04-15 14:14 ` [lttng-dev] " Mathieu Desnoyers
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Mathieu Desnoyers @ 2019-04-15 14:14 UTC (permalink / raw)
To: Nicholas Sadjoli
Cc: lttng-dev, Tom Zanussi, linux-rt-users, rostedt, dev-raspberry-pi
Re-sent as plain text.
----- On Apr 15, 2019, at 10:11 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <nick.sadjoli@gmail.com>
> wrote:
>> Dear Lttng dev Team,
>> I am currently interested in using lttng to measure the interrupt or hardware
>> real-time latencies experienced within my user-space applications.
>> Unfortunately I have been having issues installing lttng on my Raspberry Pi
>> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+. The
>> problems so far can be seen documented in the RPi Linux github forum page here:
>> [ https://github.com/raspberrypi/linux/issues/2930 |
>> https://github.com/raspberrypi/linux/issues/2930 ]
>> Initially I was about to just install the Debian package of the lttng onto the
>> RPi, and just proceeded with the commands ' sudo apt-get install
>> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to use
>> lttng package at all and found out that for the RPi, a build from source is
>> necessary for lttng, as described here: [
>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>> |
>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>> ]
>> Following their suggestion, I then tried to build the lttng packages from
>> source, and even re-building the RPi kernel locally to ensure the kernel
>> headers are in the RPi and linked up properly to the source packages.
>> Unfortunately when I tried to do this for the lttng-modules package, it instead
>> gave me the following warning in my latest attempt:
>> I n file included from
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
>> error: #error "Your kernel is known to have buggy optimized kprobes
>> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM:
>> 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>> CONFIG_OPTPROBES or upgrade your kernel."
>> # error "Your kernel is known to have buggy optimized kprobes implementation.
>> Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM: 8834/1: Fix:
>> kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>> CONFIG_OPTPROBES or upgrade your kernel."
>> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed to be
>> automatically overwritten during kernel building and is considered an important
>> part of the real-time RPi kernels. Has such problems been reported with this
>> version of the PREEMPT_RT kernel in the past (again, the current kernel version
>> that I am using is 4.14.91-rt49-v7+)? If there are I would like to know how
>> this issue was resolved so that I can properly build and use the lttng package
>> for my project.
> Hi Nicholas,
> It looks like our lttng-modules kernel version blacklist is acting exactly as it
> should. Looking at this
> git tree:
> git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
> I notice that in the git log, we find this bogus commit:
> commit f9e0bc710347fadad55910846ee36f0681d0dca8
> Author: Kees Cook <keescook@chromium.org>
> Date: Tue Oct 30 22:12:56 2018 +0100
> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
> But not the fix for it (upstream):
> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date: Tue Feb 5 16:37:40 2019 +0100
> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
> So lttng-modules refuses to build against your kernel configuration with
> CONFIG_OPTPROBES=y
> to save you the trouble of having kprobes trigger illegal instruction OOPS at
> runtime.
> I'm unusure who maintains that raspberrypi rt kernel tree. However, the upstream
> rt tree
> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
> [
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
> |
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
> ]
> (currently at 4.14.109-rt57).
> So you should contact your raspberrypi linux rt tree clone maintainer and ask
> him
> to upgrade, or, as stated in the #error generated by lttng-modules, simply
> ensure
> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
> raspberrypi community advice on how to ensure this configuration option is not
> overridden by the raspberrypi build system.
> Thanks,
> Mathieu
>> Thank you so much for your attention and looking forward to your reply soon.
>> Best Regards,
>> Nicholas Sadjoli
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [lttng-dev] Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <725895933.460.1555337504175.JavaMail.zimbra@efficios.com>
2019-04-15 14:14 ` Mathieu Desnoyers
@ 2019-04-15 14:14 ` Mathieu Desnoyers
2019-04-16 19:04 ` Nicholas Sadjoli
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Mathieu Desnoyers @ 2019-04-15 14:14 UTC (permalink / raw)
To: Nicholas Sadjoli
Cc: lttng-dev, rostedt, Tom Zanussi, linux-rt-users, dev-raspberry-pi
Re-sent as plain text.
----- On Apr 15, 2019, at 10:11 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <nick.sadjoli@gmail.com>
> wrote:
>> Dear Lttng dev Team,
>> I am currently interested in using lttng to measure the interrupt or hardware
>> real-time latencies experienced within my user-space applications.
>> Unfortunately I have been having issues installing lttng on my Raspberry Pi
>> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+. The
>> problems so far can be seen documented in the RPi Linux github forum page here:
>> [ https://github.com/raspberrypi/linux/issues/2930 |
>> https://github.com/raspberrypi/linux/issues/2930 ]
>> Initially I was about to just install the Debian package of the lttng onto the
>> RPi, and just proceeded with the commands ' sudo apt-get install
>> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to use
>> lttng package at all and found out that for the RPi, a build from source is
>> necessary for lttng, as described here: [
>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>> |
>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>> ]
>> Following their suggestion, I then tried to build the lttng packages from
>> source, and even re-building the RPi kernel locally to ensure the kernel
>> headers are in the RPi and linked up properly to the source packages.
>> Unfortunately when I tried to do this for the lttng-modules package, it instead
>> gave me the following warning in my latest attempt:
>> I n file included from
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
>> error: #error "Your kernel is known to have buggy optimized kprobes
>> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM:
>> 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>> CONFIG_OPTPROBES or upgrade your kernel."
>> # error "Your kernel is known to have buggy optimized kprobes implementation.
>> Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM: 8834/1: Fix:
>> kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>> CONFIG_OPTPROBES or upgrade your kernel."
>> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed to be
>> automatically overwritten during kernel building and is considered an important
>> part of the real-time RPi kernels. Has such problems been reported with this
>> version of the PREEMPT_RT kernel in the past (again, the current kernel version
>> that I am using is 4.14.91-rt49-v7+)? If there are I would like to know how
>> this issue was resolved so that I can properly build and use the lttng package
>> for my project.
> Hi Nicholas,
> It looks like our lttng-modules kernel version blacklist is acting exactly as it
> should. Looking at this
> git tree:
> git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
> I notice that in the git log, we find this bogus commit:
> commit f9e0bc710347fadad55910846ee36f0681d0dca8
> Author: Kees Cook <keescook@chromium.org>
> Date: Tue Oct 30 22:12:56 2018 +0100
> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
> But not the fix for it (upstream):
> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date: Tue Feb 5 16:37:40 2019 +0100
> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
> So lttng-modules refuses to build against your kernel configuration with
> CONFIG_OPTPROBES=y
> to save you the trouble of having kprobes trigger illegal instruction OOPS at
> runtime.
> I'm unusure who maintains that raspberrypi rt kernel tree. However, the upstream
> rt tree
> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
> [
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
> |
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
> ]
> (currently at 4.14.109-rt57).
> So you should contact your raspberrypi linux rt tree clone maintainer and ask
> him
> to upgrade, or, as stated in the #error generated by lttng-modules, simply
> ensure
> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
> raspberrypi community advice on how to ensure this configuration option is not
> overridden by the raspberrypi build system.
> Thanks,
> Mathieu
>> Thank you so much for your attention and looking forward to your reply soon.
>> Best Regards,
>> Nicholas Sadjoli
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <725895933.460.1555337504175.JavaMail.zimbra@efficios.com>
2019-04-15 14:14 ` Mathieu Desnoyers
2019-04-15 14:14 ` [lttng-dev] " Mathieu Desnoyers
@ 2019-04-16 19:04 ` Nicholas Sadjoli
[not found] ` <CAKUXLveyWXTsh6vG0V-isbj2YptT_r8vtMuUF-V2Aysb9SRBKg@mail.gmail.com>
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Nicholas Sadjoli @ 2019-04-16 19:04 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: lttng-dev, Tom Zanussi, linux-rt-users, rostedt, dev-raspberry-pi
[-- Attachment #1.1: Type: text/plain, Size: 5284 bytes --]
Dear Mathieu,
Thank you so much for your reply Mathieu! I have actually asked about this
to the raspberry pi github forum, but unfortunately it seems that I have
not gotten any reply from them on this matter yet. I'll be asking about how
to turn off the CONFIG_OPTPROBES from them again if that's the case.
If this still fails then I'll be trying out Tom Zanussi's upstream version
of the kernel as you suggested then. Thank you so much for your help once
again
On Mon, Apr 15, 2019 at 10:11 PM Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:
> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <
> nick.sadjoli@gmail.com> wrote:
>
> Dear Lttng dev Team,
> I am currently interested in using lttng to measure the interrupt or
> hardware real-time latencies experienced within my user-space applications.
> Unfortunately I have been having issues installing lttng on my Raspberry Pi
> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+.
> The problems so far can be seen documented in the RPi Linux github forum
> page here: https://github.com/raspberrypi/linux/issues/2930
>
> Initially I was about to just install the Debian package of the lttng onto
> the RPi, and just proceeded with the commands 'sudo apt-get install
> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to
> use lttng package at all and found out that for the RPi, a build from
> source is necessary for lttng, as described here:
> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>
> Following their suggestion, I then tried to build the lttng packages from
> source, and even re-building the RPi kernel locally to ensure the kernel
> headers are in the RPi and linked up properly to the source packages.
>
> Unfortunately when I tried to do this for the lttng-modules package, it
> instead gave me the following warning in my latest attempt:
>
> In file included from
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
> error: #error "Your kernel is known to have buggy optimized kprobes
> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
> # error "Your kernel is known to have buggy optimized kprobes
> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
>
> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed
> to be automatically overwritten during kernel building and is considered an
> important part of the real-time RPi kernels. Has such problems been
> reported with this version of the PREEMPT_RT kernel in the past (again, the
> current kernel version that I am using is 4.14.91-rt49-v7+)? If there are
> I would like to know how this issue was resolved so that I can properly
> build and use the lttng package for my project.
>
> Hi Nicholas,
>
> It looks like our lttng-modules kernel version blacklist is acting exactly
> as it should. Looking at this
> git tree:
>
> git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
>
> I notice that in the git log, we find this bogus commit:
>
> commit f9e0bc710347fadad55910846ee36f0681d0dca8
> Author: Kees Cook <keescook@chromium.org>
> Date: Tue Oct 30 22:12:56 2018 +0100
>
> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>
> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
>
> But not the fix for it (upstream):
>
> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date: Tue Feb 5 16:37:40 2019 +0100
>
> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
>
> So lttng-modules refuses to build against your kernel configuration with
> CONFIG_OPTPROBES=y
> to save you the trouble of having kprobes trigger illegal instruction OOPS
> at runtime.
>
> I'm unusure who maintains that raspberrypi rt kernel tree. However, the
> upstream rt tree
> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
>
>
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
>
> (currently at 4.14.109-rt57).
>
> So you should contact your raspberrypi linux rt tree clone maintainer and
> ask him
> to upgrade, or, as stated in the #error generated by lttng-modules, simply
> ensure
> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
> raspberrypi community advice on how to ensure this configuration option is
> not
> overridden by the raspberrypi build system.
>
> Thanks,
>
> Mathieu
>
>
>
> Thank you so much for your attention and looking forward to your reply
> soon.
>
>
>
> Best Regards,
>
>
>
> Nicholas Sadjoli
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
[-- Attachment #1.2: Type: text/html, Size: 14425 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 7+ messages in thread[parent not found: <CAKUXLveyWXTsh6vG0V-isbj2YptT_r8vtMuUF-V2Aysb9SRBKg@mail.gmail.com>]
* Re: Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <CAKUXLveyWXTsh6vG0V-isbj2YptT_r8vtMuUF-V2Aysb9SRBKg@mail.gmail.com>
@ 2019-04-16 19:05 ` Nicholas Sadjoli
0 siblings, 0 replies; 7+ messages in thread
From: Nicholas Sadjoli @ 2019-04-16 19:05 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: lttng-dev, Tom Zanussi, linux-rt-users, rostedt, dev-raspberry-pi
[-- Attachment #1.1: Type: text/plain, Size: 6149 bytes --]
Dear Mathieu,
Apoplogies but please ignore my previous email as it wasn't properly
finished.
Thank you so much for your reply Mathieu! I have actually asked about this
to the raspberry pi github forum, but unfortunately it seems that I have
not gotten any reply from them on this matter yet. I'll be asking about how
to turn off the CONFIG_OPTPROBES from them again if that's the case.
If this still fails then I'll be trying out Tom Zanussi's upstream version
of the kernel as you suggested then.
Thank you so much for your help once again and keep up the nice support
that you have for lttng!
Best Regards,
Nicholas Sadjoli
On Wed, Apr 17, 2019 at 3:04 AM Nicholas Sadjoli <nick.sadjoli@gmail.com>
wrote:
> Dear Mathieu,
>
> Thank you so much for your reply Mathieu! I have actually asked about this
> to the raspberry pi github forum, but unfortunately it seems that I have
> not gotten any reply from them on this matter yet. I'll be asking about how
> to turn off the CONFIG_OPTPROBES from them again if that's the case.
>
> If this still fails then I'll be trying out Tom Zanussi's upstream version
> of the kernel as you suggested then. Thank you so much for your help once
> again
>
> On Mon, Apr 15, 2019 at 10:11 PM Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <
>> nick.sadjoli@gmail.com> wrote:
>>
>> Dear Lttng dev Team,
>> I am currently interested in using lttng to measure the interrupt or
>> hardware real-time latencies experienced within my user-space applications.
>> Unfortunately I have been having issues installing lttng on my Raspberry Pi
>> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+.
>> The problems so far can be seen documented in the RPi Linux github forum
>> page here: https://github.com/raspberrypi/linux/issues/2930
>>
>> Initially I was about to just install the Debian package of the lttng
>> onto the RPi, and just proceeded with the commands 'sudo apt-get install
>> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable
>> to use lttng package at all and found out that for the RPi, a build from
>> source is necessary for lttng, as described here:
>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>>
>> Following their suggestion, I then tried to build the lttng packages from
>> source, and even re-building the RPi kernel locally to ensure the kernel
>> headers are in the RPi and linked up properly to the source packages.
>>
>> Unfortunately when I tried to do this for the lttng-modules package, it
>> instead gave me the following warning in my latest attempt:
>>
>> In file included from
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
>> error: #error "Your kernel is known to have buggy optimized kprobes
>> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
>> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
>> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
>> # error "Your kernel is known to have buggy optimized kprobes
>> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
>> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
>> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
>>
>> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it
>> seemed to be automatically overwritten during kernel building and is
>> considered an important part of the real-time RPi kernels. Has such
>> problems been reported with this version of the PREEMPT_RT kernel in the
>> past (again, the current kernel version that I am using is 4.14.91-rt49-v7+)?
>> If there are I would like to know how this issue was resolved so that I can
>> properly build and use the lttng package for my project.
>>
>> Hi Nicholas,
>>
>> It looks like our lttng-modules kernel version blacklist is acting
>> exactly as it should. Looking at this
>> git tree:
>>
>> git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
>>
>> I notice that in the git log, we find this bogus commit:
>>
>> commit f9e0bc710347fadad55910846ee36f0681d0dca8
>> Author: Kees Cook <keescook@chromium.org>
>> Date: Tue Oct 30 22:12:56 2018 +0100
>>
>> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>>
>> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
>>
>> But not the fix for it (upstream):
>>
>> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
>> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Date: Tue Feb 5 16:37:40 2019 +0100
>>
>> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
>>
>> So lttng-modules refuses to build against your kernel configuration with
>> CONFIG_OPTPROBES=y
>> to save you the trouble of having kprobes trigger illegal instruction
>> OOPS at runtime.
>>
>> I'm unusure who maintains that raspberrypi rt kernel tree. However, the
>> upstream rt tree
>> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
>>
>>
>> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
>>
>> (currently at 4.14.109-rt57).
>>
>> So you should contact your raspberrypi linux rt tree clone maintainer and
>> ask him
>> to upgrade, or, as stated in the #error generated by lttng-modules,
>> simply ensure
>> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
>> raspberrypi community advice on how to ensure this configuration option
>> is not
>> overridden by the raspberrypi build system.
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>> Thank you so much for your attention and looking forward to your reply
>> soon.
>>
>>
>>
>> Best Regards,
>>
>>
>>
>> Nicholas Sadjoli
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
[-- Attachment #1.2: Type: text/html, Size: 15742 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <725895933.460.1555337504175.JavaMail.zimbra@efficios.com>
` (3 preceding siblings ...)
[not found] ` <CAKUXLveyWXTsh6vG0V-isbj2YptT_r8vtMuUF-V2Aysb9SRBKg@mail.gmail.com>
@ 2019-04-28 10:11 ` Nicholas Sadjoli
[not found] ` <CAKUXLvdQ0ELiGLDg-gQGh4Ja1+q3_Jpc8GcB6Pnvh522=FE_fw@mail.gmail.com>
5 siblings, 0 replies; 7+ messages in thread
From: Nicholas Sadjoli @ 2019-04-28 10:11 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: lttng-dev, Tom Zanussi, linux-rt-users, rostedt, dev-raspberry-pi
[-- Attachment #1.1: Type: text/plain, Size: 6095 bytes --]
Hello again Mathieu,
I apologize that this is a rather late re-opening of the topic again.
However, after consulting with the RPi kernel side, it seems that even
after trying to fix for the mentioned issue (i.e. by upgrading to the
listed 'fixed' commit), I am still having issues installing the
lttng-modules. The following link is the thread that I am having with the
RPi kernel github: https://github.com/raspberrypi/linux/issues/2930
Even though it is listed that version 2.9 seems to be okay, recently I
tried to re-install the lttng-modules onto another RPi, but this time even
with the 'fixed' commit as suggested in the thread, lttng-modules still
give the same 'buggy kernel implementation' error. Please do note that it
seems CONFIG_OPTPROBES cannot be turned off at all during the kernel build,
as it will be overriden by default and seems to be integral to the build of
the RPi's RT kernel as well, hence this is not really an option either.
Is there a specific way on how to upgrade to the mentioned 'fixed' commit
without requiring a complete kernel overhaul? I am getting rather desparate
as I need this to work for an upcoming assignment deadline this week. So
I'm hoping that there is still potential solution to this.
Best Regards,
Nicholas Sadjoli
On Mon, Apr 15, 2019 at 10:11 PM Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:
> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli <
> nick.sadjoli@gmail.com> wrote:
>
> Dear Lttng dev Team,
> I am currently interested in using lttng to measure the interrupt or
> hardware real-time latencies experienced within my user-space applications.
> Unfortunately I have been having issues installing lttng on my Raspberry Pi
> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+.
> The problems so far can be seen documented in the RPi Linux github forum
> page here: https://github.com/raspberrypi/linux/issues/2930
>
> Initially I was about to just install the Debian package of the lttng onto
> the RPi, and just proceeded with the commands 'sudo apt-get install
> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to
> use lttng package at all and found out that for the RPi, a build from
> source is necessary for lttng, as described here:
> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>
> Following their suggestion, I then tried to build the lttng packages from
> source, and even re-building the RPi kernel locally to ensure the kernel
> headers are in the RPi and linked up properly to the source packages.
>
> Unfortunately when I tried to do this for the lttng-modules package, it
> instead gave me the following warning in my latest attempt:
>
> In file included from
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
> error: #error "Your kernel is known to have buggy optimized kprobes
> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
> # error "Your kernel is known to have buggy optimized kprobes
> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in
> Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
>
> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed
> to be automatically overwritten during kernel building and is considered an
> important part of the real-time RPi kernels. Has such problems been
> reported with this version of the PREEMPT_RT kernel in the past (again, the
> current kernel version that I am using is 4.14.91-rt49-v7+)? If there are
> I would like to know how this issue was resolved so that I can properly
> build and use the lttng package for my project.
>
> Hi Nicholas,
>
> It looks like our lttng-modules kernel version blacklist is acting exactly
> as it should. Looking at this
> git tree:
>
> git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
>
> I notice that in the git log, we find this bogus commit:
>
> commit f9e0bc710347fadad55910846ee36f0681d0dca8
> Author: Kees Cook <keescook@chromium.org>
> Date: Tue Oct 30 22:12:56 2018 +0100
>
> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>
> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
>
> But not the fix for it (upstream):
>
> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date: Tue Feb 5 16:37:40 2019 +0100
>
> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
>
> So lttng-modules refuses to build against your kernel configuration with
> CONFIG_OPTPROBES=y
> to save you the trouble of having kprobes trigger illegal instruction OOPS
> at runtime.
>
> I'm unusure who maintains that raspberrypi rt kernel tree. However, the
> upstream rt tree
> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
>
>
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
>
> (currently at 4.14.109-rt57).
>
> So you should contact your raspberrypi linux rt tree clone maintainer and
> ask him
> to upgrade, or, as stated in the #error generated by lttng-modules, simply
> ensure
> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
> raspberrypi community advice on how to ensure this configuration option is
> not
> overridden by the raspberrypi build system.
>
> Thanks,
>
> Mathieu
>
>
>
> Thank you so much for your attention and looking forward to your reply
> soon.
>
>
>
> Best Regards,
>
>
>
> Nicholas Sadjoli
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
[-- Attachment #1.2: Type: text/html, Size: 15434 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 7+ messages in thread[parent not found: <CAKUXLvdQ0ELiGLDg-gQGh4Ja1+q3_Jpc8GcB6Pnvh522=FE_fw@mail.gmail.com>]
* Re: Problems setting up lttng on PREEMPT_RT Raspberry Pi
[not found] ` <CAKUXLvdQ0ELiGLDg-gQGh4Ja1+q3_Jpc8GcB6Pnvh522=FE_fw@mail.gmail.com>
@ 2019-04-28 13:58 ` Mathieu Desnoyers
0 siblings, 0 replies; 7+ messages in thread
From: Mathieu Desnoyers @ 2019-04-28 13:58 UTC (permalink / raw)
To: Nicholas Sadjoli
Cc: lttng-dev, Tom Zanussi, linux-rt-users, rostedt, dev-raspberry-pi
[-- Attachment #1.1: Type: text/plain, Size: 9538 bytes --]
----- On Apr 28, 2019, at 6:11 AM, Nicholas Sadjoli <nick.sadjoli@gmail.com> wrote:
> Hello again Mathieu,
> I apologize that this is a rather late re-opening of the topic again. However,
> after consulting with the RPi kernel side, it seems that even after trying to
> fix for the mentioned issue (i.e. by upgrading to the listed 'fixed' commit), I
> am still having issues installing the lttng-modules. The following link is the
> thread that I am having with the RPi kernel github: [
> https://github.com/raspberrypi/linux/issues/2930 |
> https://github.com/raspberrypi/linux/issues/2930 ]
As replied by "pelwell" on the tracker:
"I think that the problem isn't with how you applied the back-ported patch, but with the detection of the problem by lttng. It seems that the obvious course of action is not sufficient to fix the problem, and I can't spare the time to start building lttng just to debug it for you - sorry."
Cherry-picking the kernel fix is not sufficient to let the lttng-modules kernel version blacklist mechanism know that
the kernel indeed has the fix. That's because the blacklist is based on the kernel version ranges, and you did
not change the kernel version.
> Even though it is listed that version 2.9 seems to be okay, recently I tried to
> re-install the lttng-modules onto another RPi, but this time even with the
> 'fixed' commit as suggested in the thread, lttng-modules still give the same
> 'buggy kernel implementation' error. Please do note that it seems
> CONFIG_OPTPROBES cannot be turned off at all during the kernel build, as it
> will be overriden by default and seems to be integral to the build of the RPi's
> RT kernel as well, hence this is not really an option either.
I my prior response, I never recommended to just cherry-pick the fix. I recommended that
you upgrade your kernel tree to one that contains the fix (and has a version tag which
guarantees that).
> Is there a specific way on how to upgrade to the mentioned 'fixed' commit
> without requiring a complete kernel overhaul? I am getting rather desparate as
> I need this to work for an upcoming assignment deadline this week. So I'm
> hoping that there is still potential solution to this.
If you just cherry-picked the fix without changing the kernel version, the lttng-modules blacklist will still refuse to compile
against your kernel, because the blacklist is based on kernel version range checks.
There are a few solutions here:
1) Turn off CONFIG_OPTPROBES with the help of the RPi community,
2) Upgrade your kernel tree to a kernel version that is outside of the blacklist range. Not just cherry-picking the specific fix.
3) If you choose to just cherry-pick the fix and not change the kernel version, you
can force lttng-modules to build against a blacklisted kernel version by removing
those lines in the lttng-modules source tree:
blacklist/kprobes.h:
#if LTTNG_KERNEL_RANGE(4,20,0, 4,20,13) \
|| LTTNG_KERNEL_RANGE(4,19,9, 4,19,26) \
|| LTTNG_KERNEL_RANGE(4,14,87, 4,14,104)
# if defined(CONFIG_ARM) && defined(CONFIG_OPTPROBES)
# error "Your kernel is known to have buggy optimized kprobes implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."
# endif /* #if defined(CONFIG_ARM) && defined(CONFIG_OPTPROBES) */
#endif
This will ensure lttng-modules builds against your kernel even though it's in the range
of kernel versions known to cause issues. But considering you cherry-picked the
fix, then you should be fine.
However, doing so removes the "belts and suspenders" we put in place to ensure
you don't try to use a bogus kprobes. I therefore recommend solutions 1 or 2 rather
than 3, but solution 3 might do the trick considering your short time-frame.
Good luck,
Mathieu
> Best Regards,
> Nicholas Sadjoli
> On Mon, Apr 15, 2019 at 10:11 PM Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:
>> ----- On Apr 15, 2019, at 7:07 AM, Nicholas Sadjoli < [
>> mailto:nick.sadjoli@gmail.com | nick.sadjoli@gmail.com ] > wrote:
>>> Dear Lttng dev Team,
>>> I am currently interested in using lttng to measure the interrupt or hardware
>>> real-time latencies experienced within my user-space applications.
>>> Unfortunately I have been having issues installing lttng on my Raspberry Pi
>>> that is using the PREEMPT_RT patched kernel, version 4.14.91-rt49-v7+. The
>>> problems so far can be seen documented in the RPi Linux github forum page here:
>>> [ https://github.com/raspberrypi/linux/issues/2930 |
>>> https://github.com/raspberrypi/linux/issues/2930 ]
>>> Initially I was about to just install the Debian package of the lttng onto the
>>> RPi, and just proceeded with the commands ' sudo apt-get install
>>> lttng-modules-dkms liblttng-ust-dev lttng-tools' However, I was unable to use
>>> lttng package at all and found out that for the RPi, a build from source is
>>> necessary for lttng, as described here: [
>>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>>> |
>>> https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available
>>> ]
>>> Following their suggestion, I then tried to build the lttng packages from
>>> source, and even re-building the RPi kernel locally to ensure the kernel
>>> headers are in the RPi and linked up properly to the source packages.
>>> Unfortunately when I tried to do this for the lttng-modules package, it instead
>>> gave me the following warning in my latest attempt:
>>> I n file included from
>>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
>>> /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4:
>>> error: #error "Your kernel is known to have buggy optimized kprobes
>>> implementation. Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM:
>>> 8834/1: Fix: kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>>> CONFIG_OPTPROBES or upgrade your kernel."
>>> # error "Your kernel is known to have buggy optimized kprobes implementation.
>>> Fixed by commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c \"ARM: 8834/1: Fix:
>>> kprobes: optimized kprobes illegal instruction\" in Linux. Disable
>>> CONFIG_OPTPROBES or upgrade your kernel."
>>> I then tried to turn off the CONFIG_OPTPROBES kernel config, but it seemed to be
>>> automatically overwritten during kernel building and is considered an important
>>> part of the real-time RPi kernels. Has such problems been reported with this
>>> version of the PREEMPT_RT kernel in the past (again, the current kernel version
>>> that I am using is 4.14.91-rt49-v7+)? If there are I would like to know how
>>> this issue was resolved so that I can properly build and use the lttng package
>>> for my project.
>> Hi Nicholas,
>> It looks like our lttng-modules kernel version blacklist is acting exactly as it
>> should. Looking at this
>> git tree:
>> git clone [ https://github.com/raspberrypi/linux.git |
>> https://github.com/raspberrypi/linux.git ] -b rpi-4.14.y-rt
>> I notice that in the git log, we find this bogus commit:
>> commit f9e0bc710347fadad55910846ee36f0681d0dca8
>> Author: Kees Cook < [ mailto:keescook@chromium.org | keescook@chromium.org ] >
>> Date: Tue Oct 30 22:12:56 2018 +0100
>> ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>> commit e46daee53bb50bde38805f1823a182979724c229 upstream.
>> But not the fix for it (upstream):
>> commit 0ac569bf6a7983c0c5747d6df8db9dc05bc92b6c
>> Author: Mathieu Desnoyers < [ mailto:mathieu.desnoyers@efficios.com |
>> mathieu.desnoyers@efficios.com ] >
>> Date: Tue Feb 5 16:37:40 2019 +0100
>> ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction
>> So lttng-modules refuses to build against your kernel configuration with
>> CONFIG_OPTPROBES=y
>> to save you the trouble of having kprobes trigger illegal instruction OOPS at
>> runtime.
>> I'm unusure who maintains that raspberrypi rt kernel tree. However, the upstream
>> rt tree
>> for 4.14-rt maintained by Tom Zanussi seems to contain the fix:
>> [
>> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
>> |
>> https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt
>> ]
>> (currently at 4.14.109-rt57).
>> So you should contact your raspberrypi linux rt tree clone maintainer and ask
>> him
>> to upgrade, or, as stated in the #error generated by lttng-modules, simply
>> ensure
>> CONFIG_OPTPROBES is unset when you build your kernel. You should ask the
>> raspberrypi community advice on how to ensure this configuration option is not
>> overridden by the raspberrypi build system.
>> Thanks,
>> Mathieu
>>> Thank you so much for your attention and looking forward to your reply soon.
>>> Best Regards,
>>> Nicholas Sadjoli
>>> _______________________________________________
>>> lttng-dev mailing list
>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
[-- Attachment #1.2: Type: text/html, Size: 20399 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 7+ messages in thread