[PATCH v2 0/5] systemtap improvements

[PATCH v2 0/5] systemtap improvements

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

The goal of this series is to make it possible to run compiled
systemtap probes on the target without all the tools otherwise
necessary to compile them.

This series also includes unrelated fixes to the systemtap packaging,
on which I stumbled while on the main subject.

Changes from v1:

* the last 3 patches, whole goal is to provide a systemtap.bbclass
  will be rerolled into another series later and were removed
* the "split runtime material" changes the runtime test to use the
  smaller runtime package


Yann Dirson (5):
  systemtap: split examples and python scripts out of main package
  systemtap: remove extra dependencies
  systemtap: clarify the relation between exporter and python3-probes
    feature
  systemtap: fix install when python3-probes is disabled in
    PACKAGECONFIG
  systemtap: split runtime material in its own package

 meta/lib/oeqa/selftest/cases/runtime_test.py  |  2 +-
 .../recipes-kernel/systemtap/systemtap_git.bb | 38 ++++++++++++++++---
 2 files changed, 33 insertions(+), 7 deletions(-)

-- 
2.28.0

[PATCH v2 1/5] systemtap: split examples and python scripts out of main package

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 1c9f2aed16..e98aff1851 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -38,6 +38,23 @@ RDEPENDS_${PN}-exporter = "${PN} python3-core python3-netclient"
 
 SYSTEMD_SERVICE_${PN}-exporter = "stap-exporter.service"
 
+PACKAGES =+ "${PN}-examples"
+FILES_${PN}-examples = "${datadir}/${BPN}/examples/"
+RDEPENDS_${PN}-examples += "${PN}"
+
+# don't complain that some examples involve bash, perl, php...
+INSANE_SKIP_${PN}-examples += "file-rdeps"
+
+PACKAGES =+ "${PN}-python"
+FILES_${PN}-python += "\
+ ${bindir}/dtrace \
+ ${libdir}/python*/ \
+ ${libexecdir}/${BPN}/python/ \
+"
+# python material requires sdt headers
+RDEPENDS_${PN}-python += "${PN}-dev python3-core"
+INSANE_SKIP_${PN}-python += "dev-deps"
+
 do_configure_prepend () {
     # Improve reproducibility for c++ object files
     reltivepath="${@os.path.relpath(d.getVar('STAGING_INCDIR'), d.getVar('S'))}"
-- 
2.28.0

[PATCH v2 2/5] systemtap: remove extra dependencies

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

Most of the RDEPENDS part of PACKAGECONFIG[translator] appears to be related to
examples only.
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index e98aff1851..375e570454 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -19,7 +19,7 @@ STAP_DOCS ?= "--disable-docs --disable-publican --disable-refdocs"
 EXTRA_OECONF += "${STAP_DOCS} "
 
 PACKAGECONFIG ??= "translator sqlite monitor python3-probes"
-PACKAGECONFIG[translator] = "--enable-translator,--disable-translator,boost,python3-core bash perl"
+PACKAGECONFIG[translator] = "--enable-translator,--disable-translator,boost,bash"
 PACKAGECONFIG[libvirt] = "--enable-libvirt,--disable-libvirt,libvirt"
 PACKAGECONFIG[sqlite] = "--enable-sqlite,--disable-sqlite,sqlite3"
 PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
-- 
2.28.0

[PATCH v2 3/5] systemtap: clarify the relation between exporter and python3-probes feature

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 375e570454..89f550c859 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -27,15 +27,13 @@ PACKAGECONFIG[python3-probes] = "--with-python3-probes,--without-python3-probes,
 
 inherit autotools gettext pkgconfig distutils3-base systemd
 
+# exporter comes with python3-probes
 PACKAGES =+ "${PN}-exporter"
-
 FILES_${PN}-exporter = "${sysconfdir}/stap-exporter/* \
                         ${sysconfdir}/sysconfig/stap-exporter \
                         ${systemd_unitdir}/system/stap-exporter.service \
                         ${sbindir}/stap-exporter"
-
 RDEPENDS_${PN}-exporter = "${PN} python3-core python3-netclient"
-
 SYSTEMD_SERVICE_${PN}-exporter = "stap-exporter.service"
 
 PACKAGES =+ "${PN}-examples"
-- 
2.28.0

[PATCH v2 4/5] systemtap: fix install when python3-probes is disabled in PACKAGECONFIG

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 89f550c859..74bf7cb35c 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -66,7 +66,7 @@ do_install_append () {
       rm ${D}${libexecdir}/${PN}/stap-env
    fi
 
-   if [ ${D}${prefix}/lib != `dirname ${D}${systemd_unitdir}` ]; then
+   if [ -d ${D}${prefix}/lib/systemd -a ${D}${prefix}/lib != `dirname ${D}${systemd_unitdir}` ]; then
       # Fix makefile hardcoded path assumptions for systemd (assumes $prefix)
       # without usrmerge distro feature enabled
       install -d `dirname ${D}${systemd_unitdir}`
@@ -75,7 +75,9 @@ do_install_append () {
    fi
 
    # Ensure correct ownership for files copied in
-   chown root:root ${D}${sysconfdir}/stap-exporter/* -R
+   if [ -d ${D}${sysconfdir}/stap-exporter ]; then
+       chown root:root ${D}${sysconfdir}/stap-exporter/* -R
+   fi
 }
 
 BBCLASSEXTEND = "nativesdk"
-- 
2.28.0

[PATCH v2 5/5] systemtap: split runtime material in its own package

[Yann Dirson]

From: Yann Dirson <yann@blade-group.com>

Note the _class-target qualifier, here to prevent a funky dependency of
systemtap-native on systemtap-native-runtime-native.  This possibly hints
to something deeper ?
---
 meta/lib/oeqa/selftest/cases/runtime_test.py   | 2 +-
 meta/recipes-kernel/systemtap/systemtap_git.bb | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 1bb1c4bee4..7189e4e6c5 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -384,7 +384,7 @@ KERNEL_EXTRA_FEATURES_append = " features/debug/debug-kernel.scc"
 KERNEL_EXTRA_FEATURES_append = " features/systemtap/systemtap.scc"
 
 # add systemtap run-time into target image if it is not there yet
-IMAGE_INSTALL_append = " systemtap"
+IMAGE_INSTALL_append = " systemtap-runtime"
 """
 
         def test_crosstap_helloworld(self):
diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 74bf7cb35c..8dad5b15f0 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -36,6 +36,15 @@ FILES_${PN}-exporter = "${sysconfdir}/stap-exporter/* \
 RDEPENDS_${PN}-exporter = "${PN} python3-core python3-netclient"
 SYSTEMD_SERVICE_${PN}-exporter = "stap-exporter.service"
 
+PACKAGES =+ "${PN}-runtime"
+FILES_${PN}-runtime = "\
+ ${bindir}/staprun \
+ ${bindir}/stap-merge \
+ ${bindir}/stapsh \
+ ${libexecdir}/${BPN}/stapio \
+"
+RDEPENDS_${PN}_class-target += "${PN}-runtime"
+
 PACKAGES =+ "${PN}-examples"
 FILES_${PN}-examples = "${datadir}/${BPN}/examples/"
 RDEPENDS_${PN}-examples += "${PN}"
-- 
2.28.0

Re: [OE-core] [PATCH v2 0/5] systemtap improvements

[victor.kamensky7]

[-- Attachment #1: Type: text/plain, Size: 9495 bytes --]

Hi Yann and Richard,

On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <yann.dirson@blade-group.com>
wrote:
>
> Oh I did not notice, shouldn't we be discussing this on-list ?

Very sorry about this. I did not notice that I've unintentionally removed
the mailing list.
Added it back.

I see that Richard added your patches to master-next. But I don't think
they should be
merged "as is". IMO we need to work more on them.

Richard, could you please hold off merging them?

Other thing that AUT done automatic upgrade to systemtap-4.4 and Zang have
sent patch
for it. IMO let's first merge 4.4 while we sort out what part of this
series should go in.

Let's go about slowly one issue at a time.

More responses inline.

> Another question: I can't really see what it brings, to be able to
disable the "monitor" feature.
> What should we write about this feature ?
>
> Le ven. 13 nov. 2020 à 11:57, Yann Dirson <yann.dirson@blade-group.com> a
écrit :
>>
>>
>>
>> Le jeu. 12 nov. 2020 à 21:11, Victor Kamensky <victor.kamensky7@gmail.com>
a écrit :
>>>
>>> Hi Yann,
>>>
>>> I am not sure that I understand purpose of the series.
>>
>>
>> The goal was to be able to include a precompiled probe in a readonly
image, and to get
>> it loaded during execution of a boot script (I was hunting a but that
only shows during
>> target startup).

It is an interesting feature, do you plan to merge precompiled probes in
production image, or
you want to add one for some debug case. If it is latter one probably there
are ways to do
it just by adding file built crosstap just as additional payload to the
image.

In developper case I personally deal with it by stopping system boot very
early i.e using
systemd "emergency" parameter passed as kernel boot option, dropping into
the shell
mounting some storage with precompiled probe, copy and activate probe and
exiting
emergency shell.

Also if we do with supporting such a feature one should be able to build
probe not only for
kernel but for the whole user-land. Does your proposed patch series support
probes for user-land
too? If so, how do you do that: chicken and egg problem - to build
user-land probe one need
complete sysroot of target, but since probe should be include in the image,
how do one
build sysroot first.

>>
>> It happens that I had not discovered the crosstap tool when I started
this, but it does not seem
>> to help much in my use-case.
>>
>> My approach was naturally integrated into the build, packaging the
cross-build of the .stp module,
>> finally giving birth to the systemtap.bbclass you can find in v1 of this
series (which I'm here postponing
>> to a further series).
>>
>> I also wanted to avoid installing unused stuff on target, and starting
disabling some PACKAGECONFIG
>> items, finding disabling some caused problems.
>>
>>>
>>> The current way to include SystemTap minimal runtime only into target
image
>>> that allow to run compiled on host scripts is to use
>>>
>>> PACKAGECONFIG_pn-systemtap =  openembedded-core@lists.openembedded.org
"monitor"

>>
>>
>> OK, for some reason I had missed that it does indeed achieve that.
Maybe we should add more doc about that.
>>
>> The split of a -runtime package, however, allows to still build the
"translator" part, not have it installed in the image
>> shipped on product, but still have the ability to install it from the
feed while developing, should someone prefer
>> this to cross-building the probes.  Not a big deal if you prefer not to
do that however, my own use-case would be
>> covered by the PACKAGECONFIG="monitor".

Yes, I was thinking about this use case as well. But it is quite esoteric
for OE. Furthermore,
it kind of contradicts current PACKAGECONFIG structure. If we want to go
this route we need
to reshuffle the whole thing in a different way. I.e either use
PACKAGECONFIG to
control the content systemtap package or use subpackages. Using both looks
a bit convoluted.

>>
>> But as state above it does not build on master/gatesgarth/dunfell
because of the problem fixed by the "fix install when python3-probes is
disabled in PACKAGECONFIG"
>> patch in this series.
>>
>> Furthermore, this minimal package with just runtime support still pulls
python3-core without a need for it.
>> This is coming from "inherit distutils3-base", and we should probably
not pull this one unless "python3-probes"
>> is enabled.  I had missed this one, will add another patch for this.
>>
>> BTW, does it make any sense to select "python3-probes" but not
"translator" ?

I don't quite yet follow this issue and need to play more before answering
it. Could you
please post separate patch only for this?

Thanks,
Victor


On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <yann.dirson@blade-group.com>
wrote:

> Oh I did not notice, shouldn't we be discussing this on-list ?
>
> Another question: I can't really see what it brings, to be able to disable
> the "monitor" feature.
> What should we write about this feature ?
>
> Le ven. 13 nov. 2020 à 11:57, Yann Dirson <yann.dirson@blade-group.com> a
> écrit :
>
>>
>>
>> Le jeu. 12 nov. 2020 à 21:11, Victor Kamensky <victor.kamensky7@gmail.com>
>> a écrit :
>>
>>> Hi Yann,
>>>
>>> I am not sure that I understand purpose of the series.
>>>
>>
>> The goal was to be able to include a precompiled probe in a readonly
>> image, and to get
>> it loaded during execution of a boot script (I was hunting a but that
>> only shows during
>> target startup).
>>
>> It happens that I had not discovered the crosstap tool when I started
>> this, but it does not seem
>> to help much in my use-case.
>>
>> My approach was naturally integrated into the build, packaging the
>> cross-build of the .stp module,
>> finally giving birth to the systemtap.bbclass you can find in v1 of this
>> series (which I'm here postponing
>> to a further series).
>>
>> I also wanted to avoid installing unused stuff on target, and starting
>> disabling some PACKAGECONFIG
>> items, finding disabling some caused problems.
>>
>>
>>> The current way to include SystemTap minimal runtime only into target
>>> image
>>> that allow to run compiled on host scripts is to use
>>>
>>> PACKAGECONFIG_pn-systemtap = "monitor"
>>>
>>
>> OK, for some reason I had missed that it does indeed achieve that.  Maybe
>> we should add more doc about that.
>>
>> The split of a -runtime package, however, allows to still build the
>> "translator" part, not have it installed in the image
>> shipped on product, but still have the ability to install it from the
>> feed while developing, should someone prefer
>> this to cross-building the probes.  Not a big deal if you prefer not to
>> do that however, my own use-case would be
>> covered by the PACKAGECONFIG="monitor".
>>
>> But as state above it does not build on master/gatesgarth/dunfell because
>> of the problem fixed by the "fix install when python3-probes is disabled in
>> PACKAGECONFIG"
>> patch in this series.
>>
>> Furthermore, this minimal package with just runtime support still pulls
>> python3-core without a need for it.
>> This is coming from "inherit distutils3-base", and we should probably not
>> pull this one unless "python3-probes"
>> is enabled.  I had missed this one, will add another patch for this.
>>
>> BTW, does it make any sense to select "python3-probes" but not
>> "translator" ?
>>
>> Best regards,
>> --
>> Yann
>>
>>
>>> in your local conf/local.conf file or add proper bbappend where only
>>> monitor feature would be enabled for systemtap package.
>>>
>>> That is mentioned at in crosstap script help
>>>
>>> http://git.openembedded.org/openembedded-core/tree/scripts/crosstap#n370
>>>
>>> Thanks,
>>> Victor
>>>
>>> On Thu, Nov 12, 2020 at 9:28 AM Yann Dirson <yann.dirson@blade-group.com>
>>> wrote:
>>> >
>>> > From: Yann Dirson <yann@blade-group.com>
>>> >
>>> > The goal of this series is to make it possible to run compiled
>>> > systemtap probes on the target without all the tools otherwise
>>> > necessary to compile them.
>>> >
>>> > This series also includes unrelated fixes to the systemtap packaging,
>>> > on which I stumbled while on the main subject.
>>> >
>>> > Changes from v1:
>>> >
>>> > * the last 3 patches, whole goal is to provide a systemtap.bbclass
>>> >   will be rerolled into another series later and were removed
>>> > * the "split runtime material" changes the runtime test to use the
>>> >   smaller runtime package
>>> >
>>> >
>>> > Yann Dirson (5):
>>> >   systemtap: split examples and python scripts out of main package
>>> >   systemtap: remove extra dependencies
>>> >   systemtap: clarify the relation between exporter and python3-probes
>>> >     feature
>>> >   systemtap: fix install when python3-probes is disabled in
>>> >     PACKAGECONFIG
>>> >   systemtap: split runtime material in its own package
>>> >
>>> >  meta/lib/oeqa/selftest/cases/runtime_test.py  |  2 +-
>>> >  .../recipes-kernel/systemtap/systemtap_git.bb | 38
>>> ++++++++++++++++---
>>> >  2 files changed, 33 insertions(+), 7 deletions(-)
>>> >
>>> > --
>>> > 2.28.0
>>> >
>>> >
>>> > 
>>> >
>>>
>>
>>
>> --
>> Yann Dirson <yann@blade-group.com>
>> Blade / Shadow -- http://shadow.tech
>>
>>
>
> --
> Yann Dirson <yann@blade-group.com>
> Blade / Shadow -- http://shadow.tech
>
>

[-- Attachment #2: Type: text/html, Size: 12785 bytes --]

Re: [OE-core] [PATCH v2 0/5] systemtap improvements

[Richard Purdie]

On Mon, 2020-11-16 at 20:34 -0800, Victor Kamensky wrote:
> Hi Yann and Richard,
> 
> On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <
> yann.dirson@blade-group.com> wrote:
> >
> > Oh I did not notice, shouldn't we be discussing this on-list ?
> 
> Very sorry about this. I did not notice that I've unintentionally
> removed the mailing list.
> Added it back.
> 
> I see that Richard added your patches to master-next. But I don't
> think they should be
> merged "as is". IMO we need to work more on them.
> 
> Richard, could you please hold off merging them?

Sorry, they did merge already around a day ago. Yann had proposed them
a while ago and spent quite a bit of time sorting out the automated
testing failures so it felt like we should merge them since there was
no feedback otherwise and they looked basically reasonable to me.

Obviously we can follow up with other improvements to build upon them
though.

Cheers,

Richard

Re: [OE-core] [PATCH v2 0/5] systemtap improvements

[Yann Dirson]

[-- Attachment #1: Type: text/plain, Size: 3072 bytes --]

Hi Victor, Zang,

How do you with to go forward ?

There was at least the issue quoted below to resolve wrt my patches.  Do we
want to wait for 4.4 to hit
master or is it something we can tackle already ?

Victor wrote:
>> OK, for some reason I had missed that it does indeed achieve that.
Maybe we should add more doc about that.
>>
>> The split of a -runtime package, however, allows to still build the
"translator" part, not have it installed in the image
>> shipped on product, but still have the ability to install it from the
feed while developing, should someone prefer
>> this to cross-building the probes.  Not a big deal if you prefer not to
do that however, my own use-case would be
>> covered by the PACKAGECONFIG="monitor".
>
> Yes, I was thinking about this use case as well. But it is quite esoteric
for OE. Furthermore,
> it kind of contradicts current PACKAGECONFIG structure. If we want to go
this route we need
> to reshuffle the whole thing in a different way. I.e either use
PACKAGECONFIG to
> control the content systemtap package or use subpackages. Using both
looks a bit convoluted.

It does not see contradictory to me to have PACKAGECONFIG triggering
generation of material that will
in turn de/activate packages.  Maybe we should align the naming of
PACKAGECONFIG
options and
related binary packages ?

As for myself I find the "translator" and "monitor" names not really
intuitive, they just match upstream
config option naming.  What about starting by renaming "monitor" to
"runtime" and making sure all files
generated with PACKAGECONFIG="runtime" get into ${PN}-runtime ?

As for "translator", it seems to refer to what most of us think about as a
"compiler", would that fit ?
We could then move those files into ${PN}-compiler and leave ${PN} as a
virtual package pulling
-runtime and -compiler.

Does this look correct ?


Le mar. 17 nov. 2020 à 10:24, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> On Mon, 2020-11-16 at 20:34 -0800, Victor Kamensky wrote:
> > Hi Yann and Richard,
> >
> > On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <
> > yann.dirson@blade-group.com> wrote:
> > >
> > > Oh I did not notice, shouldn't we be discussing this on-list ?
> >
> > Very sorry about this. I did not notice that I've unintentionally
> > removed the mailing list.
> > Added it back.
> >
> > I see that Richard added your patches to master-next. But I don't
> > think they should be
> > merged "as is". IMO we need to work more on them.
> >
> > Richard, could you please hold off merging them?
>
> Sorry, they did merge already around a day ago. Yann had proposed them
> a while ago and spent quite a bit of time sorting out the automated
> testing failures so it felt like we should merge them since there was
> no feedback otherwise and they looked basically reasonable to me.
>
> Obviously we can follow up with other improvements to build upon them
> though.
>

-- 
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

[-- Attachment #2: Type: text/html, Size: 4683 bytes --]

Re: [OE-core] [PATCH v2 0/5] systemtap improvements

[Yann Dirson]

[-- Attachment #1: Type: text/plain, Size: 11045 bytes --]

Hi,

Le mar. 17 nov. 2020 à 05:35, Victor Kamensky <victor.kamensky7@gmail.com>
a écrit :

> Hi Yann and Richard,
>
> On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <yann.dirson@blade-group.com>
> wrote:
> >
> > Oh I did not notice, shouldn't we be discussing this on-list ?
>
> Very sorry about this. I did not notice that I've unintentionally removed
> the mailing list.
> Added it back.
>
> I see that Richard added your patches to master-next. But I don't think
> they should be
> merged "as is". IMO we need to work more on them.
>
> Richard, could you please hold off merging them?
>
> Other thing that AUT done automatic upgrade to systemtap-4.4 and Zang have
> sent patch
> for it. IMO let's first merge 4.4 while we sort out what part of this
> series should go in.
>
> Let's go about slowly one issue at a time.
>
> More responses inline.
>
> > Another question: I can't really see what it brings, to be able to
> disable the "monitor" feature.
> > What should we write about this feature ?
> >
> > Le ven. 13 nov. 2020 à 11:57, Yann Dirson <yann.dirson@blade-group.com>
> a écrit :
> >>
> >>
> >>
> >> Le jeu. 12 nov. 2020 à 21:11, Victor Kamensky <
> victor.kamensky7@gmail.com> a écrit :
> >>>
> >>> Hi Yann,
> >>>
> >>> I am not sure that I understand purpose of the series.
> >>
> >>
> >> The goal was to be able to include a precompiled probe in a readonly
> image, and to get
> >> it loaded during execution of a boot script (I was hunting a but that
> only shows during
> >> target startup).
>
> It is an interesting feature, do you plan to merge precompiled probes in
> production image, or
> you want to add one for some debug case.
>

The idea is to be able to get probes in production images, at least shipped
to select customers
or even just internal users when we need to hunt for particulary rare
problems, and having to modify
the build chain makes things quite more painful (we use the same image
signing and publication
process for all).


> If it is latter one probably there are ways to do
> it just by adding file built crosstap just as additional payload to the
> image.
>

I'm curious how would you do that with standard tools ?  One coworker wrote
for this kind of test
an unsquashfs-stuff-resquashfs that kinda does this kind of work

In developper case I personally deal with it by stopping system boot very
> early i.e using
> systemd "emergency" parameter passed as kernel boot option, dropping into
> the shell
> mounting some storage with precompiled probe, copy and activate probe and
> exiting
> emergency shell.
>

That would not have worked for the particular problem for which I worked on
this, where the
bug did not appear on every boot, and is probably linked to a race
condition exacerbated by
loading times from squashfs :)

Also if we do with supporting such a feature one should be able to build
> probe not only for
> kernel but for the whole user-land. Does your proposed patch series
> support probes for user-land
> too? If so, how do you do that: chicken and egg problem - to build
> user-land probe one need
> complete sysroot of target, but since probe should be include in the
> image, how do one
> build sysroot first.
>

I admit I have not started to look into userland probes, as the system
probe I deployed had
already given valuable information.

>>
> >> It happens that I had not discovered the crosstap tool when I started
> this, but it does not seem
> >> to help much in my use-case.
> >>
> >> My approach was naturally integrated into the build, packaging the
> cross-build of the .stp module,
> >> finally giving birth to the systemtap.bbclass you can find in v1 of
> this series (which I'm here postponing
> >> to a further series).
> >>
> >> I also wanted to avoid installing unused stuff on target, and starting
> disabling some PACKAGECONFIG
> >> items, finding disabling some caused problems.
> >>
> >>>
> >>> The current way to include SystemTap minimal runtime only into target
> image
> >>> that allow to run compiled on host scripts is to use
> >>>
> >>> PACKAGECONFIG_pn-systemtap =  openembedded-core@lists.openembedded.org
> "monitor"
>
> >>
> >>
> >> OK, for some reason I had missed that it does indeed achieve that.
> Maybe we should add more doc about that.
> >>
> >> The split of a -runtime package, however, allows to still build the
> "translator" part, not have it installed in the image
> >> shipped on product, but still have the ability to install it from the
> feed while developing, should someone prefer
> >> this to cross-building the probes.  Not a big deal if you prefer not to
> do that however, my own use-case would be
> >> covered by the PACKAGECONFIG="monitor".
>
> Yes, I was thinking about this use case as well. But it is quite esoteric
> for OE. Furthermore,
> it kind of contradicts current PACKAGECONFIG structure. If we want to go
> this route we need
> to reshuffle the whole thing in a different way. I.e either use
> PACKAGECONFIG to
> control the content systemtap package or use subpackages. Using both looks
> a bit convoluted.
>

This would make sense, yes.


> >>
> >> But as state above it does not build on master/gatesgarth/dunfell
> because of the problem fixed by the "fix install when python3-probes is
> disabled in PACKAGECONFIG"
> >> patch in this series.
> >>
> >> Furthermore, this minimal package with just runtime support still pulls
> python3-core without a need for it.
> >> This is coming from "inherit distutils3-base", and we should probably
> not pull this one unless "python3-probes"
> >> is enabled.  I had missed this one, will add another patch for this.
> >>
> >> BTW, does it make any sense to select "python3-probes" but not
> "translator" ?
>
> I don't quite yet follow this issue and need to play more before answering
> it. Could you
> please post separate patch only for this?
>

Sending in a few minutes after rebasing (just saw Richard's email)


> Thanks,
> Victor
>
>
> On Fri, Nov 13, 2020 at 5:59 AM Yann Dirson <yann.dirson@blade-group.com>
> wrote:
>
>> Oh I did not notice, shouldn't we be discussing this on-list ?
>>
>> Another question: I can't really see what it brings, to be able to
>> disable the "monitor" feature.
>> What should we write about this feature ?
>>
>> Le ven. 13 nov. 2020 à 11:57, Yann Dirson <yann.dirson@blade-group.com>
>> a écrit :
>>
>>>
>>>
>>> Le jeu. 12 nov. 2020 à 21:11, Victor Kamensky <
>>> victor.kamensky7@gmail.com> a écrit :
>>>
>>>> Hi Yann,
>>>>
>>>> I am not sure that I understand purpose of the series.
>>>>
>>>
>>> The goal was to be able to include a precompiled probe in a readonly
>>> image, and to get
>>> it loaded during execution of a boot script (I was hunting a but that
>>> only shows during
>>> target startup).
>>>
>>> It happens that I had not discovered the crosstap tool when I started
>>> this, but it does not seem
>>> to help much in my use-case.
>>>
>>> My approach was naturally integrated into the build, packaging the
>>> cross-build of the .stp module,
>>> finally giving birth to the systemtap.bbclass you can find in v1 of this
>>> series (which I'm here postponing
>>> to a further series).
>>>
>>> I also wanted to avoid installing unused stuff on target, and starting
>>> disabling some PACKAGECONFIG
>>> items, finding disabling some caused problems.
>>>
>>>
>>>> The current way to include SystemTap minimal runtime only into target
>>>> image
>>>> that allow to run compiled on host scripts is to use
>>>>
>>>> PACKAGECONFIG_pn-systemtap = "monitor"
>>>>
>>>
>>> OK, for some reason I had missed that it does indeed achieve that.
>>> Maybe we should add more doc about that.
>>>
>>> The split of a -runtime package, however, allows to still build the
>>> "translator" part, not have it installed in the image
>>> shipped on product, but still have the ability to install it from the
>>> feed while developing, should someone prefer
>>> this to cross-building the probes.  Not a big deal if you prefer not to
>>> do that however, my own use-case would be
>>> covered by the PACKAGECONFIG="monitor".
>>>
>>> But as state above it does not build on master/gatesgarth/dunfell
>>> because of the problem fixed by the "fix install when python3-probes is
>>> disabled in PACKAGECONFIG"
>>> patch in this series.
>>>
>>> Furthermore, this minimal package with just runtime support still pulls
>>> python3-core without a need for it.
>>> This is coming from "inherit distutils3-base", and we should probably
>>> not pull this one unless "python3-probes"
>>> is enabled.  I had missed this one, will add another patch for this.
>>>
>>> BTW, does it make any sense to select "python3-probes" but not
>>> "translator" ?
>>>
>>> Best regards,
>>> --
>>> Yann
>>>
>>>
>>>> in your local conf/local.conf file or add proper bbappend where only
>>>> monitor feature would be enabled for systemtap package.
>>>>
>>>> That is mentioned at in crosstap script help
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/scripts/crosstap#n370
>>>>
>>>> Thanks,
>>>> Victor
>>>>
>>>> On Thu, Nov 12, 2020 at 9:28 AM Yann Dirson <
>>>> yann.dirson@blade-group.com> wrote:
>>>> >
>>>> > From: Yann Dirson <yann@blade-group.com>
>>>> >
>>>> > The goal of this series is to make it possible to run compiled
>>>> > systemtap probes on the target without all the tools otherwise
>>>> > necessary to compile them.
>>>> >
>>>> > This series also includes unrelated fixes to the systemtap packaging,
>>>> > on which I stumbled while on the main subject.
>>>> >
>>>> > Changes from v1:
>>>> >
>>>> > * the last 3 patches, whole goal is to provide a systemtap.bbclass
>>>> >   will be rerolled into another series later and were removed
>>>> > * the "split runtime material" changes the runtime test to use the
>>>> >   smaller runtime package
>>>> >
>>>> >
>>>> > Yann Dirson (5):
>>>> >   systemtap: split examples and python scripts out of main package
>>>> >   systemtap: remove extra dependencies
>>>> >   systemtap: clarify the relation between exporter and python3-probes
>>>> >     feature
>>>> >   systemtap: fix install when python3-probes is disabled in
>>>> >     PACKAGECONFIG
>>>> >   systemtap: split runtime material in its own package
>>>> >
>>>> >  meta/lib/oeqa/selftest/cases/runtime_test.py  |  2 +-
>>>> >  .../recipes-kernel/systemtap/systemtap_git.bb | 38
>>>> ++++++++++++++++---
>>>> >  2 files changed, 33 insertions(+), 7 deletions(-)
>>>> >
>>>> > --
>>>> > 2.28.0
>>>> >
>>>> >
>>>> > 
>>>> >
>>>>
>>>
>>>
>>> --
>>> Yann Dirson <yann@blade-group.com>
>>> Blade / Shadow -- http://shadow.tech
>>>
>>>
>>
>> --
>> Yann Dirson <yann@blade-group.com>
>> Blade / Shadow -- http://shadow.tech
>>
>>

-- 
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

[-- Attachment #2: Type: text/html, Size: 15739 bytes --]