[PATCH v2] trace: use STAP_SDT_V2 to work around symbol visibility

[PATCH v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Stefan Hajnoczi]

QEMU binaries no longer launch successfully with recent SystemTap
releases. This is because modular QEMU builds link the sdt semaphores
into the main binary instead of into the shared objects where they are
used. The symbol visibility of semaphores is 'hidden' and the dynamic
linker prints an error during module loading:

  $ ./configure --enable-trace-backends=dtrace --enable-modules ...
  ...
  Failed to open module: /builddir/build/BUILD/qemu-4.2.0/s390x-softmmu/../block-curl.so: undefined symbol: qemu_curl_close_semaphore

The long-term solution is to generate per-module dtrace .o files and
link them into the module instead of the main binary.

In the short term we can define STAP_SDT_V2 so dtrace(1) produces a .o
file with 'default' symbol visibility instead of 'hidden'. This
workaround is small and easier to merge for QEMU 5.2.

Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: wcohen@redhat.com
Cc: fche@redhat.com
Cc: kraxel@redhat.com
Cc: rjones@redhat.com
Cc: mrezanin@redhat.com
Cc: ddepaula@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
v2:
 * Define STAP_SDT_V2 everywhere [danpb]
---
 configure         | 1 +
 trace/meson.build | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index 714e75b5d8..5d91d49c7b 100755
--- a/configure
+++ b/configure
@@ -4832,6 +4832,7 @@ if have_backend "dtrace"; then
   trace_backend_stap="no"
   if has 'stap' ; then
     trace_backend_stap="yes"
+    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"
   fi
 fi
 
diff --git a/trace/meson.build b/trace/meson.build
index d5fc45c628..843ea14495 100644
--- a/trace/meson.build
+++ b/trace/meson.build
@@ -38,13 +38,13 @@ foreach dir : [ '.' ] + trace_events_subdirs
     trace_dtrace_h = custom_target(fmt.format('trace-dtrace', 'h'),
                                    output: fmt.format('trace-dtrace', 'h'),
                                    input: trace_dtrace,
-                                   command: [ 'dtrace', '-o', '@OUTPUT@', '-h', '-s', '@INPUT@' ])
+                                   command: [ 'dtrace', '-DSTAP_SDT_V2', '-o', '@OUTPUT@', '-h', '-s', '@INPUT@' ])
     trace_ss.add(trace_dtrace_h)
     if host_machine.system() != 'darwin'
       trace_dtrace_o = custom_target(fmt.format('trace-dtrace', 'o'),
                                      output: fmt.format('trace-dtrace', 'o'),
                                      input: trace_dtrace,
-                                     command: [ 'dtrace', '-o', '@OUTPUT@', '-G', '-s', '@INPUT@' ])
+                                     command: [ 'dtrace', '-DSTAP_SDT_V2', '-o', '@OUTPUT@', '-G', '-s', '@INPUT@' ])
       trace_ss.add(trace_dtrace_o)
     endif
 
-- 
2.28.0

Re: [PATCH v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Daniel P. Berrangé]

On Thu, Nov 19, 2020 at 11:27:04AM +0000, Stefan Hajnoczi wrote:
> QEMU binaries no longer launch successfully with recent SystemTap
> releases. This is because modular QEMU builds link the sdt semaphores
> into the main binary instead of into the shared objects where they are
> used. The symbol visibility of semaphores is 'hidden' and the dynamic
> linker prints an error during module loading:
> 
>   $ ./configure --enable-trace-backends=dtrace --enable-modules ...
>   ...
>   Failed to open module: /builddir/build/BUILD/qemu-4.2.0/s390x-softmmu/../block-curl.so: undefined symbol: qemu_curl_close_semaphore
> 
> The long-term solution is to generate per-module dtrace .o files and
> link them into the module instead of the main binary.
> 
> In the short term we can define STAP_SDT_V2 so dtrace(1) produces a .o
> file with 'default' symbol visibility instead of 'hidden'. This
> workaround is small and easier to merge for QEMU 5.2.

And nice for distros to backport too.

> 
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: wcohen@redhat.com
> Cc: fche@redhat.com
> Cc: kraxel@redhat.com
> Cc: rjones@redhat.com
> Cc: mrezanin@redhat.com
> Cc: ddepaula@redhat.com
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v2:
>  * Define STAP_SDT_V2 everywhere [danpb]
> ---
>  configure         | 1 +
>  trace/meson.build | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH-for-5.2 v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Philippe Mathieu-Daudé]

On 11/19/20 12:27 PM, Stefan Hajnoczi wrote:
> QEMU binaries no longer launch successfully with recent SystemTap
> releases. This is because modular QEMU builds link the sdt semaphores
> into the main binary instead of into the shared objects where they are
> used. The symbol visibility of semaphores is 'hidden' and the dynamic
> linker prints an error during module loading:
> 
>   $ ./configure --enable-trace-backends=dtrace --enable-modules ...
>   ...
>   Failed to open module: /builddir/build/BUILD/qemu-4.2.0/s390x-softmmu/../block-curl.so: undefined symbol: qemu_curl_close_semaphore
> 
> The long-term solution is to generate per-module dtrace .o files and
> link them into the module instead of the main binary.
> 
> In the short term we can define STAP_SDT_V2 so dtrace(1) produces a .o
> file with 'default' symbol visibility instead of 'hidden'. This
> workaround is small and easier to merge for QEMU 5.2.
> 
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: wcohen@redhat.com
> Cc: fche@redhat.com
> Cc: kraxel@redhat.com
> Cc: rjones@redhat.com
> Cc: mrezanin@redhat.com
> Cc: ddepaula@redhat.com
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v2:
>  * Define STAP_SDT_V2 everywhere [danpb]
> ---
>  configure         | 1 +
>  trace/meson.build | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/configure b/configure
> index 714e75b5d8..5d91d49c7b 100755
> --- a/configure
> +++ b/configure
> @@ -4832,6 +4832,7 @@ if have_backend "dtrace"; then
>    trace_backend_stap="no"
>    if has 'stap' ; then
>      trace_backend_stap="yes"

Maybe add a comment? (no need to repost if you agree):

       # Workaround to avoid dtrace(1) produces file with 'hidden'
       # symbol visibility, define STAP_SDT_V2 to produce 'default'
       # symbol visibility instead.

> +    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

>    fi
>  fi
>  
> diff --git a/trace/meson.build b/trace/meson.build
> index d5fc45c628..843ea14495 100644
> --- a/trace/meson.build
> +++ b/trace/meson.build
> @@ -38,13 +38,13 @@ foreach dir : [ '.' ] + trace_events_subdirs
>      trace_dtrace_h = custom_target(fmt.format('trace-dtrace', 'h'),
>                                     output: fmt.format('trace-dtrace', 'h'),
>                                     input: trace_dtrace,
> -                                   command: [ 'dtrace', '-o', '@OUTPUT@', '-h', '-s', '@INPUT@' ])
> +                                   command: [ 'dtrace', '-DSTAP_SDT_V2', '-o', '@OUTPUT@', '-h', '-s', '@INPUT@' ])
>      trace_ss.add(trace_dtrace_h)
>      if host_machine.system() != 'darwin'
>        trace_dtrace_o = custom_target(fmt.format('trace-dtrace', 'o'),
>                                       output: fmt.format('trace-dtrace', 'o'),
>                                       input: trace_dtrace,
> -                                     command: [ 'dtrace', '-o', '@OUTPUT@', '-G', '-s', '@INPUT@' ])
> +                                     command: [ 'dtrace', '-DSTAP_SDT_V2', '-o', '@OUTPUT@', '-G', '-s', '@INPUT@' ])
>        trace_ss.add(trace_dtrace_o)
>      endif
>  
> 

Re: [PATCH-for-5.2 v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Stefan Hajnoczi]

On Thu, Nov 19, 2020 at 11:45 AM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
> On 11/19/20 12:27 PM, Stefan Hajnoczi wrote:
> > diff --git a/configure b/configure
> > index 714e75b5d8..5d91d49c7b 100755
> > --- a/configure
> > +++ b/configure
> > @@ -4832,6 +4832,7 @@ if have_backend "dtrace"; then
> >    trace_backend_stap="no"
> >    if has 'stap' ; then
> >      trace_backend_stap="yes"
>
> Maybe add a comment? (no need to repost if you agree):
>
>        # Workaround to avoid dtrace(1) produces file with 'hidden'
>        # symbol visibility, define STAP_SDT_V2 to produce 'default'
>        # symbol visibility instead.
>
> > +    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"

Yes, that would be helpful.

Stefan

Re: [PATCH v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Miroslav Rezanina]

----- Original Message -----
> From: "Stefan Hajnoczi" <stefanha@redhat.com>
> To: qemu-devel@nongnu.org
> Cc: "Peter Maydell" <peter.maydell@linaro.org>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Daniel P . Berrangé"
> <berrange@redhat.com>, wcohen@redhat.com, fche@redhat.com, kraxel@redhat.com, rjones@redhat.com,
> mrezanin@redhat.com, ddepaula@redhat.com
> Sent: Thursday, November 19, 2020 12:27:04 PM
> Subject: [PATCH v2] trace: use STAP_SDT_V2 to work around symbol visibility
> 
> QEMU binaries no longer launch successfully with recent SystemTap
> releases. This is because modular QEMU builds link the sdt semaphores
> into the main binary instead of into the shared objects where they are
> used. The symbol visibility of semaphores is 'hidden' and the dynamic
> linker prints an error during module loading:
> 
>   $ ./configure --enable-trace-backends=dtrace --enable-modules ...
>   ...
>   Failed to open module:
>   /builddir/build/BUILD/qemu-4.2.0/s390x-softmmu/../block-curl.so: undefined
>   symbol: qemu_curl_close_semaphore
> 
> The long-term solution is to generate per-module dtrace .o files and
> link them into the module instead of the main binary.
> 
> In the short term we can define STAP_SDT_V2 so dtrace(1) produces a .o
> file with 'default' symbol visibility instead of 'hidden'. This
> workaround is small and easier to merge for QEMU 5.2.
> 

Thanks for this fix.

Reviewed-by: Miroslav Rezanina <mrezanin@redhat.com>

> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: wcohen@redhat.com
> Cc: fche@redhat.com
> Cc: kraxel@redhat.com
> Cc: rjones@redhat.com
> Cc: mrezanin@redhat.com
> Cc: ddepaula@redhat.com
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v2:
>  * Define STAP_SDT_V2 everywhere [danpb]
> ---
>  configure         | 1 +
>  trace/meson.build | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/configure b/configure
> index 714e75b5d8..5d91d49c7b 100755
> --- a/configure
> +++ b/configure
> @@ -4832,6 +4832,7 @@ if have_backend "dtrace"; then
>    trace_backend_stap="no"
>    if has 'stap' ; then
>      trace_backend_stap="yes"
> +    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"
>    fi
>  fi
>  
> diff --git a/trace/meson.build b/trace/meson.build
> index d5fc45c628..843ea14495 100644
> --- a/trace/meson.build
> +++ b/trace/meson.build
> @@ -38,13 +38,13 @@ foreach dir : [ '.' ] + trace_events_subdirs
>      trace_dtrace_h = custom_target(fmt.format('trace-dtrace', 'h'),
>                                     output: fmt.format('trace-dtrace', 'h'),
>                                     input: trace_dtrace,
> -                                   command: [ 'dtrace', '-o', '@OUTPUT@',
> '-h', '-s', '@INPUT@' ])
> +                                   command: [ 'dtrace', '-DSTAP_SDT_V2',
> '-o', '@OUTPUT@', '-h', '-s', '@INPUT@' ])
>      trace_ss.add(trace_dtrace_h)
>      if host_machine.system() != 'darwin'
>        trace_dtrace_o = custom_target(fmt.format('trace-dtrace', 'o'),
>                                       output: fmt.format('trace-dtrace',
>                                       'o'),
>                                       input: trace_dtrace,
> -                                     command: [ 'dtrace', '-o', '@OUTPUT@',
> '-G', '-s', '@INPUT@' ])
> +                                     command: [ 'dtrace', '-DSTAP_SDT_V2',
> '-o', '@OUTPUT@', '-G', '-s', '@INPUT@' ])
>        trace_ss.add(trace_dtrace_o)
>      endif
>  
> --
> 2.28.0
> 
> 

-- 
Miroslav Rezanina
Software Engineer - Virtualization Team Maintainer

Re: [PATCH-for-5.2 v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Frank Ch. Eigler]

Hi -

> > Maybe add a comment? (no need to repost if you agree):
> >
> >        # Workaround to avoid dtrace(1) produces file with 'hidden'
> >        # symbol visibility, define STAP_SDT_V2 to produce 'default'
> >        # symbol visibility instead.
> >
> > > +    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"

Please note that we don't know how long this behavior will persist.
You are relying on an accident. :-)

Much of the systemtap code doesn't support real STAP_SDT_V2 format,
and /usr/include/sys/sdt.h cannot generate it at all.  That macro
tricks only the dtrace-header-generator to suppress the "hidden"
visibility attribute, but doesn't change probe metadata format to the
old V2 (in .probes sections rather than .note.* ELF notes).

We'll try not to break it, but please move toward the more proper
per-solib or per-executable hidden copies of the semaphore objects.

- FChE

Re: [PATCH-for-5.2 v2] trace: use STAP_SDT_V2 to work around symbol visibility

[Stefan Hajnoczi]

On Thu, Nov 19, 2020 at 2:38 PM Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> > > Maybe add a comment? (no need to repost if you agree):
> > >
> > >        # Workaround to avoid dtrace(1) produces file with 'hidden'
> > >        # symbol visibility, define STAP_SDT_V2 to produce 'default'
> > >        # symbol visibility instead.
> > >
> > > > +    QEMU_CFLAGS="$QEMU_CFLAGS -DSTAP_SDT_V2"
>
> Please note that we don't know how long this behavior will persist.
> You are relying on an accident. :-)
>
> Much of the systemtap code doesn't support real STAP_SDT_V2 format,
> and /usr/include/sys/sdt.h cannot generate it at all.  That macro
> tricks only the dtrace-header-generator to suppress the "hidden"
> visibility attribute, but doesn't change probe metadata format to the
> old V2 (in .probes sections rather than .note.* ELF notes).
>
> We'll try not to break it, but please move toward the more proper
> per-solib or per-executable hidden copies of the semaphore objects.

Yes, thanks. Gerd Hoffmann has started working on per-solib semaphore
linking and this workaround will be replaced by it.

Stefan