[RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Alex Bennée]

This option is supported by both gcc (since 4.7) and clang (since
7.0). Not only does this make the linkers job easier by reducing the
amount of ELF it needs to parse it also reduces the total build size
quite considerably. In my case a default build went from 5.8G to 3.9G.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
RFC:
  - currently and RFC because it should also apply to test binaries
---
 meson.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meson.build b/meson.build
index 4588bfd864..a936d1b422 100644
--- a/meson.build
+++ b/meson.build
@@ -363,7 +363,7 @@ endforeach
 
 qemu_common_flags = [
   '-D_GNU_SOURCE', '-D_FILE_OFFSET_BITS=64', '-D_LARGEFILE_SOURCE',
-  '-fno-strict-aliasing', '-fno-common', '-fwrapv' ]
+  '-fno-strict-aliasing', '-fno-common', '-fwrapv', '-gsplit-dwarf' ]
 qemu_cflags = []
 qemu_ldflags = []
 
-- 
2.39.5

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Daniel P. Berrangé]

On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
> This option is supported by both gcc (since 4.7) and clang (since
> 7.0). Not only does this make the linkers job easier by reducing the
> amount of ELF it needs to parse it also reduces the total build size
> quite considerably. In my case a default build went from 5.8G to 3.9G.

I've not come across this option before, but the docs say

  ‘-gsplit-dwarf’
     If DWARF debugging information is enabled, separate as much
     debugging information as possible into a separate output file with
     the extension ‘.dwo’.  This option allows the build system to avoid
     linking files with debug information.  To be useful, this option
     requires a debugger capable of reading ‘.dwo’ files.

In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
of the distro RPM build config will post-process all ELF files to extract
the debug info into files that we store under /usr/lib/debug. eg for
/usr/bin/qemu-system-x86_64, we get a separate
/usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
tools like GDB, etc know to look for these separate files.

When I test our RPM build process with this -gsplit-dwarf flag added,
we fail to find any debug symbols in the QEMU binaries and thus the
whole RPM package build aborts.

IOW, if we add this to QEMU, my best guess currently is that we're
going to have to revert it downstream.

> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> RFC:
>   - currently and RFC because it should also apply to test binaries
> ---
>  meson.build | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meson.build b/meson.build
> index 4588bfd864..a936d1b422 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -363,7 +363,7 @@ endforeach
>  
>  qemu_common_flags = [
>    '-D_GNU_SOURCE', '-D_FILE_OFFSET_BITS=64', '-D_LARGEFILE_SOURCE',
> -  '-fno-strict-aliasing', '-fno-common', '-fwrapv' ]
> +  '-fno-strict-aliasing', '-fno-common', '-fwrapv', '-gsplit-dwarf' ]
>  qemu_cflags = []
>  qemu_ldflags = []
>  
> -- 
> 2.39.5
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
>> This option is supported by both gcc (since 4.7) and clang (since
>> 7.0). Not only does this make the linkers job easier by reducing the
>> amount of ELF it needs to parse it also reduces the total build size
>> quite considerably. In my case a default build went from 5.8G to 3.9G.
>
> I've not come across this option before, but the docs say
>
>   ‘-gsplit-dwarf’
>      If DWARF debugging information is enabled, separate as much
>      debugging information as possible into a separate output file with
>      the extension ‘.dwo’.  This option allows the build system to avoid
>      linking files with debug information.  To be useful, this option
>      requires a debugger capable of reading ‘.dwo’ files.
>
> In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
> of the distro RPM build config will post-process all ELF files to extract
> the debug info into files that we store under /usr/lib/debug. eg for
> /usr/bin/qemu-system-x86_64, we get a separate
> /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
> tools like GDB, etc know to look for these separate files.

A modern gdb can certainly handle fetching the debug out of the .dwo
files when debugging.

>
> When I test our RPM build process with this -gsplit-dwarf flag added,
> we fail to find any debug symbols in the QEMU binaries and thus the
> whole RPM package build aborts.

Do you know if you build the kernel the same way? That has had the
option to do split dwarf files for awhile.

The tools dwp and dwz are used to collect the split files and generate
the final (optionally compressed) dwarf symbol file.

>
> IOW, if we add this to QEMU, my best guess currently is that we're
> going to have to revert it downstream.
>
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> 
>> ---
>> RFC:
>>   - currently and RFC because it should also apply to test binaries
>> ---
>>  meson.build | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/meson.build b/meson.build
>> index 4588bfd864..a936d1b422 100644
>> --- a/meson.build
>> +++ b/meson.build
>> @@ -363,7 +363,7 @@ endforeach
>>  
>>  qemu_common_flags = [
>>    '-D_GNU_SOURCE', '-D_FILE_OFFSET_BITS=64', '-D_LARGEFILE_SOURCE',
>> -  '-fno-strict-aliasing', '-fno-common', '-fwrapv' ]
>> +  '-fno-strict-aliasing', '-fno-common', '-fwrapv', '-gsplit-dwarf' ]
>>  qemu_cflags = []
>>  qemu_ldflags = []
>>  
>> -- 
>> 2.39.5
>> 
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Daniel P. Berrangé]

On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
> >> This option is supported by both gcc (since 4.7) and clang (since
> >> 7.0). Not only does this make the linkers job easier by reducing the
> >> amount of ELF it needs to parse it also reduces the total build size
> >> quite considerably. In my case a default build went from 5.8G to 3.9G.
> >
> > I've not come across this option before, but the docs say
> >
> >   ‘-gsplit-dwarf’
> >      If DWARF debugging information is enabled, separate as much
> >      debugging information as possible into a separate output file with
> >      the extension ‘.dwo’.  This option allows the build system to avoid
> >      linking files with debug information.  To be useful, this option
> >      requires a debugger capable of reading ‘.dwo’ files.
> >
> > In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
> > of the distro RPM build config will post-process all ELF files to extract
> > the debug info into files that we store under /usr/lib/debug. eg for
> > /usr/bin/qemu-system-x86_64, we get a separate
> > /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
> > tools like GDB, etc know to look for these separate files.
> 
> A modern gdb can certainly handle fetching the debug out of the .dwo
> files when debugging.
> 
> >
> > When I test our RPM build process with this -gsplit-dwarf flag added,
> > we fail to find any debug symbols in the QEMU binaries and thus the
> > whole RPM package build aborts.
> 
> Do you know if you build the kernel the same way? That has had the
> option to do split dwarf files for awhile.

The kernel use of -gsplit-dwarf is behind the CONFIG_DEBUG_INFO_SPLIT
KConfig option, and that remains unset in Fedora/RHEL builds

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Daniel P. Berrangé]

On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
> >> This option is supported by both gcc (since 4.7) and clang (since
> >> 7.0). Not only does this make the linkers job easier by reducing the
> >> amount of ELF it needs to parse it also reduces the total build size
> >> quite considerably. In my case a default build went from 5.8G to 3.9G.
> >
> > I've not come across this option before, but the docs say
> >
> >   ‘-gsplit-dwarf’
> >      If DWARF debugging information is enabled, separate as much
> >      debugging information as possible into a separate output file with
> >      the extension ‘.dwo’.  This option allows the build system to avoid
> >      linking files with debug information.  To be useful, this option
> >      requires a debugger capable of reading ‘.dwo’ files.
> >
> > In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
> > of the distro RPM build config will post-process all ELF files to extract
> > the debug info into files that we store under /usr/lib/debug. eg for
> > /usr/bin/qemu-system-x86_64, we get a separate
> > /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
> > tools like GDB, etc know to look for these separate files.
> 
> A modern gdb can certainly handle fetching the debug out of the .dwo
> files when debugging.

See this response from one of the upstream GCC maintainers pretty much
recommending against (on by default) use of -gsplit-dwarf:

  https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RXA55BTO62NWWHCTUFYGEVHJKZPL5EKE/

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Daniel P. Berrangé]

On Tue, Mar 04, 2025 at 12:11:25PM +0000, Daniel P. Berrangé wrote:
> On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > 
> > > On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
> > >> This option is supported by both gcc (since 4.7) and clang (since
> > >> 7.0). Not only does this make the linkers job easier by reducing the
> > >> amount of ELF it needs to parse it also reduces the total build size
> > >> quite considerably. In my case a default build went from 5.8G to 3.9G.
> > >
> > > I've not come across this option before, but the docs say
> > >
> > >   ‘-gsplit-dwarf’
> > >      If DWARF debugging information is enabled, separate as much
> > >      debugging information as possible into a separate output file with
> > >      the extension ‘.dwo’.  This option allows the build system to avoid
> > >      linking files with debug information.  To be useful, this option
> > >      requires a debugger capable of reading ‘.dwo’ files.
> > >
> > > In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
> > > of the distro RPM build config will post-process all ELF files to extract
> > > the debug info into files that we store under /usr/lib/debug. eg for
> > > /usr/bin/qemu-system-x86_64, we get a separate
> > > /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
> > > tools like GDB, etc know to look for these separate files.
> > 
> > A modern gdb can certainly handle fetching the debug out of the .dwo
> > files when debugging.
> 
> See this response from one of the upstream GCC maintainers pretty much
> recommending against (on by default) use of -gsplit-dwarf:
> 
>   https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RXA55BTO62NWWHCTUFYGEVHJKZPL5EKE/

Since lists.fp.o is being rather unreliable today...

[quote]
On Tue, Mar 04, 2025 at 11:44:18AM +0000, Daniel P. Berrangé wrote:
> The QEMU community is discussing possible use of -gsplit-dwarf as a default
> option for QEMU's build system:
>
>  https://lists.nongnu.org/archive/html/qemu-devel/2025-03/msg00424.html
>
> This option causes debug symbols to be written to separate .dwo files
> instead of the .o files or final executables.

Don't.
-gsplit-dwarf is something to get faster link times at the expense of slower
debugging and more files around.
So, it can be useful for fast modify/rebuild/test cycles during development,
but is something that is really undesirable for the distro builds.
There you don't mind slightly longer link times, that is done once per
build, but having to ship larger debug info and especially when it is split
into hundreds or thousands of small files is then a price everybody who
downloads the package debuginfo pays.
debugedit can't deal with it, dwz can't either, in order to ship something
reasonable one would need to dwp the separate debug info back into the
normal debug info.

        Jakub
[/quote]

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Mar 04, 2025 at 12:11:25PM +0000, Daniel P. Berrangé wrote:
>> On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
>> > Daniel P. Berrangé <berrange@redhat.com> writes:
>> > 
>> > > On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
>> > >> This option is supported by both gcc (since 4.7) and clang (since
>> > >> 7.0). Not only does this make the linkers job easier by reducing the
>> > >> amount of ELF it needs to parse it also reduces the total build size
>> > >> quite considerably. In my case a default build went from 5.8G to 3.9G.
>> > >
>> > > I've not come across this option before, but the docs say
>> > >
>> > >   ‘-gsplit-dwarf’
>> > >      If DWARF debugging information is enabled, separate as much
>> > >      debugging information as possible into a separate output file with
>> > >      the extension ‘.dwo’.  This option allows the build system to avoid
>> > >      linking files with debug information.  To be useful, this option
>> > >      requires a debugger capable of reading ‘.dwo’ files.
>> > >
>> > > In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
>> > > of the distro RPM build config will post-process all ELF files to extract
>> > > the debug info into files that we store under /usr/lib/debug. eg for
>> > > /usr/bin/qemu-system-x86_64, we get a separate
>> > > /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
>> > > tools like GDB, etc know to look for these separate files.
>> > 
>> > A modern gdb can certainly handle fetching the debug out of the .dwo
>> > files when debugging.
>> 
>> See this response from one of the upstream GCC maintainers pretty much
>> recommending against (on by default) use of -gsplit-dwarf:
>> 
>>   https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RXA55BTO62NWWHCTUFYGEVHJKZPL5EKE/
>
> Since lists.fp.o is being rather unreliable today...
>
> [quote]
> On Tue, Mar 04, 2025 at 11:44:18AM +0000, Daniel P. Berrangé wrote:
>> The QEMU community is discussing possible use of -gsplit-dwarf as a default
>> option for QEMU's build system:
>>
>>  https://lists.nongnu.org/archive/html/qemu-devel/2025-03/msg00424.html
>>
>> This option causes debug symbols to be written to separate .dwo files
>> instead of the .o files or final executables.
>
> Don't.
> -gsplit-dwarf is something to get faster link times at the expense of slower
> debugging and more files around.
> So, it can be useful for fast modify/rebuild/test cycles during development,
> but is something that is really undesirable for the distro builds.
> There you don't mind slightly longer link times, that is done once per
> build, but having to ship larger debug info and especially when it is split
> into hundreds or thousands of small files is then a price everybody who
> downloads the package debuginfo pays.
> debugedit can't deal with it, dwz can't either, in order to ship something
> reasonable one would need to dwp the separate debug info back into the
> normal debug info.

Is this not feasible? I large part of the saving is not having the same
set of symbols defined in multiple object files as stuff gets merged
during the build.

Maybe we need a simple --debug-type which defaults to split but can be
selected as unified for the distro builds?

>
>         Jakub
> [/quote]
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Philippe Mathieu-Daudé]

On 4/3/25 13:11, Daniel P. Berrangé wrote:
> On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>>> On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
>>>> This option is supported by both gcc (since 4.7) and clang (since
>>>> 7.0). Not only does this make the linkers job easier by reducing the
>>>> amount of ELF it needs to parse it also reduces the total build size
>>>> quite considerably. In my case a default build went from 5.8G to 3.9G.
>>>
>>> I've not come across this option before, but the docs say
>>>
>>>    ‘-gsplit-dwarf’
>>>       If DWARF debugging information is enabled, separate as much
>>>       debugging information as possible into a separate output file with
>>>       the extension ‘.dwo’.  This option allows the build system to avoid
>>>       linking files with debug information.  To be useful, this option
>>>       requires a debugger capable of reading ‘.dwo’ files.
>>>
>>> In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
>>> of the distro RPM build config will post-process all ELF files to extract
>>> the debug info into files that we store under /usr/lib/debug. eg for
>>> /usr/bin/qemu-system-x86_64, we get a separate
>>> /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
>>> tools like GDB, etc know to look for these separate files.
>>
>> A modern gdb can certainly handle fetching the debug out of the .dwo
>> files when debugging.
> 
> See this response from one of the upstream GCC maintainers pretty much
> recommending against (on by default) use of -gsplit-dwarf:
> 
>    https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RXA55BTO62NWWHCTUFYGEVHJKZPL5EKE/

Jakub said:

  "it can be useful for fast modify/rebuild/test cycles during
   development, but is something that is really undesirable for
   the distro builds."

Do distro use --enable-debug by default? Otherwise it might be
useful to add it for developers.

Re: [RFC PATCH] meson.build: add -gsplit-dwarf to default cflags

[Daniel P. Berrangé]

On Tue, Mar 04, 2025 at 02:40:31PM +0100, Philippe Mathieu-Daudé wrote:
> On 4/3/25 13:11, Daniel P. Berrangé wrote:
> > On Tue, Mar 04, 2025 at 11:33:44AM +0000, Alex Bennée wrote:
> > > Daniel P. Berrangé <berrange@redhat.com> writes:
> > > 
> > > > On Mon, Mar 03, 2025 at 10:17:03PM +0000, Alex Bennée wrote:
> > > > > This option is supported by both gcc (since 4.7) and clang (since
> > > > > 7.0). Not only does this make the linkers job easier by reducing the
> > > > > amount of ELF it needs to parse it also reduces the total build size
> > > > > quite considerably. In my case a default build went from 5.8G to 3.9G.
> > > > 
> > > > I've not come across this option before, but the docs say
> > > > 
> > > >    ‘-gsplit-dwarf’
> > > >       If DWARF debugging information is enabled, separate as much
> > > >       debugging information as possible into a separate output file with
> > > >       the extension ‘.dwo’.  This option allows the build system to avoid
> > > >       linking files with debug information.  To be useful, this option
> > > >       requires a debugger capable of reading ‘.dwo’ files.
> > > > 
> > > > In Fedora and RHEL we build QEMU will full debug enabled, and then a feature
> > > > of the distro RPM build config will post-process all ELF files to extract
> > > > the debug info into files that we store under /usr/lib/debug. eg for
> > > > /usr/bin/qemu-system-x86_64, we get a separate
> > > > /usr/lib/debug/bin/qemu-system-x86_64-9.1.3-1.fc41.x86_64.debug, and
> > > > tools like GDB, etc know to look for these separate files.
> > > 
> > > A modern gdb can certainly handle fetching the debug out of the .dwo
> > > files when debugging.
> > 
> > See this response from one of the upstream GCC maintainers pretty much
> > recommending against (on by default) use of -gsplit-dwarf:
> > 
> >    https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RXA55BTO62NWWHCTUFYGEVHJKZPL5EKE/
> 
> Jakub said:
> 
>  "it can be useful for fast modify/rebuild/test cycles during
>   development, but is something that is really undesirable for
>   the distro builds."
> 
> Do distro use --enable-debug by default? Otherwise it might be
> useful to add it for developers.

We don't have --enable-debug in Fedora/RHEL, and I doubt other distros
would enable it given it turns off all optimization as well as introducing
other static debug overheads.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|