[PATCH v3] docs: Add debugging chapter to development documentation

[PATCH v3] docs: Add debugging chapter to development documentation

[Glenn Washburn]

Debugging GRUB can be tricky and require arcane knowledge. This will
help those unfamiliar with the process to get started debugging GRUB
with less effort.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
Changes from v1:
 * Add gdbinfo section
---
Interdiff against v2:
  diff --git a/docs/grub-dev.texi b/docs/grub-dev.texi
  index 188ca9c7ca6e..72470b42c61a 100644
  --- a/docs/grub-dev.texi
  +++ b/docs/grub-dev.texi
  @@ -638,7 +638,7 @@ various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
   @section i386-pc
   
   The i386-pc target is a good place to start when first debugging GRUB2
  -because in some respects its easier than EFI platforms. The reason being
  +because in some respects it's easier than EFI platforms. The reason being
   that the initial load address is always known in advance. To start
   debugging GRUB2 first QEMU must be started in GDB stub mode. The following
   command is a simple illustration:
  @@ -688,11 +688,11 @@ it does add the module symbols with the appropriate offset.
   @section x86_64-efi
   
   Using GDB to debug GRUB2 for the x86_64-efi target has some similarities with
  -the i386-pc target. Please read be familiar with the @ref{i386-pc} section
  -when reading this one. Extra care must be used to run QEMU such that it boots
  -a UEFI firmware. This usually involves either using the @samp{-bios} option
  -with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware via
  -pflash. This document will not go further into how to do this as there are
  +the i386-pc target. Please read and familiarize yourself with the @ref{i386-pc}
  +section when reading this one. Extra care must be used to run QEMU such that it
  +boots a UEFI firmware. This usually involves either using the @samp{-bios}
  +option with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware
  +via pflash. This document will not go further into how to do this as there are
   ample resource on the web.
   
   Like all EFI implementations, on x86_64-efi the (U)EFI firmware that loads
  @@ -700,7 +700,7 @@ the GRUB2 EFI application determines at runtime where the application will
   be loaded. This means that we do not know where to tell GDB to load the
   symbols for the GRUB2 core until the (U)EFI firmware determines it. There are
   two good ways of figuring this out when running in QEMU: use a @ref{OVMF debug log,
  -debug build of OVMF} and check the debug log or have GRUB2 say where it is
  +debug build of OVMF} and check the debug log, or have GRUB2 say where it is
   loaded. Neither of these are ideal because they both generally give the
   information after GRUB2 is already running, which makes debugging early boot
   infeasible. Technically, the first method does give the load address before
  @@ -734,11 +734,11 @@ application must be run via QEMU at least once prior in order to get the
   load address. Two methods for obtaining the load address are described in
   two subsections below. Generally speaking, the load address does not change
   between QEMU runs. There are exceptions to this, namely that different
  -GRUB2 EFI applications can be run at different addresses. Also, its been
  +GRUB2 EFI applications can be run at different addresses. Also, it has been
   observed that after running the EFI application for the first time, the
   second run will some times have a different load address, but subsequent
   runs of the same EFI application will have the same load address as the
  -second run. And its a near certainty that if the GRUB EFI binary has changed,
  +second run. And it's a near certainty that if the GRUB EFI binary has changed,
   eg. been recompiled, the load address will also be different.
   
   This ability to predict what the load address will be allows one to assume
  @@ -752,7 +752,7 @@ gdb -x gdb_grub -ex 'dynamic_load_symbols @var{address of .text section}'
   @end example
   
   If you load the symbols in this manner and, after continuing execution, do
  -not see output showing the loading of modules symbol, then its very likely
  +not see output showing the loading of modules symbol, then it is very likely
   that the load address was incorrect.
   
   Another thing to be aware of is how the loading of the GRUB image by the
  @@ -760,8 +760,8 @@ firmware affects previously set software breakpoints. On x86 platforms,
   software breakpoints are implemented by GDB by writing a special processor
   instruction at the location of the desired breakpoint. This special instruction
   when executed will stop the program execution and hand control to the
  -debugger, GDB. GDB will first saves the instruction bytes that will be
  -overwritten at the breakpoint, and will put them back when the breakpoint
  +debugger, GDB. GDB will first save the instruction bytes that are
  +overwritten at the breakpoint and will put them back when the breakpoint
   is hit. If GRUB is being run for the first time in QEMU, the firmware will
   be loading the GRUB image into memory where every byte is already set to 0.
   This means that if a breakpoint is set before GRUB is loaded, GDB will save

 docs/grub-dev.texi | 224 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 224 insertions(+)

diff --git a/docs/grub-dev.texi b/docs/grub-dev.texi
index 31eb99ea2994..72470b42c61a 100644
--- a/docs/grub-dev.texi
+++ b/docs/grub-dev.texi
@@ -79,6 +79,7 @@ This edition documents version @value{VERSION}.
 * Contributing Changes::
 * Setting up and running test suite::
 * Updating External Code::
+* Debugging::
 * Porting::
 * Error Handling::
 * Stack and heap size::
@@ -595,6 +596,229 @@ cp minilzo-2.10/*.[hc] grub-core/lib/minilzo
 rm -r minilzo-2.10*
 @end example
 
+@node Debugging
+@chapter Debugging
+
+GRUB2 can be difficult to debug because it runs on the bare-metal and thus
+does not have the debugging facilities normally provided by an operating
+system. This chapter aims to provide useful information on some ways to
+debug GRUB2 for some architectures. It by no means intends to be exhaustive.
+The focus will be one x86_64 and i386 architectures. Luckily for some issues
+virtual machines have made the ability to debug GRUB2 much easier, and this
+chapter will focus debugging via the QEMU virtual machine. We will not be
+going over debugging of the userland tools (eg. grub-install), there are
+many tutorials on debugging programs in userland.
+
+You will need GDB and the QEMU binaries for your system, on Debian these
+can be installed with the @samp{gdb} and @samp{qemu-system-x86} packages.
+Also it is assumed that you have already successfully compiled GRUB2 from
+source for the target specified in the section below and have some
+familiarity with GDB. When GRUB2 is built it will create many different
+binaries. The ones of concern will be in the @file{grub-core}
+directory of the GRUB2 build dir. To aide in debugging we will want the
+debugging symbols generated during the build because these symbols are not
+kept in the binaries which get installed to the boot location. The build
+process outputs two sets of binaries, one without symbols which gets executed
+at boot, and another set of ELF images with debugging symbols. The built
+images with debugging symbols will have a @file{.image} suffix, and the ones
+without a @file{.img} suffix. Similarly, loadable modules with debugging
+symbols will have a @file{.module} suffix, and ones without a @file{.mod}
+suffix. In the case of the kernel the binary with symbols is named
+@file{kernel.exec}.
+
+In the following sections, information will be provided on debugging on
+various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
+
+@menu
+* i386-pc::
+* x86_64-efi::
+@end menu
+
+@node i386-pc
+@section i386-pc
+
+The i386-pc target is a good place to start when first debugging GRUB2
+because in some respects it's easier than EFI platforms. The reason being
+that the initial load address is always known in advance. To start
+debugging GRUB2 first QEMU must be started in GDB stub mode. The following
+command is a simple illustration:
+
+@example
+qemu-system-i386 -drive file=disk.img,format=raw \
+    -device virtio-scsi-pci,id=scsi0 -S -s
+@end example
+
+This will start a QEMU instance booting from @file{disk.img}. It will pause
+at start waiting for a GDB instance to attach to it. You should change
+@file{disk.img} to something more appropriate. A block device can be used,
+but you may need to run QEMU as a privileged user.
+
+To connect to this QEMU instance with GDB, the @code{target remote} GDB
+command must be used. We also need to load a binary image, preferably with
+symbols. This can be done using the GDB command @code{file kernel.exec}, if
+GDB is started from the @file{grub-core} directory in the GRUB2 build
+directory. GRUB2 developers have made this more simple by including a GDB
+script which does much of the setup. This file at @file{grub-core/gdb_grub}
+of the build directory and is also installed via @command{make install}.
+If not building GRUB, the distribution may have a package which installs
+this GDB script along with debug symbol binaries, such as Debian's
+@samp{grub-pc-dbg} package. The GDB scripts is intended to by used
+like so, assuming:
+
+@example
+cd $(dirname /path/to/script/gdb_grub)
+gdb -x gdb_grub
+@end example
+
+Once GDB has been started with the @file{gdb_grub} script it will
+automatically connect to the QEMU instance. You can then do things you
+normally would in GDB like set a break point on @var{grub_main}.
+
+Setting breakpoints in modules is trickier since they haven't been loaded
+yet and are loaded at addresses determined at runtime. The module could be
+loaded to different addresses in different QEMU instances. The debug symbols
+in the modules @file{.module} binary, thus are always wrong, and GDB needs
+to be told where to load the symbols to. But this must happen at runtime
+after GRUB2 has determined where the module will get loaded. Luckily the
+@file{gdb_grub} script takes care of this with the @command{runtime_load_module}
+command, which configures GDB to watch for GRUB2 module loading and when
+it does add the module symbols with the appropriate offset.
+
+@node x86_64-efi
+@section x86_64-efi
+
+Using GDB to debug GRUB2 for the x86_64-efi target has some similarities with
+the i386-pc target. Please read and familiarize yourself with the @ref{i386-pc}
+section when reading this one. Extra care must be used to run QEMU such that it
+boots a UEFI firmware. This usually involves either using the @samp{-bios}
+option with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware
+via pflash. This document will not go further into how to do this as there are
+ample resource on the web.
+
+Like all EFI implementations, on x86_64-efi the (U)EFI firmware that loads
+the GRUB2 EFI application determines at runtime where the application will
+be loaded. This means that we do not know where to tell GDB to load the
+symbols for the GRUB2 core until the (U)EFI firmware determines it. There are
+two good ways of figuring this out when running in QEMU: use a @ref{OVMF debug log,
+debug build of OVMF} and check the debug log, or have GRUB2 say where it is
+loaded. Neither of these are ideal because they both generally give the
+information after GRUB2 is already running, which makes debugging early boot
+infeasible. Technically, the first method does give the load address before
+GRUB2 is run, but without debugging the EFI firmware with symbols, the author
+currently does not know how to cause the OVMF firmware to pause at that point
+to use the load address before GRUB2 is run.
+
+Even after getting the application load address, the loading of core symbols
+is complicated by the fact that the debugging symbols for the kernel are in
+an ELF binary named @file{kernel.exec} while what is in memory are sections
+for the PE32+ EFI binary. When @command{grub-mkimage} creates the PE32+
+binary it condenses several segments from the ELF kernel binary into one
+.data section in the PE32+ binary. This must be taken into account to
+properly load the other non-text sections. Otherwise, GDB will work as
+expected when breaking on functions, but, for instance, global variables
+will point to the wrong address in memory and thus give incorrect values
+(which can be difficult to debug).
+
+The calculating of the correct offsets for sections when loading symbol
+files are taken care of when loading the kernel symbols via the user-defined
+GDB command @command{dynamic_load_kernel_exec_symbols}, which takes one
+argument, the address where the text section is loaded, as determined by
+one of the methods above. Alternatively, the command @command{dynamic_load_symbols}
+with the text section address as an agrument can be called to load the
+kernel symbols and setup loading the module symbols as they are loaded at
+runtime.
+
+In the author's experience, when debugging with QEMU and OVMF, to have
+debugging symbols loaded at the start of GRUB2 execution the GRUB2 EFI
+application must be run via QEMU at least once prior in order to get the
+load address. Two methods for obtaining the load address are described in
+two subsections below. Generally speaking, the load address does not change
+between QEMU runs. There are exceptions to this, namely that different
+GRUB2 EFI applications can be run at different addresses. Also, it has been
+observed that after running the EFI application for the first time, the
+second run will some times have a different load address, but subsequent
+runs of the same EFI application will have the same load address as the
+second run. And it's a near certainty that if the GRUB EFI binary has changed,
+eg. been recompiled, the load address will also be different.
+
+This ability to predict what the load address will be allows one to assume
+the load address on subsequent runs and thus load the symbols before GRUB2
+starts. The following command illustrates this, assuming that QEMU is
+running and waiting for a debugger connection and the current working
+directory is where @file{gdb_grub} resides:
+
+@example
+gdb -x gdb_grub -ex 'dynamic_load_symbols @var{address of .text section}'
+@end example
+
+If you load the symbols in this manner and, after continuing execution, do
+not see output showing the loading of modules symbol, then it is very likely
+that the load address was incorrect.
+
+Another thing to be aware of is how the loading of the GRUB image by the
+firmware affects previously set software breakpoints. On x86 platforms,
+software breakpoints are implemented by GDB by writing a special processor
+instruction at the location of the desired breakpoint. This special instruction
+when executed will stop the program execution and hand control to the
+debugger, GDB. GDB will first save the instruction bytes that are
+overwritten at the breakpoint and will put them back when the breakpoint
+is hit. If GRUB is being run for the first time in QEMU, the firmware will
+be loading the GRUB image into memory where every byte is already set to 0.
+This means that if a breakpoint is set before GRUB is loaded, GDB will save
+the 0-byte(s) where the the special instruction will go. Then when the firmware
+loads the GRUB image and because it is unaware of the debugger, it will
+write the GRUB image to memory, overwriting anything that was there previously,
+notably in this case the instruction that implements the software breakpoint.
+This will be confusing for the person using GDB because GDB will show the
+breakpoint as set, but the brekapoint will never be hit. Furthermore, GDB
+then becomes confused, such that even deleting an recreating the breakpoint
+will not create usable breakpoints. The @file{gdb_grub} script takes care of
+this by saving the breakpoints just before they are overwritten, and then
+restores them at the start of GRUB execution. So breakpoints for GRUB can be
+set before GRUB is loaded, but be mindful of this effect if you are confused
+as to why breakpoints are not getting hit.
+
+Also note, that hardware breakpoints do not suffer this problem. They are
+implemented by having the breakpoint address in special debug registers on
+the CPU. So they can always be set freely without regard to whether GRUB has
+been loaded or not. The reason that hardware breakpoints aren't always used
+is because there are a limited number of them, usually around 4 on various
+CPUs, and specifically exactly 4 for x86 CPUs. The @file{gdb_grub} script
+goes out of its way to not use hardware breakpoints internally and when
+needed use them as short a time as possible, thus allowing the user to have a
+maximal number at their disposal.
+
+@node OVMF debug log
+@subsection OVMF debug log
+
+In order to get the GRUB2 load address from OVMF, first, a debug build
+of OVMF must be obtained (@uref{https://github.com/retrage/edk2-nightly/raw/master/bin/DEBUGX64_OVMF.fd,
+here is one} which is not officially recommended). OVMF will output debug
+messages to a special serial device, which we must add to QEMU. The following
+QEMU command will run the debug OVMF and write the debug messages to a
+file named @file{debug.log}. It is assumed that @file{disk.img} is a disk
+image or block device that is setup to boot GRUB2 EFI.
+
+@example
+qemu-system-x86_64 -bios /path/to/debug/OVMF.fd \
+    -drive file=disk.img,format=raw \
+    -device virtio-scsi-pci,id=scsi0 \
+    -debugcon file:debug.log -global isa-debugcon.iobase=0x402
+@end example
+
+If GRUB2 was started by the (U)EFI firmware, then in the @file{debug.log}
+file one of the last lines should be a log message like:
+@samp{Loading driver at 0x00006AEE000 EntryPoint=0x00006AEE756}. This
+means that the GRUB2 EFI application was loaded at @samp{0x00006AEE000} and
+its .text section is at @samp{0x00006AEE756}.
+
+@node Using the gdbinfo command
+@subsection Using the gdbinfo command
+
+On EFI platforms the command @command{gdbinfo} will output a string that
+is to be run in a GDB session running with the @file{gdb_grub} GDB script.
+
+
 @node Porting
 @chapter Porting
 
-- 
2.34.1

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Daniel Kiper]

On Tue, Jun 06, 2023 at 12:48:39AM -0500, Glenn Washburn wrote:
> Debugging GRUB can be tricky and require arcane knowledge. This will
> help those unfamiliar with the process to get started debugging GRUB
> with less effort.
>
> Signed-off-by: Glenn Washburn <development@efficientek.com>

Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>

Thank you for adding this debugging chapter!

Daniel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 20594 bytes --]

Oops, apologies for the late reply. Reading through it again, I found a
few more small nits:

On Tue, Jun 06, 2023 at 00:48:39 -0500, Glenn Washburn wrote:
> Debugging GRUB can be tricky and require arcane knowledge. This will
> help those unfamiliar with the process to get started debugging GRUB
> with less effort.
> 
> Signed-off-by: Glenn Washburn <development@efficientek.com>
> ---
> Changes from v1:
>  * Add gdbinfo section
> ---
> Interdiff against v2:
>   diff --git a/docs/grub-dev.texi b/docs/grub-dev.texi
>   index 188ca9c7ca6e..72470b42c61a 100644
>   --- a/docs/grub-dev.texi
>   +++ b/docs/grub-dev.texi
>   @@ -638,7 +638,7 @@ various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
>    @section i386-pc
>    
>    The i386-pc target is a good place to start when first debugging GRUB2
>   -because in some respects its easier than EFI platforms. The reason being
>   +because in some respects it's easier than EFI platforms. The reason being
>    that the initial load address is always known in advance. To start
>    debugging GRUB2 first QEMU must be started in GDB stub mode. The following
>    command is a simple illustration:
>   @@ -688,11 +688,11 @@ it does add the module symbols with the appropriate offset.
>    @section x86_64-efi
>    
>    Using GDB to debug GRUB2 for the x86_64-efi target has some similarities with
>   -the i386-pc target. Please read be familiar with the @ref{i386-pc} section
>   -when reading this one. Extra care must be used to run QEMU such that it boots
>   -a UEFI firmware. This usually involves either using the @samp{-bios} option
>   -with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware via
>   -pflash. This document will not go further into how to do this as there are
>   +the i386-pc target. Please read and familiarize yourself with the @ref{i386-pc}
>   +section when reading this one. Extra care must be used to run QEMU such that it
>   +boots a UEFI firmware. This usually involves either using the @samp{-bios}
>   +option with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware
>   +via pflash. This document will not go further into how to do this as there are
>    ample resource on the web.
>    
>    Like all EFI implementations, on x86_64-efi the (U)EFI firmware that loads
>   @@ -700,7 +700,7 @@ the GRUB2 EFI application determines at runtime where the application will
>    be loaded. This means that we do not know where to tell GDB to load the
>    symbols for the GRUB2 core until the (U)EFI firmware determines it. There are
>    two good ways of figuring this out when running in QEMU: use a @ref{OVMF debug log,
>   -debug build of OVMF} and check the debug log or have GRUB2 say where it is
>   +debug build of OVMF} and check the debug log, or have GRUB2 say where it is
>    loaded. Neither of these are ideal because they both generally give the
>    information after GRUB2 is already running, which makes debugging early boot
>    infeasible. Technically, the first method does give the load address before
>   @@ -734,11 +734,11 @@ application must be run via QEMU at least once prior in order to get the
>    load address. Two methods for obtaining the load address are described in
>    two subsections below. Generally speaking, the load address does not change
>    between QEMU runs. There are exceptions to this, namely that different
>   -GRUB2 EFI applications can be run at different addresses. Also, its been
>   +GRUB2 EFI applications can be run at different addresses. Also, it has been
>    observed that after running the EFI application for the first time, the
>    second run will some times have a different load address, but subsequent
>    runs of the same EFI application will have the same load address as the
>   -second run. And its a near certainty that if the GRUB EFI binary has changed,
>   +second run. And it's a near certainty that if the GRUB EFI binary has changed,
>    eg. been recompiled, the load address will also be different.
>    
>    This ability to predict what the load address will be allows one to assume
>   @@ -752,7 +752,7 @@ gdb -x gdb_grub -ex 'dynamic_load_symbols @var{address of .text section}'
>    @end example
>    
>    If you load the symbols in this manner and, after continuing execution, do
>   -not see output showing the loading of modules symbol, then its very likely
>   +not see output showing the loading of modules symbol, then it is very likely
>    that the load address was incorrect.
>    
>    Another thing to be aware of is how the loading of the GRUB image by the
>   @@ -760,8 +760,8 @@ firmware affects previously set software breakpoints. On x86 platforms,
>    software breakpoints are implemented by GDB by writing a special processor
>    instruction at the location of the desired breakpoint. This special instruction
>    when executed will stop the program execution and hand control to the
>   -debugger, GDB. GDB will first saves the instruction bytes that will be
>   -overwritten at the breakpoint, and will put them back when the breakpoint
>   +debugger, GDB. GDB will first save the instruction bytes that are
>   +overwritten at the breakpoint and will put them back when the breakpoint
>    is hit. If GRUB is being run for the first time in QEMU, the firmware will
>    be loading the GRUB image into memory where every byte is already set to 0.
>    This means that if a breakpoint is set before GRUB is loaded, GDB will save
> 
>  docs/grub-dev.texi | 224 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 224 insertions(+)
> 
> diff --git a/docs/grub-dev.texi b/docs/grub-dev.texi
> index 31eb99ea2994..72470b42c61a 100644
> --- a/docs/grub-dev.texi
> +++ b/docs/grub-dev.texi
> @@ -79,6 +79,7 @@ This edition documents version @value{VERSION}.
>  * Contributing Changes::
>  * Setting up and running test suite::
>  * Updating External Code::
> +* Debugging::
>  * Porting::
>  * Error Handling::
>  * Stack and heap size::
> @@ -595,6 +596,229 @@ cp minilzo-2.10/*.[hc] grub-core/lib/minilzo
>  rm -r minilzo-2.10*
>  @end example
>  
> +@node Debugging
> +@chapter Debugging
> +
> +GRUB2 can be difficult to debug because it runs on the bare-metal and thus
> +does not have the debugging facilities normally provided by an operating
> +system. This chapter aims to provide useful information on some ways to
> +debug GRUB2 for some architectures. It by no means intends to be exhaustive.
> +The focus will be one x86_64 and i386 architectures. Luckily for some issues
> +virtual machines have made the ability to debug GRUB2 much easier, and this
> +chapter will focus debugging via the QEMU virtual machine. We will not be
> +going over debugging of the userland tools (eg. grub-install), there are
> +many tutorials on debugging programs in userland.
> +
> +You will need GDB and the QEMU binaries for your system, on Debian these
> +can be installed with the @samp{gdb} and @samp{qemu-system-x86} packages.
> +Also it is assumed that you have already successfully compiled GRUB2 from
> +source for the target specified in the section below and have some
> +familiarity with GDB. When GRUB2 is built it will create many different
> +binaries. The ones of concern will be in the @file{grub-core}
> +directory of the GRUB2 build dir. To aide in debugging we will want the
> +debugging symbols generated during the build because these symbols are not
> +kept in the binaries which get installed to the boot location. The build
> +process outputs two sets of binaries, one without symbols which gets executed
> +at boot, and another set of ELF images with debugging symbols. The built
> +images with debugging symbols will have a @file{.image} suffix, and the ones
> +without a @file{.img} suffix. Similarly, loadable modules with debugging
> +symbols will have a @file{.module} suffix, and ones without a @file{.mod}
> +suffix. In the case of the kernel the binary with symbols is named
> +@file{kernel.exec}.
> +
> +In the following sections, information will be provided on debugging on
> +various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
> +
> +@menu
> +* i386-pc::
> +* x86_64-efi::
> +@end menu
> +
> +@node i386-pc
> +@section i386-pc
> +
> +The i386-pc target is a good place to start when first debugging GRUB2
> +because in some respects it's easier than EFI platforms. The reason being
> +that the initial load address is always known in advance. To start
> +debugging GRUB2 first QEMU must be started in GDB stub mode. The following
> +command is a simple illustration:
> +
> +@example
> +qemu-system-i386 -drive file=disk.img,format=raw \
> +    -device virtio-scsi-pci,id=scsi0 -S -s
> +@end example
> +
> +This will start a QEMU instance booting from @file{disk.img}. It will pause
> +at start waiting for a GDB instance to attach to it. You should change
> +@file{disk.img} to something more appropriate. A block device can be used,
> +but you may need to run QEMU as a privileged user.
> +
> +To connect to this QEMU instance with GDB, the @code{target remote} GDB
> +command must be used. We also need to load a binary image, preferably with
> +symbols. This can be done using the GDB command @code{file kernel.exec}, if
> +GDB is started from the @file{grub-core} directory in the GRUB2 build
> +directory. GRUB2 developers have made this more simple by including a GDB
> +script which does much of the setup. This file at @file{grub-core/gdb_grub}
> +of the build directory and is also installed via @command{make install}.

This sentence is definitely missing an "is" or similar, but I'd write
something like:

    This file is at grub-core/gdb_grub in the build directory

> +If not building GRUB, the distribution may have a package which installs
> +this GDB script along with debug symbol binaries, such as Debian's
> +@samp{grub-pc-dbg} package. The GDB scripts is intended to by used

If it's just a single script, this should be:

    The GDB script is intended to by used

> +like so, assuming:

Did you forget to state what the assumption is?

> +
> +@example
> +cd $(dirname /path/to/script/gdb_grub)
> +gdb -x gdb_grub
> +@end example
> +
> +Once GDB has been started with the @file{gdb_grub} script it will
> +automatically connect to the QEMU instance. You can then do things you
> +normally would in GDB like set a break point on @var{grub_main}.
> +
> +Setting breakpoints in modules is trickier since they haven't been loaded
> +yet and are loaded at addresses determined at runtime. The module could be
> +loaded to different addresses in different QEMU instances. The debug symbols
> +in the modules @file{.module} binary, thus are always wrong, and GDB needs
> +to be told where to load the symbols to. But this must happen at runtime
> +after GRUB2 has determined where the module will get loaded. Luckily the
> +@file{gdb_grub} script takes care of this with the @command{runtime_load_module}
> +command, which configures GDB to watch for GRUB2 module loading and when
> +it does add the module symbols with the appropriate offset.
> +
> +@node x86_64-efi
> +@section x86_64-efi
> +
> +Using GDB to debug GRUB2 for the x86_64-efi target has some similarities with
> +the i386-pc target. Please read and familiarize yourself with the @ref{i386-pc}
> +section when reading this one. Extra care must be used to run QEMU such that it
> +boots a UEFI firmware. This usually involves either using the @samp{-bios}
> +option with a UEFI firmware blob (eg. @file{OVMF.fd}) or loading the firmware
> +via pflash. This document will not go further into how to do this as there are
> +ample resource on the web.
> +
> +Like all EFI implementations, on x86_64-efi the (U)EFI firmware that loads
> +the GRUB2 EFI application determines at runtime where the application will
> +be loaded. This means that we do not know where to tell GDB to load the
> +symbols for the GRUB2 core until the (U)EFI firmware determines it. There are
> +two good ways of figuring this out when running in QEMU: use a @ref{OVMF debug log,
> +debug build of OVMF} and check the debug log, or have GRUB2 say where it is
> +loaded. Neither of these are ideal because they both generally give the
> +information after GRUB2 is already running, which makes debugging early boot
> +infeasible. Technically, the first method does give the load address before
> +GRUB2 is run, but without debugging the EFI firmware with symbols, the author
> +currently does not know how to cause the OVMF firmware to pause at that point
> +to use the load address before GRUB2 is run.
> +
> +Even after getting the application load address, the loading of core symbols
> +is complicated by the fact that the debugging symbols for the kernel are in
> +an ELF binary named @file{kernel.exec} while what is in memory are sections
> +for the PE32+ EFI binary. When @command{grub-mkimage} creates the PE32+
> +binary it condenses several segments from the ELF kernel binary into one
> +.data section in the PE32+ binary. This must be taken into account to
> +properly load the other non-text sections. Otherwise, GDB will work as
> +expected when breaking on functions, but, for instance, global variables
> +will point to the wrong address in memory and thus give incorrect values
> +(which can be difficult to debug).
> +
> +The calculating of the correct offsets for sections when loading symbol
> +files are taken care of when loading the kernel symbols via the user-defined

This sentence feels a bit clumsy. I'd write something like:

    Calculating the correct offsets for sections is taken care of
    automatically when loading the kernel symbols via the
    user-defined...

I was originally going to suggest "section offsets" here too, but I'm
not confident that it couldn't potentially mean something else in this
context.

> +GDB command @command{dynamic_load_kernel_exec_symbols}, which takes one
> +argument, the address where the text section is loaded, as determined by

I would personally drop the second comma in "argument, ... as determined
by".

> +one of the methods above. Alternatively, the command @command{dynamic_load_symbols}
> +with the text section address as an agrument can be called to load the
> +kernel symbols and setup loading the module symbols as they are loaded at

"setup" should probably be "set up".

> +runtime.
> +
> +In the author's experience, when debugging with QEMU and OVMF, to have
> +debugging symbols loaded at the start of GRUB2 execution the GRUB2 EFI
> +application must be run via QEMU at least once prior in order to get the
> +load address. Two methods for obtaining the load address are described in
> +two subsections below. Generally speaking, the load address does not change
> +between QEMU runs. There are exceptions to this, namely that different
> +GRUB2 EFI applications can be run at different addresses. Also, it has been
> +observed that after running the EFI application for the first time, the
> +second run will some times have a different load address, but subsequent

"some times" should probably be "sometimes".

> +runs of the same EFI application will have the same load address as the
> +second run. And it's a near certainty that if the GRUB EFI binary has changed,
> +eg. been recompiled, the load address will also be different.
> +
> +This ability to predict what the load address will be allows one to assume
> +the load address on subsequent runs and thus load the symbols before GRUB2
> +starts. The following command illustrates this, assuming that QEMU is
> +running and waiting for a debugger connection and the current working
> +directory is where @file{gdb_grub} resides:
> +
> +@example
> +gdb -x gdb_grub -ex 'dynamic_load_symbols @var{address of .text section}'
> +@end example
> +
> +If you load the symbols in this manner and, after continuing execution, do
> +not see output showing the loading of modules symbol, then it is very likely

Would this make more sense as "showing the module symbols loading"?

> +that the load address was incorrect.
> +
> +Another thing to be aware of is how the loading of the GRUB image by the
> +firmware affects previously set software breakpoints. On x86 platforms,
> +software breakpoints are implemented by GDB by writing a special processor
> +instruction at the location of the desired breakpoint. This special instruction
> +when executed will stop the program execution and hand control to the
> +debugger, GDB. GDB will first save the instruction bytes that are
> +overwritten at the breakpoint and will put them back when the breakpoint
> +is hit. If GRUB is being run for the first time in QEMU, the firmware will
> +be loading the GRUB image into memory where every byte is already set to 0.
> +This means that if a breakpoint is set before GRUB is loaded, GDB will save
> +the 0-byte(s) where the the special instruction will go. Then when the firmware
> +loads the GRUB image and because it is unaware of the debugger, it will
> +write the GRUB image to memory, overwriting anything that was there previously,
> +notably in this case the instruction that implements the software breakpoint.

I would probably split "notably in this case ..." off into its own
sentence.

> +This will be confusing for the person using GDB because GDB will show the
> +breakpoint as set, but the brekapoint will never be hit. Furthermore, GDB
> +then becomes confused, such that even deleting an recreating the breakpoint
> +will not create usable breakpoints. The @file{gdb_grub} script takes care of
> +this by saving the breakpoints just before they are overwritten, and then
> +restores them at the start of GRUB execution. So breakpoints for GRUB can be
> +set before GRUB is loaded, but be mindful of this effect if you are confused
> +as to why breakpoints are not getting hit.
> +
> +Also note, that hardware breakpoints do not suffer this problem. They are
> +implemented by having the breakpoint address in special debug registers on
> +the CPU. So they can always be set freely without regard to whether GRUB has
> +been loaded or not. The reason that hardware breakpoints aren't always used
> +is because there are a limited number of them, usually around 4 on various
> +CPUs, and specifically exactly 4 for x86 CPUs. The @file{gdb_grub} script
> +goes out of its way to not use hardware breakpoints internally and when
> +needed use them as short a time as possible, thus allowing the user to have a

I'd write this as:

    The gdb_grub script goes out of its way to avoid using hardware
    breakpoints internally, and when needed, uses them as briefly as
    possible, thus allowing the user...

> +maximal number at their disposal.
> +
> +@node OVMF debug log
> +@subsection OVMF debug log
> +
> +In order to get the GRUB2 load address from OVMF, first, a debug build
> +of OVMF must be obtained (@uref{https://github.com/retrage/edk2-nightly/raw/master/bin/DEBUGX64_OVMF.fd,
> +here is one} which is not officially recommended). OVMF will output debug
> +messages to a special serial device, which we must add to QEMU. The following
> +QEMU command will run the debug OVMF and write the debug messages to a
> +file named @file{debug.log}. It is assumed that @file{disk.img} is a disk
> +image or block device that is setup to boot GRUB2 EFI.

This "setup" should probably be "set up" as well.

- Oskari

> +
> +@example
> +qemu-system-x86_64 -bios /path/to/debug/OVMF.fd \
> +    -drive file=disk.img,format=raw \
> +    -device virtio-scsi-pci,id=scsi0 \
> +    -debugcon file:debug.log -global isa-debugcon.iobase=0x402
> +@end example
> +
> +If GRUB2 was started by the (U)EFI firmware, then in the @file{debug.log}
> +file one of the last lines should be a log message like:
> +@samp{Loading driver at 0x00006AEE000 EntryPoint=0x00006AEE756}. This
> +means that the GRUB2 EFI application was loaded at @samp{0x00006AEE000} and
> +its .text section is at @samp{0x00006AEE756}.
> +
> +@node Using the gdbinfo command
> +@subsection Using the gdbinfo command
> +
> +On EFI platforms the command @command{gdbinfo} will output a string that
> +is to be run in a GDB session running with the @file{gdb_grub} GDB script.
> +
> +
>  @node Porting
>  @chapter Porting
>  
> -- 
> 2.34.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Daniel Kiper]

On Thu, Jun 15, 2023 at 01:21:57AM -0500, Oskari Pirhonen wrote:
> Oops, apologies for the late reply. Reading through it again, I found a
> few more small nits:

Right now patch is in the git repo. If you want to improve that part of
doc please send a fix to grub-devel and CC interested folks.

Daniel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

On Thu, Jun 15, 2023 at 13:06:09 +0200, Daniel Kiper wrote:
> On Thu, Jun 15, 2023 at 01:21:57AM -0500, Oskari Pirhonen wrote:
> > Oops, apologies for the late reply. Reading through it again, I found a
> > few more small nits:
> 
> Right now patch is in the git repo. If you want to improve that part of
> doc please send a fix to grub-devel and CC interested folks.
> 
> Daniel

Can do.

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 480 bytes --]

Thu, 15 Jun 2023 13:06:09 +0200 Daniel Kiper <dkiper@net-space.pl>:

> Right now patch is in the git repo. If you want to improve that part of
> doc please send a fix to grub-devel and CC interested folks.

Yeah, that is very unfortunate, because it breaks the build:
https://lists.gnu.org/archive/html/bug-grub/2023-06/msg00006.html

5a3d2b4742dfe4bfe2b51f7b712bc107f75e84ed is the first bad commit

For some reason there is no --disable-docs for configure?


Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Glenn Washburn]

On Tue, 15 Aug 2023 10:14:07 +0200
Olaf Hering <olaf@aepfle.de> wrote:

> Thu, 15 Jun 2023 13:06:09 +0200 Daniel Kiper <dkiper@net-space.pl>:
> 
> > Right now patch is in the git repo. If you want to improve that part of
> > doc please send a fix to grub-devel and CC interested folks.
> 
> Yeah, that is very unfortunate, because it breaks the build:
> https://lists.gnu.org/archive/html/bug-grub/2023-06/msg00006.html
> 
> 5a3d2b4742dfe4bfe2b51f7b712bc107f75e84ed is the first bad commit
> 
> For some reason there is no --disable-docs for configure?

I'm not seeing this issue when building on Debian 11. I'm successfully
building the info, html, and pdf documentation from the texi files
using debians texinfo package at version 6.7.0.dfsg.2-6. Are you sure
you are building from clean sources? What version of texinfo are you at?

Glenn


_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 434 bytes --]

Tue, 15 Aug 2023 12:31:29 -0500 Glenn Washburn <development@efficientek.com>:

> I'm not seeing this issue when building on Debian 11. I'm successfully
> building the info, html, and pdf documentation from the texi files
> using debians texinfo package at version 6.7.0.dfsg.2-6. Are you sure
> you are building from clean sources? What version of texinfo are you at?

The sources are clean, makeinfo 4.13a is used.


Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Glenn Washburn]

On Tue, 15 Aug 2023 22:37:08 +0200
Olaf Hering <olaf@aepfle.de> wrote:

> Tue, 15 Aug 2023 12:31:29 -0500 Glenn Washburn <development@efficientek.com>:
> 
> > I'm not seeing this issue when building on Debian 11. I'm successfully
> > building the info, html, and pdf documentation from the texi files
> > using debians texinfo package at version 6.7.0.dfsg.2-6. Are you sure
> > you are building from clean sources? What version of texinfo are you at?
> 
> The sources are clean, makeinfo 4.13a is used.

Is that a 6+ year old version[1]?

 Here's my output:

$ makeinfo --version
texi2any (GNU texinfo) 6.7

Anyway, since you're the first and *so far* only person seeing this,
could you submit a patch that fixes this for you? It would be great to
support older makeinfos (if that is indeed the issue) in the upcoming
release.

Glenn

[1] https://lists.gnu.org/archive/html/bug-gnulib/2016-06/msg00017.html

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 2482 bytes --]

Wed, 16 Aug 2023 00:29:28 -0500 Glenn Washburn <development@efficientek.com>:

> Anyway, since you're the first and *so far* only person seeing this,
> could you submit a patch that fixes this for you? It would be great to
> support older makeinfos (if that is indeed the issue) in the upcoming
> release.

The change below fixes the following errors:

grub.texi:960: warning: `.' or `,' must follow @xref, not `)'.
grub.texi:960: warning: for cross-references in parentheses, use @pxref.
grub-dev.texi:815: `Using the gdbinfo command' has no Up field (perhaps incorrect sectioning?).
grub-dev.texi:791: `OVMF debug log' has no Up field (perhaps incorrect sectioning?).
grub-dev.texi:815: warning: unreferenced node `Using the gdbinfo command'.

With some rewording the first error could be fixed properly.


Olaf

--- a/docs/grub-dev.texi
+++ b/docs/grub-dev.texi
@@ -632,6 +632,8 @@ various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
 @menu
 * i386-pc::
 * x86_64-efi::
+* OVMF debug log::
+* Using the gdbinfo command::
 @end menu
 
 @node i386-pc
@@ -789,7 +791,7 @@ briefly as possible when needed, thus allowing the user to have a maximal
 number at their disposal.
 
 @node OVMF debug log
-@subsection OVMF debug log
+@section OVMF debug log
 
 In order to get the GRUB2 load address from OVMF, first, a debug build
 of OVMF must be obtained (@uref{https://github.com/retrage/edk2-nightly/raw/master/bin/DEBUGX64_OVMF.fd,
@@ -813,7 +815,7 @@ means that the GRUB2 EFI application was loaded at @samp{0x00006AEE000} and
 its .text section is at @samp{0x00006AEE756}.
 
 @node Using the gdbinfo command
-@subsection Using the gdbinfo command
+@section Using the gdbinfo command
 
 On EFI platforms the command @command{gdbinfo} will output a string that
 is to be run in a GDB session running with the @file{gdb_grub} GDB script.
--- a/docs/grub.texi
+++ b/docs/grub.texi
@@ -957,7 +957,7 @@ information.
 
 GRUB can be run in userspace by invoking the grub2-emu tool. It will
 read all configuration scripts as if booting directly (see @xref{Loading
-an operating system directly}). With the @code{--kexec} flag, and
+an operating system directly}.) With the @code{--kexec} flag, and
 kexec(8) support from the operating system, the @command{linux} command
 will directly boot the target image. For systems that lack working
 systemctl(1) support for kexec, passing the @code{--kexec} flag twice

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Glenn Washburn]

On Wed, 16 Aug 2023 10:41:40 +0200
Olaf Hering <olaf@aepfle.de> wrote:

> Wed, 16 Aug 2023 00:29:28 -0500 Glenn Washburn <development@efficientek.com>:
> 
> > Anyway, since you're the first and *so far* only person seeing this,
> > could you submit a patch that fixes this for you? It would be great to
> > support older makeinfos (if that is indeed the issue) in the upcoming
> > release.
> 
> The change below fixes the following errors:
> 
> grub.texi:960: warning: `.' or `,' must follow @xref, not `)'.
> grub.texi:960: warning: for cross-references in parentheses, use @pxref.
> grub-dev.texi:815: `Using the gdbinfo command' has no Up field (perhaps incorrect sectioning?).
> grub-dev.texi:791: `OVMF debug log' has no Up field (perhaps incorrect sectioning?).
> grub-dev.texi:815: warning: unreferenced node `Using the gdbinfo command'.
> 
> With some rewording the first error could be fixed properly.

Thanks for working on this. My comments below.

> 
> 
> Olaf
> 
> --- a/docs/grub-dev.texi
> +++ b/docs/grub-dev.texi
> @@ -632,6 +632,8 @@ various targets using @command{gdb} and the @samp{gdb_grub} GDB script.
>  @menu
>  * i386-pc::
>  * x86_64-efi::
> +* OVMF debug log::
> +* Using the gdbinfo command::

I specifically wanted these two sections to be subsections of
x86_64-efi and not at the same level as i386-pc, hence why I have them
as subsections. I suspect that the issue could be fixed by adding a
@menu in the x86_64-efi section with these two lines. I didn't really
want a menu there, but I'm fine with it if it fixes it for you.

>  @end menu
>  
>  @node i386-pc
> @@ -789,7 +791,7 @@ briefly as possible when needed, thus allowing the user to have a maximal
>  number at their disposal.
>  
>  @node OVMF debug log
> -@subsection OVMF debug log
> +@section OVMF debug log
>  
>  In order to get the GRUB2 load address from OVMF, first, a debug build
>  of OVMF must be obtained (@uref{https://github.com/retrage/edk2-nightly/raw/master/bin/DEBUGX64_OVMF.fd,
> @@ -813,7 +815,7 @@ means that the GRUB2 EFI application was loaded at @samp{0x00006AEE000} and
>  its .text section is at @samp{0x00006AEE756}.
>  
>  @node Using the gdbinfo command
> -@subsection Using the gdbinfo command
> +@section Using the gdbinfo command
>  
>  On EFI platforms the command @command{gdbinfo} will output a string that
>  is to be run in a GDB session running with the @file{gdb_grub} GDB script.
> --- a/docs/grub.texi
> +++ b/docs/grub.texi
> @@ -957,7 +957,7 @@ information.
>  
>  GRUB can be run in userspace by invoking the grub2-emu tool. It will
>  read all configuration scripts as if booting directly (see @xref{Loading
> -an operating system directly}). With the @code{--kexec} flag, and
> +an operating system directly}.) With the @code{--kexec} flag, and

Actually I think the correction should be:

  (@pxref{Loading an operating system directly}).

@xref is for use at the beginning of a sentence and @pxref is for
use inside parenthesis[1].

Glenn

[1]
https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Cross-Reference-Commands.html

>  kexec(8) support from the operating system, the @command{linux} command
>  will directly boot the target image. For systems that lack working
>  systemctl(1) support for kexec, passing the @code{--kexec} flag twice

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: [PATCH v3] docs: Add debugging chapter to development documentation

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 599 bytes --]

Wed, 16 Aug 2023 12:53:48 -0500 Glenn Washburn <development@efficientek.com>:

> I specifically wanted these two sections to be subsections of
> x86_64-efi and not at the same level as i386-pc, hence why I have them
> as subsections. I suspect that the issue could be fixed by adding a
> @menu in the x86_64-efi section with these two lines. I didn't really
> want a menu there, but I'm fine with it if it fixes it for you.

Yeah, this is my understanding as well.

Now that I have found a way to avoid building the info files,
this change is not required from my perspective.


Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel